This changes allows fetching the file size more easily (for supported
files) in order to afterwards calculate the multipart part size
accordingly.
Change-Id: Idabba4c2ee794ee471973889f5843174a7acad35
This change allows the uplink to bump the part size based on the
content length that is being copied. This ensures we are staying
below the 10k part limit currently enforced on the satellites.
If the user specifies the flag, it will error out if the value
chosen by the user is too low. Otherwise it will use it.
Change-Id: I00d30f603d941c2f7703ba19d5923e668629a7b9
recursive copy had a bug with relative local paths.
this fixes that bug and changes the test framework
to use more of the code that actually runs in uplink
and only mocks out the direct interaction with the
operating system.
Change-Id: I9da2a80bfda8f86a8d05879b87171f299f759c7e
With this change users can use the uplink cli in
scripts (ie. bash) more easily, since the output
can be switched to an easier processable json format.
It keeps the default of tabbed output.
Change-Id: I37e2c55f75c2250c3119fd8df8b66a766ff9096b
When ctx is cancelled limiter won't start a new goroutine.
The code didn't immediately return an error in that case.
The dst.Commit(ctx) would fail anyways due to a cancelled ctx.
However, we can make the behavior clearer by returning immediately.
Change-Id: I65df7ca85de55813f3200a50db2eaaa7a297ba2c
It was possible for the a previous write / part to fail or be aborted
and the next part write still happened. This causes a data ordering
corruption.
The whole write to parallel stdout fails, so there shouldn't be
confusion with regards to the output acceptability. However, it would
be clearer, if we avoided writing out-of-order data... mainly to be
clear that we didn't corrupt the data, just that it's incomplete.
Change-Id: I97b0d14404f29e8615e7d29b10cbd61ccb861e40
Also ensure that abort is given at least 5 seconds to clear up any
pending uploads on cancellation.
Change-Id: I814aa407ee5783f2609a76b54de2879dcd5f89bb
If the cp command is executed with higher level of parallelism, it would
open more connections to storage nodes at the same time. Therefore, the
connection pool capacity should be expanded accordingly.
The pool capacity is set to 100 * parallelism.
Change-Id: Ia8b3ab6a99340d8cbb87a7b80c3354b2b21c1958
I don't think it should matter for correctness whether this matches the
segment size or not, so I think there is something else wrong. However,
making this change seems to eliminate the "corruption when ulimit -n is
too low" problem we're seeing right now.
Change-Id: I232fe0d0a371b86ddf902e8c2d4778e140b2f1fc
When copying an object from cli you can now set the expiry.
It uses the same datetime format as restricting access grants.
Closes https://github.com/storj/storj/issues/4595
Change-Id: Icab73a64a9589817d6bc6d702b765b166ca1350d
some old configs had a value like
access: <data>
in the yaml. this would end up causing migration to
create a json file where it had no access values and
a default name of the data. that's not what the command
expects to operate on, so now we fix that during
migration and add a little mini migration for any
users that may have hit it.
Change-Id: I4c98ca5d09d043fe9338738ef6b4f930f933892c
Change the implementation of register and share so that it uses the
uplink method to contact the Auth Service. The network protocol switches
from HTTP to DRPC.
Closes https://github.com/storj/storj/issues/4324
Change-Id: Ib8fdb1665c6385bb39a546ba46a8df43a136df9c
We want to monitor traffic and tools that are used
to interact with our network so we need to append
its user agent.
The same user agent is appended for current uplink
and uplinkng as eventually we will remove first.
Change-Id: I116080d6c2c6c85d591771facf01356de02a9392
In uplink we have command uplink share and we need
to port it to uplinkng to have command with same functionality.
Command could be executed with parameter or without.
without any flags we share as readonly.
Change-Id: I973b11d00da237358834acf5a863ebab37e684cc
In uplink we have command uplink access inspect and we need
a command with the same functionality for uplinkng.
Command could be executed with parameter or without.
without parameter - we should show default access.
If parameter exists - it could be access name or value.
If access name or value is wrong - we show error.
F.e.
uplinkng access inspect
uplinkng access inspect accessName
uplinkng access inspect accessValue
https://storjlabs.atlassian.net/browse/PG-318
Change-Id: I85fd961283850feb8684db2d126441f6b9bf0270
https://storjlabs.atlassian.net/browse/PG-305
we should extend method move of cmd/uplink/cmd/mv.go
if both parameters end with slash -
i should list all files and call move method in loop
parameters:
uplink mv sj://bucket/a/prefix/ sj://new-bucket/a/new-prefix/
Change-Id: Ic24c2af83153ea60ec74393e65736af094877151
New command for cli to move object to different
location.
uplink mv sj://bucket/your-object sj://bucket/moved-object
Change-Id: I85a4961aa59f250819954e78f20363ac3c570938
Use multipart upload to upload single object in parts in
parallel. Its using parallelism flag added earlier.
Change-Id: I45b531a5db43c86f0112a5e3bb4a83bc1d65650f
Adds support for new uplink method DownloadObjectAt which
gives ability to download single object in parallel.
Change-Id: I8388653429992b0d24c383d17d7e90904203fe77
The auth service has no way to remove access grant registrations that
lack expiration dates. We want to encourage people to set them, so as
to slow the rate at which the auth service DB fills up.
Change-Id: I1ccf629cd995dc184d2d90333166eab34d34ae07
This enables use of the --progress, --expires, and --metadata flags with
'uplink put'. These flags work similarly to their counterparts in the
'uplink cp' command.
Small difference from cp: "progress" is on by default in 'uplink cp',
but (for backwards compatibility) off by default in 'uplink put'.
Why: Requested by @Toyoo on the forum:
https://forum.storj.io/t/explicit-data-expiration-questions/13854/2
..and it's an easy addition to make.
Change-Id: Id20aedd3a280db43e4883338f92f6beec7a400de
This expanded format shows expiration times for objects and how much
custom metadata each object has.
This commit also organizes output formatting code together in one
section for simpler adjustments in the future.
Change-Id: Ica041c8a1de6ee73c104a0554c5c259e447536c4
Initially there were pkg and private packages, however for all practical
purposes there's no significant difference between them. It's clearer to
have a single private package - and when we do get a specific
abstraction that needs to be reused, we can move it to storj.io/common
or storj.io/private.
Change-Id: Ibc2036e67f312f5d63cb4a97f5a92e38ae413aa5
Currently os.Create was leaving a file open causing atomic write file to
fail with access denied.
Also add a specific test for importing an access.
Change-Id: Id188bc480e795849ec7fdc72b1fc86433d76c47a
should be
https://link.tardigradeshare.io/s/<access>/<bucket>/<path>
legacy URLs have the /s/ missing but a redirect is issued
Change-Id: Ic2a3dc092ff68d7706fd888a9fbfc27716877c6e
