Use multipart upload to upload single object in parts in
parallel. Its using parallelism flag added earlier.
Change-Id: I45b531a5db43c86f0112a5e3bb4a83bc1d65650f
Adds support for new uplink method DownloadObjectAt which
gives ability to download single object in parallel.
Change-Id: I8388653429992b0d24c383d17d7e90904203fe77
* integration tests have now a trap that will display where script failed, line and message
* functions from uplink tests moved to utils.sh to reuse them
Change-Id: Ib2311775fd70ce784aa986328969d75eefc5ac36
This PR updates `uplink rb --force` command to use the new libuplink API
`DeleteBucketWithObjects`.
It also updates `DeleteBucket` endpoint to return a specific error
message when a given bucket has concurrent writes while being deleted.
Change-Id: Ic9593d55b0c27b26cd8966dd1bc8cd1e02a6666e
this is going to make all the tests slower but it is what it is
test-sim-aws.sh is removed because it was moved to storj/gateway repo.
Change-Id: I10727e747a4c3740b1c9054ce7d17313b4fa310b
We want to make using uplink as easy as possible. That's why we wan't to
avoid requiring setup or import command before normal usage if user
specified --access flag. If this flag is set then rest flags should be
set as defaults.
https://storjlabs.atlassian.net/browse/V3-3490
Change-Id: I95a7bd77a3f00b8d9981fee513e9e77aef298bca
* add integration tests to jenksin
* have jenkins run storj-sim integration tests w/crdb
Change-Id: I696d55c5894aaf630dcd7a566e1dd705ee88486b
* rm crdb integration tests to see if postgres passes
Change-Id: I1727a027ff802acbff5fc55961a0d605faefcf2d
* comment out aws tests to see if that is the error
Change-Id: I456c3d36f6a4ce7760ea0b6c402b6ea16cfe77e3
* add aws profile to integration tests
Change-Id: Ic01185dbc7b84ac48dfb846f8f272b34b50379b6
* add tmp path for aws profile and creds
Change-Id: I7b82ee5a99937edd3f66ae01bfb5cb21028a62cf
* change linux KiB syntax to bytes to support osx
Change-Id: Ia1f1027ba8da64a6ba537062deb9b3519973621f
* uplink: Mark encryption key config field for setup
Set the "setup" property to the `EncryptionConfig.EncrptionKey` for
avoiding to save it in the configuration file.
This field is only meant for using in the command line parameters which
need to use a different encryption key than the one present in the key
file or use it when there is not set any encryption key file path.
* cmd/uplink: Setup non-interactive accept enc key
Change the uplink CLI setup command non-interactive to save the
encryption key into a file when it's passed through the flag
--enc.encryption-key
Previous to this change it wasn't possible to create an key file despite
of that the flag was provided, so it was useless on the setup command.
* cmd/uplink: Reuse logic to read pwd from terminal
Reuse the logic which is already implemented in the pkg/cfgstruct for
reading a password from the terminal on interactive mode, rather than
duplicating it in the setup command.
* cmd/gateway: Use encryption key file flags
The cmd/gateway was still using the `enc.key` configuration field which
doesn't exist anymore and its setup command wasn't using the
`enc.key-filepath` with combination of the `enc.encryption-key` for
generating a file with the encryption key.
This commit update the cmd/gateway appropriately and move to the uplink
package the function used by cmd/uplink to save the encryption key for
allowing to also be used by the cmd/gateway without duplicating the
logic.
* cmd/storj-sim: Adapt gateway config cmd changes
Adapt the cmd/storj-sim to correctly pass the parameters to the
cmd/gateway setup and run command.
* scripts: Don't pass the --enc.encryption-key flag
uplink configuration has changed to only support the
`--enc.encryption-key` flag for setup commands and consequently the
cmd/uplink and cmd/gateway don't accept this flag over other commands,
hence the test for the uplink had to be updated for no passing the
flag on the multiples calls that the test do to cmd/uplink.
* uplink: Remove func which aren't useful anymore
Remove the function which allows to user or load an encryption key
because it isn't needed anymore since the `--enc.encryption-key` flag is
only available for the setup command.
Consequently remove its usage from cmd/uplink and cmd/gateway, because
such flag will always be empty because in case that's passed Cobra will
return an error due to a "unknown flag".
* uplink: Add a new flag to set the filepath of the file which is used for
saving the encryption key and rename the one that hold the encryption key and
establish that it has priority over the key stored in the file to make the
configuration usable without having a huge refactoring in test-sim.
* cmd/uplink: Adapt the setup subcommand for storing the user input key to a file
and adapt the rest of the subcommands for reading the key from the key-file when
the key isn't explicitly set with a command line flag.
* cmd/gateway: Adapt it to read the encryption key from the key-file or use the
one passed by a command line flag.
* pkg/process: Export the default configuration filename so other packages which
use the same value can reference to it rather than having it hardcoded.
* Adapt several integrations (scripts, etc.) to consider the changes applied in uplink and cmd packages.
* test-network-stalls tests... network stalls!
in particular, right now, it just tests whether an uplink correctly
times out after some amount of time when one of the nodes it's talking
to suddenly goes offline.
This tool is meant to be run under `storj-sim network test`.
Also included here:
* fix storj-sim-related test scripts on Mac
the default storj config dir on Mac has a space in it
('~/Library/Application Support/Storj'), which breaks everywhere it
shows up in an unquoted variable in a sh/bash script. easy enough to fix
as a one-off, but quoting bash vars avoids a dozen other potential
problems too.
change a few things using `head -c` to use `dd`. `head -c` works,
but is not as widely understood (as evidenced by each of these scripts
getting through code review, one at a time, with the comments not
matching the numbers actually used).
* storj-sim reports PIDs of worker processes to test
so that the tests can cause unfortunate "accidents" to befall the worker
processes in the course of the test, and find out whether everything
reacts correctly.
* initial wireup of test script
* making some progress
* working on getting gateway config to work
* get script working for testing uplink
* removes unnecessary stuff
* more removal of unnecessary change
* remove unnecessary pointerdb set in setup
* fix dest dir in test-aws script
