Commit Graph

782 Commits

Author SHA1 Message Date
Kaloyan Raev
091b49b921 cmd/satellite: command to move unspent bonuses to Stripe balance
Jira: https://storjlabs.atlassian.net/browse/USR-821

The `migrate-credits` billing command checks the available credits
balance for all users and moves it to the Stripe balance by creating a
new credit balance transaction.

Change-Id: Iafc7b95a4edad47f7c145a22e210f8c821ac183d
2020-06-24 13:20:24 +00:00
Ivan Fraixedes
37cfc01ac6 cmd/storagenode: Add long description to ge command
Add a long description to the graceful exit command to clearly mention
that the command is interactive asking which satellite the SNO wants to
exit.

Change-Id: Icd4056a470e707322f600133e63d9dc56eb877b7
2020-06-24 08:47:52 +00:00
Isaac Hess
cad21f11e5 cmd/uplink: Add revoke command
Change-Id: I8ae74da5c9171429b970d76598793a3227ed4d11
2020-06-22 13:50:07 -06:00
Isaac Hess
2d727bb14e satellite: Check macaroon revocation
When a request comes in on the satellite api and we validate the
macaroon, we now also check if any of the macaroon's tails have been
revoked.

Change-Id: I80ce4312602baf431cfa1b1285f79bed88bb4497
2020-06-22 13:50:07 -06:00
Stefan Benten
433fc91054
cmd/satellite: clean up entrypoint (#3910) 2020-06-22 21:06:31 +02:00
JT Olio
2bd9067ad5 cmd/uplink: tighter linksharing security
when a user runs `uplink share`, they get a bunch of results back,
given their configuration and existing access. one of the results
is a URL for in-browser sharing and hosting of the file.

first off, we want to make sure this URL is read only. we want to
avoid a situation where someone post this URL to some public
location, not realizing the access allows writes or deletes. if
a user really wants a URL with write/delete access, they can
construct it themselves.

secondly, we want to make sure the url is sharing a single path or
path prefix. having a url for multiple paths/path prefixes
indepedently again can be constructed of course, but should not
be the default behavior

Change-Id: I2ca2ebeea9f1c7d4bfbd7a437a32dc7a3b2a32cc
2020-06-16 15:02:53 -06:00
Michal Niewrzal
7a0778fac4 cmd/satellite: choose correct Stripe client also for commands
Commands were using always real Strip client but for e.g. storj-sim
should use mock by default.

Change-Id: Ifd3c02028312d2e8d6d73d67b2dadadcd69077c8
2020-06-15 11:03:14 +00:00
Kaloyan Raev
adb5a25b63 cmd/satellite: revert 'organize subcommands'
Reverts https://review.dev.storj.io/c/storj/storj/+/2074

That change broke all billing commands with `sql: database is closed`.
Fixing the issue does not make sense as the whole setup of billing
commands is now error-prone. It's better to revert it.

Change-Id: Id13f817b73c7313ba60181f740b0712e4bdce077
2020-06-15 10:06:30 +00:00
stefanbenten
44792f7b45 cmd/satellite: add finalize invoices command
Change-Id: Ie57dedc9b9c8fe133dede573c921dfee0883e5ff
2020-06-10 14:48:15 +00:00
stefanbenten
fdbb2c3ed6 cmd/satellite: organize subcommands
Change-Id: If45575656e5e16642bdee4b6e6867e4f1d824f9a
2020-06-10 14:35:21 +00:00
Qweder93
e52809d53e cmd/storagenode: add check if satellites available to gracefulexit
Change-Id: I8747507593d810bbdec0d140de0600ee147011c3
2020-06-10 13:38:36 +00:00
JT Olio
d9b1c4520e storagenode-updater: fix logic to use minimum version correctly
See:
  https://review.dev.storj.io/c/storj/storj/+/2052
  https://review.dev.storj.io/c/storj/private/+/2053

Change-Id: I30952c6e3893194ca876c12ecb7e53944e66aa2e
2020-06-08 17:43:10 +00:00
Yingrong Zhao
175f048aa7 scripts;cmd/storj-sim: include satellite node id in satellite address
Expose satellite ID from storj-sim so we can have access to it when
changing satellite address.

Change-Id: Ife816f4d35eae2d0bc9f9ad592fe75d73d93d9ff
2020-06-05 09:44:19 -04:00
Egon Elbre
34db4a80fd ci: fix staticcheck failures
Change-Id: I176fb24214755a1940a0a1a4e9cc8e39f184870b
2020-06-05 13:15:34 +00:00
Egon Elbre
0de1b5fa9f ci: fix golangci-lint failures
Change-Id: Ia8c4dd03439ce4e3f85890d45255b8ea1c94b50a
2020-06-05 12:55:35 +00:00
Michal Niewrzal
056ae7ffa8 scripts/test/rollingupgrade: set correct satellite address for imported
access

Addition: use always latest gateway release

Change-Id: I5e5231e6da4b6f7900cb71bb6e227901474270ea
2020-06-04 15:04:10 +00:00
Michal Niewrzal
79a562c3ed cmd/uplink/cmd: improve uplink remove bucket command with force flag
* don't stop on single delete failure
* display number of successful and unsuccessful deletes

Change-Id: Id7f29055f250aad515e46622c2bfd3596055b93a
2020-06-01 07:56:14 +00:00
Michal Niewrzal
48e67a29a9 cmd/storj-sim: remove old libuplink from storj-sim
Change-Id: I146a2d7947d1f358e7bdd8acba68eb3fee68539d
2020-05-29 08:39:37 +00:00
Michal Niewrzal
84892631c8 private/testplanet: remove old libuplink from testplanet
Change-Id: Ib1553f84d0b3ae12a5b00382f0f53357b6a273e2
2020-05-28 13:50:23 +00:00
Michal Niewrzal
a4c19b3359 cmd/uplink/cmd: remove port detection when selecting satellite while
setup

We don't have default ports as a part of configuration anymore because
satellite-addr flag was removed.

Change-Id: Ibf9fc4b399beaf51ebb9461de2d8994a322f9686
2020-05-26 13:25:51 +02:00
Stefan Benten
3da100c242
cmd/satellite: Install curl per default to docker image (#3899) 2020-05-25 02:39:21 +02:00
Michal Niewrzal
340700eda9 cmd/uplink/cmd: remove old libuplink from Uplink CLI code
Change-Id: I38444a67e3bdec5f63c56895b5fc362464e7dce1
2020-05-22 09:08:58 +00:00
Bill Thorp
bcb867f1de cmd/uplink: add UserAgent flag for partner value attribution
This changeset allows a user agent string to be set during uplink setup, which
is thereafter used for partner value attribution.  EG

uplink setup --client.user-agent ”MyCompany”

Change-Id: Iefa8755fccc06acb8a303a342b943cece44a81f7
2020-05-22 07:39:14 +00:00
Ivan Fraixedes
963db8c738 cmd/segment-reaper: Remove unneeded test cases
Remove unneeded observer's tests cases after the changes that enhanced
the segment-reaper not to be limited to objects up to 65 segments.

Also remove some tests cases that were already covered by the
TestObserver_processSegment_from_to test function.

Change-Id: I2385b406b50f7810ed22d3297b903a993f83fcfe
2020-05-20 10:10:51 +00:00
Natalie Villasana
c7cbdcbccf cmd/internal: request lowercase access name
Viper has a feature/bug where all YAML config keys are cast
to lowercase (see https://github.com/spf13/viper/issues/260
and https://github.com/spf13/viper/issues/411).
Since we use Viper to load our config values, it doesn't seem
like there's an easy way to preserve case sensitivity for the
access names chosen by users right now. Adding this prompt
should help user experience by clarifying that all access
names must be lowercase.

Change-Id: I47e8344bb0ca7e78458405496f20e78e3c9f9a88
2020-05-19 16:04:35 +00:00
Ivan Fraixedes
cbdd3bfd7b cmd/segment-reaper: Rename test name to what it tests
Rename a test function to indicate the function that it actually tests
rather than indicating the function that calls the function under test.

Change-Id: I2c1fe165e3acf952993a9a65dfbfbdcda0d1cccc
2020-05-19 10:06:56 +00:00
Michal Niewrzal
ac375d37bc satellite/payments: remove mockpayments and add Stripe client mock
instead

Change-Id: If3496f6abc16da90d2b43fa0c5be356847a39507
2020-05-19 09:35:37 +02:00
Stefan Benten
1de813d22e
cmd/storj-admin: initial commit of new tool (#3890) 2020-05-18 22:16:15 +02:00
Ivan Fraixedes
5ea82dfb2e cmd/segment-reaper: Report zombie segment's size
Makes the detect command of the segments reaper to report the size of
each zombie segment.

Change-Id: Ibded5f4d1b53f6d2805dac96fd306452af7f2524
2020-05-18 18:25:14 +00:00
Ivan Fraixedes
d27cb60f81 cmd/segment-reaper: Remove unneeded test logic
After the segment-reaper was enhanced to support objects with any number
of segments, the test helper function to generate test data doesn't have
to support to generate objects with a number of segments up to or over
the previous limit.

The commit remove that unnecessary test logic.

Change-Id: I2897c5a96fb133f61de2cdb0ef7d13ee5e0151e8
2020-05-18 16:39:48 +00:00
Kaloyan Raev
aac1e3c45f satellite/payments: move inspector commands to satellite cli
This allows to seeing logs in the output of the invoice commands.

Existing ensure-stripe-customer commands is moved from the 'reports' to
the new 'billing' root command.

Change-Id: I752c7ab6ca59bfac8e0f174a45d2ab45fc18e467
2020-05-18 16:12:45 +00:00
Stefan Benten
26f36d590e
cmd/satellite: ensure we only create stripe customers for activated users (#3884) 2020-05-15 16:00:46 +02:00
Michal Niewrzal
2eb2c25e51 satellite/payments/stripecoinpayments: add StripeClient interface to be
able to cover more testing scenarios

Currently, its hard to implement test suite for payments because
mockpayments is on to high level and we cannot emulate many things e.g.
adding credit card. This change is first to be able to add mock for
Stripe client and do more granular tests.

Change-Id: Ied85d4bd0642debdffe1161657c1e475202e9d23
2020-05-15 10:52:44 +02:00
Kaloyan Raev
49571f1a23 satellite/payments: all invoice commands require period
To avoid including multiple months in a single invoice, we need all
inspector's invoice commands to run in for specific period.

See https://storjlabs.atlassian.net/browse/USR-725

Change-Id: I3637dc189234f02350daca8d897c21765762ea55
2020-05-14 11:50:19 +00:00
Ethan
159df8b2e4 Add logging listener for retrieving and setting log levels
See https://storjlabs.atlassian.net/browse/SM-752

These changes allow us to change the log level at runtime through a handler off of the debug endpoint.

Examples of changing the log level on storj-sim

To get the current level for the satellite api process:
curl -XGET 'http://127.0.0.1:10009/logging' --header 'Content-Type: text/plain'

To change the log level:
curl -XPUT 'http://127.0.0.1:10009/logging' --header 'Content-Type: text/plain' --data-raw '{"level":"error"}'

Change-Id: I05d164b290929fa06b6d78c01075ee41f8238044
2020-05-12 16:38:06 -04:00
Ivan Fraixedes
c58dccb7a8
cmd/segment-reaper: Remove segments of skipped objects
Reorganize some operations in the observer.processSegment method:

* to minimal reduce the memory usage removing the segments of the
  objects marked as skipped. Skipped objects aren't discarded of the
  analysis stage so the segments aren't needed.
* returns earlier when an object is skipped because isn't needed a
  further processing.

Change-Id: I210a26c394477ee411ff7f640507dcc07733a47f
2020-05-12 16:12:44 +02:00
Ivan Fraixedes
9314092841
cmd/segment-reaper: Rename Bitmask to BitArray
Rename a test name and an error class because the bitmask was renamed to
bitArray in a previous commit, however, they weren't updated.

Change-Id: I92afcf747dec9e0a15d4c59b896e07b942f3518b
2020-05-12 15:22:30 +02:00
Ivan Fraixedes
50e8314279 cmd/segment-reaper: Fix bitArray.IsSequence method
The bitArray.IsSequence method had a bug that caused that sometimes the
test "Bitmask/IsSequence/sequence starts at other index than 0" to fail
due to it uses random values.

The test mostly found a corner case, I've added new tests to catch the
bug and then apply the fix.

Change-Id: If01dc07006d261ba40bbd99d36ef776e09ed3efc
2020-05-12 13:02:52 +00:00
Egon Elbre
ec589a8289 all: fix comments about grpc
Change-Id: Id830fbe2d44f083c88765561b6c07c5689afe5bd
2020-05-11 13:05:34 +03:00
Egon Elbre
e6d5ce6b77 all: remove grpc
It seems everyone has migrated to drpc.

Change-Id: Ica6b2d0bdef68c6603083f2963458843eca71e9e
2020-05-10 06:36:09 +00:00
Bill Thorp
4b612a7906 cmd/segment-reaper: add support for >65 segments
Change the bitmask used by segment reaper to use []byte rather than uint64

This passes tests but I have literally no clue how to integration test this.

Change-Id: I393f4598b27cae6e427da2190dd3109bca721c34
2020-05-09 10:45:26 +00:00
Matt Robinson
94503db5a6
cmd/satellite: Enable admin service like other services, also sort environment variables (#3874)
Change-Id: I67592b7bf34a0e2ce9a445b58b7fa34396aa8c5d

Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2020-05-08 07:38:21 -04:00
Kaloyan Raev
b5433322b4 cmd/storj-sim: enable open user registration
Change-Id: I695664279f42209245a82cc5bf103d037b844c32
2020-05-05 13:03:02 +00:00
Jeff Wendling
f93d764998 cmd/uplink: properly path escape shares
Fixes #3829

Change-Id: I0f900a392239fcc064cea968fc7a5bc373ee26a1
2020-05-04 17:00:25 +00:00
Yingrong Zhao
0b80d60491 cmd/uplink: disable tlemetry if user opt out
Change-Id: I793e6428d04e4f422d1f8292e99b233c70d9dd6e
2020-05-04 14:46:49 +00:00
Egon Elbre
c630cf2490 storagenode/pieces: implement buffering for writing
Currently uploads can cause a lot of IOPS, reduce this by introducing a
in-memory buffer on-top of the file.

Change-Id: I5f4e3e01c0a36258271d180b922107de447bcb59
2020-05-04 06:01:32 +00:00
Yaroslav Vorobiov
516b8cf2be storagenode-updater: add recovery for windows service restart
Reimplement windows service restart part using svc, add recovery
for failed service startup. Added restart-service cmd, to execute
self restart in a separate process.

Addressed issues:
https://storjlabs.atlassian.net/browse/SG-49
https://storjlabs.atlassian.net/browse/SG-136
https://storjlabs.atlassian.net/browse/SG-137

Change-Id: Ic51d9a99e8c1c10800c6c60ff4e218321c674fea
2020-05-01 09:07:05 +00:00
Egon Elbre
8928399d02 all: rename CreateTables to MigrateToLatest
CreateTables hasn't been quite true for a while now, rename to
MigrateToLatest to be clearer in it's behavior.

Change-Id: Ida48e95122a5d9b7a814e922d3698e00024a2ba7
2020-04-30 07:21:17 +00:00
Michal Niewrzal
b2acd93a78 storagenode/inspector: adjust Uptime type change in protobuf
Change-Id: I4f5110b534ac4f419b74f1a3dd72f8600e0a53a8
2020-04-29 09:28:37 +00:00
Yingrong Zhao
c5309a3f91 cmd/uplink: set sample rate for tracing to be 1 when tracing is enabled
when tracing is enabled, we should also set sampling rate to
a non-zero value. For now, we will set it to 1.
Uplink CLI users should be able to override it with the sample
flag.

Change-Id: I8bcf514fb14c2a1c4349b7957dd24ec23e4a85e5
2020-04-28 20:15:28 +00:00