Commit Graph

801 Commits

Author SHA1 Message Date
Cameron Ayer
ca0c1a5f0c storagenode/{monitor,pieces}, storage/filestore: add loop to check storage directory writability
periodically create and delete a temp file in the storage directory
to verify writability. If this check fails, shut the node down.

Change-Id: I433e3a8d1d775fc779ae78e7cf3144a05ffd0574
2020-08-31 21:20:49 +00:00
stefanbenten
086a3d5348 satellite/{payments,admin}: add deletion of user creditcards on account deletion
Change-Id: I38bf7e3995846150268f7b88a70f75b0ac871b62
2020-08-27 10:18:19 +00:00
Cameron Ayer
3e343b683b cmd/segment-reaper: add metrics for zombie segments count
Change-Id: I106c6795946283165ba3de8465e5898346da1a3f
2020-08-26 18:42:59 +00:00
Egon Elbre
94a09ce20b all: add missing dots
Change-Id: I93b86c9fb3398c5d3c9121b8859dad1c615fa23a
2020-08-11 17:50:01 +03:00
Qweder93
6e90ca8b0e cmd/storagenode: CLI dashboard last contact renamed to status
Change-Id: I29bcd62d0d2970d6294dc642b442b60a3331bbd6
2020-08-06 13:54:07 +00:00
Kaloyan Raev
edfd3d7661 satellite/payments: delete credits and credits_spendings db tables
Jira: https://storjlabs.atlassian.net/browse/USR-822

This the last step of dropping these 2 db tables. It also deletes all
code associate with them.

Change-Id: I8be840dc2a7be255cf6308c9434b729fe4d9391e
2020-07-30 12:19:57 +03:00
JT Olio
c89746a383 storagenode: use hardcoded metric application name
Change-Id: If11f226d03bc4bf70e2465569f8d578750718ae7
2020-07-28 16:24:15 +00:00
Bill Thorp
b265b7f555 satellite/console: make paywall optional
Add a config so that some percent of users require credit cards /
account balances
in order to create a project or have a promotional coupon applied

UI was updated to match needed paywall status

At this point we decided not to use a field to store if a user is in an
A/B
test, and instead just use math to see if they're in a test.  We decided
to use MD5 (because its in Postgres too) and User UUID for that math.

Change-Id: I0fcd80707dc29afc668632d078e1b5a7a24f3bb3
2020-07-28 10:57:49 +00:00
Kaloyan Raev
cb0caa2e25 satellites/payments: resolve data race in StripeMock
Change-Id: I0dbcf043e88edab12704742ad9653c62fd7b08f4
2020-07-28 09:53:50 +00:00
Yaroslav Vorobiov
4d2a505788 storagenode/db: explicitly open and create dbs
To prevent storagenode from implicitly recreating missing dbs and storage,
as such behaviour leads to audit failures. Do not allow storagenode to
start if any of dbs or storage is missing, corrupted, or dedicated storage disk is
unmounted, to get downtime instead.

Change-Id: Ic64e1f0ff4d8ef5b2fddbe7a7e53df4f4bd8652e
2020-07-24 14:08:47 +03:00
Egon Elbre
b67d7ecbc5 cmd/storagenode,storage/cockroachkv: better error handling
Change-Id: I6646aa046dc365c0dee38f23041be4fc2defb759
2020-07-16 20:03:50 +03:00
Egon Elbre
d8dcae3075 all: fix error checking
Change-Id: Ia0da1bbd6ce695139922f94096c2419281905e32
2020-07-16 19:13:14 +03:00
Egon Elbre
080ba47a06 all: fix dots
Change-Id: I6a419c62700c568254ff67ae5b73efed2fc98aa2
2020-07-16 14:58:28 +00:00
stefanbenten
257855b5de all: replace == comparison with errors.Is
Change-Id: I05d9a369c7c6f144b94a4c524e8aea18eb9cb714
2020-07-14 15:50:25 +00:00
stefanbenten
1149417615 satellite/admin: cleanup parameter handling
We passed in revocationDB and metainfoDB for no reason.
Lets remove it from the dependency list to further reduce the footprint.

Change-Id: Ic0317bb92670fbd305d4a8b0ed1cb82858e2f6d3
2020-07-14 13:53:09 +02:00
Yaroslav Vorobiov
75ab3507dd storagenode-updater: process storagenode config
Change-Id: I452dd4422ae660ca46d7f4525358c8e809ba0d75
2020-07-14 13:30:14 +03:00
Egon Elbre
29ccda5d47 cmd/uplink/cmd: remove spf13/cast dependency
Change-Id: I8013014b0b37966128e673155d851705ee1d43db
2020-07-13 11:10:48 +00:00
Kaloyan Raev
12a15e5a6a satellite/payments: remove migare-credits billing command
This was a one-time command and it has been already executed on all
production satellites.

Change-Id: Ic58a151c3e5da9c139df875f6517f37275a1ebf5
2020-07-07 11:47:53 +03:00
Egon Elbre
c40d5043e1 cmd/uplink: pass useragent flag to library
Change-Id: I8cf12c978c01eca968086560be0c28809eb2ba7d
2020-07-03 16:41:35 +03:00
Kaloyan Raev
091b49b921 cmd/satellite: command to move unspent bonuses to Stripe balance
Jira: https://storjlabs.atlassian.net/browse/USR-821

The `migrate-credits` billing command checks the available credits
balance for all users and moves it to the Stripe balance by creating a
new credit balance transaction.

Change-Id: Iafc7b95a4edad47f7c145a22e210f8c821ac183d
2020-06-24 13:20:24 +00:00
Ivan Fraixedes
37cfc01ac6 cmd/storagenode: Add long description to ge command
Add a long description to the graceful exit command to clearly mention
that the command is interactive asking which satellite the SNO wants to
exit.

Change-Id: Icd4056a470e707322f600133e63d9dc56eb877b7
2020-06-24 08:47:52 +00:00
Isaac Hess
cad21f11e5 cmd/uplink: Add revoke command
Change-Id: I8ae74da5c9171429b970d76598793a3227ed4d11
2020-06-22 13:50:07 -06:00
Isaac Hess
2d727bb14e satellite: Check macaroon revocation
When a request comes in on the satellite api and we validate the
macaroon, we now also check if any of the macaroon's tails have been
revoked.

Change-Id: I80ce4312602baf431cfa1b1285f79bed88bb4497
2020-06-22 13:50:07 -06:00
Stefan Benten
433fc91054
cmd/satellite: clean up entrypoint (#3910) 2020-06-22 21:06:31 +02:00
JT Olio
2bd9067ad5 cmd/uplink: tighter linksharing security
when a user runs `uplink share`, they get a bunch of results back,
given their configuration and existing access. one of the results
is a URL for in-browser sharing and hosting of the file.

first off, we want to make sure this URL is read only. we want to
avoid a situation where someone post this URL to some public
location, not realizing the access allows writes or deletes. if
a user really wants a URL with write/delete access, they can
construct it themselves.

secondly, we want to make sure the url is sharing a single path or
path prefix. having a url for multiple paths/path prefixes
indepedently again can be constructed of course, but should not
be the default behavior

Change-Id: I2ca2ebeea9f1c7d4bfbd7a437a32dc7a3b2a32cc
2020-06-16 15:02:53 -06:00
Michal Niewrzal
7a0778fac4 cmd/satellite: choose correct Stripe client also for commands
Commands were using always real Strip client but for e.g. storj-sim
should use mock by default.

Change-Id: Ifd3c02028312d2e8d6d73d67b2dadadcd69077c8
2020-06-15 11:03:14 +00:00
Kaloyan Raev
adb5a25b63 cmd/satellite: revert 'organize subcommands'
Reverts https://review.dev.storj.io/c/storj/storj/+/2074

That change broke all billing commands with `sql: database is closed`.
Fixing the issue does not make sense as the whole setup of billing
commands is now error-prone. It's better to revert it.

Change-Id: Id13f817b73c7313ba60181f740b0712e4bdce077
2020-06-15 10:06:30 +00:00
stefanbenten
44792f7b45 cmd/satellite: add finalize invoices command
Change-Id: Ie57dedc9b9c8fe133dede573c921dfee0883e5ff
2020-06-10 14:48:15 +00:00
stefanbenten
fdbb2c3ed6 cmd/satellite: organize subcommands
Change-Id: If45575656e5e16642bdee4b6e6867e4f1d824f9a
2020-06-10 14:35:21 +00:00
Qweder93
e52809d53e cmd/storagenode: add check if satellites available to gracefulexit
Change-Id: I8747507593d810bbdec0d140de0600ee147011c3
2020-06-10 13:38:36 +00:00
JT Olio
d9b1c4520e storagenode-updater: fix logic to use minimum version correctly
See:
  https://review.dev.storj.io/c/storj/storj/+/2052
  https://review.dev.storj.io/c/storj/private/+/2053

Change-Id: I30952c6e3893194ca876c12ecb7e53944e66aa2e
2020-06-08 17:43:10 +00:00
Yingrong Zhao
175f048aa7 scripts;cmd/storj-sim: include satellite node id in satellite address
Expose satellite ID from storj-sim so we can have access to it when
changing satellite address.

Change-Id: Ife816f4d35eae2d0bc9f9ad592fe75d73d93d9ff
2020-06-05 09:44:19 -04:00
Egon Elbre
34db4a80fd ci: fix staticcheck failures
Change-Id: I176fb24214755a1940a0a1a4e9cc8e39f184870b
2020-06-05 13:15:34 +00:00
Egon Elbre
0de1b5fa9f ci: fix golangci-lint failures
Change-Id: Ia8c4dd03439ce4e3f85890d45255b8ea1c94b50a
2020-06-05 12:55:35 +00:00
Michal Niewrzal
056ae7ffa8 scripts/test/rollingupgrade: set correct satellite address for imported
access

Addition: use always latest gateway release

Change-Id: I5e5231e6da4b6f7900cb71bb6e227901474270ea
2020-06-04 15:04:10 +00:00
Michal Niewrzal
79a562c3ed cmd/uplink/cmd: improve uplink remove bucket command with force flag
* don't stop on single delete failure
* display number of successful and unsuccessful deletes

Change-Id: Id7f29055f250aad515e46622c2bfd3596055b93a
2020-06-01 07:56:14 +00:00
Michal Niewrzal
48e67a29a9 cmd/storj-sim: remove old libuplink from storj-sim
Change-Id: I146a2d7947d1f358e7bdd8acba68eb3fee68539d
2020-05-29 08:39:37 +00:00
Michal Niewrzal
84892631c8 private/testplanet: remove old libuplink from testplanet
Change-Id: Ib1553f84d0b3ae12a5b00382f0f53357b6a273e2
2020-05-28 13:50:23 +00:00
Michal Niewrzal
a4c19b3359 cmd/uplink/cmd: remove port detection when selecting satellite while
setup

We don't have default ports as a part of configuration anymore because
satellite-addr flag was removed.

Change-Id: Ibf9fc4b399beaf51ebb9461de2d8994a322f9686
2020-05-26 13:25:51 +02:00
Stefan Benten
3da100c242
cmd/satellite: Install curl per default to docker image (#3899) 2020-05-25 02:39:21 +02:00
Michal Niewrzal
340700eda9 cmd/uplink/cmd: remove old libuplink from Uplink CLI code
Change-Id: I38444a67e3bdec5f63c56895b5fc362464e7dce1
2020-05-22 09:08:58 +00:00
Bill Thorp
bcb867f1de cmd/uplink: add UserAgent flag for partner value attribution
This changeset allows a user agent string to be set during uplink setup, which
is thereafter used for partner value attribution.  EG

uplink setup --client.user-agent ”MyCompany”

Change-Id: Iefa8755fccc06acb8a303a342b943cece44a81f7
2020-05-22 07:39:14 +00:00
Ivan Fraixedes
963db8c738 cmd/segment-reaper: Remove unneeded test cases
Remove unneeded observer's tests cases after the changes that enhanced
the segment-reaper not to be limited to objects up to 65 segments.

Also remove some tests cases that were already covered by the
TestObserver_processSegment_from_to test function.

Change-Id: I2385b406b50f7810ed22d3297b903a993f83fcfe
2020-05-20 10:10:51 +00:00
Natalie Villasana
c7cbdcbccf cmd/internal: request lowercase access name
Viper has a feature/bug where all YAML config keys are cast
to lowercase (see https://github.com/spf13/viper/issues/260
and https://github.com/spf13/viper/issues/411).
Since we use Viper to load our config values, it doesn't seem
like there's an easy way to preserve case sensitivity for the
access names chosen by users right now. Adding this prompt
should help user experience by clarifying that all access
names must be lowercase.

Change-Id: I47e8344bb0ca7e78458405496f20e78e3c9f9a88
2020-05-19 16:04:35 +00:00
Ivan Fraixedes
cbdd3bfd7b cmd/segment-reaper: Rename test name to what it tests
Rename a test function to indicate the function that it actually tests
rather than indicating the function that calls the function under test.

Change-Id: I2c1fe165e3acf952993a9a65dfbfbdcda0d1cccc
2020-05-19 10:06:56 +00:00
Michal Niewrzal
ac375d37bc satellite/payments: remove mockpayments and add Stripe client mock
instead

Change-Id: If3496f6abc16da90d2b43fa0c5be356847a39507
2020-05-19 09:35:37 +02:00
Stefan Benten
1de813d22e
cmd/storj-admin: initial commit of new tool (#3890) 2020-05-18 22:16:15 +02:00
Ivan Fraixedes
5ea82dfb2e cmd/segment-reaper: Report zombie segment's size
Makes the detect command of the segments reaper to report the size of
each zombie segment.

Change-Id: Ibded5f4d1b53f6d2805dac96fd306452af7f2524
2020-05-18 18:25:14 +00:00
Ivan Fraixedes
d27cb60f81 cmd/segment-reaper: Remove unneeded test logic
After the segment-reaper was enhanced to support objects with any number
of segments, the test helper function to generate test data doesn't have
to support to generate objects with a number of segments up to or over
the previous limit.

The commit remove that unnecessary test logic.

Change-Id: I2897c5a96fb133f61de2cdb0ef7d13ee5e0151e8
2020-05-18 16:39:48 +00:00
Kaloyan Raev
aac1e3c45f satellite/payments: move inspector commands to satellite cli
This allows to seeing logs in the output of the invoice commands.

Existing ensure-stripe-customer commands is moved from the 'reports' to
the new 'billing' root command.

Change-Id: I752c7ab6ca59bfac8e0f174a45d2ab45fc18e467
2020-05-18 16:12:45 +00:00