Commit Graph

6342 Commits

Author SHA1 Message Date
Vitalii Shpital
1def7b0ec3 web/satellite, testsuite/ui/satellite: added tests for invalid sign up credentials and satellites dropdown
Added tests for signup with invalid email or password.
Added test for signup screen's satellites dropdown.

Change-Id: I76d975039543e315b3e9c9416e3ec1f2a3331a6a
2021-09-29 15:20:24 +03:00
Egon Elbre
f52f5931da ci: don't fail when nothing to delete
Change-Id: Ided795c3b60f053dfaff89be6e944cf2cce5faba
2021-09-29 13:41:50 +03:00
Clement Sam
0d58172c38 storagenode: add doc.go files for sno packages
Change-Id: I23d4b8b462e1b03718d0c4801cc2aaff520e7356
2021-09-29 08:24:56 +00:00
paul cannon
c053bdbd70 satellite/satellitedb: prepare to remove big.Float from db
Why: big.Float is not an ideal type for dealing with monetary amounts,
because no matter how high the precision, some non-integer decimal
values can not be represented exactly in base-2 floating point. Also,
storing gob-encoded big.Float values in the database makes it very hard
to use those values in meaningful queries, making it difficult to do
any sort of analysis on billing.

Now that we have amounts represented using monetary.Amount, we can
simply store them in the database using integers (as given by the
.BaseUnits() method on monetary.Amount).

We should move toward storing the currency along with any monetary
amount, wherever we are storing amounts, because satellites might want
to deal with currencies other than STORJ and USD. Even better, it
becomes much clearer what currency each monetary value is _supposed_ to
be in (I had to dig through code to find that out for our current
monetary columns).

Deployment
----------

Getting rid of the big.Float columns will take multiple deployment
steps. There does not seem to be any way to make the change in a way
that lets existing queries continue to work on CockroachDB (it could be
done with rules and triggers and a stored procedure that knows how to
gob-decode big.Float objects, but CockroachDB doesn't have rules _or_
triggers _or_ stored procedures). Instead, in this first step, we make
no changes to the database schema, but add code that knows how to deal
with the planned changes to the schema when they are made in a future
"step 2" deployment. All functions that deal with the
coinbase_transactions table have been taught to recognize the "undefined
column" error, and when it is seen, to call a separate "transition shim"
function to accomplish the task. Once all the services are running this
code, and the step 2 deployment makes breaking changes to the schema,
any services that are still running and connected to the database will
keep working correctly because of the fallback code included here. The
step 2 deployment can be made without these transition shims included,
because it will apply the database schema changes before any of its code
runs.

Step 1:

    No schema changes; just include code that recognizes the
    "undefined column" error when dealing with the
    coinbase_transactions or stripecoinpayments_tx_conversion_rates
    tables, and if found, assumes that the column changes from Step
    2 have already been made.

Step 2:

    In coinbase_transactions:

     * change the names of the 'amount' and 'received' columns to
       'amount_gob' and 'received_gob' respectively
     * add new 'amount_numeric' and 'received_numeric' columns with
       INT8 type.

    In stripecoinpayments_tx_conversion_rates:

     * change the name of the 'rate' column to 'rate_gob'
     * add new 'rate_numeric' column with NUMERIC(8, 8) type

    Code reading from either of these tables must query both the X_gob
    and X_numeric columns. If X_numeric is not null, its value should
    be used; otherwise, the gob-encoded big.Float in X_gob should be
    used. A chore might be included in this step that transitions values
    from X_gob to X_numeric a few rows at a time.

Step 3:

    Once all prod satellites have no values left in the _gob columns, we
    can drop those columns and add NOT NULL constraints to the _numeric
    columns.

Change-Id: Id6db304b404e6fde44f5a8c23cdaeeaaa2324f20
2021-09-29 00:23:44 +00:00
paul cannon
a16aecfa96 satellite/payments: specialized type for monetary amounts
Why: big.Float is not an ideal type for dealing with monetary amounts,
because no matter how high the precision, some non-integer decimal
values can not be represented exactly in base-2 floating point. Also,
storing gob-encoded big.Float values in the database makes it very hard
to use those values in meaningful queries, making it difficult to do
any sort of analysis on billing.

For better accuracy, then, we can just represent monetary values as
integers (in whatever base units are appropriate for the currency). For
example, STORJ tokens or Bitcoins can not be split into pieces smaller
than 10^-8, so we can store amounts of STORJ or BTC with precision
simply by moving the decimal point 8 digits to the right. For USD values
(assuming we don't want to deal with fractional cents), we can move the
decimal point 2 digits to the right.

To make it easier and less error-prone to deal with the math involved, I
introduce here a new type, monetary.Amount, instances of which have an
associated value _and_ a currency.

Change-Id: I03395d52f0e2473cf301361f6033722b54640265
2021-09-28 23:27:44 +00:00
Yingrong Zhao
c911360eb5 satellite/metainfo: separate burst limit from rate limit config
This PR utilize the new burst limit column from projects table to allow
control on the limit for request per seconds and token bucket size
When no burst limit is explicitly set, rate limit is applied to both so
we don't limit how quickly request can be made in a second.

Change-Id: I883235c60c5d6416aeadd1c80ed2ebd193aa4d9f
2021-09-28 22:47:41 +00:00
dlamarmorgan
ab7e81c279 satellite/accounting: update GetProjectBandwidthTotals function to use the less expensive project_bandwidth_rollups table
Verbose

Change-Id: I450360cbd51feb3afa8f5f63d5727e08acafbeaa
2021-09-28 12:17:38 -07:00
Egon Elbre
6d3fd33ca3 satellite/metabase/segmentloop: start immediately on manual trigger
Currently loops wait for the coalesce duration for TriggerWait.
Let's skip the coalesce when we trigger it manually.

Change-Id: If5bacd4e263d233f1f3ea41b989922d2ed5a48d4
2021-09-28 18:32:03 +00:00
Yingrong Zhao
4db80773ef satellite/satellitedb: add burst_limit for project
In order to limit the amount of overall requests a user can issue in a
time span, we need to have the ability to define such limit separate
from per second request rate.
This PR adds a new column on the projects table to store the burst limit
per project.

Change-Id: I7efc2ccdda4579252347cc6878cf846b85146dc7
2021-09-28 13:25:03 -04:00
Artur M. Wolff
cc9b845a83 ci: switch back to pulling gateway@main
Switch back to pulling gateway@main for continuous integration because
https://review.dev.storj.io/c/storj/gateway/+/5834 was merged. This
change is a partial revert of
https://review.dev.storj.io/c/storj/storj/+/5832.

Change-Id: I0a87f6f18a9a863fe92003b76c830335f9253ced
2021-09-28 15:01:04 +00:00
Egon Elbre
9c232cebe1 ci: make deletion a separate step
Deletion as a separate step so it's more understandable what's
actually happening.

Change-Id: Icb3cefe7e4093c56604af633ac0e02226bae42ac
2021-09-28 10:24:43 +00:00
Michał Niewrzał
512d0d61fb satellite/metrics: speedup test
Metrics observer is joining loop only for monitoring
and because of that it waits until non-monitoring
observer will join loop.
That was causing some delays in tests. To fix that
we are triggering non-monitoring observer manually.

Change-Id: Id93d59c9fd8d637dac0beea82ff2c40247d94b10
2021-09-28 11:39:37 +02:00
Michał Niewrzał
40084d51a1 satellite/metainfo: reduce number of testplanet runs
This should speedup deletion tests a little bit.

Change-Id: If7bc6863362ddbb1de6672c676cb08cf7e50beb4
2021-09-28 08:38:21 +00:00
Clement Sam
d397f6e2be docs: move contribution guide to repository root
Instead of having a guide for only the sno development, we need a main contributing guide which will show up when a contributor visits https://github.com/storj/storj/contribute.

Change-Id: I4477c3c124ad0d15b17b4733304333179f3d4084
2021-09-28 07:38:11 +00:00
Michał Niewrzał
1ed5db1467 satellite/metainfo: simplifying limits code
Its a very simple change to reduct code duplication.

Change-Id: Ia135232e3aefd094f76c6988e82e297be028e174
2021-09-28 06:22:13 +00:00
Clement Sam
118e64fa15 jenkinsfile: bump build timeout
Jenkins build consistently times out due to the enormous tests we have now.
Bumping the timeout to 40 fixes it for now.

Change-Id: Ie96cc5ec3e452744d5a7873853d14befe367bd7e
2021-09-27 18:31:16 +00:00
Egon Elbre
8036644422 ci: try fix builds
This chmod is combined with "Clean before checkout" behaviour
that deletes everything, except .git. This should give us
the sufficient ordering to ensure that everything has the
correct permissions.

Story so far:

Checkout with "Wipe out repository & force clone" didn't work,
because the chmod would've run after trying to delete.

Deleting as a post action doesn't work, because during timeouts
the always steps don't get executed sometimes.

We cannot easily run docker with non root, due to how our ci
image works.

Change-Id: Id4c1605d39ae8d7722ba3dd1ab5df8618de3309b
2021-09-27 16:18:04 +00:00
Egon Elbre
7fd3466925 ci: remove deleting workspace steps
Moved the deleting workspace to pipeline configuration.

Change-Id: I7c94d1a9bac514f1824988d21dfb3a144b10ca7e
2021-09-27 15:49:34 +03:00
Michał Niewrzał
0330871a3e satellite/metabase: add missing FinishMoveObject parameter
FinishMoveObject requires NewBucket parater
to be set and we missed it.

Change-Id: Ia8e66fbef3c0b9bcc7f20b5baa66aed380f8c8c5
2021-09-27 10:41:13 +02:00
Qweder93
4fefa36a79 satellite/metabase: NewBucket added to metabase/metainfo FinishMoveObject methods
server-side move extended with moving between buckets, for this reason
we change bucket name for object in db.

Change-Id: Ie21bcccc170e6ff14dcd8053fdb86fdf6d8438a0
2021-09-24 17:53:02 +03:00
Vitalii Shpital
9da3de1a89 web/satellite, testsuite/satellite/ui: removed onb CLI flow's Generate AG step and updated tests
Removed Generate access grant step.
Updated according test and added Back button click test logic.

Change-Id: I736eb4ecb4c3a2437c78af3f42da160b0f147d26
2021-09-24 13:21:40 +00:00
Clement Sam
7d90770f8e .github: add issue templates
Is this a good idea?

Change-Id: I0e36bcbd1aea1d20c0b55ce8856c607a8c775e61
2021-09-24 08:42:48 +00:00
Yaroslav Vorobiov
469ae72c19 satellite/repair: update audit records during repair
Change-Id: I788b2096968f043601aba6502a2e4e784f1f02a0
2021-09-24 00:48:13 +00:00
Moby von Briesen
030ab669fa web/satellite: Route all child routes to object browser
Also disable some warning/error checks in jestSetup.ts that are causing
build to fail.

Change-Id: Iafaa05af4058bd0f6cc1ec94f6d85b11672612f3
2021-09-23 18:22:27 -04:00
Clement Sam
0e1c3cb811 docs: add contributing guide for SNO development
Change-Id: I85d84ce9dcf975287b30900e4b895cf7e1461530
2021-09-23 20:38:22 +00:00
Jeremy Wharton
b97479e36b web/satellite: Reset reCAPTCHA response token on error
Resets the reCAPTCHA response token when an error is returned from
the satellite. This prevents old response tokens from being used
in later requests.

Change-Id: Icabb53bb6b307d22c21ae960075d552ba739aa86
2021-09-23 14:35:34 +00:00
Egon Elbre
8ef03b0967 ci: cleanup ws before build, fix gateway install
Jenkins uses the same folder for different PR-s. Sometimes other runs do
not cleanup after themselves (e.g. timeout), hence add a cleanWs step to
ensure we delete files in the workspace.

gateway-st introduced a replace directive in go.mod, which does not work
with go install. Hardcode to the last version without the directive.
Using this fix to unblock ci builds.

Change-Id: I5e5d75bf47e30a5a8b6d835867c0c9176f25e08a
2021-09-23 15:57:47 +03:00
Egon Elbre
bd4017b2a6 ci: fix logical races between tools
Currently some building may generate some additional files,
similary, we want to run check-clean-directory after everything
has finished.

Change-Id: I4d700896094257e65cc5197c468f6d752024ddd9
2021-09-23 12:16:31 +03:00
Vitalii Shpital
99914dfc02 web/satellite: fix for button label uppercase issue
Added prop for VButton component to make button labels uppercase.
It uses text-transform css property which is better for screen readers.

Change-Id: I2a771af954a3bb9846a58dac9c2a1769e13f6e5c
2021-09-22 18:51:05 +00:00
Qweder93
cae08d816c satellite/metabase: FinishMoveObject segment query improved
Change-Id: I198033d5763e6f386d3090fed49edcbfd95ff536
2021-09-22 12:51:48 +03:00
Egon Elbre
71eb184ef3 storagenode/piecestore: simplify TestTooManyRequests
Change-Id: I1452735bef206bc8cf9bde72fc503fec3ae5b067
2021-09-21 17:18:37 +03:00
Egon Elbre
36911b44a3 satellite/accounting/tally: make tests faster
Change-Id: I706399043479fc4c4ae3abc0bc70ae2db0318096
2021-09-21 17:18:37 +03:00
Egon Elbre
df09e7d1f7 satellite/metainfo: ensure storagenodes finish work for test
Some processing inside storagenodes is async compared to uplink upload
and download, hence we need to explicitly wait for storagenodes to
finish their pending work before flushing orders to the satellite.

Hopefully this fixes TestAttributionReport flakiness.

Change-Id: I77c651ab6471ae094b5c21d1ab3860c96cb0d039
2021-09-21 17:18:37 +03:00
Vitalii Shpital
8b91c55ec4 web/satellite, satellite/console: return old onboarding flow with feature flag
Added new feature flag for switching between different flows.
Got back old onboarding flow.

Change-Id: Idbf23ba8ff36900a47e3b4ebeb34b78351202e29
2021-09-21 11:37:30 +00:00
Michał Niewrzał
38366914b2 satellite/metainfo: add position to begin move results
BeginMoveObject response was missing Position.

Change-Id: I2a91f0b1ac39c476731a2755754e46970c8657be
2021-09-21 10:23:33 +00:00
Egon Elbre
32cee1e572 satellite/metabase/segmentloop: ensure we shutdown on ctx err
Not all errors from RunOnce can be retried. The context can be cancelled
with several different errors, e.g. timeout. Ensure we stop the loop
when context has errored, because none of the queries will succeed
when it has failed.

Change-Id: If3ff11f11a6f43c0d67633be1cfaf23e3e9e55f3
2021-09-21 10:02:00 +00:00
Vitalii Shpital
69d3a839fc web/satellite: fix for onb CLI upload step image
Fixed SVG image usage ID for onb CLI upload step

Change-Id: Ieb94902de5702594edb20817541a8d968c0ad4f1
2021-09-20 20:07:48 +03:00
Vitalii Shpital
91c69a8afc web/satellite: fix for encrypt your data screen
Fixed issue for Encrypt your data screen. It was impossible to proceed further into object browser after selecting Generate passphrase option

Change-Id: Ibddda3c6bd7fce34bd1c629bbbb2d87529964b8a
2021-09-20 19:34:34 +03:00
crawter
5e4b196b11 satellite/metainfo: finish move object
Second method needed to perform server-side move. It updates
metadata key and nonce and all segments key and nonces.

Change-Id: Ia43b26622a13048269f0ae9e1524b345db112adb
2021-09-20 16:21:01 +03:00
Vitalii Shpital
9153b221fd testsuite/ui/satellite: updated pnboarding CLI tests and improved selectors
Updated onb CLI flow test.
Updated tests to use aria-roledescription as selector.

Change-Id: I1442592e1938fbe730064977dc9b78266e403706
2021-09-17 15:11:47 +00:00
Vitalii Shpital
853a7536f6 web/satellite: reworked onb CLI flow to correspond v1.1
Removed useless screens and fixed routing.
Removed useless logic from common component.

Change-Id: I73a1af5803f866526d3ae327e832654501e4f8bd
2021-09-17 15:11:41 +00:00
Yingrong Zhao
0b500a30e4 satellite/audit: move audit metrics out of reporter
Since we are sharing the reporting logic between repair and audit. We
need to remove metric reporting logic in reporter.

Change-Id: Ib87295ab19079329e7438327d785a7f5c21d3b21
2021-09-16 17:58:56 +00:00
Malcolm Bouzi
252b78580d satellite/console: add status check to user authorization to ensure deleted accounts cannot perform actions
Change-Id: I2b37101335d154bf4bc93ec85c06ff62e555c9ec
2021-09-16 17:06:34 +00:00
Egon Elbre
09e1ff7fc4 web/satellite: fix promise usages
Change-Id: I6496ebdd57667e8222b2bfbe6174860997b6bb3f
2021-09-16 16:32:21 +00:00
Egon Elbre
6e660cecdd Jenkinsfile: test cross-compile and bump deps
Change-Id: I47091de2e80bd96fcada616f75e1db07a59fb0c1
2021-09-16 18:59:31 +03:00
Egon Elbre
b2d724962c cmd/storagenode-updater: avoid depending on the storagenode code
Change-Id: If81927aee43ccdd678a0e42f91632f85e9fdefab
2021-09-16 11:44:55 +00:00
Egon Elbre
6b3718f33e Makefile: skip darwin build for the time being
Mac builds need to be reworked to work with latest Go version.

Change-Id: Ifbe37cf8fb41c5e0d806c518a3be32e093b448b0
2021-09-16 12:53:41 +03:00
Yingrong Zhao
f0b73b4f45 mod: bump drpc, uplink and common
Updates:
- drpc to v0.0.25
- uplink to latest
- common to latest

This will fix the crash on UDP network temporary error for windows and a
rare deadlock during shutdown

Change-Id: I7e9c4ace22665c5471f19f9b8bf63aec4d68ceb3
2021-09-16 07:42:32 +00:00
Jeff Wendling
b160ec4c1b satellite/orders: bound RollupsWriteCache flushes
In the situation where the flushes take longer than the incoming
rate of writes, the RollupsWriteCache will take every connection
in the database pool and use them forever. Instead of doing that
and taking down satellite availability, bound the number of flush
operations that it will perform and drop incoming writes earlier
to keep memory usage constant.

Adds monitoring events for if any flushes or updates are lost.

Change-Id: I81b169b73501ee9b999f4b03d1e79645fc56f167
2021-09-15 19:14:39 +00:00
Vitalii Shpital
86845351c3 web/satellite: reworked upgrade to Pro account modal
Reworked the modal to correspond to latest designs.
Added prices and updated copies.

Change-Id: I6e59fdfe4222e02fcc35ac5d594bdc0bdf52a63c
2021-09-15 19:47:56 +03:00