Commit Graph

6082 Commits

Author SHA1 Message Date
Egon Elbre
9c232cebe1 ci: make deletion a separate step
Deletion as a separate step so it's more understandable what's
actually happening.

Change-Id: Icb3cefe7e4093c56604af633ac0e02226bae42ac
2021-09-28 10:24:43 +00:00
Michał Niewrzał
512d0d61fb satellite/metrics: speedup test
Metrics observer is joining loop only for monitoring
and because of that it waits until non-monitoring
observer will join loop.
That was causing some delays in tests. To fix that
we are triggering non-monitoring observer manually.

Change-Id: Id93d59c9fd8d637dac0beea82ff2c40247d94b10
2021-09-28 11:39:37 +02:00
Michał Niewrzał
40084d51a1 satellite/metainfo: reduce number of testplanet runs
This should speedup deletion tests a little bit.

Change-Id: If7bc6863362ddbb1de6672c676cb08cf7e50beb4
2021-09-28 08:38:21 +00:00
Clement Sam
d397f6e2be docs: move contribution guide to repository root
Instead of having a guide for only the sno development, we need a main contributing guide which will show up when a contributor visits https://github.com/storj/storj/contribute.

Change-Id: I4477c3c124ad0d15b17b4733304333179f3d4084
2021-09-28 07:38:11 +00:00
Michał Niewrzał
1ed5db1467 satellite/metainfo: simplifying limits code
Its a very simple change to reduct code duplication.

Change-Id: Ia135232e3aefd094f76c6988e82e297be028e174
2021-09-28 06:22:13 +00:00
Clement Sam
118e64fa15 jenkinsfile: bump build timeout
Jenkins build consistently times out due to the enormous tests we have now.
Bumping the timeout to 40 fixes it for now.

Change-Id: Ie96cc5ec3e452744d5a7873853d14befe367bd7e
2021-09-27 18:31:16 +00:00
Egon Elbre
8036644422 ci: try fix builds
This chmod is combined with "Clean before checkout" behaviour
that deletes everything, except .git. This should give us
the sufficient ordering to ensure that everything has the
correct permissions.

Story so far:

Checkout with "Wipe out repository & force clone" didn't work,
because the chmod would've run after trying to delete.

Deleting as a post action doesn't work, because during timeouts
the always steps don't get executed sometimes.

We cannot easily run docker with non root, due to how our ci
image works.

Change-Id: Id4c1605d39ae8d7722ba3dd1ab5df8618de3309b
2021-09-27 16:18:04 +00:00
Egon Elbre
7fd3466925 ci: remove deleting workspace steps
Moved the deleting workspace to pipeline configuration.

Change-Id: I7c94d1a9bac514f1824988d21dfb3a144b10ca7e
2021-09-27 15:49:34 +03:00
Michał Niewrzał
0330871a3e satellite/metabase: add missing FinishMoveObject parameter
FinishMoveObject requires NewBucket parater
to be set and we missed it.

Change-Id: Ia8e66fbef3c0b9bcc7f20b5baa66aed380f8c8c5
2021-09-27 10:41:13 +02:00
Qweder93
4fefa36a79 satellite/metabase: NewBucket added to metabase/metainfo FinishMoveObject methods
server-side move extended with moving between buckets, for this reason
we change bucket name for object in db.

Change-Id: Ie21bcccc170e6ff14dcd8053fdb86fdf6d8438a0
2021-09-24 17:53:02 +03:00
Vitalii Shpital
9da3de1a89 web/satellite, testsuite/satellite/ui: removed onb CLI flow's Generate AG step and updated tests
Removed Generate access grant step.
Updated according test and added Back button click test logic.

Change-Id: I736eb4ecb4c3a2437c78af3f42da160b0f147d26
2021-09-24 13:21:40 +00:00
Clement Sam
7d90770f8e .github: add issue templates
Is this a good idea?

Change-Id: I0e36bcbd1aea1d20c0b55ce8856c607a8c775e61
2021-09-24 08:42:48 +00:00
Yaroslav Vorobiov
469ae72c19 satellite/repair: update audit records during repair
Change-Id: I788b2096968f043601aba6502a2e4e784f1f02a0
2021-09-24 00:48:13 +00:00
Moby von Briesen
030ab669fa web/satellite: Route all child routes to object browser
Also disable some warning/error checks in jestSetup.ts that are causing
build to fail.

Change-Id: Iafaa05af4058bd0f6cc1ec94f6d85b11672612f3
2021-09-23 18:22:27 -04:00
Clement Sam
0e1c3cb811 docs: add contributing guide for SNO development
Change-Id: I85d84ce9dcf975287b30900e4b895cf7e1461530
2021-09-23 20:38:22 +00:00
Jeremy Wharton
b97479e36b web/satellite: Reset reCAPTCHA response token on error
Resets the reCAPTCHA response token when an error is returned from
the satellite. This prevents old response tokens from being used
in later requests.

Change-Id: Icabb53bb6b307d22c21ae960075d552ba739aa86
2021-09-23 14:35:34 +00:00
Egon Elbre
8ef03b0967 ci: cleanup ws before build, fix gateway install
Jenkins uses the same folder for different PR-s. Sometimes other runs do
not cleanup after themselves (e.g. timeout), hence add a cleanWs step to
ensure we delete files in the workspace.

gateway-st introduced a replace directive in go.mod, which does not work
with go install. Hardcode to the last version without the directive.
Using this fix to unblock ci builds.

Change-Id: I5e5d75bf47e30a5a8b6d835867c0c9176f25e08a
2021-09-23 15:57:47 +03:00
Egon Elbre
bd4017b2a6 ci: fix logical races between tools
Currently some building may generate some additional files,
similary, we want to run check-clean-directory after everything
has finished.

Change-Id: I4d700896094257e65cc5197c468f6d752024ddd9
2021-09-23 12:16:31 +03:00
Vitalii Shpital
99914dfc02 web/satellite: fix for button label uppercase issue
Added prop for VButton component to make button labels uppercase.
It uses text-transform css property which is better for screen readers.

Change-Id: I2a771af954a3bb9846a58dac9c2a1769e13f6e5c
2021-09-22 18:51:05 +00:00
Qweder93
cae08d816c satellite/metabase: FinishMoveObject segment query improved
Change-Id: I198033d5763e6f386d3090fed49edcbfd95ff536
2021-09-22 12:51:48 +03:00
Egon Elbre
71eb184ef3 storagenode/piecestore: simplify TestTooManyRequests
Change-Id: I1452735bef206bc8cf9bde72fc503fec3ae5b067
2021-09-21 17:18:37 +03:00
Egon Elbre
36911b44a3 satellite/accounting/tally: make tests faster
Change-Id: I706399043479fc4c4ae3abc0bc70ae2db0318096
2021-09-21 17:18:37 +03:00
Egon Elbre
df09e7d1f7 satellite/metainfo: ensure storagenodes finish work for test
Some processing inside storagenodes is async compared to uplink upload
and download, hence we need to explicitly wait for storagenodes to
finish their pending work before flushing orders to the satellite.

Hopefully this fixes TestAttributionReport flakiness.

Change-Id: I77c651ab6471ae094b5c21d1ab3860c96cb0d039
2021-09-21 17:18:37 +03:00
Vitalii Shpital
8b91c55ec4 web/satellite, satellite/console: return old onboarding flow with feature flag
Added new feature flag for switching between different flows.
Got back old onboarding flow.

Change-Id: Idbf23ba8ff36900a47e3b4ebeb34b78351202e29
2021-09-21 11:37:30 +00:00
Michał Niewrzał
38366914b2 satellite/metainfo: add position to begin move results
BeginMoveObject response was missing Position.

Change-Id: I2a91f0b1ac39c476731a2755754e46970c8657be
2021-09-21 10:23:33 +00:00
Egon Elbre
32cee1e572 satellite/metabase/segmentloop: ensure we shutdown on ctx err
Not all errors from RunOnce can be retried. The context can be cancelled
with several different errors, e.g. timeout. Ensure we stop the loop
when context has errored, because none of the queries will succeed
when it has failed.

Change-Id: If3ff11f11a6f43c0d67633be1cfaf23e3e9e55f3
2021-09-21 10:02:00 +00:00
Vitalii Shpital
69d3a839fc web/satellite: fix for onb CLI upload step image
Fixed SVG image usage ID for onb CLI upload step

Change-Id: Ieb94902de5702594edb20817541a8d968c0ad4f1
2021-09-20 20:07:48 +03:00
Vitalii Shpital
91c69a8afc web/satellite: fix for encrypt your data screen
Fixed issue for Encrypt your data screen. It was impossible to proceed further into object browser after selecting Generate passphrase option

Change-Id: Ibddda3c6bd7fce34bd1c629bbbb2d87529964b8a
2021-09-20 19:34:34 +03:00
crawter
5e4b196b11 satellite/metainfo: finish move object
Second method needed to perform server-side move. It updates
metadata key and nonce and all segments key and nonces.

Change-Id: Ia43b26622a13048269f0ae9e1524b345db112adb
2021-09-20 16:21:01 +03:00
Vitalii Shpital
9153b221fd testsuite/ui/satellite: updated pnboarding CLI tests and improved selectors
Updated onb CLI flow test.
Updated tests to use aria-roledescription as selector.

Change-Id: I1442592e1938fbe730064977dc9b78266e403706
2021-09-17 15:11:47 +00:00
Vitalii Shpital
853a7536f6 web/satellite: reworked onb CLI flow to correspond v1.1
Removed useless screens and fixed routing.
Removed useless logic from common component.

Change-Id: I73a1af5803f866526d3ae327e832654501e4f8bd
2021-09-17 15:11:41 +00:00
Yingrong Zhao
0b500a30e4 satellite/audit: move audit metrics out of reporter
Since we are sharing the reporting logic between repair and audit. We
need to remove metric reporting logic in reporter.

Change-Id: Ib87295ab19079329e7438327d785a7f5c21d3b21
2021-09-16 17:58:56 +00:00
Malcolm Bouzi
252b78580d satellite/console: add status check to user authorization to ensure deleted accounts cannot perform actions
Change-Id: I2b37101335d154bf4bc93ec85c06ff62e555c9ec
2021-09-16 17:06:34 +00:00
Egon Elbre
09e1ff7fc4 web/satellite: fix promise usages
Change-Id: I6496ebdd57667e8222b2bfbe6174860997b6bb3f
2021-09-16 16:32:21 +00:00
Egon Elbre
6e660cecdd Jenkinsfile: test cross-compile and bump deps
Change-Id: I47091de2e80bd96fcada616f75e1db07a59fb0c1
2021-09-16 18:59:31 +03:00
Egon Elbre
b2d724962c cmd/storagenode-updater: avoid depending on the storagenode code
Change-Id: If81927aee43ccdd678a0e42f91632f85e9fdefab
2021-09-16 11:44:55 +00:00
Egon Elbre
6b3718f33e Makefile: skip darwin build for the time being
Mac builds need to be reworked to work with latest Go version.

Change-Id: Ifbe37cf8fb41c5e0d806c518a3be32e093b448b0
2021-09-16 12:53:41 +03:00
Yingrong Zhao
f0b73b4f45 mod: bump drpc, uplink and common
Updates:
- drpc to v0.0.25
- uplink to latest
- common to latest

This will fix the crash on UDP network temporary error for windows and a
rare deadlock during shutdown

Change-Id: I7e9c4ace22665c5471f19f9b8bf63aec4d68ceb3
2021-09-16 07:42:32 +00:00
Jeff Wendling
b160ec4c1b satellite/orders: bound RollupsWriteCache flushes
In the situation where the flushes take longer than the incoming
rate of writes, the RollupsWriteCache will take every connection
in the database pool and use them forever. Instead of doing that
and taking down satellite availability, bound the number of flush
operations that it will perform and drop incoming writes earlier
to keep memory usage constant.

Adds monitoring events for if any flushes or updates are lost.

Change-Id: I81b169b73501ee9b999f4b03d1e79645fc56f167
2021-09-15 19:14:39 +00:00
Vitalii Shpital
86845351c3 web/satellite: reworked upgrade to Pro account modal
Reworked the modal to correspond to latest designs.
Added prices and updated copies.

Change-Id: I6e59fdfe4222e02fcc35ac5d594bdc0bdf52a63c
2021-09-15 19:47:56 +03:00
littleskunk
3c3df0f362
storage/filestore: test interaction between VerifyDirReadable and VerifyDirWriteable 2021-09-15 18:57:56 +03:00
Jeremy Wharton
9dab10e1d9 web/satellite: Remove Advanced Report from GUI
Advanced Report is broken and it's causing support tickets.
The Advanced Report page must not be linked to until it is fixed.

Change-Id: I51d6afeee7584022ec920f311bb4c0d31174fd2c
2021-09-15 14:33:02 +00:00
Vitalii Shpital
2e6b4e90f8 web/satellite: add warning message to EncryptYourData component
Added warning copy to EncryptYourData component to let user know that this approach uses server-side encryption.

Change-Id: I864b8d5272e3d54a5ef0fafe464cd0e1ccaa9e83
2021-09-15 09:11:46 +00:00
Yingrong Zhao
e805dce704 satellite/satellitedb: drop audit reputation score related columns from
nodes table

Change-Id: I090dc69999531b8d33e979d42ae9aa4d04ff664e
2021-09-14 16:41:02 +00:00
crawter
07311cdd6f satellite/metainfo: begin move object added
First from two methods needed to perform server-side move. It gets
metadata key and nonce and all segments key and nonces and returns
all of that to uplink.

Change-Id: Ied2c79559e77d3f63091c4d61948f2d6a2147d67
2021-09-14 13:18:21 +00:00
Fadila Khadar
c00ecae75c satellite/gracefulexit: stop using gracefulexit_transfer_queue
Remove the logic associated to the old transfer queue.
A new transfer queue (gracefulexit_segment_transfer_queue) has been created for migration to segmentLoop.
Transfers from the old queue were not moved to the new queue.
Instead, it was still used for nodes which have initiated graceful exit before migration.
There is no such node left, so we can remove all this logic.
In a next step, we will drop the table.

Change-Id: I3aa9bc29b76065d34b57a73f6e9c9d0297587f54
2021-09-14 11:52:34 +00:00
crawter
e1a4195a35 satellite/metainfo/endpoint: MaxObjectKey length validation for BeginObject and CommitObject added
Change-Id: I6a766415a84b1e3c44ce65052ffa8ed4511d00dd
2021-09-13 15:07:16 +03:00
Egon Elbre
1aec831d98 satellite/audit,storage: increase sleep delay in TestMaxVerifyCount
Currently TextMaxVerifyCount flakes in some tests, try increasing the
sleep time to ensure that things are slow enough to trigger the error
condition.

Also pass ctx to all the funcs so we can handle sleep better.

Change-Id: I605b6ea8b14a0a66d81a605ce3251f57a1669c00
2021-09-10 15:30:37 +00:00
Vitalii Shpital
b472925c1e web/satellite: hot fix for Onb CLI flow's Create Bucket component's back button
There is a bug with back button of Create Bucket step of new onb CLI flow when proceeding further into flow and trying to come back.

Change-Id: I663a05021824164617ba933815973d8be856539f
2021-09-10 12:36:40 +00:00
Artur M. Wolff
7f595445ac satellite/metainfo: make subsequent auth validations not perform rate-limiting
Currently, requests that were successfully passed through the metainfo
endpoints rate-limiter might still fail in the middle of the
corresponding response. The problem is that we perform rate-limiting a
second time, which means other requests would influence whether the
current (already rate-checked) request will fail. This also has other
unintended effects, like responding with rpcstatus.PermissionDenied for
requests that were successfully rate-checked and did not lack
permissions but were rate-checked again in the middle of
(*Endpoint).BeginObject. This situation has been happening on the
gateway side and might affect other uplink clients. This change, where
appropriate, swaps subsequent validateAuth with validateAuthN that
performs rate-limiting once.

Change-Id: I6fc26dedb8c442dd20acaab5942f751279020b08
2021-09-10 12:07:06 +00:00