Commit Graph

5138 Commits

Author SHA1 Message Date
VitaliiShpital
0771cdb0b1 web/satellite: create access grant: generate gateway credentials step
WHAT:
generate gateway credentials step for create access grant flow

WHY:
part of the flow

Change-Id: I6496712b43f78a818ba0582b586cfae3a44683e6
2020-11-30 10:36:29 +00:00
VitaliiShpital
bb7677a85f web/satellite: get gateway credentials request using url from config
WHAT:
POST request to get gateway credentials using access grant.
Put request url to config and use it for request.

WHY:
to show gateway credentials on UI

Change-Id: I15ef43ecdeed69b0961d5796aacb47f36d560b1b
2020-11-30 10:36:23 +00:00
VitaliiShpital
d3d6e0c67c web/satellite: create access grant: set duration restrictions
WHAT:
date range picked on permissions step is applied now

WHY:
limit access grant by date range

Change-Id: I75b1a63a4e19110ce6cb913b3743d1be9023f9c4
2020-11-30 10:36:17 +00:00
Michal Niewrzal
21602e0494 satellite/metainfo: enable commented test
Test was commented to make uplink refactoring possible. Now we can bring
back this test.

Change-Id: I0511b76073efaafed8aac97f8e845dcec93dd059
2020-11-30 10:49:23 +01:00
JT Olio
71e11b27f3 satellite/dbx: only retry with cockroach
Change-Id: Id3630c26dbfda36dcbece2849e2353d5ab2882af
2020-11-29 18:10:07 -07:00
JT Olio
bd23d12bb9 satellite/dbx: add cockroach retries for other QueryContext operations
Change-Id: Ia30fbba55c926892702fa96fb9dd01b75347d351
2020-11-29 18:09:56 -07:00
JT Olio
ea2f39ca7f satellite/dbx: add retries for QueryRowContext-based operations
Change-Id: Ie2527b673dd4ce5250cf5c0cbf8f14921262f665
2020-11-29 18:09:46 -07:00
JT Olio
d3b0691bbd satellite/dbx: import dbx templates
these are unchanged from storj.io/dbx.

we're importing them because in a later commit we
will change them, and it'd be nice to see that
diff as a separate commit.

Change-Id: I8315130ed6bab397bd65b9a1a90c29d130b8c02d
2020-11-29 18:09:33 -07:00
JT Olio
5d8a67a4f7 satellitedb: retry GetBandwidthSince on cockroach
Change-Id: I2bf20f3a19e7f3af97630d8a679410feba70661e
2020-11-29 16:36:15 -07:00
Ethan
5dc013d3bd satellite/overlay: Add retry to all selects in overlaycache
Change-Id: I0356d71a35701f8e0ca04a34b2bb2aea666c1394
2020-11-29 16:46:57 -05:00
JT Olio
6bce907cb0 satellite: try to stream rollups to aggregation function to use less memory
this change tries really hard to never have all of the storage node
rollups in memory at the same time, up until the rollups are actually
getting summed together.

Change-Id: If67f49e7d71106798d996a6850b3e48671bd9e18
2020-11-29 10:26:32 -07:00
JT Olio
6aae21541f satellitedb: do saverollup in batches
Change-Id: I78278a192cba60541eee2986f54a88d5a479bd3e
2020-11-28 19:26:46 -07:00
JT Olio
0ba516d405 satellite: support pointing db components at different databases
the immediate need is to be able to move the repair queue back out
of cockroach if we can't save it.

Change-Id: If26001a4e6804f6bb8713b4aee7e4fd6254dc326
2020-11-28 18:39:16 +00:00
Moby von Briesen
75f0f713a3 satellite/repair/checker/checker.go: Use number of healthy pieces instead of SegmentHealth for injured segments queue.
We did not test the SegmentHealth function with actual production
values, and it turns out that values such as 52 healthy, 35 minimum
result in +Inf segment health - so pretty much all segments put into the
repair queue have the same health, which means we effectively aren't
sorting by health.

This change inserts numHealthy as segment health into the database so
the segments are ordered as they were before. We need to refine the
SegmentHealth function before we can support multi RS.

Change-Id: Ief19bbfee3594c5dfe94ca606bc930f05f85ff74
2020-11-28 12:16:32 -05:00
nerdatwork
5c34b62bde
Fixed typos in downtime tracking with audits doc (#3977)
Co-authored-by: Ivan Fraixedes <ivan@fraixed.es>
2020-11-27 17:25:21 +01:00
Ivan Fraixedes
7eb3b2d6d0
satellite/gc: Init map with an aprox size
Because the PieceTracker receives a piece count per nodes which is an
approximation of the number of nodes that they are going to be reported
by the metainfo loop so we can use as a good guess of the map's size and
initialized with it.

Change-Id: I644db40926c03e4c457457fb41d2ec1da059cea6
2020-11-27 10:44:19 +01:00
Ivan Fraixedes
319d2cad11
satellite: Fix typo in a comment
Change-Id: I151b824e868db1cc1e8b8e8af9f35b027db1e6ff
2020-11-26 15:44:49 +01:00
Michal Niewrzal
cfb45a785c satellite/metainfo/metabase: add TotalPlainSize to objects table
We need to know how large is object in general and we shouldn't query
segments for that.

Change-Id: Ie316e1ec38bfc39065e6046b79b55d58f3ce9d53
2020-11-25 14:35:28 +01:00
Michal Niewrzal
8ceef9f357 satellite/metainfo: temporary disable one assertion in test
This is need to merge https://review.dev.storj.io/c/storj/uplink/+/3208
, after that this code will be back.

Change-Id: If9f2f1db95c7a1bba64a41c45a39bd3096a519e7
2020-11-25 13:21:41 +00:00
Michal Niewrzal
983b1737c2 satellite/metainfo: implement IsBucketEmpty with metabase
Use objects iterator to check if bucket is empty.

Change-Id: I590bb7258a47dd816f079039a1151df809dd2637
2020-11-25 12:20:30 +00:00
Egon Elbre
3792e2921c satellite/accounting/tally: make test less fragile
MetadataSize can slightly vary and checking for exact value makes
difficult to change what's being encoded in metadata.

Change-Id: I5f1ade41bc26d115e6743367ee35cf1ba74795c9
2020-11-25 13:33:24 +02:00
Malcolm Bouzi
c6626748d0
web/satellite: create acces grant: duration selection logic (#3978) 2020-11-24 19:18:02 +02:00
Michal Niewrzal
46102c1942 satellite/metainfo: change ListSegments required permission to
Read/Download

We should treat listing segments as a part of download process, not
listing.

Change-Id: Iae72c97c77976c427f668a96486af613d58da3b9
2020-11-24 17:50:33 +01:00
Egon Elbre
dad36179c6 satellite/metainfo/metabase: fix segment listing with cursor.Part
Change-Id: I910888d435b11d2d4687ea0042998a1078cd6443
2020-11-24 15:31:57 +02:00
VitaliiShpital
257c8682d3 web/satellite: create access grant: result step
WHAT:
result step for access grant flow

WHY:
to show user access gratn key

Change-Id: I0426691ce670efd1d280ba819ecb0281b514452a
2020-11-24 12:48:08 +00:00
VitaliiShpital
84fb8eee11 web/satellite: create access grant: generate access grant in web worker
WHAT:
generate access grant in web worker to return back to UI

WHY:
get access grant

Change-Id: If8bf6389d3bfabd8a8a12fb0bbbc46cd9ecd6e74
2020-11-24 12:48:01 +00:00
Kaloyan Raev
67c210a64c satellite/metainfo: set list cursor to version 1
Otherwise, if left to default version 0, the iterator will include the
cursor item in the result, which fails some tests.

Change-Id: I85103a36852477f371ec46c673a82c2e129978b7
2020-11-24 10:17:42 +00:00
Kaloyan Raev
53b7fd7b00 satellite/{audit,gracefulexit}: remove logic for PieceHashesVerified
We now have the piece hashes verified for all segments on all production
satellites. We can remove the code that handles the case where piece
hashes are not verified. This would make easier the migration of
services from PointerDB to the new metabase.

For consistency, PieceHashesVerified is still set to true in PointerDB
for new segments.

Change-Id: Idf0ccce4c8d01ae812f11e8384a7221d90d4c183
2020-11-24 11:09:48 +02:00
Michal Niewrzal
efaba85c73 Merge 'master' branch
Change-Id: I3520b3e327732929f5167b07a15ddb92d26cae1b
2020-11-24 10:03:20 +01:00
Egon Elbre
9de1617db0 satellite/orders: ensure encryption keys handles set twice
Currently flag parsing seems to call Set twice, which causes problems
with encryption keys. We can clear for every set for now.

Change-Id: Id5c695b4020194ac1c50a2da9c7d2a896cb9216f
2020-11-23 19:47:22 +00:00
Moby von Briesen
575f50df84 satellite/repair: Update repair override config to support multiple RS schemes.
Rather than having a single repair override value, we will now support
repair override values based on a particular segment's RS scheme.

The new format for RS override values is
"k/o/n-override,k/o/n-override..."

Change-Id: Ieb422638446ef3a9357d59b2d279ee941367604d
2020-11-23 18:01:15 +00:00
Egon Elbre
55d5e1fd7d satellite/orders: ensure that expired deletion doesn't stall
Add checks to ensure that when somebody uses empty options, the deletion
doesn't loop infinitely.

Change-Id: I1738fb1e7e1f8efbbb954c491cb6489f7bcdc2db
2020-11-23 14:52:40 +02:00
Jessica Grebenschikov
5beb2f5737 satellite/orders: add factory function to encryption key
Change-Id: I9a1020c63e4ebc6d73683cf1749366e9b9f20f07
2020-11-20 11:40:15 -08:00
Ethan
2b92bba563 satellite/satellitedb/orders: Handle serial_numbers deletes in smaller increments on CRDB
CRDB doesn't like large deletes. While testing in the POC environment we found that deletes on the serial_numbers table could take hours.  This change limits deletes to 1000 at a time (configurable) to avoid blocking other queries.

Change-Id: I08455e25db1574579dd4d7b7125a08e9c913dff1
2020-11-20 13:44:52 +00:00
Michal Niewrzal
5be4d413b0 satellite/metainfo: set PlainSize for inline segment
Plain size is required by metabase implementation.

Change-Id: I6739744bff4ce485e0066f2a49acdf38560d31fb
2020-11-20 13:54:14 +01:00
VitaliiShpital
2a981b86d5 web/satellite: create access grant: enter passphrase step
WHAT:
enter passphrase step for create access grant flow

WHY:
if user already has access grant, passphrase could be used one more time

Change-Id: I7ceb6ca5b59a1d72510c739e268c73c511bbedef
2020-11-20 12:18:07 +00:00
VitaliiShpital
4dddb6e668 web/satellite: create access grant: create passphrase step
WHAT:
create passphrase step for access grant flow

WHY:
passphare is needed to generate access grant

Change-Id: I9d0540826744d192b2e00a30fda907cc581a6e00
2020-11-20 12:18:00 +00:00
VitaliiShpital
005b089c01 web/satellite: create access grant: get restricted api key from web worker
WHAT:
get new api key with restrictions from web worker

WHY:
apply restrictions

Change-Id: I5747c9ede0c508b29f635a07895738715bd54621
2020-11-20 12:17:50 +00:00
Malcolm Bouzi
c22ae05bbe
web/satellite:access grant date picker (#3972)
* add create button

* menu toggle function

* add date buttons

* add date pick

* styling edit

* center date picker

* remove date picker from page

* edit spacing

* revert dropdown

* re-add spacing

* re-add spacing

* extra-gap

* re-add spacing

* remove gap

* add picker to permissions step

* add package-lock

* space edits

* indent fix

* space fix

* indent fixes

* indent edits

* duration picker indent edit

* picker component indent

* input class change

* add datepicker to jest config

Co-authored-by: Vitalii Shpital <46756926+VitaliiShpital@users.noreply.github.com>
2020-11-19 14:07:25 -05:00
Kaloyan Raev
05fe497a6f satellite/metainfo: add stream id when listing pending objects
Change-Id: Idcdffd70b82c0fe4f79635fc193604b66101ef76
2020-11-19 14:22:09 +02:00
Jeff Wendling
54ae9b040e cmd/uplink: allow public access registration
Change-Id: I1316f10158d8a466bc80d6548ad0773016bfb916
2020-11-18 17:40:30 -05:00
Moby von Briesen
a8b66dce17 satellite/accounting: account for old orders that can be submitted in satellite rollup
With the new phase 3 order submission, orders can be added to the
storage and bandwidth rollup tables at timestamps before the most recent
rollup was run. This change shifts the start time of each new rollup
window to account for any unexpired orders that might have been added
since the previous rollup.

A satellitedb migration is necessary to allow upserts in the
accounting_rollups table when entries with identical node_ids and
start_times are inserted.

Change-Id: Ib3022081f4d6be60cfec8430b45867ad3c01da63
2020-11-18 14:46:00 -05:00
Egon Elbre
e19fabc880 scripts/tests/rollingupgrade: fix typo in flag
Change-Id: Ia3e8d076741a30fb2a42af9b2621796a814c75ae
2020-11-18 19:15:42 +00:00
Bill Thorp
5fe3d2dea7 cmd/uplink Allow use of named accesses in uplink register
Previously uplink register only accepted a fully serialized access grant.
This is kind of annoying, I changed it so that it could also use access names.

Change-Id: If6d4d1baa8d4fb3d87fdedb895d459fa12743f1a
2020-11-18 12:23:57 -05:00
Michal Niewrzal
1465539220 satellite/metainfo: ignore empty inline segments for backward
compatibility

Change-Id: I2567d77e172391945dee24cea8e110fa1963d663
2020-11-18 14:12:03 +00:00
Egon Elbre
aeb801604e {satellite,storagenode}/orders: fix flaky tests
Before manipulating order information on storagenodes we need to wait
for the orders to propagate to the database. Some of that happens
async with uplink.

Change-Id: Iaacfd7db0909ab5d2831d06388e5fb27b6d4778f
2020-11-18 13:44:02 +00:00
Michal Niewrzal
70b1c7aa17 satellite/metainfo: check if EncryptedMetadataNonce was set
It possible that old uplinks won't set it.

Change-Id: Ifd3c4d4a45c5636cfeb70b0a37bcf1305a097727
2020-11-18 14:04:31 +01:00
Moby von Briesen
41d86c0985 storagenode/orders/ordersfile: Add reasonable size caps for orders/limits when detecting file corruption.
Define constants of 32 KiB as the upper limit of the marshalled order
and limit protobuf sizes. This value gives lots of buffer in case the
protobufs ever change, but is not as extreme as what we were doing
before in V0 files, which was to use the Uint32 max value.

Change-Id: I0914d17dde3b044b2611af33f931d46d55f81e98
2020-11-18 12:33:26 +00:00
Egon Elbre
8da5e6a554 scripts/tests/rollingupgrade: use wait-for instead of sleep
Change-Id: Ie879e061d3b312705726375953767d420e922073
2020-11-18 12:00:16 +00:00
Kaloyan Raev
a30f5d7ec0 satellite/metainfo: use Object.Status from ObjectListRequest
Change-Id: Ie2e48d2eedf69bd0f04b7c9d6e7e217fec6b0267
2020-11-18 13:31:52 +02:00