Commit Graph

4679 Commits

Author SHA1 Message Date
JT Olio
bd23d12bb9 satellite/dbx: add cockroach retries for other QueryContext operations
Change-Id: Ia30fbba55c926892702fa96fb9dd01b75347d351
2020-11-29 18:09:56 -07:00
JT Olio
ea2f39ca7f satellite/dbx: add retries for QueryRowContext-based operations
Change-Id: Ie2527b673dd4ce5250cf5c0cbf8f14921262f665
2020-11-29 18:09:46 -07:00
JT Olio
d3b0691bbd satellite/dbx: import dbx templates
these are unchanged from storj.io/dbx.

we're importing them because in a later commit we
will change them, and it'd be nice to see that
diff as a separate commit.

Change-Id: I8315130ed6bab397bd65b9a1a90c29d130b8c02d
2020-11-29 18:09:33 -07:00
JT Olio
5d8a67a4f7 satellitedb: retry GetBandwidthSince on cockroach
Change-Id: I2bf20f3a19e7f3af97630d8a679410feba70661e
2020-11-29 16:36:15 -07:00
Ethan
5dc013d3bd satellite/overlay: Add retry to all selects in overlaycache
Change-Id: I0356d71a35701f8e0ca04a34b2bb2aea666c1394
2020-11-29 16:46:57 -05:00
JT Olio
6bce907cb0 satellite: try to stream rollups to aggregation function to use less memory
this change tries really hard to never have all of the storage node
rollups in memory at the same time, up until the rollups are actually
getting summed together.

Change-Id: If67f49e7d71106798d996a6850b3e48671bd9e18
2020-11-29 10:26:32 -07:00
JT Olio
6aae21541f satellitedb: do saverollup in batches
Change-Id: I78278a192cba60541eee2986f54a88d5a479bd3e
2020-11-28 19:26:46 -07:00
JT Olio
0ba516d405 satellite: support pointing db components at different databases
the immediate need is to be able to move the repair queue back out
of cockroach if we can't save it.

Change-Id: If26001a4e6804f6bb8713b4aee7e4fd6254dc326
2020-11-28 18:39:16 +00:00
Moby von Briesen
75f0f713a3 satellite/repair/checker/checker.go: Use number of healthy pieces instead of SegmentHealth for injured segments queue.
We did not test the SegmentHealth function with actual production
values, and it turns out that values such as 52 healthy, 35 minimum
result in +Inf segment health - so pretty much all segments put into the
repair queue have the same health, which means we effectively aren't
sorting by health.

This change inserts numHealthy as segment health into the database so
the segments are ordered as they were before. We need to refine the
SegmentHealth function before we can support multi RS.

Change-Id: Ief19bbfee3594c5dfe94ca606bc930f05f85ff74
2020-11-28 12:16:32 -05:00
nerdatwork
5c34b62bde
Fixed typos in downtime tracking with audits doc (#3977)
Co-authored-by: Ivan Fraixedes <ivan@fraixed.es>
2020-11-27 17:25:21 +01:00
Ivan Fraixedes
7eb3b2d6d0
satellite/gc: Init map with an aprox size
Because the PieceTracker receives a piece count per nodes which is an
approximation of the number of nodes that they are going to be reported
by the metainfo loop so we can use as a good guess of the map's size and
initialized with it.

Change-Id: I644db40926c03e4c457457fb41d2ec1da059cea6
2020-11-27 10:44:19 +01:00
Ivan Fraixedes
319d2cad11
satellite: Fix typo in a comment
Change-Id: I151b824e868db1cc1e8b8e8af9f35b027db1e6ff
2020-11-26 15:44:49 +01:00
Michal Niewrzal
8ceef9f357 satellite/metainfo: temporary disable one assertion in test
This is need to merge https://review.dev.storj.io/c/storj/uplink/+/3208
, after that this code will be back.

Change-Id: If9f2f1db95c7a1bba64a41c45a39bd3096a519e7
2020-11-25 13:21:41 +00:00
Egon Elbre
3792e2921c satellite/accounting/tally: make test less fragile
MetadataSize can slightly vary and checking for exact value makes
difficult to change what's being encoded in metadata.

Change-Id: I5f1ade41bc26d115e6743367ee35cf1ba74795c9
2020-11-25 13:33:24 +02:00
Malcolm Bouzi
c6626748d0
web/satellite: create acces grant: duration selection logic (#3978) 2020-11-24 19:18:02 +02:00
VitaliiShpital
257c8682d3 web/satellite: create access grant: result step
WHAT:
result step for access grant flow

WHY:
to show user access gratn key

Change-Id: I0426691ce670efd1d280ba819ecb0281b514452a
2020-11-24 12:48:08 +00:00
VitaliiShpital
84fb8eee11 web/satellite: create access grant: generate access grant in web worker
WHAT:
generate access grant in web worker to return back to UI

WHY:
get access grant

Change-Id: If8bf6389d3bfabd8a8a12fb0bbbc46cd9ecd6e74
2020-11-24 12:48:01 +00:00
Kaloyan Raev
53b7fd7b00 satellite/{audit,gracefulexit}: remove logic for PieceHashesVerified
We now have the piece hashes verified for all segments on all production
satellites. We can remove the code that handles the case where piece
hashes are not verified. This would make easier the migration of
services from PointerDB to the new metabase.

For consistency, PieceHashesVerified is still set to true in PointerDB
for new segments.

Change-Id: Idf0ccce4c8d01ae812f11e8384a7221d90d4c183
2020-11-24 11:09:48 +02:00
Egon Elbre
9de1617db0 satellite/orders: ensure encryption keys handles set twice
Currently flag parsing seems to call Set twice, which causes problems
with encryption keys. We can clear for every set for now.

Change-Id: Id5c695b4020194ac1c50a2da9c7d2a896cb9216f
2020-11-23 19:47:22 +00:00
Moby von Briesen
575f50df84 satellite/repair: Update repair override config to support multiple RS schemes.
Rather than having a single repair override value, we will now support
repair override values based on a particular segment's RS scheme.

The new format for RS override values is
"k/o/n-override,k/o/n-override..."

Change-Id: Ieb422638446ef3a9357d59b2d279ee941367604d
2020-11-23 18:01:15 +00:00
Egon Elbre
55d5e1fd7d satellite/orders: ensure that expired deletion doesn't stall
Add checks to ensure that when somebody uses empty options, the deletion
doesn't loop infinitely.

Change-Id: I1738fb1e7e1f8efbbb954c491cb6489f7bcdc2db
2020-11-23 14:52:40 +02:00
Jessica Grebenschikov
5beb2f5737 satellite/orders: add factory function to encryption key
Change-Id: I9a1020c63e4ebc6d73683cf1749366e9b9f20f07
2020-11-20 11:40:15 -08:00
Ethan
2b92bba563 satellite/satellitedb/orders: Handle serial_numbers deletes in smaller increments on CRDB
CRDB doesn't like large deletes. While testing in the POC environment we found that deletes on the serial_numbers table could take hours.  This change limits deletes to 1000 at a time (configurable) to avoid blocking other queries.

Change-Id: I08455e25db1574579dd4d7b7125a08e9c913dff1
2020-11-20 13:44:52 +00:00
VitaliiShpital
2a981b86d5 web/satellite: create access grant: enter passphrase step
WHAT:
enter passphrase step for create access grant flow

WHY:
if user already has access grant, passphrase could be used one more time

Change-Id: I7ceb6ca5b59a1d72510c739e268c73c511bbedef
2020-11-20 12:18:07 +00:00
VitaliiShpital
4dddb6e668 web/satellite: create access grant: create passphrase step
WHAT:
create passphrase step for access grant flow

WHY:
passphare is needed to generate access grant

Change-Id: I9d0540826744d192b2e00a30fda907cc581a6e00
2020-11-20 12:18:00 +00:00
VitaliiShpital
005b089c01 web/satellite: create access grant: get restricted api key from web worker
WHAT:
get new api key with restrictions from web worker

WHY:
apply restrictions

Change-Id: I5747c9ede0c508b29f635a07895738715bd54621
2020-11-20 12:17:50 +00:00
Malcolm Bouzi
c22ae05bbe
web/satellite:access grant date picker (#3972)
* add create button

* menu toggle function

* add date buttons

* add date pick

* styling edit

* center date picker

* remove date picker from page

* edit spacing

* revert dropdown

* re-add spacing

* re-add spacing

* extra-gap

* re-add spacing

* remove gap

* add picker to permissions step

* add package-lock

* space edits

* indent fix

* space fix

* indent fixes

* indent edits

* duration picker indent edit

* picker component indent

* input class change

* add datepicker to jest config

Co-authored-by: Vitalii Shpital <46756926+VitaliiShpital@users.noreply.github.com>
2020-11-19 14:07:25 -05:00
Jeff Wendling
54ae9b040e cmd/uplink: allow public access registration
Change-Id: I1316f10158d8a466bc80d6548ad0773016bfb916
2020-11-18 17:40:30 -05:00
Moby von Briesen
a8b66dce17 satellite/accounting: account for old orders that can be submitted in satellite rollup
With the new phase 3 order submission, orders can be added to the
storage and bandwidth rollup tables at timestamps before the most recent
rollup was run. This change shifts the start time of each new rollup
window to account for any unexpired orders that might have been added
since the previous rollup.

A satellitedb migration is necessary to allow upserts in the
accounting_rollups table when entries with identical node_ids and
start_times are inserted.

Change-Id: Ib3022081f4d6be60cfec8430b45867ad3c01da63
2020-11-18 14:46:00 -05:00
Egon Elbre
e19fabc880 scripts/tests/rollingupgrade: fix typo in flag
Change-Id: Ia3e8d076741a30fb2a42af9b2621796a814c75ae
2020-11-18 19:15:42 +00:00
Bill Thorp
5fe3d2dea7 cmd/uplink Allow use of named accesses in uplink register
Previously uplink register only accepted a fully serialized access grant.
This is kind of annoying, I changed it so that it could also use access names.

Change-Id: If6d4d1baa8d4fb3d87fdedb895d459fa12743f1a
2020-11-18 12:23:57 -05:00
Egon Elbre
aeb801604e {satellite,storagenode}/orders: fix flaky tests
Before manipulating order information on storagenodes we need to wait
for the orders to propagate to the database. Some of that happens
async with uplink.

Change-Id: Iaacfd7db0909ab5d2831d06388e5fb27b6d4778f
2020-11-18 13:44:02 +00:00
Moby von Briesen
41d86c0985 storagenode/orders/ordersfile: Add reasonable size caps for orders/limits when detecting file corruption.
Define constants of 32 KiB as the upper limit of the marshalled order
and limit protobuf sizes. This value gives lots of buffer in case the
protobufs ever change, but is not as extreme as what we were doing
before in V0 files, which was to use the Uint32 max value.

Change-Id: I0914d17dde3b044b2611af33f931d46d55f81e98
2020-11-18 12:33:26 +00:00
Egon Elbre
8da5e6a554 scripts/tests/rollingupgrade: use wait-for instead of sleep
Change-Id: Ie879e061d3b312705726375953767d420e922073
2020-11-18 12:00:16 +00:00
Qweder93
a17cd9aa3e storageode/apikey: added service, CLI issue api key
Change-Id: I840cd0fdbd8dca884eefbd111f21fd3990c11e68
2020-11-18 10:40:17 +00:00
paul cannon
2b59640f18 cmd/satellite: ignore Canceled in exit from repair worker
Firstly, this changes the repair functionality to return Canceled errors
when a repair is canceled during the Get phase. Previously, because we
do not track individual errors per piece, this would just show up as a
failure to download enough pieces to repair the segment, which would
cause the segment to be added to the IrreparableDB, which is entirely
unhelpful.

Then, ignore Canceled errors in the return value of the repair worker.
Apparently, when the worker returns an error, that makes Cobra exit the
program with a nonzero exit code, which causes some piece of our
deployment automation to freak out and page people. And when we ask the
repair worker to shut down, "canceled" errors are what we _expect_, not
an error case.

Change-Id: Ia3eb1c60a8d6ec5d09e7cef55dea523be28e8435
2020-11-17 21:37:59 +00:00
VitaliiShpital
f5a5308cc0 web/satellite: upload data step
WHAT:
access grant flow: upload data page

WHY:
info page telling what could be done next

Change-Id: I9bd5d558c97d5cf643f9746169952b8424c3294f
2020-11-17 18:44:22 +00:00
VitaliiShpital
1c13065b0b web/satellite: create access grant: continue in CLI step
WHAT:
continue in CLI step which returns regular API key

WHY:
in case user wants to create access grants in CLI

Change-Id: I8a0fa15f07e553628bda3a3e871506295230f0a2
2020-11-17 18:44:12 +00:00
VitaliiShpital
6f35ee98e6 web/satellite: create access grant: bucket names selection logic
WHAT:
bucket names selection logic for create access grant flow

WHY:
bucket based access grant restriction

Change-Id: I922811ce43afbc0bf0c2c9bcaea755657257f26f
2020-11-17 18:44:04 +00:00
VitaliiShpital
4e49b00c6c web/satellite: create access grant permission step, regular permissions, buckets dropdown
WHAT:
permissions step page for create access grant flow. Regular permissions and buckets dropdown

WHY:
to configure access grant permissions

Change-Id: Ia5571556a7fb83fd9a508e6aabfcdf5b57e9bc96
2020-11-17 18:43:39 +00:00
VitaliiShpital
a332b3d811 web/satellite: create access grant name step
WHAT:
name step for create access grant flow

WHY:
give access grant a name

Change-Id: Ic1819dcc6565b2ca20008459f0a33ece61930165
2020-11-17 18:43:11 +00:00
VitaliiShpital
278e29c1c7 web/satellite: create access grant progress bar
WHAT:
progress bar for create access grant flow

WHY:
progress bar to show user current step of the flow

Change-Id: Ia3665fee91ac9b3c27eed5d5190e69d7ea5b3e8a
2020-11-17 18:43:03 +00:00
VitaliiShpital
6517315ff8 web/satellite: create access grant base container
WHAT:
create access grant base container

WHY:
base container - first step of access grant flow

Change-Id: Id31e25333eadbe6a40cdce972de5cb87413a299e
2020-11-17 18:42:54 +00:00
VitaliiShpital
6664a129b0 web/satellite: add all needed methods to access grant webworker
WHAT:
all needed methods added to webworker

WHY:
to generate correct access grant

Change-Id: I700f24840d5bbe1515dbafa7f4e71e505205f903
2020-11-17 18:42:04 +00:00
VitaliiShpital
b60939e483 web/satellite: delete access grant flow
WHAT:
delete process for access grant flow. Including popup

WHY:
ability to remove access grant

Change-Id: Idf9f4659863a2004ce8b74976525b05103329b9a
2020-11-17 20:18:58 +02:00
VitaliiShpital
e16f02b70d web/satellite: access grant list page
WHAT:
access grants list page where all the created access grants will be visible/deletable

WHY:
initial page of new access grant flow

Change-Id: I0b99f15e47295bd0d307dd3aebd9f6dea3ffbb25
2020-11-17 17:50:00 +00:00
Ivan Fraixedes
fa95c6bbb9
storagenode/orders/ordersfile: Fix error message wrong var
Fix the error message reported by a wrong order size due to passing the
wrong variable to the interpolation pattern.

Change-Id: Ic0059615c60cfa33a26d4aeb0ebda5e586f0df05
2020-11-17 15:22:27 +01:00
NickolaiYurchenko
ac94333422 web/storagenode: notifications components unit tests
Change-Id: I2b0249eff73572b3bf401c5f920440f095bc3978
2020-11-17 12:00:00 +00:00
Ivan Fraixedes
9740da6508 storagenode/orders: Don't panic if size is over MaxInt32
`make` built function to build a new slice with a negative
length panics.
`make` length parameter is of `int` type.

These changes avoid that `make` panics on 32 bits architecture due to
the fact that `int` type is a `int32` an uint32 value can be over the
maximum `int32`, and when that happens the length parameter value
becomes negative and makes `make` to panic.

Change-Id: Ife9ab5993916d6dcf5584b37c208272269cb2b45
2020-11-17 10:35:21 +00:00
Qweder93
c409194d43 storagenode/payouts: estimation payout heldamount rounding removed
Change-Id: I9fdc7cda15de0df8875436b0b376f0e6479d3aeb
2020-11-17 10:06:11 +00:00