Commit Graph

4710 Commits

Author SHA1 Message Date
VitaliiShpital
c8e8fea052 web/satellite: replacing api keys with access grants
WHAT:
api keys appearence is replaced with access grants

WHY:
last step of access grants implementing

Change-Id: Ibef391849c7185fa56627b482218c76fb2d31b46
2020-12-08 13:34:59 +02:00
VitaliiShpital
70795c68ff web/satellite: update onboarding flow: create access grant
WHAT:
create access grant flow for onboarding tour

WHY:
new onboarding tour

Change-Id: I75e84b00edb71cac07f8e6623ce06655ae771894
2020-12-08 10:39:15 +00:00
VitaliiShpital
55b495faa2 web/satellite: update onboarding flow: payment step update
WHAT:
added untitled project creation after adding payment method.
UI was slightly changed.

WHY:
new onboarding flow

Change-Id: I6cca1c784bd0462f20f2b06cc6595b9920d4f992
2020-12-08 10:39:08 +00:00
VitaliiShpital
701474acd5 web/satellite: update onboarding tour: overview step
WHAT:
updated overview step of onboarding tour. It shows upload data methods instead of tour steps now

WHY:
new onboarding tour

Change-Id: I7ffe9b2b91c2e17dd0c27e5e80a15301f6de16aa
2020-12-08 10:39:01 +00:00
VitaliiShpital
d5f1245cc7 web/satellite: update onboarding flow: removed redundant steps, routing applied
WHAT:
redundant steps removed from onboarding tour, new routing applied

WHY:
flow simplification

Change-Id: I153c5fe30914317ddd9bf7008d4e5e6a24f77f9a
2020-12-08 10:38:54 +00:00
VitaliiShpital
aaec8bfc5c web/satellite: update onboarding tour: routing
WHAT:
first step of updating onboarding tour - adding routes

WHY:
onboarding your is being reworked so adding routes will make it easier to operate over it's states

Change-Id: Ide830989e39a6222e975bd2a6106b0efbb3839f9
2020-12-08 10:38:47 +00:00
Stefan Benten
a71c908c1e
cmd: add ca-certificates to Docker images (#3986) 2020-12-08 01:38:33 +01:00
Michal Niewrzal
cdeea1c999 private/testplanet: add helper OpenProject method to testplanet uplink
This will simplify opening pure uplink.Project in tests.

Change-Id: I076875e15e21608f49dc875bb445412f34609bdb
2020-12-07 13:45:47 +00:00
VitaliiShpital
7eccfccfda web/satellite: create access grant: bucket search feature
WHAT:
search for bucket names during creating access grant flow

WHY:
make user be able to search for needed buckets. Just in case if the number of buckets is too large.

Change-Id: I73bcaa160c7a1f433d8f0f7213999e7e40543bbc
2020-12-07 11:54:05 +00:00
Stefan Benten
494bd5db81
all: golangci-lint v1.33.0 fixes (#3985) 2020-12-05 17:01:42 +01:00
Yingrong Zhao
746315672f scripts/tests/testversions: fix indentation
Change-Id: Iaa5aec27f0ad78e1d8bf1a68aa5a62762c8ab537
2020-12-04 21:54:55 +00:00
Yingrong Zhao
0faf7d5293 scripts/tests/testversions: fix race in install_sim
Change-Id: I0792686d99a222d5977fd913425e2b94d100c40e
2020-12-04 18:18:14 +00:00
Caleb Case
c1bdb88d3c
Upgrade to uplink v1.4.1
Change-Id: Ie823f9b72d41f3da8e929d96a870fffc2e7ec12b
2020-12-04 11:38:19 -05:00
Ethan Adams
f90ea10a4a
Allow for DB application names per process. (#3983) 2020-12-04 11:24:39 +01:00
Moby von Briesen
d75e4be11f satellite/{accounting, contact}: Remove periods and spaces from metrics.
Change-Id: I84179c2931293e3a1eb0ff8050416d25e481ce07
2020-12-03 15:33:01 +00:00
Yingrong Zhao
13555f3983 scripts/tests/testversions: enable concurrent installation for each
version

The test-versions test currently takes 1h 40min to run each time. By
running each installation concurrently, hopefully, it will reduce the execution
time for the whole test.

Change-Id: I680c7d9945e982894b11825c9075c167f754e087
2020-12-03 15:01:37 +00:00
Moby von Briesen
3fc76f4ffe satellite/downtime: Remove deprecated downtime tracking service.
We are no longer planning on implementing downtime penalization using
the method described in
docs/blueprints/archive/storage-node-downtime-tracking-deprecated.md.
Now, we are implementing the design described in
docs/blueprints/storage-node-downtime-tracking-with-audits.md.

This change removes the downtime estimation chores from the satellite
core as well as the package satellite/downtime. A future change will
remove the database table.

Change-Id: I1a1d3cf9dceeba36255d25243294865b89925518
2020-12-02 15:16:13 -05:00
Caleb Case
6ba8dd58e6 Update to uplink v1.4.0
Change-Id: Ia01fec012031ceb2867b80a37e6a6c2a7d179cca
2020-12-02 13:44:32 +00:00
Bill Thorp
47e008b714 cmd/uplink: Write to AWS Credentials file more safely from uplink access register
Jeff provided feedback on https://review.dev.storj.io/c/storj/storj/+/3176 after
the changeset was already merged.  I attempt to address that feedback here.

Change-Id: Ibc7dba3e4e2c73736042fe4b4ee49ce679ba7f44
2020-12-01 17:47:14 +00:00
JT Olio
1728c3a992 satellite/dbx: standardize on assignment
Change-Id: I8f87bc8391e765e4480b0590d92d3601248e1f93
2020-12-01 16:10:18 +00:00
Jessica Grebenschikov
b261110352 satellite/orders: get bucketID from encrypted metadata in order instead of serial_numbers table
We want to stop using the serial_numbers table in satelliteDB. One of the last places using the serial_numbers table is when storagenodes settle orders, we look up the bucket name and project ID from the serial number from the serial_numbers table.

Now that we have support to add encrypted metadata into the OrderLimit, this PR makes use of that and now attempts to read the project ID and bucket name from the encrypted orderLimit metadata instead of from the serial_numbers table. For backwards compatibility and to ensure no errors, we will still fallback to the old way of getting that info from the serial_numbers table, but this will be removed in the next release as long as there are no errors.

All processes that create orderLimits must have an orders.encryption-keys set. The services that create orderLimits (and thus need to encrypt the order metadata) are the satellite apiProcess, the repair process, audit service (core process), and graceful exit (core process). Only the satellite api process decrypts the order metadata when storagenodes settle orders. This means that the same encryption key needs to be provided in the config for the satellite api process, repair process, and the core process like so:
orders.include-encrypted-metadata=true
orders.encryption-keys="<"encryptionKeyID>=<encryptionKey>"

Change-Id: Ie2c037971713d6fbf69d697bfad7f8b672eedd66
2020-12-01 15:29:32 +00:00
JT Olio
70b91aac54 satellitedb: remove cruft caused by https://review.dev.storj.io/c/storj/storj/+/3223
Change-Id: I198bb2f869cc7177b9ecafdd8932bbf2b58be5b8
2020-12-01 00:16:26 +00:00
Yingrong Zhao
d8ba7b3057 satellite/console: only allow project member to get all bucket names
Change-Id: I8ceb0b7eb19e221072b4ff3411a4ec1a7817d16f
2020-11-30 15:41:35 -05:00
Bill Thorp
65919f9f7d cmd/uplink: add --aws-profile flag to uplink access register
Added flag to append a new profile to ~/.aws/credentials using
the provided profile name.  This is handy for the AWS CLI, so
you can do things like 'aws configure get aws_access_key_id --profile=me'

Change-Id: I0469a18ca76e078624ed455a06bd7aabd95a1b97
2020-11-30 09:39:01 -05:00
Egon Elbre
f456d7ce03 satellite: remove implementation detail from DB interface
Which database access and how it internally does migrations is an
implementation detail and does not belong in the requirements interface.

Change-Id: Ia4a6994f39470063a96a8e5f3a1bd27aa79fe5cd
2020-11-30 13:29:20 +02:00
Egon Elbre
28ea63be92 satellite/repair: avoid TestDBAccess
Change-Id: I34adb58cd67fba5917032f2f328d75b1c4afdbbf
2020-11-30 13:29:08 +02:00
VitaliiShpital
0771cdb0b1 web/satellite: create access grant: generate gateway credentials step
WHAT:
generate gateway credentials step for create access grant flow

WHY:
part of the flow

Change-Id: I6496712b43f78a818ba0582b586cfae3a44683e6
2020-11-30 10:36:29 +00:00
VitaliiShpital
bb7677a85f web/satellite: get gateway credentials request using url from config
WHAT:
POST request to get gateway credentials using access grant.
Put request url to config and use it for request.

WHY:
to show gateway credentials on UI

Change-Id: I15ef43ecdeed69b0961d5796aacb47f36d560b1b
2020-11-30 10:36:23 +00:00
VitaliiShpital
d3d6e0c67c web/satellite: create access grant: set duration restrictions
WHAT:
date range picked on permissions step is applied now

WHY:
limit access grant by date range

Change-Id: I75b1a63a4e19110ce6cb913b3743d1be9023f9c4
2020-11-30 10:36:17 +00:00
Michal Niewrzal
21602e0494 satellite/metainfo: enable commented test
Test was commented to make uplink refactoring possible. Now we can bring
back this test.

Change-Id: I0511b76073efaafed8aac97f8e845dcec93dd059
2020-11-30 10:49:23 +01:00
JT Olio
71e11b27f3 satellite/dbx: only retry with cockroach
Change-Id: Id3630c26dbfda36dcbece2849e2353d5ab2882af
2020-11-29 18:10:07 -07:00
JT Olio
bd23d12bb9 satellite/dbx: add cockroach retries for other QueryContext operations
Change-Id: Ia30fbba55c926892702fa96fb9dd01b75347d351
2020-11-29 18:09:56 -07:00
JT Olio
ea2f39ca7f satellite/dbx: add retries for QueryRowContext-based operations
Change-Id: Ie2527b673dd4ce5250cf5c0cbf8f14921262f665
2020-11-29 18:09:46 -07:00
JT Olio
d3b0691bbd satellite/dbx: import dbx templates
these are unchanged from storj.io/dbx.

we're importing them because in a later commit we
will change them, and it'd be nice to see that
diff as a separate commit.

Change-Id: I8315130ed6bab397bd65b9a1a90c29d130b8c02d
2020-11-29 18:09:33 -07:00
JT Olio
5d8a67a4f7 satellitedb: retry GetBandwidthSince on cockroach
Change-Id: I2bf20f3a19e7f3af97630d8a679410feba70661e
2020-11-29 16:36:15 -07:00
Ethan
5dc013d3bd satellite/overlay: Add retry to all selects in overlaycache
Change-Id: I0356d71a35701f8e0ca04a34b2bb2aea666c1394
2020-11-29 16:46:57 -05:00
JT Olio
6bce907cb0 satellite: try to stream rollups to aggregation function to use less memory
this change tries really hard to never have all of the storage node
rollups in memory at the same time, up until the rollups are actually
getting summed together.

Change-Id: If67f49e7d71106798d996a6850b3e48671bd9e18
2020-11-29 10:26:32 -07:00
JT Olio
6aae21541f satellitedb: do saverollup in batches
Change-Id: I78278a192cba60541eee2986f54a88d5a479bd3e
2020-11-28 19:26:46 -07:00
JT Olio
0ba516d405 satellite: support pointing db components at different databases
the immediate need is to be able to move the repair queue back out
of cockroach if we can't save it.

Change-Id: If26001a4e6804f6bb8713b4aee7e4fd6254dc326
2020-11-28 18:39:16 +00:00
Moby von Briesen
75f0f713a3 satellite/repair/checker/checker.go: Use number of healthy pieces instead of SegmentHealth for injured segments queue.
We did not test the SegmentHealth function with actual production
values, and it turns out that values such as 52 healthy, 35 minimum
result in +Inf segment health - so pretty much all segments put into the
repair queue have the same health, which means we effectively aren't
sorting by health.

This change inserts numHealthy as segment health into the database so
the segments are ordered as they were before. We need to refine the
SegmentHealth function before we can support multi RS.

Change-Id: Ief19bbfee3594c5dfe94ca606bc930f05f85ff74
2020-11-28 12:16:32 -05:00
nerdatwork
5c34b62bde
Fixed typos in downtime tracking with audits doc (#3977)
Co-authored-by: Ivan Fraixedes <ivan@fraixed.es>
2020-11-27 17:25:21 +01:00
Ivan Fraixedes
7eb3b2d6d0
satellite/gc: Init map with an aprox size
Because the PieceTracker receives a piece count per nodes which is an
approximation of the number of nodes that they are going to be reported
by the metainfo loop so we can use as a good guess of the map's size and
initialized with it.

Change-Id: I644db40926c03e4c457457fb41d2ec1da059cea6
2020-11-27 10:44:19 +01:00
Ivan Fraixedes
319d2cad11
satellite: Fix typo in a comment
Change-Id: I151b824e868db1cc1e8b8e8af9f35b027db1e6ff
2020-11-26 15:44:49 +01:00
Michal Niewrzal
8ceef9f357 satellite/metainfo: temporary disable one assertion in test
This is need to merge https://review.dev.storj.io/c/storj/uplink/+/3208
, after that this code will be back.

Change-Id: If9f2f1db95c7a1bba64a41c45a39bd3096a519e7
2020-11-25 13:21:41 +00:00
Egon Elbre
3792e2921c satellite/accounting/tally: make test less fragile
MetadataSize can slightly vary and checking for exact value makes
difficult to change what's being encoded in metadata.

Change-Id: I5f1ade41bc26d115e6743367ee35cf1ba74795c9
2020-11-25 13:33:24 +02:00
Malcolm Bouzi
c6626748d0
web/satellite: create acces grant: duration selection logic (#3978) 2020-11-24 19:18:02 +02:00
VitaliiShpital
257c8682d3 web/satellite: create access grant: result step
WHAT:
result step for access grant flow

WHY:
to show user access gratn key

Change-Id: I0426691ce670efd1d280ba819ecb0281b514452a
2020-11-24 12:48:08 +00:00
VitaliiShpital
84fb8eee11 web/satellite: create access grant: generate access grant in web worker
WHAT:
generate access grant in web worker to return back to UI

WHY:
get access grant

Change-Id: If8bf6389d3bfabd8a8a12fb0bbbc46cd9ecd6e74
2020-11-24 12:48:01 +00:00
Kaloyan Raev
53b7fd7b00 satellite/{audit,gracefulexit}: remove logic for PieceHashesVerified
We now have the piece hashes verified for all segments on all production
satellites. We can remove the code that handles the case where piece
hashes are not verified. This would make easier the migration of
services from PointerDB to the new metabase.

For consistency, PieceHashesVerified is still set to true in PointerDB
for new segments.

Change-Id: Idf0ccce4c8d01ae812f11e8384a7221d90d4c183
2020-11-24 11:09:48 +02:00
Egon Elbre
9de1617db0 satellite/orders: ensure encryption keys handles set twice
Currently flag parsing seems to call Set twice, which causes problems
with encryption keys. We can clear for every set for now.

Change-Id: Id5c695b4020194ac1c50a2da9c7d2a896cb9216f
2020-11-23 19:47:22 +00:00