Commit Graph

6564 Commits

Author SHA1 Message Date
Malcolm Bouzi
a7ca78a519 web/satellite: Add user account tabs to signup ui
- toggle logic to trigger visual tab change
- toggling tabs renders appropriate fields
- error handling for professional account inputs
- successfully create professional user account

Goal is to be able to target users with specific professional focused communication.

Change-Id: Iffbeb712dac24ea1a83bb374740e0c1cd2100663
2021-03-02 15:02:40 +00:00
Michał Niewrzał
c51ea68ad3 satellite/metainfo/metabase: reduce number of fields for LoopSegmentEntry
For metainfo loop we need only some of Segment fields. By removing some of them we will reduce memory consumption during loop.

Change-Id: I4af8baab58f7de8ddf5e142380180bb70b1b442d
2021-03-02 15:04:54 +01:00
Natalie Villasana
c290e5ac9a satellite/orders: decrease FlushBatchSize default to 1000
The previous default FlushBatchSize of 10000 was causing major
slow down in select and insert statements on bucket_bandwidth_rollups.
We saw on the saltlake satellite that a FlushBatchSize of 1000 helped
reduce contention and query latency.

Change-Id: Ib95e73482219bc5aedc11925b1849fa5999774ba
2021-03-02 14:00:48 +00:00
Michał Niewrzał
c223c2d845 satellite/metainfo/metabase: rename ListObjectsSegments to ListLoopSegmentEntries
This method will be used only with metainfo loop and we need to customize query to consume less memory.

Change-Id: Iaa97392f483c5df5609d501b3847b80eb1ea2583
2021-03-02 12:24:59 +00:00
Michał Niewrzał
8e97111513 satellite/metainfo/metabase: reduce fields number for LoopObjectEntry
We want to read from DB only those fields that are used by metainfo loop so we need to remove most of fields from LoopObjectEntry.

Change-Id: I14ecae288f631dc0ff54f4c560ce43b736eccdcf
2021-03-02 12:21:19 +00:00
Egon Elbre
7e72a231c2 satellite/metainfo/metabase: cast bucket_name to []byte
Currently our metabase assumption is that it may contain arbitrary
bucket names and endpoint applies the naming constraints as it sees fit.
However by passing bucket_name as TEXT pg and crdb automatically try to
convert it to []byte, which may or not may work as intended... or in
some cases not work at all.

Cast all bucket name arguments to []byte to make it work.

Change-Id: I44650f5c873010997398bb0163d7f56ff6d9b5cf
2021-03-02 13:22:49 +02:00
JT Olio
932039ebdf blueprint: layer2 support for zksync
Change-Id: I130ce4c6d04b6f56acdbaefff507c694e8bf8260
2021-03-01 23:41:11 +00:00
Egon Elbre
ba0197a9b7 satellite/metainfo/metabase: avoid full table scan
Change-Id: Id47ffb5e1287d9303ce6fb530e87dbdc23cf8307
2021-03-01 23:34:53 +02:00
Cameron Ayer
aeac6264cd sallite/satellitedb: add metric stray_nodes_dq_count
Add metric so we can see how many nodes are DQd due to
this.

Change-Id: Ie4bdd1375fb9bd948af14fed9a2962b783b6a526
2021-03-01 21:06:36 +00:00
Michał Niewrzał
646cf229a2 satellite/metainfo/metabase: rename FullIterateObjects method to IterateLoopObjects
We want to have custom loop iterator to avoid reading all object fields to reduce memory consumpion. This is first step to just rename existing iterator to IterateLoopObjects.

Change-Id: I8878ff21a49ba224db2d497cc8f9076e75c7609e
2021-03-01 17:19:05 +01:00
Natalie Villasana
856db68fd9 satellite/gracefulexit: extend GE data cleanup to include exit_progress
The new 'consistency ge-cleanup-orphaned-data' cli command deleted
orphaned transfer queue items, but not entries in the
graceful_exit_progress table. This will delete orphaned entries
from the exit progress table too.

Change-Id: I5f927aac1f258490678deaf179be92ccfe10fcd8
2021-03-01 15:52:43 +00:00
Cameron Ayer
411a7ad0bc satellite/satellitedb: drop uptime_reputation_alpha and uptime_reputation_beta from nodes table
Change-Id: Ib46e783bf1a5c036394b4cac281382d0380bb1be
2021-03-01 15:30:51 +00:00
Egon Elbre
994854dcda satellite/metainfo/metabase: use key with prefix as the next cursor
We shouldn't use a trimmed object key as the next cursor.

Change-Id: I8513f7d94ed366dff9e9eca732c0e1ded6f5cd71
2021-02-28 22:05:42 +02:00
Egon Elbre
261a4c1c09 satellite/metainfo/metabase: fix iterator boundaries
Currently the old encrypted keys may not match the path component
encoding. Change the iterator such that the prefixes handle arbitrary
byte sequences.

Change-Id: I0a50049f4ef9887e1c4df6f9692f967a054430eb
2021-02-28 21:19:51 +02:00
Moby von Briesen
0370d1553e storagenode/console: Fix TestStorageNodeApi flakiness
TestStorageNodeApi is failing due to very slight differences in
float values. This change rounds these values to 3 decimal places before
comparing them.

Change-Id: Ic7fae3a5e0a0a942c03d982bfa7b19357f2e3d2e
2021-02-26 15:12:35 -05:00
Stefan Benten
3a533640ce
Makefile: downgrade to go v1.15.7 (#4048) 2021-02-26 17:34:57 +01:00
Fadila Khadar
cf0c88c679 cmd/uplink: fix no result when listing single file
uplink ls sj://bucket/file.ext gives a result if file.ext exists.

Change-Id: Ife741926acacf510e4e098009ce1724386c7522f
2021-02-26 11:43:02 +00:00
Michał Niewrzał
68605f32ed satellite/metainfo: limit number of batched segments in metainfo loop
New metainfo loop can have memory issues when in one batch we will have object with many segments. This change limits number of batched segments to defined limit. Solution is not perfect as if we will have single object with extreme large segments count it can cross defined limit a lot. We need to prepare safer solution soon.

Change-Id: Iefcf466d5bac76513d4219b1a9d99adc361c54ae
2021-02-26 12:32:43 +01:00
Egon Elbre
4a2c6f6208 private/dbutil/pgutil: add UUIDArray helper
Change-Id: I469fa2381b7c1a731880a742787bb4842af9b902
2021-02-26 09:56:15 +02:00
Egon Elbre
1d28dc314c private/dbutil/pgutil: move type helpers to separate file
Change-Id: I6357894434b565b9e48dbe5ac9b87542dcd5fc38
2021-02-26 09:56:15 +02:00
Vitalii Shpital
4eb12eb70b web/satellite: create a project on AG name step
WHAT:
right now we've got accounts with PM but without projects.
So when user tries to create AG they get uuid error which means that there is no uuid of a project provided.
So we decided to create default project for such users before actual creating of AG.

WHY:
backwards compatibility

Change-Id: Icab857e711f1020fdcf8a12279153014d9daf11c
2021-02-25 17:41:27 +02:00
Fadila Khadar
8bbabe57db cmd/uplink: allow deletion of pending objects
Basic command for deleting a pending object by object key.

Change-Id: I47f68acf79cb0568a42a65b6d8195ca309b185a7
2021-02-25 14:23:56 +00:00
Qweder93
f2be856864 storagenode/payouts: estimated payouts calculations upgraded
Added recalculations of estimated income in current month in case
node joined in current month

Change-Id: Iac925fd8bcf2be8f1089e5f001a01448ba7650be
2021-02-25 14:11:44 +00:00
Michał Niewrzał
1af9400a23 satellite/satellitedb/dbx: remove unused methods
Turns out that many methods generated with dbx where not used at all.
Lets remove them.

As a next step we can think about dropping tables like:
* user_credit
* offer

Change-Id: Id6cda81a701348db2a6b8b26daa22ae9c4f87cb4
2021-02-25 13:50:50 +00:00
Fadila Khadar
e96ed289e5 cmd/uplink: add --pending for listing pending objects
Change-Id: I19c1965117c386f63e7b1c71ba70402d312329f0
2021-02-25 13:50:10 +00:00
Michał Niewrzał
a0cc7e8c5e satellite/metainfo/metabase: use segment.Pieces to check if segment is inline
It looks that we cannot use root piece id as indicator if segment is inline as we have case in SLC satellite that inline segment have root piece id set. Pieces should be better thing to check.

Change-Id: I2377ff88861390342273f5e71871373eaf462615
2021-02-24 22:14:18 +01:00
NickolaiYurchenko
badd6d12d4 web/satellite: distributed amount added to tables, content changed
Change-Id: Ib0eb3fa3f4f9d451f4b4048a6409d19968a41956
2021-02-24 16:01:40 +00:00
Vitalii Shpital
300e88f9a7 web/satellite: config flag for satellites in beta
WHAT:
config flag to indicate if satellite is in beta

WHY:
to avoid using hardcoded satellite names which may cause issues

Change-Id: If92eb7417c340bf343a9a91e2f6b11f0349020c5
2021-02-24 12:29:07 +02:00
Malcolm Bouzi
f3e75c754f satellite/console: add new professional tab fields to create user function in service
Change-Id: I746062ef6f938fa145a3eb29a491399c2cf35806
2021-02-23 18:25:49 +00:00
Michał Niewrzał
846bb895e0 satellite/metainfo: loop cleanups
Change-Id: I89b4f519ce18cb0ce34a61a4cbe1675d93741637
2021-02-23 17:48:36 +01:00
Vitalii Shpital
c3e13b9b38 web/satellite: change info banner for US2 satellites
WHAT:
added message for us2 users that their data may be deleted at any point of time

WHY:
to make users aware

Change-Id: I4369fc4ec780ab66e7355c664740f8f6f5a32392
2021-02-23 13:53:40 +00:00
Michał Niewrzał
95b78e8011 satellite/metainfo: select segments in batches for metainfo loop
Segments are not read in batches. For each batch of objects
we are reading all segments for those objects.

Change-Id: Idaf19bbe4d4b095065d59399dd326e22c57499a6
2021-02-23 11:38:44 +00:00
Michał Niewrzał
d995fb497f Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I367da03351ab80f7343332420490dde9282aa47a
2021-02-23 12:31:31 +01:00
Egon Elbre
0178ec7771 satellite/satellitedb: clearer testdata version parsing
Change-Id: I93a23625b30472e826b962dab31ff3e1081b1b8f
2021-02-23 10:22:19 +02:00
Egon Elbre
32afbd0d15 satellite/satellitedb: faster test database setup
Pregenerate the database schema we should use for most tests.
Currently, Cockroach is slow with regards to migration and it's
better if it happens in as few transactions as possible.

This reduces test time from ~21min to ~15min.

Change-Id: Ife8117053e6b9ecf3c93fe63677edf15d4d7c254
2021-02-22 21:13:00 +02:00
Egon Elbre
a34da8531b ci: bump test timeout
Currently testing with coverage exceeds the timeout very easily.

Change-Id: I4e77aa98d28bebe373416ac05ee39b4c70c3fed2
2021-02-22 17:31:21 +02:00
Vitalii Shpital
fce5c95eb2 web/satellite: logic for beta satellites terms confirmation
WHAT:
this logic is used on us2 sign up page to accept additional beta terms.

WHY:
to make US2 users know that their data can be removed.

Change-Id: I43f25bc945a5957bacc6662499830cb2dd6d4079
2021-02-22 14:54:51 +00:00
Egon Elbre
21a6dccd36 ci: set specific line coverage targets
Currently the auto-updating keeps moving the target upwards without
allowing for variations in the pipeline.

Let's try these settings:

* >=70% lines covered, to pass
* <=60% lines covered, to mark as unhealthy
* <=50% lines covered, to mark as unstable

Change-Id: Idc75e682345f6d62af34379362c75d321e564d2c
2021-02-22 15:13:18 +02:00
Egon Elbre
bc68898846 ci: enable coverage for multipart-upload branch
Change-Id: I4797d16bbc90a3fc7fc9627386b63124755f4730
2021-02-22 13:43:32 +02:00
Cameron Ayer
549033f2e6 satellite/satellitedb: don't include DQd and exited nodes in DQStrayNodes
Don't update DQ time of already DQd nodes. Don't DQ nodes who exited.

Change-Id: I4528a9ba9f8e278987165ad337a9b34dadb9788b
2021-02-19 15:12:30 -05:00
Egon Elbre
3c49f0ff73 cmd/satellite: restore-trash, cleanup workers
Use a limiter instead of a custom work queue.

Change-Id: I1506daaaaeb2935f6011ddaeb83a0904abb69c98
2021-02-19 19:08:10 +02:00
Ethan
79ecd80c0a satellite/metabase: Remove subquery on delete objects for CRDB
The subquery for DELETE FROM obects returns a stream_id field for filtering. Unfortunately stream_id is not indexed. This change removed the subquery from the CockroachDB delete bucket query.

Change-Id: If1abe21668c593e6d4bdc3ba8cdbad26c09d234e
2021-02-19 16:51:01 +00:00
Egon Elbre
630718c392 satellite/gracefulexit: fix test DeleteAllFinishedTransferQueueItems
Using random piece num may generate the exact same key or piecenum.
Instead use fixed piecenum and key.

Change-Id: I54b7bc1a6698149bf99608dd46501ea963cec084
2021-02-19 18:15:22 +02:00
Egon Elbre
62a2d05ae6 satellite/accounting: try fix test ResetLimitsFirstDayOfNextMonth
TestProjectUsage_ResetLimitsFirstDayOfNextMonth seems to be flaky.
It doesn't seem to reproduce easily, however it should flush orders from
storagenodes to satellites, that way the orders chore flushing on the
satellite makes sense.

Change-Id: I586d9d8ce10b5f6320e79324f6128b4f8c2cac5f
2021-02-19 17:29:15 +02:00
Egon Elbre
1137620baf satellite/satellitedb: move tests to their domains
Testing interfaces is slightly clearer when it's in the package needing
the database rather than each individual implementation.

Change-Id: I10334c214a205f7e510b939b4359a2214c4e060a
2021-02-19 17:29:15 +02:00
JT Olio
e23f423107 uplink: add api key back to inspect
Change-Id: I9aa46f3910a76a81a399597d7512056b218e55af
2021-02-19 14:46:35 +00:00
Kaloyan Raev
e0f15130a2 satellite/metainfo: ensure list prefix is prepended in satStreamID
When listing pending objects with prefix, the prefix should be prepended
to the EncryptedPath in satStreamID. Otherwise, listing multipart
uploads may display different UploadID than expected.

Change-Id: I27e9f9af9348783e053ad123121b6ddd051739e4
2021-02-19 16:32:55 +02:00
Michał Niewrzał
8b9da01817 satellite/metainfo/metabase: allow committing empty inline segments
We need to keep empty inline segments as we did it with pointerDB because otherwise old uplinks after uploading data won't be able to download such file. To reduce number of empty inline segments on uplink side we need to implement skipping empty last inline segments for multipart upload.

Change-Id: Ice86c805babba1ad17149754cbd6b3f4fd652722
2021-02-19 13:41:34 +00:00
Egon Elbre
4c0ea717eb satellite/metainfo: remove unneeded dependencies from Loop
metainfo.Loop doesn't require buckets nor pointerdb anymore.

Also:
* fix comments
* update full iterator limit to 2500

Change-Id: I6604402868f5c34079197c407f969ac8015e63c5
2021-02-19 15:11:16 +02:00
Egon Elbre
a25e35f0b0 cmd/metainfo-loop-benchmark: add benchmark
Change-Id: I0745cfcf9f8c9d73fd025dcba6ee8a7480273fe2
2021-02-19 10:59:41 +02:00