Commit Graph

7478 Commits

Author SHA1 Message Date
Egon Elbre
b14019b8c5 satellite/{metabase/rangedloop,metainfo/piecedeletion}: fix flaky tests
TestLoopContinuesAfterObserverError was failing due to system
granularity measuring the duration as 0.

TestDialer_DialTimeout was failing due to connection failure came with a
delay and wasn't being handled.

Change-Id: I4638c86f5d021a86c3d3529fab13cf3608f35c40
2023-02-09 16:07:00 +02:00
Vitalii
450772f034 web/satellite: first step of new access grant flow
Added icons and typings.
Added a first step (Create a new access) of the new flow.

Change-Id: Ic60073dd20fd9ac8728151eb9a2f182366c46e99
2023-02-09 14:39:59 +02:00
Kaloyan Raev
b1d4a159a6 satellite/metainfo: ListUploads returns incorrect UploadID
ListUploads returns incorrect UploadID if Expires was set in
BeginUpload. DB is truncating expiration date to microseconds precision
so we need to do this also in code.

Change-Id: Iee0cf45cb705342f6bb9a2f745acca91cce6ff52
2023-02-09 10:33:08 +00:00
Vitalii
65e3cfb9c6 satellite/{web, console}: feature flag for new access grant flow
Added a feature flag for new create access grant flow.
Also added some initial setup.

Change-Id: I7f738181c8a83f5a724f9e562427445cae146b6f
2023-02-09 00:35:35 +00:00
Cameron
38275bd710 satellite/nodeevents: deduplicate node IDs
Remove dupulicate node IDs when sending notifications.

Change-Id: I76c642e342081f8fdf8248443de8383b9e88eed2
2023-02-08 23:14:18 +00:00
Lizzy Thomson
f733cb255b satellite/satellitedb: drop unused last_verification_reminder column
(updated to migration step 225) remove users table column
last_verification_reminder and update migrate_test.go

Issue: https://github.com/storj/storj/issues/4560

Change-Id: Icbd8b70a8243b75a11c481a3b87df0ebe9402c5f
2023-02-08 16:23:09 +00:00
Egon Elbre
dd3d051170 satellite/satellitedb/dbx: add docs for user.dbx
Change-Id: I346ceaedf6963169bb9d6d1435454016239d4cb1
2023-02-08 17:30:29 +02:00
Egon Elbre
a1bd36bb64 satellite/satellitedb/dbx: add docs for repair.dbx
Change-Id: Icffcdebb7dcc1d10eb8c97212acc6ae309b2f92c
2023-02-08 17:30:27 +02:00
Egon Elbre
0e00c7b8da certificate/authorization,cmd/certificates: remove gob code
Now that we've migrated data we can remove gob handling.

Change-Id: I55f772b7a0dda71c51db0683bad3db66b89867ac
2023-02-08 15:05:40 +00:00
Michal Niewrzal
252c437b0e satellite/orders: ignore context canceled when updating bucket bandwidth
Orders from storage nodes are received by SettlementWithWindowFinal method. There is a stream which receives all orders and after getting
all orders we are inserting into DB storagenode and bucket bandwidth. Problem is with bucket bandwidth which is stored through cache which is often using context from SettlementWithWindowFinal stream to perform DB inserts and its doing this in separate goroutine. Because of that is possible that  SettlementWithWindowFinal is finished before flushing was finished and context is canceled while doing insert into DB

Change-Id: I3a72c86390e9aedc060f6b082bb059f1406231ee
2023-02-08 13:21:42 +00:00
Márton Elek
51a0d43eef earthfile: use latest storj-up base container for ad-hoc containers
note: $IMAGE:latest is also removed to avoid any confusion when containers are locally build. CI can push the latest tag anyway.

Change-Id: I653a1f3df0f2d09f10dec6c37c68ac7761f36bad
2023-02-08 12:23:27 +00:00
Wilfred Asomani
8ec3bc5803 web/satellite: move banners into page content
The banners on the web satellite previously hovered over the page
and overlapped other content as a result. This change moves them
into the page content so it overlaps nothing and is scrollable off-screen
This change also makes the upgrade banner only show on the dashboard and
only if the user joined more than seven days ago.

Issue: https://github.com/storj/storj/issues/5525

Change-Id: I7278c31201f09d3515d907b833622b04c6de8557
2023-02-08 09:22:58 +00:00
Jeremy Wharton
b5f4385278 web/satellite: update table row selected & hovered colors
Resolves #5478

Change-Id: Iaa8f0c5a0e6ca3239316a18abf32494beef4a2a1
2023-02-07 23:07:54 +00:00
Jeremy Wharton
49916b02f0 satellite/{console,satellitedb}: migration, methods for user settings
This change adds a database migration for a new table that stores
configurations for a user, the first of which is the session duration.
Database methods are implemented to interact with this table.

Resolves #5472

Change-Id: I01049265f385ea5de65907da1bc3bcf426d3c577
2023-02-07 21:48:29 +00:00
JT Olio
522aed083d private/server,satellite/contact,misc: use new storj/common noise helpers
this change uses the new storj/common noise helpers, which:
 * add a security fix (require an expected node id for validating
   noise key attestations)
 * stops doing an unnecessary order signature validation (it's
   already been done inside of PutPiece)
 * removes some duplicate code

Change-Id: I5e67a08ff216cd9c5b0b82e40b4d9de664b6b0fc
2023-02-07 09:53:45 -05:00
Egon Elbre
dcebcf770d satellite/metabase: add database comments
Change-Id: I78e19fb9301044d076e3eabf7e92fa6b715ddf03
2023-02-07 11:42:21 +02:00
paul cannon
d6f8be1ec6 satellite/audit: add ContainmentSyncChore
We will be needing an infrequent chore to check which nodes are in the
reverify queue and synchronize that set with the 'contained' field in
the nodes db, since it is easily possible for them to get out of sync.
(We can't require that the reverification queue table be in the same
database as the nodes table, so maintaining consistency with SQL
transactions is out. Plus, even if they were in the same database, using
such SQL transactions to maintain consistency would be slow and
unwieldy.)

This commit adds the actual chore.

Refs: https://github.com/storj/storj/issues/5431
Change-Id: Id78b40bf69fae1ac39010e3b553315db8a1472bd
2023-02-07 01:18:49 +00:00
Clement Sam
3d3f9d133a storagenode: fix B*h to bytes disk usage conversion
The used space graph values are correct when a single satellite is
selected but wrong for 'All satellites'. This is related to the
queries for getting the individual disk usages for all satellites
per day and the summary and average for all satellites per day:

1. dividing the sum of at_rest_total by the total_hours is wrong.
Simply put, we were assuming that, for example (4/2)+(6/3) equals
to (4+6)/(2+3), assuming we had 4 and 6 at_rest_total values with
2 and 3 respective hours.

2. To get the average, we need to first find the sum of the
at_rest_total_bytes for each timestamp across all satellites
before taking the average of the sums instead of just taking the
average from the individual satellite values.

Closes https://github.com/storj/storj/issues/5519

Change-Id: Ib1314e238b695a6c1ecd9f9171ee86dd56bb3b24
2023-02-06 18:50:31 +00:00
Egon Elbre
3146ad7f2e satellite/satellitedb: cleanup testing access
Previously we were exposing the testing facilities via interface casting
the necessary parts, however, when things are not part of the main
satellite.DB interface they need to be manually propagated. Rather than
relying on using hidden methods lets expose things as long as they don't
create a direct dependency to the database driver.

Change-Id: I2eb7d8b60f4b64de1320c2d32581f7be267c0f57
2023-02-06 14:36:11 +02:00
Wilfred Asomani
e8cd096eec satellite/{console,web}: make grapql mutations support publicId
This updates project related graphql mutations and queries to support
project publicId while maintaining support for project ID. The frontend
is updated to use only publicId when using these mutations/queries.

Issues:
https://github.com/storj/storj/issues/5409
https://github.com/storj/storj/issues/5413

Change-Id: Ib6241db157de3b37c86a4a98c9f682bf4a047b62
2023-02-06 11:57:44 +00:00
paul cannon
0e2fef977f satellite/overlay: add SetAllContainedNodes method to overlay.DB
We will be needing an infrequent chore to check which nodes are in the
reverify queue and synchronize that set with the 'contained' field in
the nodes db, since it is easily possible for them to get out of sync.
(We can't require that the reverification queue table be in the same
database as the nodes table, so maintaining consistency with SQL
transactions is out. Plus, even if they were in the same database, using
such SQL transactions to maintain consistency would be slow and
unwieldy.)

This commit adds a method to the overlay allowing the caller to set the
contained status of all nodes in the nodes table at once. This is valid
because our definition of "contained" now depends solely on whether a
node appears at least once in the reverification queue. Only rows whose
contained field does not match the expectation will be updated; the
contained timestamp will not be updated for a node which is supposed to
be contained and was already contained.

Change-Id: I8cabe56ad897b6027e11aa5b17175295391aa3ac
2023-02-06 10:18:54 +00:00
Wilfred Asomani
12f884e802 web/satellite: fix account menu visibility on tablets
This change fixes an issue where the "My Account" menu on the side nav
will not be visible on certain tablets.

Issue: https://github.com/storj/storj/issues/5445

Change-Id: I15d90851bc39fd1ab37e7e127c48d3de6c7e811e
2023-02-06 09:39:39 +00:00
Moby von Briesen
395426977f satellite/reputation: Alter logic for hasReputationChanged
This change modifies "reputation change trigger" logic for dq and
suspension in order to:
* count only nil -> not-nil and not-nil -> nil as "reputation changes"
* do not count the timestamp updating (not-nil -> not-nil) as a
  reputation change

Change-Id: Ie57c97e80e5f3b368e2f2dc004e680a2f9416d1c
2023-02-03 21:17:57 -05:00
Jeremy Wharton
add3034b43 satellite/payments/stripecoinpayments: forbid replacing partner coupons
Users with a partner package plan should be unable to replace their
plan's coupon. This change enforces this behavior by rejecting coupon
application attempts from users that meet this criteria.

Change-Id: I6383d19f2c7fbd9e1a2826473b2f867ea8a8ea3e
2023-02-03 17:15:01 +00:00
Jeff Wendling
f3f22d8443 cmd/uplink/ulfs: fix listing root on windows
listing "/" on windows was not returning files from
the root because it was adding an extra separator
unconditionally. the docs for filepath.Clean say

    The returned path ends in a slash only if it represents
    a root directory, such as "/" on Unix or `C:\` on Windows.

so we need to add the slash only if it doesn't already have
one to avoid the double slash problem while still ensuring
the path ends with a slash.

Change-Id: I98afc1f1a06bb06035c7647ecb0da3214080162d
2023-02-03 11:02:17 -05:00
Egon Elbre
873a202530 mod: bump storj.io/common
This bumps common, such that things build with Go 1.20.

Also, adds `go vet` checks for testsuite/storjscan and testsuite/ui.

The latest golang.org/x/bcrypt has a check that the new password is less
than 72 bytes, because bcrypt silently discarded them. This means our
own password validation has the same limitation. Old passwords should
still work fine.

Change-Id: Ibb8735b15eeb91460145906b81ae4e365e9ac418
2023-02-03 16:49:41 +02:00
Michal Niewrzal
36cbf4f0b8 private/testplanet: enable full table scan detection
Testplanet tests will print into logs (WARN) if full table scan will
be detected. Test won't be failed automatically. That's because
currently we have multiple queries which are doing full table scan and it's not trivial to change.

https://github.com/storj/storj/issues/5471

Change-Id: Ia2fcbfb9102424d58f95e00071329454a8c1066e
2023-02-03 12:11:18 +00:00
Michal Niewrzal
828654f98b gc: remove segment loop dependency
While splitting GC into two parts, bloom filter generation and bloom
filter sending, we missed to remove segments loop dependency from boom
filter sending part.

Change-Id: I48f213b0c0f8583e24fbb2bdcbfced518abe74cb
2023-02-03 10:31:16 +00:00
Michal Niewrzal
0b519fbd05 satellite/satellitedb: create index for interval_day for project_bandwidth_daily_rollup
To improve deletion of old entries in project_bandwidth_daily_rollup
we need index on `interval_day` column which is used to find those old
entries.

As an addition we are changing interval how often deletion is executed
from 7 to 1 day. We would like to have smaller portion of data to
delete.

Fixes https://github.com/storj/storj/issues/5465

Change-Id: Ie18ebe859887b93d6e4e6065a61fb9214c7ad27a
2023-02-03 10:02:58 +00:00
Moby von Briesen
d45b65bcfc cmd/tools/generate-missing-project-salt: Clean up test
Move global variables to be local for each test to reduce the likelihood
of unexpected bugs. Also parallelize the different db tests and clean up
unnecessary lines/checks.

Change-Id: I9dc3894d0945430908b10af5aeeba2f9246caf2a
2023-02-02 16:36:38 +00:00
JT Olio
686faeedbd satellite/overlay: return noise info with selected nodes
we have two more fields in the database (noise_proto and
noise_public_key) that now need to go into pb.NodeAddress when
returning AddressedOrderLimits.

the only real complication is making sure type conversions between
database types and NodeURLs and so on don't lose this new
pb.NodeAddress field (NoiseInfo). otherwise this is a relatively
straightforward commit

Change-Id: I45b59d7b2d3ae21c2e6eb95497f07cd388d454b3
2023-02-02 15:46:27 +00:00
Cameron
de44c6ab58 satellite/admin: update README
add descriptions for freeze and unfreeze user endpoints and regenerate
table of contents. This additionally added the oauth endpoints and one project limits POST parameter to the ToC

Change-Id: I05025f1f3a11c3775a4f59b01569ccb419d72858
2023-02-02 14:40:24 +00:00
Stefan Benten
b979bb3232 Jenkinsfile: remove outdated email step
Change-Id: Ie0c953def5f832bbd6a971fdb960564cd88cc6fd
2023-02-02 13:51:52 +00:00
Michal Niewrzal
c1fffe881a satellite/satellitedb/satellitedbtest: enable full table scan detection
Satellite DB tests will print into logs (WARN) if full table scan will
be detected. Test won't be failed automatically. That's because currently
we have multiple queries which are doing full table scan and it's not
trivial to change.

We may change that behavior when we will figure out how to skip
specific query from detection or we will fix all problematic queries.

https://github.com/storj/storj/issues/5471

Change-Id: Icafe782257a0d353e8bcdf6fa8a19c20b1091a0b
2023-02-02 12:14:05 +00:00
Michal Niewrzal
4e0c7b2d90 satellite/metabase/metabasetest: fix race while running tests
Change-Id: I4fdb328443617bc9710ee6b9168b31870fe336d9
2023-02-02 10:35:28 +00:00
Michal Niewrzal
bfdc60f3d8 update CODEOWNERS
Change-Id: I66bf195bab7ae98a17535c114b582a67a1eb0ab3
2023-02-02 10:01:50 +00:00
JT Olio
2753d5a32f satellite/overlay: keep track of noise info per node
Change-Id: Icef04c3e87dbf4bb57d3837274c323bf6dd2c81f
2023-02-01 23:03:35 -05:00
paul cannon
7652a598d8 satellite/audit: add GetAllContainedNodes method to ReverifyQueue
We will be needing an infrequent chore to check which nodes are in the
reverify queue and synchronize that set with the 'contained' field in
the nodes db, since it is easily possible for them to get out of sync.
(We can't require that the reverification queue table be in the same
database as the nodes table, so maintaining consistency with SQL
transactions is out. Plus, even if they were in the same database, using
such SQL transactions to maintain consistency would be slow and
unwieldy.)

This commit adds a method to the class representing the reverify queue
in the database, allowing us to get the list of every node that has at
least one record in the reverification queue.

Refs: https://github.com/storj/storj/issues/5431
Change-Id: Idce2633b3d63f2645170365e5cdeb2ea749fa9cb
2023-02-02 00:39:29 +00:00
Jeremy Wharton
091ed29935 satellite/payments/stripecoinpayments: make price overrides per-bucket
This change causes the bucket's partner info to be used rather than the
user's when calculating project usage prices. This ensures that users
who own differently-partnered buckets will be charged correctly for
usage based on the specific bucket they are utilizing.
according to the bucket's partner.

Related to storj/storj-private#90

Change-Id: Ieeedfcc5451e254216918dcc9f096758be6a8961
2023-02-01 23:17:25 +00:00
Cameron
0596651580 satellite/satellitedb: fix updating nodes.last_software_update_email
The CASE expression used to determine which value to set
last_software_update_email to did not have an ELSE clause. Therefore,
when the node is both below the minimum version and did not receive a
version update email (no condition is true), the value would be set to
NULL.

Additionally, replace `time.Now()` with `timestamp` in the check to
determine if the email cooldown has passed.

Change-Id: I2e2e93f1a865e123ed8b665be9621cebfb72236f
2023-02-01 17:25:58 -05:00
Wilfred Asomani
e751ee3c1b web/satellite: dismiss access grant modal tooltip
This change fixes an issue where the tooltip on the create access grant
modal will not dismiss even after hovering/clicking outside of it.

Issue: https://github.com/storj/storj/issues/5502

Change-Id: Ib6c042f3b18566bfeb4b05cb50acfa5e5ca1fbb9
2023-02-01 18:36:55 +00:00
Moby von Briesen
b4dbfa0cf8 web/satellite: Fix permission setting for access grants modal
Every case for the check/uncheck logic was not covered to actually
update the permissions list that gets sent to the worker to create the
access. This commit fixes these cases.

Solves https://github.com/storj/storj/issues/5512

Change-Id: I15f30f9f989039df110930b5a6e8914f5bf9d91e
2023-02-01 17:52:20 +00:00
Egon Elbre
6e298cca90 cmd/tools/node-cleanup: include tz information in test
Change-Id: Ic31b585b7df3fcb65fa37055ee62888fa43fa418
2023-02-01 17:21:38 +00:00
Jeremy Wharton
897de167a6 satellite/console,web/satellite: remove user input from welcome and invite emails
Emails should not contain user input that could be used by malicious
agents to deliver a message. Usernames have been removed from
account activation emails, and project names have been removed from
project invitation emails.

References storj-private#133

Change-Id: Ic05921149b409145df109c0966ea5dfd86d86eb1
2023-02-01 16:29:57 +00:00
Michal Niewrzal
15508d270c satellite/orders: don't store non user bandwidth actions for bucket
For bucket_bandwidth_rollups we are trying to insert lots of entries
with empty bucket name and project id. Those are inserts from orders
created by repair, audit and GE. High load on the same primary key
(the same range) is causing many retries and that's affect all inserts
as we are putting 1000 entries into DB a once.

This change solves this problem by not storing into
bucket_bandwidth_rollups other actions then GET and PUT. Those actions
are only important from bucket bandwidth usage perspective because
those are actions performed by users. Other actions (repair, audit or
GE) are also stored in storagenode_bandwdith_rollups so we will still
have access to them e.g. for statistic purposes.

https://github.com/storj/storj/issues/5332

Change-Id: Ibb5bf0a4c869b0439dc65da1c9342a38ca2890ba
2023-02-01 15:38:48 +00:00
Vitalii
7dfa379d77 web/satellite: show prompt for project passphrase on project create/switch
Show modal to enter project passphrase when switching between projects or creating new one.
Also fixed a bug when create project passphrase modal is shown twice after onboarding 'continue in web' click.

Change-Id: Iab64cf9f457b9739314610612608a5d8070810e3
2023-02-01 16:41:46 +02:00
Michal Niewrzal
3b6e1123b8 satellite/orders: fix sorting rollups before inserting
Sorting by primary key before inserting data into DB is fixed.
Earlier we were sorting input slice of BucketBandwidthRollup but then
we were putting all entries into map to rollup input data. Iteration
over map with a range loop doesn't guarantee any specific order so we
were loosing sorted order when we were creating with this map slices to
use with DB insert.

New code is also using map but when map is full its sorting map keys
separately and iterates over them to get data from map.

https://github.com/storj/storj/issues/5332

Change-Id: I5bf09489b0eecb6858bf854ab387b660124bf53f
2023-02-01 12:17:25 +00:00
Egon Elbre
9a6be7f1e7 private/server: avoid relying on timing
The previous test relied on timing, but instead let's try dialing
the server and see whether we can do something with the connection.

We probably should test all the supported protocols instead of just tcp.

Change-Id: I9217494859faea0a7b93515aad706da4fdd8a140
2023-02-01 12:33:56 +02:00
Egon Elbre
3976a2fd1d go.mod: bump common to latest
Temporarily allow larger wasm size, while we migrate to picobuf.

Change-Id: I55916f47486465e4223cbd79f412ab2876fee042
2023-02-01 12:33:37 +02:00
Egon Elbre
b8c7dcbf7b certificate: improve gob migration
`storage.KeyValueStore` requires ordered iteration, which redis
doesn't support natively. This would require loading all the keys
into memory and then processing them, rather than iterating over them
one-by-one.

This adds a temporary `IterateUnordered` to handle the migrations
more gracefully.

Change-Id: I55b763500523077c7ab8fdfad175c32cc7788e47
2023-02-01 10:21:48 +00:00