Commit Graph

2697 Commits

Author SHA1 Message Date
Clement Sam
f2f046af7a satellite/satellitedb: add interval_end_time column to accounting_rollups table
Adding an interval_end_time column to the accounting_rollups table
to keep the last interval_end_time for each daily storage tallies.

Updates https://github.com/storj/storj/issues/4178

Change-Id: If7a8210c5e9fe2fc9df84b137a8b6e3db2471c58
2022-07-27 17:53:27 +00:00
Qweder93
2b24632610 {satellite/metainfo, satellite/accounting}: remove ValidateSegmentLimit config value and check
removed segment limit validation and checks in metainfo endpoint and accounting/projectusage
since feature is live and has always has segment limitation now

Resolves: https://github.com/storj/storj/issues/4470

Change-Id: I8cf87cbbc40ac61262f9f05e52573d3ae6410611
2022-07-27 15:36:53 +00:00
Yaroslav Vorobiov
7194973b56 satellite/payments/storjscan: add usd value to cached payments
Add USD value to storjscan cached payment struct to store and
retrieve payments from payments DB.

Change-Id: Ic0b2b4a198e4ea8230bc9a01865b429343f52767
2022-07-27 14:55:45 +00:00
Michał Niewrzał
7390f389cd satellite/metabase: non recursive listing optimization
With pointerdb listing objects operation was optimized to skip
objects from prefixes for non recursive listing. This change it
adopting this optimiaztion from old code.

Main change is to drop current page results if we detect a prefix
that needs to be skipped and jump with next listing query after this
prefix by setting cursor to "some-prefix(byte('/')+1) which is
effectively "some-prefix0".

Benchmark:
name                                                 old time/op  new time/op  delta
NonRecursiveListing/Postgres/listing_no_prefix-8      960µs ±11%   257µs ±12%  -73.19%  (p=0.008 n=5+5)
NonRecursiveListing/Postgres/listing_with_prefix-8    945µs ±11%   671µs ±12%  -28.97%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_no_prefix-8    4.31ms ± 8%  1.19ms ± 7%  -72.44%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8  4.97ms ± 8%  3.35ms ±15%  -32.67%  (p=0.008 n=5+5)

Fixes https://github.com/storj/team-metainfo/issues/115

Change-Id: Iafdf3600d058abbaf441f792d32a7fc17cc08696
2022-07-27 11:26:35 +00:00
Erik van Velzen
b5fc04af89 satellite/metainfo: usage limits for copy
Previously there was no realtime administration of the storage usage
during copies. Now there is.

Closes https://github.com/storj/storj/issues/4719

Change-Id: I0d536bf551d16208116c3aceac89ed590ec473bf
2022-07-25 12:35:27 +02:00
Moby von Briesen
c96ff2fdef satellite/analytics: Track have_sales_contact differently
When signing up, a user can opt in to having sales contact them. This
change alters the way this flag is passed to Hubspot and Segment.

Hubspot sends a form submission request to create the user, followed by
a "custom behavioral event" with some additional user info.

Segment sends an "Identify" call followed by a "create user" event.

This change moves "have_sales_contact" to the form submission for
Hubspot, and the Identify call for Segment.

This simplifies the process of applying this field to a contact/user in
Segment and Hubspot.

Change-Id: I5e6871b3926a76fb24f97fb2d835a26720275072
2022-07-22 19:22:05 +00:00
prerna-parashar
ed2ee8443f
satellite/analytics: Modified property update in HubSpot (#4998)
satellite/analytics: Update lifecyclestage = "other" in segment identify calls and createUser Api in Hubspot
2022-07-22 09:59:53 -07:00
Cameron
4815cfc09b satellite/{console,satellitedb}: add PublicID to Project, db method GetByPublicID
github issue: https://github.com/storj/storj/issues/4861

Change-Id: Ia83635c0de751a77cd5a49d641da19ed76132c46
2022-07-21 19:49:26 +00:00
Lizzy Thomson
c237468ac9 satellite/console: allow user to update project when limits are above paid defaults
When a user's bandwidth/storage limits are manually set to exceed the
paid tier defaults, attempting to update their project via the satellite
UI (e.g. to change the name/description) would result in an error.
This change modifies the limit checks for updating a project to remove
this issue.

https://github.com/storj/storj/issues/4892

Change-Id: I48853a3289b0ac51587f268a18c1b25743123fcf
2022-07-21 08:40:18 -06:00
Michał Niewrzał
d72f9525d4 satellite/metainfo/piecedeletion: use nodes cache
Piece deletion service was using KnownReliable method from
overlaycache to get nodes addresses to send delete request.
KnownReliable was always hitting DB because this method was
not using cache. This change is using new DownloadSelectionCache
to avoid direct DB calls.

Change is not perfect because DownloadSelectionCache is not as
precise as KnownReliable method and can select few more nodes
to which we will send delete request but difference should be
small and we can improve it later.

Updates https://github.com/storj/storj/issues/4959

Change-Id: I4c3d91089a18ac35ebcb469a56536c33f76e44ea
2022-07-21 00:43:24 +00:00
Egon Elbre
82fede2132 satellite/satellitedb: batch delete storage node tallies
Currently we have a significant number of tallies that need to be
deleted together. Add a limit (by default 10k) to how many will
be deleted at the same time.

Change-Id: If530383f19b4d3bb83ed5fe956610a2e52f130a1
2022-07-20 15:42:35 +03:00
Erik van Velzen
87e1528104 satellite/metabase: error if source object changed
An object copy/move is done by 2 DRPC calls. It's possible a new object was uploaded ore moved to the source location between these calls. For copy, in that case the segments end up with the wrong keys. This change adds an explicit check for that by comparing the streamId supplied by the user with the streamId in the database.

Fixes https://github.com/storj/storj/issues/4930

Change-Id: Id600456ce78fb4069b93644828a0b3eb85e23e16
2022-07-19 13:27:55 +02:00
Qweder93
cf12802a12 satellite/metainfo: ListBuckets extended with value attribution assigned to bucket
We need to provide the ability to see bucket attribution on the gateway side
so customers can validate if bucket is attributed to them. Extendet metainfo.ListBuckets
request with UserAgent.

Fixes https://github.com/storj/storj/issues/4965

Change-Id: I5624874a7faa14cda06183ad44013e9ebb385b63
2022-07-18 18:23:50 +00:00
Ivan Fraixedes
7aaab3c4c4 satellite/console/consoleweb: Trace HTTP endpoints
Trace all the requests that the HTTP API endpoints receive.

We want to trace them with Monkit because we want to break them down by
request type and response code for seeing if they succeeded or failed.

Also log them with DEBUG level with the IP client.

Change-Id: Ia7b013351c788f131e775818f27091f3014ea861
2022-07-18 13:27:50 +00:00
paul cannon
726c95160b satellite/repair: avoid retrying GET_REPAIR incorrectly
We retry a GET_REPAIR operation in one case, and one case only (as far
as I can determine): when we are trying to connect to a node using its
last known working IP and port combination rather than its supplied
hostname, and we think the operation failed the first time because of a
Dial failure.

However, logs collected from storage node operators along with logs
collected from satellites are strongly indicating that we are retrying
GET_REPAIR operations in some cases even when we succeeded in connecting
to the node the first time. This results in the node complaining loudly
about being given a duplicate order limit (as it should), whereupon the
satellite counts that as an unknown error and potentially penalizes the
node.

See discussion at
https://forum.storj.io/t/get-repair-error-used-serial-already-exists-in-store/17922/36
.

Investigation into this problem has revealed that
`!piecestore.CloseError.Has(err)` may not be the best way of determining
whether a problem occurred during Dial. In fact, it is probably
downright Wrong. Handling of errors on a stream is somewhat complicated,
but it would appear that there are several paths by which an RPC error
originating on the remote side might show up during the Close() call,
and would thus be labeled as a "CloseError".

This change creates a new error class, repairer.ErrDialFailed, with
which we will now wrap errors that _really definitely_ occurred during
a Dial call. We will use this class to determine whether or not to retry
a GET_REPAIR operation. The error will still also be wrapped with
whatever wrapper classes it used to be wrapped with, so the potential
for breakage here should be minimal.

Refs: https://github.com/storj/storj/issues/4687
Change-Id: Ifdd3deadc8258f34cf3fbc42aff393fa545794eb
2022-07-18 05:11:56 +00:00
paul cannon
2f20bbf4d8 satellite/reputation: add a reputation write cache
This should lower the amount of database load coming from
reputation updates.

Change-Id: Iaacfb81480075261da77c5cc93e08b24f69f8949
2022-07-14 21:40:16 +00:00
Yaroslav Vorobiov
de5951331a satellite: wire storjscan chore to core process
Adds storjscan chore and storjscan client to satellite chore.
Storjscan chore interval added to storjscan config.

Change-Id: I13bb8d83b14ca90f4735c9a10c361ad24a56b256
2022-07-14 15:07:52 +00:00
Vitalii
69dc9a4731 satellite/console: added new email which is sent on unknown password reset
Added new email html template.
It is sent when user tries to reset password with unknown or unverified account.
Made a couple of minor config changes.

Issue: https://github.com/storj/storj/issues/4913

Change-Id: I730f48b3478e302d1e38e1f8a27c75f66a8ba6fd
2022-07-14 14:32:59 +00:00
Jennifer Johnson
61649d2850 cmd/node-cleanup: command to delete uncontacted nodes
Some nodes were added to the nodes table due to a bug in quic
based storagenode contact code. This is a tool to clean up these nodes.

Delete with batch-size 1k seems to take ~400ms on local CockroachDB.

Change-Id: Ic0c1180528c27952e19c431fc9cc327292a10a5f
2022-07-12 12:15:49 +00:00
Yaroslav Vorobiov
6d95e34e39 satellite/payments: add payments method for deposit wallets
Add payments method to payments to DepositWallets interface.
Exposes payments retrieval API for a particular wallet to
other systems such as console billing API.

Change-Id: Ifcb3a35514aab50be00f6360007954980b5d8b38
2022-07-12 11:25:30 +00:00
Egon Elbre
51d4e5c275 satellite/{orders,overlay}: use cache for downloads
Use DownloadSelectionCache to avoid querying database for every
download.
This change only addresses downloads from users. The download selection
cache is not currently used for audit and repair.

Change-Id: I96a49e121dac0b4204f97592a63131edabd73fb5
2022-07-12 11:04:34 +00:00
Egon Elbre
48b0a65fbd satellite/overlay: use ReadCache in Download/UploadSelectionCache
sync2.ReadCache implements preemptive refreshing preventing stalling
while it's being updated.

Change-Id: Iee9ef36049b986f0e426c14a139b2bc9ac17fb53
2022-07-12 13:52:48 +03:00
Egon Elbre
15dc797efd web/{multinode,satellite,storagenode}: revert go.mod
Adding go.mod into node_modules is not sufficient, because npm install
wipes them quite often out. Similarly, when running npm install locally
it will remove it, causing the git state to be dirty.

Rather than having them committed, add them after running npm install.

Change-Id: Iaf21a9c6e198dc31fe50345ec5dee85b44617176
2022-07-08 19:51:51 +03:00
Michał Niewrzał
43286dbc37 satellite/metainfo: use metaclient type instead of storj
This just cleanup change to unblock libuplink to reorganize types
which are aliases to storj types.

Change-Id: Id3edf13f1b0aef52d7606d545aa7a6594cf8d13f
2022-07-08 14:17:19 +02:00
Egon Elbre
4e31c96836 satellite/console: don't use global log
Change-Id: I2464f2aca3cdb97c19de29dac59499cc9d5ff2bb
2022-07-06 16:50:04 +00:00
Cameron
023a1c6271 satellite/satellitedb: add public_id column and index to projects table
github issue: https://github.com/storj/storj/issues/4861

Change-Id: I3d1928ca51c825a4c1589db5ae55902e57f00c3b
2022-07-05 18:39:29 +00:00
Qweder93
417df4d73c satellite/metabase: don't return expired objects
Resolves: https://github.com/storj/team-metainfo/issues/112

Change-Id: If48f865826c22764fc1ff83745c34524cd396aac
2022-07-04 21:32:30 +03:00
Ivan Fraixedes
1fbc8f1f40 satellite/console/consoleweb: Delete old TODO comment
Delete an old TODO comment which doesn't apply anymore.

Closes https://github.com/storj/storj/issues/4958

Change-Id: Ie4be51afb6a39b3f5e5822ac04d1c8fd9d86ee3a
2022-07-01 11:31:19 +00:00
Egon Elbre
97e6e5a6a3 build: improve npm build
Add go.mod to node_modules folder, that way Go compiler doesn't
need to scan the node_module directories for any Go code.

Change-Id: I747909416490c847d6b4bfa3438fea66660fcd53
2022-07-01 10:51:53 +00:00
Egon Elbre
05e165283f storagenode/console/consoleapi: use fixed time.Now()
It seems the tests relied on time.Now(), which might cause some
discrepancies in calculations. Use a fixed time.Now() rather than
recalculating.

As a sidefix, remove "Test" prefix from t.Run. These are unnecessary.

Change-Id: I1de903fcf0fcf46fc8e3acf2463e17239b8e3cc6
2022-07-01 12:36:01 +03:00
Erik van Velzen
f23d5eb5a1 satellite/repair: remove superfluous conditional
Change-Id: If80ae0a1a4ee436763ed437fc77b0ed26db17a68
2022-06-30 18:09:17 +00:00
littleskunk
cf750716ea
satellite/console: enable new access grant flow (#4934) 2022-06-30 00:06:39 +02:00
Egon Elbre
b8006e192b satellite/audit: use a larger delay in the test
The MinDownloadTimeout 950ms and delay of 1s were quiet close, possibly
causing flaky behavior in TestVerifierSlowDownload.

Change-Id: I4f6c1554a118b21427357642abe39986fd0af38d
2022-06-28 17:03:23 +03:00
Ivan Fraixedes
0051298eec satellite/console: Classify errors activation tokens
Classify errors related to invalid tokens for activating user accounts
for returning 400 status code rather than 500 status code.

Don't log all the errors with "error" level, only the ones related to
internal server errors and the rest log them with "debug" level because
they pollute the production satellite errors with errors that are
misguiding.

Change-Id: Id2bd737edba8550ce08965b51b8bf2540bd13ca4
2022-06-28 11:07:57 +00:00
Egon Elbre
65b5a0fe82 satellite/{overlay,satellitedb}: add AS OF to download selection
This should reduce the load caused by download selection queries.

Change-Id: Ic1a89d9c3eb5418f0792eb20ec2aece18dc63f2c
2022-06-28 05:18:01 +00:00
Erik van Velzen
74f4f6e765 satellite/metabase: fix copy to ancestor location
Previously copying an object to it's ancestor location (copy of copy)
broke the object and all copies.

This fixes this by calling the existing delete method rather than a
custom one when there is an existing object at the copy destination.

The check for existing object at destination has been moved to an
earlier point in FinishCopy.

metabase.DeleteObject exposes a transaction parameter so that it can be
reused within metabase.

Closes https://github.com/storj/storj/issues/4707

Uplink test at https://review.dev.storj.io/c/storj/uplink/+/7557

Change-Id: I418fc3337fa9f30146ccc1db456af168ae41c326
2022-06-28 03:53:01 +02:00
Jeff Wendling
3a63939fc9 satellite/metabase: add defensive check on copy object
- instead of closing over the outer err variable, potentially
  overwriting some errors or something, declare local variables.

- double check that we got the number of rows we expected to get
  and error otherwise. this prevents a possible source of inserting
  bogus rows into the database.

Change-Id: I30662be2727afe0a90e4215a182fedc2648d1169
2022-06-27 12:30:44 -04:00
Erik van Velzen
4da61aa257 satellite/metabase: segment_copies full table scan
Part of the delete query cause a full table scan of segment_copies. This
slowed down the system. This change should have the same semantics but
improved performance.

Part of https://github.com/storj/storj/issues/4898

Change-Id: I4afe23df05467eafc9c91591f47a7251a0f3dd31
2022-06-27 11:03:15 +00:00
Yaroslav Vorobiov
91ea659298 satellite/payments/storjscan: payment API type add USD value
Change-Id: I6e06a5557df795b4f91dd52b191384d7e6b4b07c
2022-06-24 22:30:30 +00:00
JT Olio
18666b078a metabase: support disabling serverside copy when already enabled
Change-Id: I99b0b5597537f69fbdd8cab479e059ba4bf310bd
2022-06-24 17:56:43 +00:00
Erik van Velzen
77fea6137f satellite/metabase: copy inside transaction
Read the source object and write the destination object in the same
transaction, to prevent breaking the object because it was deleted
simultaneously.

This is probably the root cause of the metainfo loop halting from
2022-06-21 onwards, where 2 objects lost their root_piece_id during
copying.

Part of https://github.com/storj/storj/issues/4930

Change-Id: I9c45d56a7bfb48ecd5f4906ee1cca42922901e90
2022-06-24 01:32:09 +02:00
Ivan Fraixedes
6e8e86a0b7 satellite/accounting: Fix typos in doc comments
Change-Id: I497bebe152848e023d927337c61e3eee73b87697
2022-06-22 14:45:21 +00:00
Vitalii
0dba785798 satellite/admin: return only owned projects for get user endpoint
Returns only user's own projects when we hit GET user endpoint.

Fix for this issue
https://github.com/storj/storj/issues/4820

Change-Id: I546268fa3e5983a72f11f998803da5455c0035b4
2022-06-22 13:09:36 +03:00
Ivan Fraixedes
f626cd1868 satellite/accounting: Correctly cache project bandwidth
Satellite caches the project bandwidth in Redis when it doesn't have it
because was not set or the key expired, however, it doesn't perform the
check and set if not exists in a transaction. It also uses the increase
function which increases the value if it exists otherwise it sets it.

This provokes that multiple concurrent request to the same project may
increase the total project by multiples of the bandwidth usage
registered in the database rather than setting it because they may check
if the key exists before any other has executed the increase and then
the first one executing it will set the value but the others will
increased causing that Redis has a wrong bandwidth usage value which is
N magnitude of the real one and making the satellite to deny the
downloading if it surpasses the project limit.

This commit changes the "update"" project bandwidth usage by an "insert"
but using a Redis function that only sets the value if the key doesn't
exists for solving the increase issue but also not overriding the value
due to may contain updates of other downloading requests which aren't
already registered in the DB.

Change-Id: I33e2fe462930b2fdb4061fc94002bd3544476f94
2022-06-21 08:31:41 +00:00
prerna-parashar
6a1d7c8747
satellite/analytics: Added tracks calls for product activity metrics (#4907)
satellite/analytics: Added tracks calls for product activity metrics
2022-06-17 12:57:10 -07:00
paul cannon
4e81a60838 satellite/overlay: fix TestNodeSelectionGracefulExit
This test had an effective config.Reputation.AuditCount = 0, meaning all
nodes that had _any_ positive audit results were considered vetted.
Because of that, only one node in the test setup was "new". And that
node was marked as being in GE, so could not be returned by node
selection.

The reason the tests still worked is because of the node selection rule
that says "if there are no new nodes at all, just get all reputable
nodes to satisfy the request".

This commit makes it so half of the nodes are vetted and half new, which
makes the test somewhat more interesting (and means we aren't
concentrating too much on testing details of behavior when AuditCount is
0).

Change-Id: I09157b7dc20ecaddd2a6e60cfe146e9186e3603b
2022-06-17 18:28:04 +00:00
Jeremy Wharton
5ce7d980af private/apigen: Make API generation deterministic
This change fixes the issue where the API generator would produce
different Go code for the same API definition upon each invocation
due to the random nature of map iteration.

Change-Id: I6770a10faf06311c24f541611c25d0b2b0f8e521
2022-06-17 12:06:08 -05:00
Qweder93
cc0dc2e262 satellite/metabase: FinishMoveObject cheks for empty metadata
To avoid regression with old versions of uplink objects move we need to
remove FinishMoveObject check for key and nonce, in SQL
in FinishMoveObject do check if metadata is nil then
don't set key and nonce. The same UPDATE clause should return metadata and
if metadata != nil we should do the same validation for key and nonce
to avoid putting broken key and nonce while doing move

Resolves: https://github.com/storj/team-metainfo/issues/108

Change-Id: If723dfad899e9235f53559b71ee1c7fe49deb8b8
2022-06-17 08:24:24 +00:00
Yaroslav Vorobiov
13be4ad0dd satellite/payments/storjscan: add chore to periodically ask for new payments
Change-Id: Ia6ad182184596fe1fe1f9a40f909a3af4be8feef
2022-06-16 09:38:37 +01:00
Fadila Khadar
ffcd54719a satellite/satellitedb: use tx instead of db.db in transactions
Change-Id: Icee8871af76233651f007e03173660905eafc0e3
2022-06-14 17:35:44 +00:00
Cameron
240b70b828 satellite/console: use new type UpdateUserRequest as arg to db users.Update
The users.Update method in the satellitedb package takes a console.User
as an argument. It reads some of the fields on this struct and assigns
the value to dbx.User_Update_Fields. However, you cannot optionally
update only some of the fields. They all will always be updated. This means
that if you only want to update FullName, you still need to read the
user info from the DB to avoid updating the rest of the fields to zero.
This is not good because concurrent updates can overwrite each other.

This change introduces a new struct type, UpdateUserRequest, which
contains pointers for all the fields that are updated by satellite db
users.Update. Now the update method will check if a field is nil before
assigning the value to be updated in the db, so you only need to set the
field you want updated. For nullable columns, the respective field is a
double pointer. This allows us to update a column to NULL if the outer
pointer is not nil, but the inner pointer is.

Change-Id: I27f842d283c2711e24d51dcab622e57eeb9157f1
2022-06-14 09:28:03 -04:00
Fadila Khadar
3ae325462c satellite/metabase: fix move transaction
Change-Id: Id1b6c721e7373dffd3f40aafa15109fc2cb82474
2022-06-14 11:04:44 +02:00
Erik van Velzen
b828b6ec34 satellite/metabase: fix copy transaction
Change-Id: I4921d3e6bc17b0c6aff00eba7c76266cb36e78a0
2022-06-13 19:47:07 +02:00
Jeremy Wharton
58c5d44f44 satellite/console: integrate sessions into satellite UI
This change integrates the session management database functionality
with the web application. Claim-based authentication has been removed
in favor of session token-based authentication.

Change-Id: I62a4f5354a3ed8ca80272814aad2448f901eab1b
2022-06-13 08:02:02 +00:00
Artur M. Wolff
8c1caea5db satellite/contact: swap net.IP.IsPrivateIP with isPrivateIP
This change swaps net.IP.IsPrivateIP usages with custom isPrivateIP to
unbreak the build as we want to build for earlier than Go 1.17.

Change-Id: I44badbb487f35e43b8b0433ad0f3b9c87af718d4
2022-06-13 01:01:44 +02:00
Yaroslav Vorobiov
da97fafb0c satellite/payments/storjscan: add payments DB
Change-Id: Ia6f0f62912c9a60d2c513459959634179eb9506c
2022-06-10 13:44:27 +01:00
prerna-parashar
cc0518f473
satellite/analytics: Added segment.io page calls to track all the pages (#4880)
satellite/analytics: send analytics 'page visit' api requests when the user navigates around the UI
2022-06-09 11:54:23 -07:00
Cameron
5cfa7ca460 satellite/console: add cutoff to email reminders
There are multiple entries in the users table with the same email
address. This is because in the past users were able to register
multiple times if the email was not verified. This is no longer
the case. If a user tries to register with an unverified email
already in the DB, we send a verification email instead of
creating another entry. However, since these old entries in the
table with duplicate emails were never cleaned up, the email
reminder chore will send out email verification reminders to them.
A single person will get one separate email per entry in the DB
with their email and where status = 0.

Since the multiple entries with the same email problem was solved
a while ago, just add a constraint to GetUnverifiedNeedingReminder
to only select users created after a cutoff. Once the DB is
migrated to remove the duplicate emails, we can remove the cutoff.

github issue: https://github.com/storj/storj/issues/4853

Change-Id: I07d77d43109bcacc8909df61d4fb49165a99527c
2022-06-09 16:45:35 +00:00
Márton Elek
35c022ebf9 satellite/accounting: don't use testplanet in rollup_test if not necessary
TestRollupNoDeletes is very flaky (passes locally but fails in the main branch build).

The exact reason is not clear, but stopping the loop seems to be async, the following lines may not stop the loops immediatelly which is a potential problem:

```
satellitePeer.Accounting.Rollup.Loop.Pause()
satellitePeer.Accounting.Tally.Loop.Pause()
```

Fortunatelly these test check only the database interfaces. Instead of testplanet.Run we can run only satellitedbtest.Run which is faster and more predictable (no background loops).

Other potential problem: comment claims that the default of DeleteTallies is false:

```
// In testplanet the setting config.Rollup.DeleteTallies defaults to false.
```

But it seems to be true (rollup.go):

```
DeleteTallies bool          `help:"option for deleting tallies after they are rolled up" default:"true"`
```

This is also fixed in the patch (as we need set it explicit), but TBH it can be fixed with testplanet, too.

Change-Id: Id7ec80d5c069bed2c556f4d001c71aa23fc5af23
2022-06-09 13:11:45 +00:00
Egon Elbre
11dd0687c9 satellite/metabase: add missing monitoring
Change-Id: I8b1049aafc0d2c6029a68cac0e6fdc3d7b25b30d
2022-06-08 12:34:00 +00:00
Moby von Briesen
5b79970575 satellite/metainfo: Use project-level attribution at bucket level
If project.UserAgent is set, use this for bucket.UserAgent on bucket
creation. Otherwise, set bucket attribution as before (getting UserAgent
from request headers).

Tests were updated to create the bucket with a different user, added as
a project member. Otherwise, the tests do not catch the bug.

Change-Id: I7ecf79a8eac5957eed361cbea94823190f58b776
2022-06-07 17:08:50 +00:00
paul cannon
737d7c7dfc satellite/reputation: new ApplyUpdates() method
The ApplyUpdates() method on the reputation.DB interface acts like the
similar Update() method, but can allow for applying the changes from
multiple audit events, instead of only one.

This will be necessary for the reputation write cache, which will batch
up changes to each node's reputation in order to flush them
periodically.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I44cc47767ea2d9423166bb8fed080c8a11182041
2022-06-07 15:22:25 +00:00
Yaroslav Vorobiov
62774768d9 satellite/payments/storjscan: update config and wrap service errors
Change-Id: I4f75375e9499119cbcce78c5e77d1950e4bae38b
2022-06-07 14:21:42 +00:00
Vitalii
f56504de2a apigen: project delete endpoint
Implemented project delete endpoint for REST API.
Added project usage status check service method to indicate if project can be deleted.
Updated project invoice status check method to indicate if project can be deleted.

Change-Id: I57dc96efb072517144252001ab5405446c9cdeb4
2022-06-07 12:23:24 +03:00
Paul Willoughby
911cc1e163 satellite/contact: reject privateIPs in PingMe and CheckIn endpoints
prevent network enumeration by rejecting privateIPs in PingMe and
Checkin endpoints

Closes storj/storj-private#32

Change-Id: I63f00483ff4128ebd5fa9b7b8da826a5706748c9
2022-06-07 08:09:14 +00:00
Vitalii
f0b28d6326 apigen: endpoint to get user
Implemented new GET user by request context endpoint.
Updated docs.

Change-Id: Iebb493e55f9456b89d7dbd234bb0b939b82b0ced
2022-06-06 16:31:19 +00:00
Vitalii
ba58530089 {satellite}/web,console,testsuite: remove old navigation structure
Removed old satellite UI navigation structure.
Removed old feature flag.

Change-Id: Ic998886cf2e30ebd44e67a20fc53888103fe4b8d
2022-06-06 16:43:05 +03:00
Michał Niewrzał
18ef3d3881 satellite/accounting: add metrics for total pending objects
We don't have metric to track how many pending objects we have in the
system. This change is using tally objects loop to collect pending
objects per bucket and at the end its combining all buckets values
into single metric for all pending objects in a system.

Change-Id: Iac7a6bfb48854f7e70127d275ea8fdd60c4eb8b7
2022-06-06 12:04:46 +00:00
dlamarmorgan
270204f352 satellite/{payments/storjscan,satellitedb}: Add wallet implementation
Add storjscan wallets implementation to the satellite. The wallets interface allows you to add and claim new wallets as called by the API. The storjscan specific implementation of this interface uses a wallets DB to associate the user to a wallet address, as well as a storjscan client to request and associate new wallets to the satellite.

Change-Id: I54081edb5545d4e3ee07cf1cce3d3e87cc00c4a1
2022-06-03 11:45:47 +00:00
cl-mitch
cbaca8b17e
web/satellite added new billing screen feature flag (#4836)
New feature flag implemented to enable the new billing flow.

Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
2022-06-02 09:30:27 -05:00
Ivan Fraixedes
261a6edf4f
satellite/admin/ui: Update NPM dependencies
Update all the NPM dependencies used by the Admin UI.

The dev dependencies correspond to the ones that are currently used by
an svelte app generated with the last svelte-kit version. They
deprecated some configuration options and changed some svelte
directives.

The only non-dev dependency is also updated to the last published.

Change-Id: I5f2192cab41e00efc3239237f8dc8f3d07816b63
2022-06-02 13:24:15 +02:00
Egon Elbre
763c04770f satellite/console/consoleweb: fix CSP values
script-src-elem is preferred over script-src in certain scenarios.
If it's absent, then the browser always uses script-src. By adding
script-src-elem it ended up blocking google recaptcha.

Change-Id: I9cf96e71e69054c4a034ca189db84fbe8903a59b
2022-06-02 10:44:48 +03:00
Vitalii
a61f0f6be3 satellite/projectaccounting: make daily usage query return correct allocated traffic
Updated daily project usage query to return correct allocated traffic.
If allocated egress has expired then we return settled egress.
If not then we return allocated egress - dead egress.

Fix for this issue
https://github.com/storj/storj/issues/4563

Change-Id: Ia15a50d3bb8d8cb1106936e17dbe0f1f5a40fa87
2022-06-01 15:24:52 +00:00
Vitalii
07e65cd338 satellite/projectaccounting: sum up bucket usages for daily usage query
Fixed daily usage query returning single bucket usage.
We sum up bucket usages now.
Also fixed https://github.com/storj/storj/issues/4559.

Change-Id: I2eb6299f1ef500d68150879195011b6fbb5f37ed
2022-06-01 12:50:10 +00:00
Márton Elek
84d02f7fbf metabase: use predefined snapshot for unit tests avoid migration
Change-Id: I6225ae32c312030e3559d8ed0faa137ffc3cc5b7
2022-06-01 12:07:00 +00:00
littleskunk
6cdd250019
satellite/console: enable new object flow / onboarding (#4851) 2022-06-01 14:04:03 +02:00
littleskunk
3d1b469f35
satellite/console: disable account activation reminder email (#4844)
* satellite/console: disable account activation reminder email
2022-05-31 17:36:40 +02:00
Márton Elek
9d8eaf3d7e metabasetest: use DROP instead of TRUNCATE
TRUNCATE requires table recreation which involves 'online schema change' with crdb.
(with psql it might be fater than DROP, that was the motivation of the original change)

`online schema change` is an async operation with crdb and it's eventually very slow therefore we try to avoid it.

This·reverts·commit·15bed0ed0e81d54fe4ffac9928bdf648f5e06ec6.¬

Change-Id: I93e1ab470962be77e3458d74c8787442c9d7bee0
2022-05-31 12:16:02 +00:00
Ivan Fraixedes
2438c92d56
satellite/admin/ui: Add disable MFA endpoint
Add the admin API endpoint for disabling an user's multifacdtor
authentication to the satellite admin UI.

Remove a couple of commented code lines too.

Change-Id: Iaee7efe7a3d4d38bdd6541311447a9726806f0f1
2022-05-30 15:23:35 +02:00
Stefan Benten
25b45c47ee
satellite/admin: allow disabling of MFA
We have a couple of support tickets so far that require us to
disable the mfa on accounts. Since we currently had no other
way than doing a SQL War Crime, it makes sense to add it to the
admin API.

Change-Id: Ib16735c1961380b04345a3495d4eebee5fa0bc41
2022-05-30 13:56:11 +02:00
Stefan Benten
62a64bae3b
satellite/admin: allow deletion of paid tier user projects when invoiced
Currently we have a bug in which we would require that a project of
a paid tier user needs to be two months unused before we can delete it.
This change fixes it and reduces it back to the normal next billing cycle.

Change-Id: I28610b6c45c68943fd4f2621233bccc06cab28a0
2022-05-30 13:54:51 +02:00
Stefan Benten
8673583582 satellite/admin: fix console config handling
An older change plummed the full console config as subconfig of
the admin api configuration in. This bloated the generated satellite
configuration unnecessarily while also allow for confusion/mistakes.

Change-Id: Icf49cc1f147711e37e85f6eac1143fab8ddf1659
2022-05-27 22:26:06 +00:00
Fadila Khadar
792c49e5ad satellite/metabase: simplify delete query
when deleting an object that has been copied multiple times, we look for an ancestor_stream_id by taking the min of all copies stream_id.

This change simplifies this process by picking any stream_id as a new ancestor by using 'distinct on'.

Fixes https://github.com/storj/storj/issues/4745

Change-Id: Iffb519b82d2ae2ed73af48fa0e86f87384e0158f
2022-05-27 20:09:50 +00:00
Cameron
b379fe89d2 satellite/core.go: hook up flag to enable email reminders
Change-Id: I973bb3a709e806e7744679b3543217c1cab85f9a
2022-05-27 13:16:14 -04:00
Vitalii
69ad49f473 satellite/server: fixed CSP for hcaptcha
Fixed CSP errors for hcaptcha

Change-Id: Ie928e206c652c97d36bcbdaf8436ae4a33afed8d
2022-05-27 14:28:31 +00:00
dlamarmorgan
007d4190c2 satellite/{payments/billing,satellitedb}: Add payment billing DB
Add billing DB to the satellite. This DB will hold all transactions on the users account and can be used to compute the users current usable account balance.

Change-Id: I056416efc169e5e5e30c9f30cd8bc766b7bc8073
2022-05-27 08:56:31 +00:00
Vitalii
d916b26e0e apigen: create api key endpoint
Implemented new service method for generating API keys.
Implemented new endpoint.
Improved multiple endpoint groups handling.

Change-Id: Iba26fbf9123707b5b4c2d5e8c5a35d507404f24a
2022-05-26 16:00:23 +00:00
Cameron
55821605e8 satellite/console: add monkit metrics around user registraion/login
github issue: https://github.com/storj/storj/issues/4807

Change-Id: Id56ec73ec91b07b639b8011f0f916b4adbb01be6
2022-05-26 10:44:47 -04:00
Michał Niewrzał
c934f45bfc satellite/metabase: drop IterateObjectsAllVersions
We are not using this method and most probably we
won't need to list objects with all statuses at once.
Removing for now.

Change-Id: I7aa0468c5f635ee2fb1fe51db382595c6343dd9c
2022-05-26 14:12:20 +00:00
Fadila Khadar
4725a3878c satellite/metainfo: parallel deletion tests
- parallel deletion of 50 objects and their 50 copies (one copy per object)
This test is skipped because it's creating deadlocks that are not automatically retried on postgres
- parallel deletion of 1 object and its 50 copies.

Fixes https://github.com/storj/storj/issues/4745

Change-Id: Id7a28251c06bb12b5edcc88721f60bf7a4bc0492
2022-05-26 11:00:42 +00:00
Márton Elek
c136796308 test: make http client in TestActivationRouting thread-safe
testplanet executes cockroach and postgress tests parallel, therefore using http.DefaultClient is safe only as long as we don't modify it.

TestActivationRouting modifies it (client.CheckRedirect=...), therefore it should use a local version instead of the default one.

Problem reported by a jenkins build:

```
==================
WARNING: DATA RACE
Write at 0x000003486af0 by goroutine 143:
  storj.io/storj/satellite/console/consoleweb_test.TestActivationRouting.func1()
      /home/jenkins/workspace/storj-testing-experiments/satellite/console/consoleweb/server_test.go:66 +0x378
  storj.io/storj/private/testplanet.Run.func1.1()
...

Previous read at 0x000003486af0 by goroutine 104:
  net/http.(*Client).checkRedirect()
      /usr/local/go/src/net/http/client.go:494 +0xd73
  net/http.(*Client).do()
      /usr/local/go/src/net/http/client.go:691 +0xd31
  net/http.(*Client).Do()
      /usr/local/go/src/net/http/client.go:593 +0x204
  storj.io/storj/satellite/console/consoleweb_test.TestActivationRouting.func1.1()
      /home/jenkins/workspace/storj-testing-experiments/satellite/console/consoleweb/server_test.go:48 +0x1e5
  storj.io/storj/satellite/console/consoleweb_test.TestActivationRouting.func1()
      /home/jenkins/workspace/storj-testing-experiments/satellite/console/consoleweb/server_test.go:74 +0x49d
  storj.io/storj/private/testplanet.Run.func1.1()
...

```

Change-Id: I73319a5a593e067b906ec1fda70a44ca1e5a49a2
2022-05-25 15:27:02 +00:00
Michał Niewrzał
6759ba831c satellite/metainfo: skip TestUserAgentTransferStats
It's hard to make this test stable on CI but still can be
useful for local testing.

Change-Id: Iac5368d1a9179b67ea2409ee99bcf189db862897
2022-05-25 11:15:23 +00:00
paul cannon
5bc4c254d4 satellite/reputation: move Config fields into a Config struct
This has been a cause of some confusion, even though the fields are
labeled as being copies of config values.

Having them be under a field explicitly named "Config" makes this
clearer, plus, allows the values to be passed in simply as a copy
of the Config struct from the satellite, rather than copying the fields
individually (which can be error-prone, particularly as the AuditCount
field in UpdateRequest is apparently not the same thing as the
AuditCount field in reputation.Config).

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I386953347b71068596618616934aa28e3245cdc1
2022-05-24 20:57:51 +00:00
Márton Elek
5e8266d1de satellite/{payments/storjscan,satellitedb}: Add storjscan wallet DB
Add storjscan wallets DB to the satellite. For now this DB is a one to one mapping of the users account to a storjscan wallet that can be used by the account holder to make payments on their Storj account.

relates to https://github.com/storj/storj/issues/4347

Change-Id: I6e65b15817b90ceb75641244f9bf173c3b4228a7
2022-05-24 08:42:02 +00:00
paul cannon
2ee9463e34 satellite/reputation: don't need 3 identical AuditHistory types
The two protobuf types are identical except that one is in our common/pb
package, and the other is in internalpb. Since the type is public
already, and there is no difference in the internal one, it seems better
to use the public one for all satellite needs.

There is also another type which is essentially identical, but which is
not a protobuf type, also called "AuditHistory". It looks like we don't
ever actually need to have a separate type from the protobuf one.

This change makes us use "storj/common/pb".AuditHistory for all of our
AuditHistory needs.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: If845fde21bb31c801db6d67ffc9a146d1617b991
2022-05-24 05:48:46 +00:00
Cameron
87f6a3dcda {web/satelliite, satellite/console}: logo redirects to homepage
logo redirects to homepage on login, signup, forgot password, reset
password, and activate account pages

Change-Id: I992aeae197004d620addd8d515cae1c1ca80a778
2022-05-23 16:26:19 -04:00
Márton Elek
55de50eea7 console: stub endpoints for native token payments
Change-Id: I3f99ab44332eb8c9e2a3834a932eee72f44bd490
2022-05-23 13:28:00 +00:00
Márton Elek
31560c94f8 satellite/metabase/test: extend default time diff for comparison
Tests are intermittently fail with similar error:

```
    --- FAIL: TestDeletePendingObject/Cockroach (15.85s)

    test.go:380:
        	Error Trace:	test.go:380
        	           				delete_test.go:221
        	Error:      	Should be zero, but was   metabase.DeleteObjectResult{
        	           	 	Objects: []metabase.Object{
        	           	 		{
        	           	 			ObjectStream: {ProjectID: {0x0f, 0x40, 0x70, 0x41, ...}, BucketName: "txxywyg4", ObjectKey: "\xbb+$\x17\x80\xc6\xcaC\xa3\xdb\xc3z*\xa8\xbe\xaf", Version: 1, ...},
        	           	- 			CreatedAt:    s"2022-05-20 14:40:15.995376773 +0200 CEST",
        	           	+ 			CreatedAt:    s"2022-05-20 14:40:21.04949 +0200 CEST",
        	           	 			ExpiresAt:    nil,
        	           	 			Status:       1,
        	           	 			... // 9 identical fields
        	           	 		},
        	           	 	},
        	           	 	Segments: {{RootPieceID: {0x01, 0x00, 0x00, 0x00, ...}, Pieces: {{...}}}, {RootPieceID: {0x01, 0x00, 0x00, 0x00, ...}, Pieces: {{...}}}},
        	           	 }
        	Test:       	TestDeletePendingObject/Cockroach/with_segments
        --- FAIL: TestDeletePendingObject/Cockroach/with_segments (0.68s)
```

Looks like we shouldn't have an assumption that all tests can be finished in 5 seconds, especially not in highly parallel environment.

These tests use `time.Now` at the beginning and compare the time saved in the database (usually filled by the database).

The difference shouldn't be higher than 20 seconds (before this commit: 5 seconds) which assumes that the records are saved in this timeframe...

Change-Id: Ia6f52897d13f88c6857c05d728bf8e72ab863c9b
2022-05-23 10:24:58 +00:00
NickolaiYurchenko
99237d5c78 web/satellite: new bucket creation flow
old bucket creation flow removed
new flow added
name and passphrase splitted into separate views
demo bucket will not be created automatically
bucket creation progress bar added

Change-Id: I2a1d7d77c3038caaafb3c06bdb0ac5dd1ad17599
2022-05-22 11:55:55 +00:00
paul cannon
951a842fd4 satellite/{reputation,satellitedb}: move addAudit
This functionality will be needed in both packages, so here we move it
into the more general reputation-code package and export it for use in
satellitedb.

This also removes the related UpdateAuditHistory() signature from the
reputation DB interface, since it doesn't have anything to do with the
db. It doesn't need to be a method, either.

Finally, this changes the test for addAudit to be a plain test function
instead of using testplanet.Run(). It didn't need a whole testplanet
setup or any databases.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I90f6a909e5404f03ad776b95cfa2f248308c57c1
2022-05-20 15:37:31 +00:00