Commit Graph

2630 Commits

Author SHA1 Message Date
Michał Niewrzał
18ef3d3881 satellite/accounting: add metrics for total pending objects
We don't have metric to track how many pending objects we have in the
system. This change is using tally objects loop to collect pending
objects per bucket and at the end its combining all buckets values
into single metric for all pending objects in a system.

Change-Id: Iac7a6bfb48854f7e70127d275ea8fdd60c4eb8b7
2022-06-06 12:04:46 +00:00
dlamarmorgan
270204f352 satellite/{payments/storjscan,satellitedb}: Add wallet implementation
Add storjscan wallets implementation to the satellite. The wallets interface allows you to add and claim new wallets as called by the API. The storjscan specific implementation of this interface uses a wallets DB to associate the user to a wallet address, as well as a storjscan client to request and associate new wallets to the satellite.

Change-Id: I54081edb5545d4e3ee07cf1cce3d3e87cc00c4a1
2022-06-03 11:45:47 +00:00
cl-mitch
cbaca8b17e
web/satellite added new billing screen feature flag (#4836)
New feature flag implemented to enable the new billing flow.

Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
2022-06-02 09:30:27 -05:00
Ivan Fraixedes
261a6edf4f
satellite/admin/ui: Update NPM dependencies
Update all the NPM dependencies used by the Admin UI.

The dev dependencies correspond to the ones that are currently used by
an svelte app generated with the last svelte-kit version. They
deprecated some configuration options and changed some svelte
directives.

The only non-dev dependency is also updated to the last published.

Change-Id: I5f2192cab41e00efc3239237f8dc8f3d07816b63
2022-06-02 13:24:15 +02:00
Egon Elbre
763c04770f satellite/console/consoleweb: fix CSP values
script-src-elem is preferred over script-src in certain scenarios.
If it's absent, then the browser always uses script-src. By adding
script-src-elem it ended up blocking google recaptcha.

Change-Id: I9cf96e71e69054c4a034ca189db84fbe8903a59b
2022-06-02 10:44:48 +03:00
Vitalii
a61f0f6be3 satellite/projectaccounting: make daily usage query return correct allocated traffic
Updated daily project usage query to return correct allocated traffic.
If allocated egress has expired then we return settled egress.
If not then we return allocated egress - dead egress.

Fix for this issue
https://github.com/storj/storj/issues/4563

Change-Id: Ia15a50d3bb8d8cb1106936e17dbe0f1f5a40fa87
2022-06-01 15:24:52 +00:00
Vitalii
07e65cd338 satellite/projectaccounting: sum up bucket usages for daily usage query
Fixed daily usage query returning single bucket usage.
We sum up bucket usages now.
Also fixed https://github.com/storj/storj/issues/4559.

Change-Id: I2eb6299f1ef500d68150879195011b6fbb5f37ed
2022-06-01 12:50:10 +00:00
Márton Elek
84d02f7fbf metabase: use predefined snapshot for unit tests avoid migration
Change-Id: I6225ae32c312030e3559d8ed0faa137ffc3cc5b7
2022-06-01 12:07:00 +00:00
littleskunk
6cdd250019
satellite/console: enable new object flow / onboarding (#4851) 2022-06-01 14:04:03 +02:00
littleskunk
3d1b469f35
satellite/console: disable account activation reminder email (#4844)
* satellite/console: disable account activation reminder email
2022-05-31 17:36:40 +02:00
Márton Elek
9d8eaf3d7e metabasetest: use DROP instead of TRUNCATE
TRUNCATE requires table recreation which involves 'online schema change' with crdb.
(with psql it might be fater than DROP, that was the motivation of the original change)

`online schema change` is an async operation with crdb and it's eventually very slow therefore we try to avoid it.

This·reverts·commit·15bed0ed0e81d54fe4ffac9928bdf648f5e06ec6.¬

Change-Id: I93e1ab470962be77e3458d74c8787442c9d7bee0
2022-05-31 12:16:02 +00:00
Ivan Fraixedes
2438c92d56
satellite/admin/ui: Add disable MFA endpoint
Add the admin API endpoint for disabling an user's multifacdtor
authentication to the satellite admin UI.

Remove a couple of commented code lines too.

Change-Id: Iaee7efe7a3d4d38bdd6541311447a9726806f0f1
2022-05-30 15:23:35 +02:00
Stefan Benten
25b45c47ee
satellite/admin: allow disabling of MFA
We have a couple of support tickets so far that require us to
disable the mfa on accounts. Since we currently had no other
way than doing a SQL War Crime, it makes sense to add it to the
admin API.

Change-Id: Ib16735c1961380b04345a3495d4eebee5fa0bc41
2022-05-30 13:56:11 +02:00
Stefan Benten
62a64bae3b
satellite/admin: allow deletion of paid tier user projects when invoiced
Currently we have a bug in which we would require that a project of
a paid tier user needs to be two months unused before we can delete it.
This change fixes it and reduces it back to the normal next billing cycle.

Change-Id: I28610b6c45c68943fd4f2621233bccc06cab28a0
2022-05-30 13:54:51 +02:00
Stefan Benten
8673583582 satellite/admin: fix console config handling
An older change plummed the full console config as subconfig of
the admin api configuration in. This bloated the generated satellite
configuration unnecessarily while also allow for confusion/mistakes.

Change-Id: Icf49cc1f147711e37e85f6eac1143fab8ddf1659
2022-05-27 22:26:06 +00:00
Fadila Khadar
792c49e5ad satellite/metabase: simplify delete query
when deleting an object that has been copied multiple times, we look for an ancestor_stream_id by taking the min of all copies stream_id.

This change simplifies this process by picking any stream_id as a new ancestor by using 'distinct on'.

Fixes https://github.com/storj/storj/issues/4745

Change-Id: Iffb519b82d2ae2ed73af48fa0e86f87384e0158f
2022-05-27 20:09:50 +00:00
Cameron
b379fe89d2 satellite/core.go: hook up flag to enable email reminders
Change-Id: I973bb3a709e806e7744679b3543217c1cab85f9a
2022-05-27 13:16:14 -04:00
Vitalii
69ad49f473 satellite/server: fixed CSP for hcaptcha
Fixed CSP errors for hcaptcha

Change-Id: Ie928e206c652c97d36bcbdaf8436ae4a33afed8d
2022-05-27 14:28:31 +00:00
dlamarmorgan
007d4190c2 satellite/{payments/billing,satellitedb}: Add payment billing DB
Add billing DB to the satellite. This DB will hold all transactions on the users account and can be used to compute the users current usable account balance.

Change-Id: I056416efc169e5e5e30c9f30cd8bc766b7bc8073
2022-05-27 08:56:31 +00:00
Vitalii
d916b26e0e apigen: create api key endpoint
Implemented new service method for generating API keys.
Implemented new endpoint.
Improved multiple endpoint groups handling.

Change-Id: Iba26fbf9123707b5b4c2d5e8c5a35d507404f24a
2022-05-26 16:00:23 +00:00
Cameron
55821605e8 satellite/console: add monkit metrics around user registraion/login
github issue: https://github.com/storj/storj/issues/4807

Change-Id: Id56ec73ec91b07b639b8011f0f916b4adbb01be6
2022-05-26 10:44:47 -04:00
Michał Niewrzał
c934f45bfc satellite/metabase: drop IterateObjectsAllVersions
We are not using this method and most probably we
won't need to list objects with all statuses at once.
Removing for now.

Change-Id: I7aa0468c5f635ee2fb1fe51db382595c6343dd9c
2022-05-26 14:12:20 +00:00
Fadila Khadar
4725a3878c satellite/metainfo: parallel deletion tests
- parallel deletion of 50 objects and their 50 copies (one copy per object)
This test is skipped because it's creating deadlocks that are not automatically retried on postgres
- parallel deletion of 1 object and its 50 copies.

Fixes https://github.com/storj/storj/issues/4745

Change-Id: Id7a28251c06bb12b5edcc88721f60bf7a4bc0492
2022-05-26 11:00:42 +00:00
Márton Elek
c136796308 test: make http client in TestActivationRouting thread-safe
testplanet executes cockroach and postgress tests parallel, therefore using http.DefaultClient is safe only as long as we don't modify it.

TestActivationRouting modifies it (client.CheckRedirect=...), therefore it should use a local version instead of the default one.

Problem reported by a jenkins build:

```
==================
WARNING: DATA RACE
Write at 0x000003486af0 by goroutine 143:
  storj.io/storj/satellite/console/consoleweb_test.TestActivationRouting.func1()
      /home/jenkins/workspace/storj-testing-experiments/satellite/console/consoleweb/server_test.go:66 +0x378
  storj.io/storj/private/testplanet.Run.func1.1()
...

Previous read at 0x000003486af0 by goroutine 104:
  net/http.(*Client).checkRedirect()
      /usr/local/go/src/net/http/client.go:494 +0xd73
  net/http.(*Client).do()
      /usr/local/go/src/net/http/client.go:691 +0xd31
  net/http.(*Client).Do()
      /usr/local/go/src/net/http/client.go:593 +0x204
  storj.io/storj/satellite/console/consoleweb_test.TestActivationRouting.func1.1()
      /home/jenkins/workspace/storj-testing-experiments/satellite/console/consoleweb/server_test.go:48 +0x1e5
  storj.io/storj/satellite/console/consoleweb_test.TestActivationRouting.func1()
      /home/jenkins/workspace/storj-testing-experiments/satellite/console/consoleweb/server_test.go:74 +0x49d
  storj.io/storj/private/testplanet.Run.func1.1()
...

```

Change-Id: I73319a5a593e067b906ec1fda70a44ca1e5a49a2
2022-05-25 15:27:02 +00:00
Michał Niewrzał
6759ba831c satellite/metainfo: skip TestUserAgentTransferStats
It's hard to make this test stable on CI but still can be
useful for local testing.

Change-Id: Iac5368d1a9179b67ea2409ee99bcf189db862897
2022-05-25 11:15:23 +00:00
paul cannon
5bc4c254d4 satellite/reputation: move Config fields into a Config struct
This has been a cause of some confusion, even though the fields are
labeled as being copies of config values.

Having them be under a field explicitly named "Config" makes this
clearer, plus, allows the values to be passed in simply as a copy
of the Config struct from the satellite, rather than copying the fields
individually (which can be error-prone, particularly as the AuditCount
field in UpdateRequest is apparently not the same thing as the
AuditCount field in reputation.Config).

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I386953347b71068596618616934aa28e3245cdc1
2022-05-24 20:57:51 +00:00
Márton Elek
5e8266d1de satellite/{payments/storjscan,satellitedb}: Add storjscan wallet DB
Add storjscan wallets DB to the satellite. For now this DB is a one to one mapping of the users account to a storjscan wallet that can be used by the account holder to make payments on their Storj account.

relates to https://github.com/storj/storj/issues/4347

Change-Id: I6e65b15817b90ceb75641244f9bf173c3b4228a7
2022-05-24 08:42:02 +00:00
paul cannon
2ee9463e34 satellite/reputation: don't need 3 identical AuditHistory types
The two protobuf types are identical except that one is in our common/pb
package, and the other is in internalpb. Since the type is public
already, and there is no difference in the internal one, it seems better
to use the public one for all satellite needs.

There is also another type which is essentially identical, but which is
not a protobuf type, also called "AuditHistory". It looks like we don't
ever actually need to have a separate type from the protobuf one.

This change makes us use "storj/common/pb".AuditHistory for all of our
AuditHistory needs.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: If845fde21bb31c801db6d67ffc9a146d1617b991
2022-05-24 05:48:46 +00:00
Cameron
87f6a3dcda {web/satelliite, satellite/console}: logo redirects to homepage
logo redirects to homepage on login, signup, forgot password, reset
password, and activate account pages

Change-Id: I992aeae197004d620addd8d515cae1c1ca80a778
2022-05-23 16:26:19 -04:00
Márton Elek
55de50eea7 console: stub endpoints for native token payments
Change-Id: I3f99ab44332eb8c9e2a3834a932eee72f44bd490
2022-05-23 13:28:00 +00:00
Márton Elek
31560c94f8 satellite/metabase/test: extend default time diff for comparison
Tests are intermittently fail with similar error:

```
    --- FAIL: TestDeletePendingObject/Cockroach (15.85s)

    test.go:380:
        	Error Trace:	test.go:380
        	           				delete_test.go:221
        	Error:      	Should be zero, but was   metabase.DeleteObjectResult{
        	           	 	Objects: []metabase.Object{
        	           	 		{
        	           	 			ObjectStream: {ProjectID: {0x0f, 0x40, 0x70, 0x41, ...}, BucketName: "txxywyg4", ObjectKey: "\xbb+$\x17\x80\xc6\xcaC\xa3\xdb\xc3z*\xa8\xbe\xaf", Version: 1, ...},
        	           	- 			CreatedAt:    s"2022-05-20 14:40:15.995376773 +0200 CEST",
        	           	+ 			CreatedAt:    s"2022-05-20 14:40:21.04949 +0200 CEST",
        	           	 			ExpiresAt:    nil,
        	           	 			Status:       1,
        	           	 			... // 9 identical fields
        	           	 		},
        	           	 	},
        	           	 	Segments: {{RootPieceID: {0x01, 0x00, 0x00, 0x00, ...}, Pieces: {{...}}}, {RootPieceID: {0x01, 0x00, 0x00, 0x00, ...}, Pieces: {{...}}}},
        	           	 }
        	Test:       	TestDeletePendingObject/Cockroach/with_segments
        --- FAIL: TestDeletePendingObject/Cockroach/with_segments (0.68s)
```

Looks like we shouldn't have an assumption that all tests can be finished in 5 seconds, especially not in highly parallel environment.

These tests use `time.Now` at the beginning and compare the time saved in the database (usually filled by the database).

The difference shouldn't be higher than 20 seconds (before this commit: 5 seconds) which assumes that the records are saved in this timeframe...

Change-Id: Ia6f52897d13f88c6857c05d728bf8e72ab863c9b
2022-05-23 10:24:58 +00:00
NickolaiYurchenko
99237d5c78 web/satellite: new bucket creation flow
old bucket creation flow removed
new flow added
name and passphrase splitted into separate views
demo bucket will not be created automatically
bucket creation progress bar added

Change-Id: I2a1d7d77c3038caaafb3c06bdb0ac5dd1ad17599
2022-05-22 11:55:55 +00:00
paul cannon
951a842fd4 satellite/{reputation,satellitedb}: move addAudit
This functionality will be needed in both packages, so here we move it
into the more general reputation-code package and export it for use in
satellitedb.

This also removes the related UpdateAuditHistory() signature from the
reputation DB interface, since it doesn't have anything to do with the
db. It doesn't need to be a method, either.

Finally, this changes the test for addAudit to be a plain test function
instead of using testplanet.Run(). It didn't need a whole testplanet
setup or any databases.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I90f6a909e5404f03ad776b95cfa2f248308c57c1
2022-05-20 15:37:31 +00:00
Yaroslav Vorobiov
0bf12523e1 satellite/payments/storjscan: add client and list all payments API call
Change-Id: I1f5065b3d15cc93f4b42868941e82e04af364565
2022-05-20 15:10:17 +00:00
Michał Niewrzał
7a2d2a36ca satellite: use more optimal monkit call for loop observers methods
Recently we applied this optimization to metrics observer and time
used by its method dropped from 12m to 3m for us1 (220m segments).
It looks that it make sense to apply the same code to all observers.

Change-Id: I05898aaacbd9bcdf21babc7be9955da1db57bdf2
2022-05-20 11:03:41 +00:00
Michał Niewrzał
c5edf77237 satellite/metabase: fix flaky TestCommitInlineSegment
If TestCommitInlineSegment tests are taking longer time
then zombieDeadline created at the beginning of test
can be too far in the past. Creating zombieDeadline for each case
should avoid flakines.

Change-Id: Ieb011e8e470f6f1c32cf9365c8ae819317de6738
2022-05-20 10:20:56 +00:00
Egon Elbre
5d6d5331f5 private/testplanet: don't cleanup cockroach
Márton found out that DROP DATABASE is rather slow on CRDB, and it makes
a significant impact when running the whole testsuite. In sum of test
times it's ~2.5h compared to ~2h. And the end-to-end ~20m to ~16m.

This adds a new flag STORJ_TEST_COCKROACH_NODROP for enabling this
behavior in the CI environment.

Fixes https://github.com/storj/dev-enablement/issues/6

Change-Id: I5a6616c32dc6596a96ba3d203f409368307d7438
2022-05-19 15:27:37 +03:00
Michał Niewrzał
456aea727e satellite: use PieceIDDeriver for derivation
We can use PieceIDDeriver in all places where we are deriving id from
the same id multiple times. We have serveral such places: gc, segment
deletion, segment validation, order limit creation. Using it should
save some resources.

Change-Id: I24668d516c0f7cea4aec6470614067734149501d
2022-05-19 06:31:42 +00:00
prerna-parashar
14ebba0d77
satellite/analytics: Track events generated from navigation and dashboard (#4795)
satellite/analytics: Track events generated from navigation and dashboard
2022-05-18 12:49:10 -07:00
paul cannon
ac7752ac3c satellite/reputation: return full Info from reputation update
This will let us update our reputation cache when writing through to the
db.

Since the information is already being fetched from the db and returned
to the application, the extra cpu load here should be minimal.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I2b8619f2c0d541893c7d3e7d33b1863b96775ebd
2022-05-18 17:22:06 +00:00
Malcolm Bouzi
087e57d037 web/satellite: create chore that will resend verification emails to unverified users
We want to remind unverified users to verify their emails:
once after 24 hours has passed and again after 5 days has passed.

Add mailservice.Service to satellite core because it is needed by the
chore for sending emails. To add the mailservice.Service to the core,
we create a helper function in satellite/peer.go to avoid duplicating
the code in both api.go and core.go. In addition to the chore, this
change adds methods to users.DB to get unverified users in need of
reminder.

Change-Id: I4e515bdf43f922788b4f965b2efb34fa32288bd1
2022-05-18 08:08:33 +00:00
Qweder93
50b826e582 satellite/metabase: drop GetProjectSegmentCount and related code
Resolves: https://github.com/storj/storj/issues/4744

Change-Id: I66b12e17de461242745e785dedd397fbe3b386fa
2022-05-17 19:01:51 +00:00
Qweder93
a447b27ffc satellite/accounting: modify tally.go to use returned segments values
modify tally to calculate how we need to update segments in the live accounting
cache with UpdateProjectSegmentUsage method. adjust accounting.ExceedsUploadLimits
to use only cache for segment validation, if cache returns 0 or key not found then
we shouldn't reject such project as its possible that we won't have this value before first object iteration
https://github.com/storj/storj/issues/4744

Change-Id: I32c22d7fb71236e354653ba8719e029fc71f04c7
2022-05-17 17:20:02 +00:00
paul cannon
aa728bd6ea satellite/satellitedb: add reputations.disqualification_reason
We added nodes.disqualification_reason recently, but we didn't add a
corresponding column in the reputations table (despite having a
corresponding `disqualified` column there).

Without this change, the (very useful and informative) assignments to
updateFields.DisqualificationReason in reputations.go have no effect.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I77404902ca64b56aed72f1de76b303fe82b76aab
2022-05-17 10:09:36 -05:00
Michał Niewrzał
230424eace satellite/metainfo: add IPFS datastore user agent to metrics
Change-Id: I2ac14c97c64542669acef58ca6ded79984d0bd11
2022-05-17 13:09:22 +02:00
Michał Niewrzał
3ebb05ecca satellite/metainfo: add user-agent bandwidth metric
The existing versionCollector metrics can tell us how many times various
metainfo endpoints are called, but they don't tell us how many bytes a
client is transferring.  We currently can't collect precise information
on this, but we can collect information on how much planned traffic is
requested via order limits.

The implementation as provided is intended to measure objects sizes
before erasure encoding is taken into account.

Change-Id: I2f1d2a7831630e8439ecf5342e933df259151792
2022-05-17 09:57:39 +00:00
Michał Niewrzał
7b253075bb satellite/accounting: test if server-side copy affects bandwidth
Doing server-side copy operation should not affect user monthly
bandwidth. This test covers that case.

Fixes https://github.com/storj/storj/issues/4717

Change-Id: I84ffab96b84851f395ea3a34d88f7dba424ec440
2022-05-17 06:24:15 +00:00
JT Olio
5fb9ee3cfa uplink, satellite: use bgp hostnames
Change-Id: I58f5011d3019f8267fa8cbd3096b2cfe42eb5f8b
2022-05-16 18:14:37 +00:00
Cameron
bfad64a6f0 satellite/satellitedb: add verification_reminders column to users table
When a new user registers, we send a verification request to their
email. Currently, if they do not verify their email, we take no further
action. We want to send these users reminders: one after about one day
and one after about 5 days. To do this we will use this new
verification_reminders column.

It will look something like this:

```
SELECT email FROM users
WHERE status = 0
   AND (
          (verification_reminders = 0 AND created_at < now() - 'INTERVAL 1d')
          OR (verification_reminders = 1 AND created_at < now() - 'INTERVAL 5d')
   )
```

Change-Id: If0620e08c97e9e337c9563481d665c5bd462693b
2022-05-16 16:37:16 +00:00
Cameron
772397172f satellite/console: implement WebappSessions db
Create WebappSessions interface in consoleauth package.
Interface implements the DB for webapp_sessions table.

https://github.com/storj/storj/blob/main/docs/blueprints/webapp-session-management.md

Change-Id: Ib56f238c20b58f2877046fc384232add253ee82b
2022-05-16 15:06:04 +00:00
Cameron
18eedaa62f satellite/satellitedb/dbx: add method to delete webapp sessions by user ID
Change-Id: I4552622fdf40c6ecd53046e597250bbe713f510c
2022-05-16 14:37:34 +00:00
Michał Niewrzał
277f8d2ae0 satellite/accounting: don't expire segments usage key
Initialy we wanted to put segment usage into cache values retrieved
directly from metabase but it cause performance issues. Now we
will be collecting segment usage from tally object loop and those
values will be put into a cache but becuse we cannot get those values
on demand we shouldn't expire cache value at all because objects
loop requires sustencial amount of time to be executed.

Part of solution for https://github.com/storj/storj/issues/4744

Change-Id: I3b37e23badeecebed0c95064156e85b38038bfe2
2022-05-16 13:00:16 +00:00
Cameron
0633aca607 satellite/console: create new consoleauth service
We want to send email verification reminders to users from the satellite
core, but some of the functionality required to do so exists in the
satellite console service. We could simply import the console service
into the core to achieve this, but the service requires a lot of
dependencies that would go unused just to be able to send these emails.

Instead, we break out the needed functionality into a new service which
can be imported separately by the console service and the future email
chore.

The consoleauth service creates, signs, and checks the expiration of auth
tokens.

Change-Id: I2ad794b7fd256f8af24c1a8d73a203d508069078
2022-05-13 16:27:07 +00:00
Moby von Briesen
763bfc0913 satellite/console,web/satellite: Implement hCaptcha
Adds a new configuration for hcaptcha enabled, secretkey, and sitekey.
If both reCAPTCHA and hCaptcha are configured as "enabled", reCAPTCHA
will be used.

Change-Id: I73cc6e133d8da3555e0ed8b2b377cf9eb263e6dc
2022-05-13 14:57:45 +00:00
Vitalii
95ae739d9c satellite/projectaccounting: fetch bucket usage since bucket creation instead of project creation
Fix for this customer issue
https://github.com/storj/customer-issues/issues/34

By this change we fetch bucket usage since its creation instead of using project's createdAt timestamp.

Change-Id: Ic0ea5d169056a5bd64ed143d13954d794da6e1d2
2022-05-13 13:27:42 +00:00
Erik van Velzen
db1cc8ca95 satellite/repair/checker: buffer repair queue
Integrate previous changes. Speed up the segment loop by batch inserting
into repair queue.

Change-Id: Ib9f4962d91960d21bad298f7771345b0dd270276
2022-05-12 16:28:05 +00:00
Egon Elbre
4791ba5d50 satellite/{mailservice,oidc}: improvements to debugging
Things that make debugging easier.
* Added logging to automatic link clicking to make it obvious, when it
  fails.
* Added monitoring to oidc.
* Made dbx create calls noreturn for oauth_*

Change-Id: I37397b4e84ce5bfd82954aed9c38fdfd52595f24
2022-05-11 19:59:42 +00:00
paul cannon
41c5879f7c satellite: more detailed goroutine labels
This will apply an appropriate "subsystem" label to goroutines which are
part of the core, api, repairer, admin, or gc subsystems.

It will also label goroutines whose job it is to watch for slow shutdown
of lifecycle groups (there are a lot of these).

Finally, this will also label goroutines whose job it is to wait on the
toplevel errgroup of a subsystem.

Change-Id: I560b5fff4a0101300d6c9a67609c2d80d7424486
2022-05-11 17:50:55 +00:00
Vitalii
dedccbd2e4 satellite/console, web/satellite: limit failed login attempts
Added account locking on 3 or more login attempts.
Includes both password and MFA failed attempts on login.
Unlock account on successful password reset.

Change-Id: If4899b40ab4a77d531c1f18bfe22cee2cffa72e0
2022-05-11 14:49:11 +00:00
Erik van Velzen
928375a67c satellite/repair/queue: buffer batch insert
Implement a buffer for inserting repair items into the queue in a batch.

Part of https://github.com/storj/storj/issues/4727

Change-Id: I718472b2f2b1f4993c3d6f15c44923776407155a
2022-05-11 09:02:20 +00:00
Kaloyan Raev
15bed0ed0e satellite/metabase: use TRUNCATE in TestingDeleteAll
TRUNCATE is faster than DELETE when deleting all rows.

As almost every metabase test case calls TestingDeleteAll, this change
should give some slight test speed-up.

Change-Id: Ib477962b6deb93edd60d6db2f1be6ede1b4b2381
2022-05-10 19:13:04 +00:00
prerna-parashar
490254213e
satellite/analytics: Track events generated from Upgrade Account Banner (#4770)
satellite/analytics: Track events generated from Upgrade Account banner
2022-05-10 10:01:27 -07:00
paul cannon
fd01c6cc25 satellite/{repair,audit}: simplify reputation reporter
Also, make it an interface so that the upcoming write cache can be
dropped in to the same place.

Change-Id: I2c286743825e647c0cef5b6578245391851fa10c
2022-05-10 14:04:43 +00:00
Ivan Fraixedes
648d6bf0a7 satellite/metabase: Classify pending object error
Create an error class for the "pending object error" for distinguishing
it from other errors for allowing to return it as a "Not Found" DRPC
status code instead an "Internal" status code.

"Internal" errors are logged in the satellite error so this was
polluting the server logs aside of returning an inappropriate status
code.

Change-Id: I10a81adfc887c030c08a228158adc8815834b23c
2022-05-10 10:54:17 +00:00
Michał Niewrzał
e4a4248337 satellite/metainfo: add aws cli user agent for tracking
Change-Id: Ibb00a836dd1dc22b5e20eeb1d1df593ca6ae7d8e
2022-05-09 21:09:33 +00:00
Moby von Briesen
9fc6484600 private/apigen: Reorganize and rename files
Change-Id: I430d620bb0940f2186cb884b00cb5eb9729b9744
2022-05-09 15:30:54 +00:00
Erik van Velzen
26f495f717 satellite/repair: implementation of batch insert
Part of https://github.com/storj/storj/issues/4727

Change-Id: I44990a7614af26f8ee0be9c7aed496a1dd9e5df7
2022-05-09 12:41:22 +00:00
Erik van Velzen
10d71a8a3c satellite/satellitedb: outline for batch insert
Part of https://github.com/storj/storj/issues/4727

Change-Id: I1a9ad3b009f363e37f5e68e810074eecb7448db3
2022-05-09 11:39:52 +00:00
Ivan Fraixedes
05d9c7940d
satellite/.../consoleapi: Respond with 401 on unauth req (#4781)
Respond with the appropriate HTTP status code when a request to the
analytics trigger event handler receive an authorized request.

A part of fixing the response status code this will stop to log these
response with ERROR level in our satellite logs.

Example of error message found in our satellite logs:

  {
    "insertId": "0ljf1cfn4xroxfd6",
    "jsonPayload": {
      "N": "console:endpoint",
      "T": "2022-05-06T13:31:35.415Z",
      "errorVerbose": "unauthorized: http: named cookie not present\n\tstorj.io/storj/satellite/console.GetAuth:72\n\tstorj.io/storj/satellite/console/consoleweb/consoleapi.(*Analytics).EventTriggered:60\n\tnet/http.HandlerFunc.ServeHTTP:2047\n\tstorj.io/storj/satellite/console/consoleweb.(*Server).withAuth.func1:488\n\tnet/http.HandlerFunc.ServeHTTP:2047\n\tgithub.com/gorilla/mux.(*Router).ServeHTTP:210\n\tstorj.io/storj/satellite/console/consoleweb.(*Server).withRequest.func1:495\n\tnet/http.HandlerFunc.ServeHTTP:2047\n\tnet/http.serverHandler.ServeHTTP:2879\n\tnet/http.(*conn).serve:1930",
      "L": "ERROR",
      "error": "unauthorized: http: named cookie not present",
      "message": "unauthorized: http: named cookie not present",
      "code": 500,
      "S": "storj.io/storj/satellite/console/consoleweb/consoleapi.serveCustomJSONError\n\t/go/src/storj.io/storj/satellite/console/consoleweb/consoleapi/common.go:37\nstorj.io/storj/satellite/console/consoleweb/consoleapi.serveJSONError\n\t/go/src/storj.io/storj/satellite/console/consoleweb/consoleapi/common.go:23\nstorj.io/storj/satellite/console/consoleweb/consoleapi.(*Analytics).serveJSONError\n\t/go/src/storj.io/storj/satellite/console/consoleweb/consoleapi/analytics.go:75\nstorj.io/storj/satellite/console/consoleweb/consoleapi.(*Analytics).EventTriggered\n\t/go/src/storj.io/storj/satellite/console/consoleweb/consoleapi/analytics.go:62\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2047\nstorj.io/storj/satellite/console/consoleweb.(*Server).withAuth.func1\n\t/go/src/storj.io/storj/satellite/console/consoleweb/server.go:488\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2047\ngithub.com/gorilla/mux.(*Router).ServeHTTP\n\t/go/pkg/mod/github.com/gorilla/mux@v1.8.0/mux.go:210\nstorj.io/storj/satellite/console/consoleweb.(*Server).withRequest.func1\n\t/go/src/storj.io/storj/satellite/console/consoleweb/server.go:495\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2047\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2879\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1930",
      "M": "returning error to client"
    },
    "resource": {
      "type": "k8s_container",
      "labels": {
        "location": "us-central1",
        "pod_name": "us-central1-satellite-api-77c47f5c5-dzrpj",
        "project_id": "storj-prod",
        "namespace_name": "satellite",
        "container_name": "satellite",
        "cluster_name": "us-central1-gke-manatee"
      }
    },
    "timestamp": "2022-05-06T13:31:35.416050390Z",
    "severity": "ERROR",
    "labels": {
      "k8s-pod/version": "v3",
      "k8s-pod/app": "us-central1-satellite-api",
      "compute.googleapis.com/resource_name": "gke-us-central1-gke--terraform-202110-97ff1891-t0fv",
      "k8s-pod/service": "api",
      "k8s-pod/pod-template-hash": "77c47f5c5"
    },
    "logName": "projects/storj-prod/logs/stderr",
    "receiveTimestamp": "2022-05-06T13:31:37.419991630Z"
  }

Change-Id: I7cfcfb500b7878c59b1d259683c92e8963e2dc3f

Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2022-05-08 12:35:42 +02:00
Michał Niewrzał
307295977d satellite/{audit,metrics}: optimize loop methods
What was applied:
* avoid extra map lookups
* reorganize monikit for less cpu usage

Change-Id: I70575f404f717f7905b27d43888cbd7489f0176d
2022-05-05 15:10:56 +00:00
Qweder93
a3c9ca653f satellite/accounting: GetAllProjectTotals returns storage and segment totals combined
return storage and segment totals as a single result, instead of returning only storage
and bandwidth and segment values are filtered out, https://github.com/storj/storj/issues/4744

Change-Id: I624d67ed5205ae21ecd5a2f39775f63ed042e629
2022-05-05 13:38:33 +00:00
Moby von Briesen
24bbb7f236 satellite/satellitedb: Add webapp sessions table
Enables us to better manage sessions for satellite UI users.

Change-Id: I6695f8f1174d2e5fb970efddb0a8b14725e2b57e
2022-05-04 20:37:56 +00:00
Fadila Khadar
af1f0aa943 satellite/repair: lighten tests covering excluded countries
TestSegmentInExcludedCountriesRepair and TestSegmentInExcludedCountriesRepairIrreparable are using 20 storage nodes.
This change make them use 7 by adjusting the test redundancy scheme.

Change-Id: I1a44aa8b997d6edcc9a3305fdd0dac57e4d525b5
2022-05-04 07:51:59 +00:00
Qweder93
384118ee1e satellite/metabase: use pgutil.NullTimestampTZArray
This code cleanup to use helper DB method.

Change-Id: Idc438234cb4f7da23d6ade2d69f5d889f15e9009
2022-05-03 19:30:21 +00:00
hovex023
58f957bd1d
web/satellite: Add new access grants flow (#4765)
* Added new feature Flag for new Access Grant Flow. 

* Added 3 cards to access grant view for S3, CLI and Access grant to replace old header

* Added new formatting, text and Icon for Access Grant Delete Popup modal
2022-04-29 10:31:52 -05:00
Vitalii
96411ba56a rest-api: endpoint reworkings
Added documentation.
Replaced PUT request with POST request.
Added inline param support for PATCH request.
Replaced unix timestamps handling with RFC-3339 timestampts handling.
Added 'Bearer' method requirement for Authorization header.

Change-Id: I4faa3864051dd18826c2c583ada53666d4aaec44
2022-04-28 18:17:54 +00:00
prerna-parashar
894b7b1cf3
satellite/analytics: Track event "Credit Card Added" (#4749)
satellite/analytics: Track event "Credit Card Added"
2022-04-28 11:12:42 -07:00
Qweder93
04b22f24eb satellite/metabase: copy expires_at to segments while FinishCopyObject
Resolves: https://github.com/storj/storj/issues/4726

Change-Id: I0bb222b7d5d3c66fe96c78a8aa1ea8f3b0a33f2b
2022-04-27 18:39:54 +00:00
Mya
5cebbdee03 web/satellite: add consent screen for oauth
When an application wants to interact with resources on behalf of
an end-user, it needs to be granted access. In OAuth, this is done
when a user submits the consent screen.

Change-Id: Id838772f76999f63f5c9dbdda0995697b41c123a
2022-04-27 14:33:07 +00:00
Cameron
dd6ab25cbd {cmd/satellite/reports, satellite/attribution}: type and variable name adjustments
Change-Id: I553b7ed9cb702479911baaf54d77d8f2662faaa2
2022-04-26 20:12:38 +00:00
nadimhq
dd6aac9b39
satellite/metainfo: enable server-side copy in production (#4708)
these changes enable server-side copy in production
2022-04-26 09:58:16 -04:00
Michał Niewrzał
db5d781735 satellite/metainfo: simplify version collector logging
Version collector previously returned errors and logged them in the
calling code.  It is cleaner to log inside version collector.

Change-Id: I52cb49a1ef53f3f1f51692ddb26ec095cfd0f100
2022-04-26 10:16:20 +00:00
Michał Niewrzał
e66beb2429 satellite/metainfo: set metadata with BeginObject
This change makes possible to set metadata at the beginning
of upload.

https://github.com/storj/team-metainfo/issues/105

Change-Id: If5b6b90243d9c9318a8b72c2af3fbde400c5d8e7
2022-04-26 08:04:27 +00:00
Michał Niewrzał
6e5a94698e satellite/metabase: add option to override metadata with CommitObject
We were already able to override (or not) metadata with this method
but to be explicit we are introducting new option to control storing
metadata with object. Separate option should be less error prone.

https://github.com/storj/team-metainfo/issues/105

Change-Id: I4c5bce953a633a0009b05c5ca84266ca6ceefc26
2022-04-26 08:03:52 +00:00
Moby von Briesen
ed5ebb2527 satellite: Rename "acct mgmt api" to "rest api"
"REST API" is a more accurate descriptor of the generated API in the
console package than "account management API". The generated API is very
flexible and will allow us to implement many more endpoints outside the
scope of "account management", and "account management" is not very well
defined to begin with.

Change-Id: Ie87faeaa3c743ef4371eaf0edd2826303d592da7
2022-04-25 18:51:46 +00:00
Vitalii
318f4dc688 satellite/admin: extend user update query
Extended user update query so prod owner can change user's paid tier status, bandwidth, storage and segment limits.

Change-Id: I82768afd1e50f653a50f7020310ce1e91578d746
2022-04-25 13:44:25 +00:00
JT Olio
abc9e2493f satellite/compensation: add a code that crypthopper-go now uses
it's a bit weird that these code definitions are in storj/storj
instead of storj/crypthopper-go, but as it stands, we should make
sure this package knows about all the codes in use.

Change-Id: I8df4666a015098e2d2e536d2f6c8ca5317a4369c
2022-04-25 10:46:51 +00:00
Qweder93
cb1bf52794 satellite/metainfo: add expireddeletion test for copied objects
We implemented server-side copy feature and we would like to
confirm that it is not affecting expired deletion service.

Resolves: https://github.com/storj/storj/issues/4698

Change-Id: Ia8ca27a7ab7764a48a0c85dc7be80a58bfc83729
2022-04-21 12:07:00 +00:00
Qweder93
7e6618cfad satellite/accounting: test tally on copied object flaky fix
Resolves: https://github.com/storj/storj/issues/4732

Change-Id: I2abf50aafc62dd97be12b0c0573db2f95d560796
2022-04-21 10:30:17 +00:00
Yaroslav Vorobiov
553ef9d640 satellite/satellitedb: suspended column removal migration
Last part of backwards compatible db migration to remove "suspended" column.
Removes exeption which removes "suspended" column in tests from `migrate_test.go`.
Adds DB migration to remove "suspended" column from 'nodes' and 'reputations' tables.

Change-Id: I02051279f6f4181e966c567919af0e774583f165
2022-04-20 17:59:11 +00:00
Michał Niewrzał
671923cc60 satellite/admin: add mising segment limit to top menu
We missed menu entry for segment limits.

Change-Id: I977515acb282582b0730e581662ca262cc75f0b5
2022-04-20 16:36:14 +00:00
Yaroslav Vorobiov
3f47d19aa6 satellite/overlay: add disqualification reason
Add disqualification reason to NodeDossier.
Extend DB.DisqualifyNode with disqualification reason.
Extend reputation Service.TestDisqualifyNode with disqualification reason.

Change-Id: I8611b6340c7f42ac1bb8bd0fd7f0648ad650ab2d
2022-04-20 13:29:31 +00:00
Yaroslav Vorobiov
4223fa01f8 satellite/reputation: add disqualification reason for status update
Set disqualification reason when reputations stats are updated on DB.Update.
Added tests for DisqualifyNode and for disqualification cases which happens during Update.

Change-Id: I00130ab5d9722422805159ad2f183c205de60f7e
2022-04-20 13:29:10 +00:00
paul cannon
3f3f028c88 satellite/gracefulexit: don't mark GE done when it's not done
When an api server is processing a graceful exit (node is connected and
getting lists of pieces to transfer), and the api server is shut down,
it was incorrectly marking all pending graceful exits as complete. The
GE then either passed or failed depending on the ratio of successfully
transferred pieces to unsuccessful pieces. In at least one case, _no_
pieces were transferred at all before the GE was marked a success.

Change-Id: I62cfab54a2296572c2e654eb460b62f772b7a60b
2022-04-19 14:00:29 +00:00
Cameron
48fb3e947c cmd/satellite/reports: sum attribution data by user agent
Attribution is attached to bucket usage, but that's more granular than
necessary for the attribution report. This change iterates over the
bucket attributions, parses the user agent, converts the first entry
to lower case, and uses that as the key to a map which holds the
attribution totals for each unique user agent.

Change-Id: Ib2962ba0f57daa8a7298f11fcb1ac44a8bb97875
2022-04-18 13:23:37 +00:00
NickolaiYurchenko
c32ca6e67f apigen: endpint to update project
Implemented new endpoint for project update using apigen.
Implemented new service method compatible with new generated api.

Change-Id: Ic0a7e0bbf3ea942275bd927d6e30cfb7e721e9c1
2022-04-14 22:21:08 +00:00
Michał Niewrzał
6188c4c0dd satellite/metainfo: enable segment limit by default
Change-Id: I26cc0bbd97d6cc2ffc96213788f261efb749d028
2022-04-14 06:12:09 +00:00
Vitalii
513ad99756 satellite/satellitedb: migrate wrongly affected users to have default limits
Migrate free tier users to have default limits if their limits were set to 0.
They were affected by incorrect working of Update user query.

Change-Id: I4c49c8d99b12dba2b9b0ab61b2175085976dcc95
2022-04-13 13:33:03 +00:00
Qweder93
8e0196a6cc satellite/accounting: test accounting/tally with object copies
We implemented server-side copy feature and we would like to
confirm that it is not affecting accounting/tally service.

Resolves https://github.com/storj/storj/issues/4697

Change-Id: I3944ea52c0acc68107ec15c1911750dc7d947501
2022-04-13 09:16:43 +00:00
Michał Niewrzał
623cb16b6e satellite/gracefulexit: test GE with copies
Test case to verify if server-side copy doesn't affect
GE in any negative way.

Fixes https://github.com/storj/storj/issues/4699

Change-Id: I8c385767cca61499d46d9cb8de7318c56e5d7397
2022-04-12 15:59:14 +00:00