Commit Graph

490 Commits

Author SHA1 Message Date
paul cannon
2f20bbf4d8 satellite/reputation: add a reputation write cache
This should lower the amount of database load coming from
reputation updates.

Change-Id: Iaacfb81480075261da77c5cc93e08b24f69f8949
2022-07-14 21:40:16 +00:00
Yaroslav Vorobiov
de5951331a satellite: wire storjscan chore to core process
Adds storjscan chore and storjscan client to satellite chore.
Storjscan chore interval added to storjscan config.

Change-Id: I13bb8d83b14ca90f4735c9a10c361ad24a56b256
2022-07-14 15:07:52 +00:00
Jeremy Wharton
5ce7d980af private/apigen: Make API generation deterministic
This change fixes the issue where the API generator would produce
different Go code for the same API definition upon each invocation
due to the random nature of map iteration.

Change-Id: I6770a10faf06311c24f541611c25d0b2b0f8e521
2022-06-17 12:06:08 -05:00
Jeremy Wharton
bd0d717f4c private/apigen: Remove unnecessary log message when generating API code
A message intended to diagnose issues in a previous change was
accidentally allowed to remain in finalized code and merged into
the codebase.

Change-Id: I7df662a81e3790de065beeb6db54be5280ac2fa1
2022-06-15 15:58:45 -05:00
Jeremy Wharton
905654956a private/apigen: Resolve import issues
This change resolves several issues:
* Imports can no longer be duplicated
* The import statement now conforms to Storj's code style requirements
* Packages are imported as needed, eliminating extraneous imports

Change-Id: Ie021e06bb0d472c65aa3575db94bc81ccac108e3
2022-06-15 19:51:54 +00:00
Jeremy Wharton
58c5d44f44 satellite/console: integrate sessions into satellite UI
This change integrates the session management database functionality
with the web application. Claim-based authentication has been removed
in favor of session token-based authentication.

Change-Id: I62a4f5354a3ed8ca80272814aad2448f901eab1b
2022-06-13 08:02:02 +00:00
JT Olio
c761acc0b5 web/ratelimiter: support disabling
Change-Id: I6f2d77016c4917389514b75b5eab64223cc69b2d
2022-06-09 15:33:35 +00:00
Vitalii
f56504de2a apigen: project delete endpoint
Implemented project delete endpoint for REST API.
Added project usage status check service method to indicate if project can be deleted.
Updated project invoice status check method to indicate if project can be deleted.

Change-Id: I57dc96efb072517144252001ab5405446c9cdeb4
2022-06-07 12:23:24 +03:00
Márton Elek
84d02f7fbf metabase: use predefined snapshot for unit tests avoid migration
Change-Id: I6225ae32c312030e3559d8ed0faa137ffc3cc5b7
2022-06-01 12:07:00 +00:00
Cameron
b379fe89d2 satellite/core.go: hook up flag to enable email reminders
Change-Id: I973bb3a709e806e7744679b3543217c1cab85f9a
2022-05-27 13:16:14 -04:00
Vitalii
d916b26e0e apigen: create api key endpoint
Implemented new service method for generating API keys.
Implemented new endpoint.
Improved multiple endpoint groups handling.

Change-Id: Iba26fbf9123707b5b4c2d5e8c5a35d507404f24a
2022-05-26 16:00:23 +00:00
Márton Elek
5e8266d1de satellite/{payments/storjscan,satellitedb}: Add storjscan wallet DB
Add storjscan wallets DB to the satellite. For now this DB is a one to one mapping of the users account to a storjscan wallet that can be used by the account holder to make payments on their Storj account.

relates to https://github.com/storj/storj/issues/4347

Change-Id: I6e65b15817b90ceb75641244f9bf173c3b4228a7
2022-05-24 08:42:02 +00:00
Yaroslav Vorobiov
0bf12523e1 satellite/payments/storjscan: add client and list all payments API call
Change-Id: I1f5065b3d15cc93f4b42868941e82e04af364565
2022-05-20 15:10:17 +00:00
Malcolm Bouzi
087e57d037 web/satellite: create chore that will resend verification emails to unverified users
We want to remind unverified users to verify their emails:
once after 24 hours has passed and again after 5 days has passed.

Add mailservice.Service to satellite core because it is needed by the
chore for sending emails. To add the mailservice.Service to the core,
we create a helper function in satellite/peer.go to avoid duplicating
the code in both api.go and core.go. In addition to the chore, this
change adds methods to users.DB to get unverified users in need of
reminder.

Change-Id: I4e515bdf43f922788b4f965b2efb34fa32288bd1
2022-05-18 08:08:33 +00:00
Márton Elek
82b1450a46 private/blockchain: generic types for wallet/transactions
Change-Id: I32a77c58ee519059bf6cd42e0db829d5104843a4
2022-05-17 12:05:11 +00:00
paul cannon
41c5879f7c satellite: more detailed goroutine labels
This will apply an appropriate "subsystem" label to goroutines which are
part of the core, api, repairer, admin, or gc subsystems.

It will also label goroutines whose job it is to watch for slow shutdown
of lifecycle groups (there are a lot of these).

Finally, this will also label goroutines whose job it is to wait on the
toplevel errgroup of a subsystem.

Change-Id: I560b5fff4a0101300d6c9a67609c2d80d7424486
2022-05-11 17:50:55 +00:00
Michał Niewrzał
6175731166 private/testplanet: use simpler access for tests
This change created access grant manually to save some cpu
by avoiding root key derivation and avoid dialing satellite
for project id. It affects only testplanet tests.

Change-Id: I6742bcf699cca51e658f147e6df72c6b3db78d10
2022-05-10 20:10:10 +00:00
paul cannon
fd01c6cc25 satellite/{repair,audit}: simplify reputation reporter
Also, make it an interface so that the upcoming write cache can be
dropped in to the same place.

Change-Id: I2c286743825e647c0cef5b6578245391851fa10c
2022-05-10 14:04:43 +00:00
Moby von Briesen
9fc6484600 private/apigen: Reorganize and rename files
Change-Id: I430d620bb0940f2186cb884b00cb5eb9729b9744
2022-05-09 15:30:54 +00:00
Vitalii
96411ba56a rest-api: endpoint reworkings
Added documentation.
Replaced PUT request with POST request.
Added inline param support for PATCH request.
Replaced unix timestamps handling with RFC-3339 timestampts handling.
Added 'Bearer' method requirement for Authorization header.

Change-Id: I4faa3864051dd18826c2c583ada53666d4aaec44
2022-04-28 18:17:54 +00:00
NickolaiYurchenko
c32ca6e67f apigen: endpint to update project
Implemented new endpoint for project update using apigen.
Implemented new service method compatible with new generated api.

Change-Id: Ic0a7e0bbf3ea942275bd927d6e30cfb7e721e9c1
2022-04-14 22:21:08 +00:00
Kaloyan Raev
de6852510f private/testplanet: configurable timeout
Testplanet limits the execution of a single test case to 3 minutes.

This change adds a Timeout field to the Testplanet's config, so test
cases can configure their timeout. This is helpful when executing larger
3rd party test suite on top of Testplanet.

Change-Id: Ibbf7c5ffdc0a9e723e7e28b885eac084f04c6ca1
2022-04-11 13:19:08 +00:00
Vitalii
3b39399905 apigen: endpoint to create new Project
Implemented new endpoint for project creation using apigen.
Implemented new service method compatible with new generated api.

Change-Id: I2bae22c8b046f21ec5bb6522f09b9c4e74bdba0c
2022-04-06 17:49:46 +00:00
Vitalii
67b5b07730 apigen: api key authentication implemented
Implemented account management api key authentication.
Extended IsAuthenticated service method to include both cookie and api key authorization.

Change-Id: I6f2d01fdc6115cb860f2e49c74980a39155afe7e
2022-04-01 15:17:38 +00:00
Egon Elbre
7aa001d62f private/version: document kebabToPascal
Change-Id: Id7bdf969072c330956c093c24b662f4330fc4d4c
2022-03-31 13:52:27 +03:00
Egon Elbre
d9b14f7902 private/version: replace custom stringsTitle
cases package has an option cases.NoLower for the behavior we need.

Change-Id: Ie5bd4c52384fff45228321e02ab2478b5fecf245
2022-03-31 13:28:10 +03:00
Egon Elbre
e5972d8920 all: fix linting errors
strings.Title is deprecated in Go 1.18, replace it with
golang.org/x/text/cases.

Change-Id: I6185b97d37309dbe4a6715f794383ab259a0658b
2022-03-30 18:45:23 +03:00
Egon Elbre
19a2555126 private/testplanet: try using multiple localhosts
Rather than starting all servers on 127.0.0.1 start them
on a random local host to try avoid port exhaustion.
The port exhaustion is just a guess.

Change-Id: Ibf31d6a017852238d836291d703642b44ff66c0c
2022-03-29 05:24:35 +00:00
Egon Elbre
0d2d59f884 all: fix linting issues
Change-Id: Idfc93948e59a181321d79b365e638d63e256a16f
2022-03-21 15:26:42 +00:00
Jeff Wendling
1bd43c1064 private/server: monitor incoming connection kinds
Change-Id: Ia90d4fba58d87b3d5324fbe480b57c6381b9f873
2022-03-15 21:20:32 +00:00
Fadila Khadar
29fd36a20e satellite/repairer: handle excluded countries
For nodes in excluded areas, we don't necessarily want to remove them
from the pointer, but we do want to increase the number of pieces in the
segment in case those excluded area nodes go down. To do that, we
increase the number of pieces repaired by the number of pieces in
excluded areas.

Change-Id: I0424f1bcd7e93f33eb3eeeec79dbada3b3ea1f3a
2022-03-14 10:59:36 -04:00
Egon Elbre
5f7ea1358d private/web: make caching headers reusable
Move storagnode/console caching headers to private/web. Also,
start using them in multinode/console/server.

Change-Id: I1f0f3c9833a183476009737cece515ae7537fb83
2022-03-11 11:19:11 +02:00
Vitalii Shpital
1245283637 apigen: new endpoint to get project's single bucket rollup
Added new endpoint to get project's single bucket usage rollup.
Extended generation code to handle service method args.

Change-Id: Ief768632a801c047c66e0617056fbd7b30427b33
2022-03-04 17:33:38 +00:00
Vitalii Shpital
9b5904cd49 satellite/{projectaccounting, console}:query to get single bucket rollup
Added new projectaccounting query to get project's single bucket usage rollup.
Added new service method to call new query.
Added implementation for IsAuthenticated method which is used by new generated API.

Change-Id: I7cde5656d489953b6c7d109f236362eb465fa64a
2022-03-03 12:04:29 +00:00
Vitalii Shpital
ba6956db0f console/server, apigen: feature flag for new generated console api
Added a feture flag which will be used to indicate if new generated console api is used.
Fixed some comments from previous PR.

Change-Id: Ice31c998b0b347028a491c971a648fd1269bfd49
2022-02-28 23:00:12 +00:00
Michał Niewrzał
fbe2680500 satellite/metainfo: add feature flag for server side copy code
We would like to disable in production those parts of code
which are now mixed with new server-side copy logic.

Change-Id: Iff50682bc9545207330f58dd19b5eee53d404d7f
2022-02-24 10:43:49 +00:00
Stefan Benten
321bf26c85
cmd,private,satellite: add application_name to metabase connections
Currently the metainfo/metabase DB connections are missing the proper
application_name in order to differentiate and filter queries on the DB
side for analytics.
Without it, it is very time-consuming to correlate processes and their load.

This change adds the "check" on DB connection init and passes the fallbacks
in all places to catch connection strings, that do not set it.

Change-Id: Iea5cea8658bc63778ff89038e5c1c352bf482cfd
2022-02-20 22:29:41 +01:00
Mya
56c668a2e3 private/testplanet: use info level logs in jenkins
Change-Id: Ie500a7bfcd23ca7470139341e961b2e9986d6a33
2022-02-10 11:47:06 -06:00
Vitalii Shpital
07c71e34c2 satellite/console{gen}: GetUsersProjects endpoint
Initial implementation of auto-generated GetUsersProjects endpoint

Change-Id: If41bff2ea3ff9cfc87afeda9e5e5b3f586cbab33
2022-02-10 08:03:02 +00:00
Márton Elek
2ae78db660 satellite/email: add delimiter to close the last part of Multipart emails
The existing implementation doesn't send proper rfc1341 compatible mails
according to https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html as
the closing `--boundary_id--` is missing from our mails. (see 7.2.1 from
the standard).

Mailservers which are not very flexible, deny the mails.

Example mail log: (see the
missing boundary delimiter at the end).

```
Subject: Activate your email
From: "Storj DCS - EU1" <noreply@eu1.storj.io>
To: "..." <...>
...

--26d7220f6f1c9f6fb47b535319eb15dce513bb6e1d941a0efddf25e96712
Content-Type: text/html; charset=UTF-8

....

</body>
</html>
.
smtp: DATA error	{"msg_id":"6d82c9e3","reason":"unexpected EOF"}
smtp: 554 5.0.0 Internal server error (msg ID = 6d82c9e3)
```

This patch moves all the Multipart writing to a function to make sure
that the `wr.Close()` (which writes out the last part) is executed
BEFORE `body.Bytes()` (defer added it AFTER `body.Bytes()` was calculated)

Change-Id: I8f18fc81b1857b646470eab32e73d6cbdc50d2ad
2022-02-01 09:55:29 +00:00
Egon Elbre
a1aa9f24ed private/testplanet: expose NewLogger for other tests
NewLogger contains bunch of logic that is useful for creating
a nice logger.

Change-Id: Idad57525d334f40b3faa70dbc4737b114992b0a4
2022-01-27 13:56:27 +02:00
Vitalii Shpital
5d6ee506b0 private/apigen: initial typings and methods
Initial typings and methods for the future API Management implementation.
Blueprint - https://review.dev.storj.io/c/storj/storj/+/6341

Change-Id: I9afe66a7610a3a80e764b317ddd211f125b1ea43
2022-01-26 14:39:22 +00:00
Malcolm Bouzi
c30c7def44 satellite/{console,payments},web/satellite: save signup promo code to newly registered users
Users signing up through a url containing a promo code will have that code applied to their stripe account instead of the free tier coupon.

Change-Id: I071041b0934648ef3f5bdb05b6ec97c400f89ae4
2022-01-21 15:02:22 +00:00
Clement Sam
8b40a071a0 storagenode: check if QUIC is properly configured
This change adds a check to confirm if UDP port if properly configured for QUIC

Resolves https://github.com/storj/storj/issues/4332
Partly resolves https://github.com/storj/storj/issues/4358

Change-Id: I9a66f26a115e48b4fcd168f50a7d0b4d81712f4e
2022-01-20 12:04:04 +00:00
Clement Sam
589c82f23e storagenode: display quic status on SNO dashboard
This change includes storagenode QUIC status on SNO dashboard.
If disabled, it displays warning for SNOs to foward their
UDP port for quic.

Change-Id: I8d28c9c0f5f1e90d80b7c18b9e1e7b78c5e45609
2021-12-23 12:22:24 +00:00
Ethan
185617326c satellite/email: Move MIME-Version header to the last element before Content-Type
Some MTAs are marking our emails as SPAM because of the current location of the header.

Change-Id: I3f12c36b184f69b643a8299f5ccbc609f4599ef9
2021-12-17 11:26:25 -06:00
TungHoang
f9b630b0f4
Fix monthly earning estimation (#4282) 2021-11-17 18:26:21 -05:00
Mya
814e3126fa satellite/buckets: add new buckets service
The main motivation is to wrap the bucket DB and metainfo DB, so we
could check if a bucket is empty before applying geofencing config.

Change-Id: I8bac21555e01d51a663fb557bc1acfc8106bc2e1
2021-11-16 12:36:17 +02:00
Egon Elbre
dfd2977a01 testsuite/ui: fixes to setup
* Remove "enable-logging", because it ends up spawning consoles on
  Windows.
* Remove "disable-gpu", if we have a GPU, then let's use it.
* Create custom client, so we can attach logging to CDP.
* Ignore potential context.Canceled.
* Fix onboarding wizard test for new objects browser.
* Return an error on a context cancellation.
* Wire all loggers to planet.

Change-Id: I67eb138ba31252f55ac5b383679d033bcf71f1b2
2021-11-15 15:56:35 +02:00
Qweder93
65d139fb96 private/testplanet: create metabaseDB after reconfigure apply
Moved CreateMetabaseDB after applying reconfigure changes to affect them during testing

Change-Id: I51034a6671728645dcc52a0d8c808b9dfc2fba7c
2021-10-25 15:55:45 +00:00