Commit Graph

461 Commits

Author SHA1 Message Date
Jeff Wendling
1bd43c1064 private/server: monitor incoming connection kinds
Change-Id: Ia90d4fba58d87b3d5324fbe480b57c6381b9f873
2022-03-15 21:20:32 +00:00
Fadila Khadar
29fd36a20e satellite/repairer: handle excluded countries
For nodes in excluded areas, we don't necessarily want to remove them
from the pointer, but we do want to increase the number of pieces in the
segment in case those excluded area nodes go down. To do that, we
increase the number of pieces repaired by the number of pieces in
excluded areas.

Change-Id: I0424f1bcd7e93f33eb3eeeec79dbada3b3ea1f3a
2022-03-14 10:59:36 -04:00
Egon Elbre
5f7ea1358d private/web: make caching headers reusable
Move storagnode/console caching headers to private/web. Also,
start using them in multinode/console/server.

Change-Id: I1f0f3c9833a183476009737cece515ae7537fb83
2022-03-11 11:19:11 +02:00
Vitalii Shpital
1245283637 apigen: new endpoint to get project's single bucket rollup
Added new endpoint to get project's single bucket usage rollup.
Extended generation code to handle service method args.

Change-Id: Ief768632a801c047c66e0617056fbd7b30427b33
2022-03-04 17:33:38 +00:00
Vitalii Shpital
9b5904cd49 satellite/{projectaccounting, console}:query to get single bucket rollup
Added new projectaccounting query to get project's single bucket usage rollup.
Added new service method to call new query.
Added implementation for IsAuthenticated method which is used by new generated API.

Change-Id: I7cde5656d489953b6c7d109f236362eb465fa64a
2022-03-03 12:04:29 +00:00
Vitalii Shpital
ba6956db0f console/server, apigen: feature flag for new generated console api
Added a feture flag which will be used to indicate if new generated console api is used.
Fixed some comments from previous PR.

Change-Id: Ice31c998b0b347028a491c971a648fd1269bfd49
2022-02-28 23:00:12 +00:00
Michał Niewrzał
fbe2680500 satellite/metainfo: add feature flag for server side copy code
We would like to disable in production those parts of code
which are now mixed with new server-side copy logic.

Change-Id: Iff50682bc9545207330f58dd19b5eee53d404d7f
2022-02-24 10:43:49 +00:00
Stefan Benten
321bf26c85
cmd,private,satellite: add application_name to metabase connections
Currently the metainfo/metabase DB connections are missing the proper
application_name in order to differentiate and filter queries on the DB
side for analytics.
Without it, it is very time-consuming to correlate processes and their load.

This change adds the "check" on DB connection init and passes the fallbacks
in all places to catch connection strings, that do not set it.

Change-Id: Iea5cea8658bc63778ff89038e5c1c352bf482cfd
2022-02-20 22:29:41 +01:00
Mya
56c668a2e3 private/testplanet: use info level logs in jenkins
Change-Id: Ie500a7bfcd23ca7470139341e961b2e9986d6a33
2022-02-10 11:47:06 -06:00
Vitalii Shpital
07c71e34c2 satellite/console{gen}: GetUsersProjects endpoint
Initial implementation of auto-generated GetUsersProjects endpoint

Change-Id: If41bff2ea3ff9cfc87afeda9e5e5b3f586cbab33
2022-02-10 08:03:02 +00:00
Márton Elek
2ae78db660 satellite/email: add delimiter to close the last part of Multipart emails
The existing implementation doesn't send proper rfc1341 compatible mails
according to https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html as
the closing `--boundary_id--` is missing from our mails. (see 7.2.1 from
the standard).

Mailservers which are not very flexible, deny the mails.

Example mail log: (see the
missing boundary delimiter at the end).

```
Subject: Activate your email
From: "Storj DCS - EU1" <noreply@eu1.storj.io>
To: "..." <...>
...

--26d7220f6f1c9f6fb47b535319eb15dce513bb6e1d941a0efddf25e96712
Content-Type: text/html; charset=UTF-8

....

</body>
</html>
.
smtp: DATA error	{"msg_id":"6d82c9e3","reason":"unexpected EOF"}
smtp: 554 5.0.0 Internal server error (msg ID = 6d82c9e3)
```

This patch moves all the Multipart writing to a function to make sure
that the `wr.Close()` (which writes out the last part) is executed
BEFORE `body.Bytes()` (defer added it AFTER `body.Bytes()` was calculated)

Change-Id: I8f18fc81b1857b646470eab32e73d6cbdc50d2ad
2022-02-01 09:55:29 +00:00
Egon Elbre
a1aa9f24ed private/testplanet: expose NewLogger for other tests
NewLogger contains bunch of logic that is useful for creating
a nice logger.

Change-Id: Idad57525d334f40b3faa70dbc4737b114992b0a4
2022-01-27 13:56:27 +02:00
Vitalii Shpital
5d6ee506b0 private/apigen: initial typings and methods
Initial typings and methods for the future API Management implementation.
Blueprint - https://review.dev.storj.io/c/storj/storj/+/6341

Change-Id: I9afe66a7610a3a80e764b317ddd211f125b1ea43
2022-01-26 14:39:22 +00:00
Malcolm Bouzi
c30c7def44 satellite/{console,payments},web/satellite: save signup promo code to newly registered users
Users signing up through a url containing a promo code will have that code applied to their stripe account instead of the free tier coupon.

Change-Id: I071041b0934648ef3f5bdb05b6ec97c400f89ae4
2022-01-21 15:02:22 +00:00
Clement Sam
8b40a071a0 storagenode: check if QUIC is properly configured
This change adds a check to confirm if UDP port if properly configured for QUIC

Resolves https://github.com/storj/storj/issues/4332
Partly resolves https://github.com/storj/storj/issues/4358

Change-Id: I9a66f26a115e48b4fcd168f50a7d0b4d81712f4e
2022-01-20 12:04:04 +00:00
Clement Sam
589c82f23e storagenode: display quic status on SNO dashboard
This change includes storagenode QUIC status on SNO dashboard.
If disabled, it displays warning for SNOs to foward their
UDP port for quic.

Change-Id: I8d28c9c0f5f1e90d80b7c18b9e1e7b78c5e45609
2021-12-23 12:22:24 +00:00
Ethan
185617326c satellite/email: Move MIME-Version header to the last element before Content-Type
Some MTAs are marking our emails as SPAM because of the current location of the header.

Change-Id: I3f12c36b184f69b643a8299f5ccbc609f4599ef9
2021-12-17 11:26:25 -06:00
TungHoang
f9b630b0f4
Fix monthly earning estimation (#4282) 2021-11-17 18:26:21 -05:00
Mya
814e3126fa satellite/buckets: add new buckets service
The main motivation is to wrap the bucket DB and metainfo DB, so we
could check if a bucket is empty before applying geofencing config.

Change-Id: I8bac21555e01d51a663fb557bc1acfc8106bc2e1
2021-11-16 12:36:17 +02:00
Egon Elbre
dfd2977a01 testsuite/ui: fixes to setup
* Remove "enable-logging", because it ends up spawning consoles on
  Windows.
* Remove "disable-gpu", if we have a GPU, then let's use it.
* Create custom client, so we can attach logging to CDP.
* Ignore potential context.Canceled.
* Fix onboarding wizard test for new objects browser.
* Return an error on a context cancellation.
* Wire all loggers to planet.

Change-Id: I67eb138ba31252f55ac5b383679d033bcf71f1b2
2021-11-15 15:56:35 +02:00
Qweder93
65d139fb96 private/testplanet: create metabaseDB after reconfigure apply
Moved CreateMetabaseDB after applying reconfigure changes to affect them during testing

Change-Id: I51034a6671728645dcc52a0d8c808b9dfc2fba7c
2021-10-25 15:55:45 +00:00
Qweder93
50baefa10e satellite/metabase: limit maximum number of parts and size
Multipart upload limits added. Last part has no size limit.
Max number of parts: 10000, min part size: 5 MiB

Change-Id: Ic2262ce25f989b34d92f662bde720d4c4d0dc93d
2021-10-22 10:16:37 +00:00
Egon Elbre
bdadc4b460 private/testplanet: add stack traces related to the planet
It's useful to see only the goroutines that are related to the current
testplanet.

Change-Id: Id77c62dec42c09d44c523b019f7bdc940905fcf1
2021-10-20 23:12:07 +03:00
Clement Sam
29599dd7cd testsuite/ui/multinode: add first node test
Change-Id: Ibeaee92e6d1a41e408659a5045f52e1908f73089
2021-10-18 16:07:01 +00:00
Moby von Briesen
5b729779a2 satellite/console: Automatically log a user in after verifying email
When an email is verified, insert an auth cookie so that when the user
is redirected after verifying their email, they are immediately taken to
the onboarding flow.

Change-Id: I557d8a2805b24dd8039ada255522bc1b56cc8b53
2021-10-13 13:08:27 +00:00
Clement Sam
b8dd35ceaf private/testplanet: extend testplanet with multinode instance
Testplanet does not have a multinode instance, hence,
makes it difficult to run ui tests for the multinode dashboard.

Instead of using storj-sim for multinode ui tests,
it's better to use the same approach (testplanet) fir all UI tests.

This changes adds a multinode instance to testplanet.

Change-Id: I58aa8c4597e789275f9e7ea7059703c742903492
2021-10-13 10:19:45 +00:00
Egon Elbre
4020e9e2ce private/testmonkit: move monkit test helper
Some tests do not use testplanet and hence testmonkit should be
reusable.

Change-Id: If6b7194a199d9ffba5eb1a91f38d3a792f883336
2021-10-12 14:00:01 +03:00
Jeremy Wharton
3b751a35c5 satellite/{payments,satellitedb}: Remove custom coupon implementation
Removes database tables and functionality related to our custom
coupon implementation because it has been superseded by the Stripe
coupon and promo code system. Requires implementations of the
payments Invoices interface to return coupon usages along with
invoices.

Change-Id: Iac52d2ff64afca8cc4dbb2d1f20e6ad4b39ddfde
2021-10-11 19:47:00 +00:00
Egon Elbre
7ebd35337b private/testplanet: document env variables
Change-Id: I122d56b90af3df1f1b17716a31ddca7c115fca7c
2021-10-01 07:18:50 +00:00
Egon Elbre
6511bb91fb private/testplanet: support writing monitoring spans
This allows to set `STORJ_TEST_MONKIT` to either
`svg` or `json` to write individual testplanet test
traces to disk.

It also allows to specify an absolute directory:

  STORJ_TEST_MONKIT=svg,dir:/abs/dir/path

It requires an absolute path, because from the context of
tests, there's no easy way to find the folder where tests
were called.

Change-Id: I6fe008a4d4237d221cf5a5bede798b46399ee197
2021-09-30 09:03:35 +03:00
Vitalii Shpital
1def7b0ec3 web/satellite, testsuite/ui/satellite: added tests for invalid sign up credentials and satellites dropdown
Added tests for signup with invalid email or password.
Added test for signup screen's satellites dropdown.

Change-Id: I76d975039543e315b3e9c9416e3ec1f2a3331a6a
2021-09-29 15:20:24 +03:00
Yaroslav Vorobiov
469ae72c19 satellite/repair: update audit records during repair
Change-Id: I788b2096968f043601aba6502a2e4e784f1f02a0
2021-09-24 00:48:13 +00:00
crawter
e1a4195a35 satellite/metainfo/endpoint: MaxObjectKey length validation for BeginObject and CommitObject added
Change-Id: I6a766415a84b1e3c44ce65052ffa8ed4511d00dd
2021-09-13 15:07:16 +03:00
Egon Elbre
1aec831d98 satellite/audit,storage: increase sleep delay in TestMaxVerifyCount
Currently TextMaxVerifyCount flakes in some tests, try increasing the
sleep time to ensure that things are slow enough to trigger the error
condition.

Also pass ctx to all the funcs so we can handle sleep better.

Change-Id: I605b6ea8b14a0a66d81a605ce3251f57a1669c00
2021-09-10 15:30:37 +00:00
Michał Niewrzał
495e530933 satellite/metainfo: drop metainfo.Service
Drop a service that in fact don't make sense and
its just a wrapper for direct DB methods.

Change-Id: I1cb76f3ecc2d8765964d919c88541179957645c1
2021-09-09 17:30:10 +02:00
Michał Niewrzał
c258f4bbac private/testplanet: move Metabase outside Metainfo for satellite
At some point we moved metabase package outside Metainfo
but we didn't do that for satellite structure. This change
refactors only tests.
When uplink will be adjusted we can remove old entries in
Metainfo struct.

Change-Id: I2b66ed29f539b0ec0f490cad42c72840e0351bcb
2021-09-09 07:15:51 +00:00
Michał Niewrzał
b7555980c4 satellite/metainfo: add zombiedeletion chore
This chore will be deleting zombie objects and it's segments.

Change-Id: Ia5d92dd6c3f52e178d4d43d9603732310761e027
2021-08-24 08:22:06 +00:00
Jeremy Wharton
6a6cc28fc1 satellite/console,private/web: Rate limit coupon code application
Rate limits application of coupon codes by user ID to prevent
brute forcing. Refactors the rate limiter to allow limiting based
on arbitrary criteria and not just by IP.

Change-Id: I99d6749bd5b5e47d7e1aeb0314e363a8e7259dba
2021-08-23 17:00:31 +00:00
Artur M. Wolff
2cd68bf4fb private/lrucache: import from common
Change-Id: Ia1f43d0440fef21122b071b05da59b4cf2689d6c
2021-08-16 10:04:32 +00:00
Yingrong Zhao
077ec96d94 private/server: use quic implementation from storj.io/common
Change-Id: I820cf6444a3ddccee0d7c647dc84c80b2752068c
2021-08-10 13:32:21 +00:00
Yingrong Zhao
ae02f6deda satellite/reputation: return default reputation stats when node is not
found

Change-Id: I587d0ab36ffa0efaf345a6a6e221ae5d2068e1c5
2021-08-04 19:34:54 +00:00
Yingrong Zhao
f8914ccce0 satellite/{repair, overlay}: use reputation store in repair
Change-Id: I48db9e68f48239d48621ccc77d33618ecb83ce1a
2021-07-28 13:22:05 -04:00
Yingrong Zhao
58238d850c satellite/{audit, accounting}: use reputation store in tests
Change-Id: I86a8ccf5dcee8d108196a9f67a476fe0ccbd8257
2021-07-28 13:21:55 -04:00
Yingrong Zhao
55a77d04bc satellite/satellitedb,private: add initial value on testplanet startup
Currently, reputation table is only populated when a node has been
audited. This is ok in production, however a lot of our tests doesn't
upload any data or trigger audits.
This PR adds an initialization step in testplanet to populate reputation
table with zero value for nodes reputation.

Change-Id: I11b381236669db346dc68a48a6d4a27334a0a8b8
2021-07-28 13:20:32 -04:00
Yingrong Zhao
6c7bf357cd satellite/{reputation,audit,overlay}: replace overlay with reputation
package in audit

This PR implements reputation store and replace overlay in audit service
to use such store for storing node's audit stats.

In order to keep the changeset smaller, most of the changes in this PR is for copying audit logic in overlay to
reputation package. In a following PR, the duplicating code will be
removed from overlay.

Change-Id: I16c12494a0970f44c422b26cf603c1dc489e5bc1
2021-07-28 13:10:48 -04:00
Jeremy Wharton
dae6ed7d03 satellite/console: Implement MFA backend
Added MFA passcode and recovery code field for token requests.
Added endpoints for MFA-related activity: enabling MFA,
disabling MFA, generating a new MFA secret key, and
generating new MFA recovery codes.

Change-Id: Ia1443f05d3a2fecaa7f170f56d73c7a4e9b69ad5
2021-07-26 16:37:05 +00:00
Michał Niewrzał
b12d29935a satellite/metabase: remove metaloop package
We moved everything to segment loop so we can now
remove metaloop from code.

Change-Id: I9bd8d2349e5638d7cdad50f2f313f9bd89a8165c
2021-07-22 13:00:45 +00:00
Michał Niewrzał
cbbbfca439 satellite/accounting: move storage node tally to separate service
Current tally is calculating storage both for buckets and
storage nodes. This change is moving nodes storage
calculation to separate service that will be using
segment loop.

Change-Id: I9e68bfa0bc751c82ff738c71ca58d311f257bd8d
2021-07-08 14:02:33 +00:00
Jeremy Wharton
a5f6bb9cc0 satellite/console: Add reCAPTCHA verification step to registration
The user must complete a reCAPTCHA in order to register.
ReCAPTCHA verification failure results in rejection of the
registration attempt.

Change-Id: I34ba7db414d756fd1aaebdc3d19cccbfc7fc1ea3
2021-07-07 21:34:07 +00:00
Yaroslav Vorobiov
cbb4cd3fc3 multinode/reputation: add vetted at timestamp
Change-Id: Id35cb6cfdabf4bf2762e4a162cf3157afb0ff170
2021-07-07 18:11:54 +03:00