Commit Graph

444 Commits

Author SHA1 Message Date
TungHoang
f9b630b0f4
Fix monthly earning estimation (#4282) 2021-11-17 18:26:21 -05:00
Mya
814e3126fa satellite/buckets: add new buckets service
The main motivation is to wrap the bucket DB and metainfo DB, so we
could check if a bucket is empty before applying geofencing config.

Change-Id: I8bac21555e01d51a663fb557bc1acfc8106bc2e1
2021-11-16 12:36:17 +02:00
Egon Elbre
dfd2977a01 testsuite/ui: fixes to setup
* Remove "enable-logging", because it ends up spawning consoles on
  Windows.
* Remove "disable-gpu", if we have a GPU, then let's use it.
* Create custom client, so we can attach logging to CDP.
* Ignore potential context.Canceled.
* Fix onboarding wizard test for new objects browser.
* Return an error on a context cancellation.
* Wire all loggers to planet.

Change-Id: I67eb138ba31252f55ac5b383679d033bcf71f1b2
2021-11-15 15:56:35 +02:00
Qweder93
65d139fb96 private/testplanet: create metabaseDB after reconfigure apply
Moved CreateMetabaseDB after applying reconfigure changes to affect them during testing

Change-Id: I51034a6671728645dcc52a0d8c808b9dfc2fba7c
2021-10-25 15:55:45 +00:00
Qweder93
50baefa10e satellite/metabase: limit maximum number of parts and size
Multipart upload limits added. Last part has no size limit.
Max number of parts: 10000, min part size: 5 MiB

Change-Id: Ic2262ce25f989b34d92f662bde720d4c4d0dc93d
2021-10-22 10:16:37 +00:00
Egon Elbre
bdadc4b460 private/testplanet: add stack traces related to the planet
It's useful to see only the goroutines that are related to the current
testplanet.

Change-Id: Id77c62dec42c09d44c523b019f7bdc940905fcf1
2021-10-20 23:12:07 +03:00
Clement Sam
29599dd7cd testsuite/ui/multinode: add first node test
Change-Id: Ibeaee92e6d1a41e408659a5045f52e1908f73089
2021-10-18 16:07:01 +00:00
Moby von Briesen
5b729779a2 satellite/console: Automatically log a user in after verifying email
When an email is verified, insert an auth cookie so that when the user
is redirected after verifying their email, they are immediately taken to
the onboarding flow.

Change-Id: I557d8a2805b24dd8039ada255522bc1b56cc8b53
2021-10-13 13:08:27 +00:00
Clement Sam
b8dd35ceaf private/testplanet: extend testplanet with multinode instance
Testplanet does not have a multinode instance, hence,
makes it difficult to run ui tests for the multinode dashboard.

Instead of using storj-sim for multinode ui tests,
it's better to use the same approach (testplanet) fir all UI tests.

This changes adds a multinode instance to testplanet.

Change-Id: I58aa8c4597e789275f9e7ea7059703c742903492
2021-10-13 10:19:45 +00:00
Egon Elbre
4020e9e2ce private/testmonkit: move monkit test helper
Some tests do not use testplanet and hence testmonkit should be
reusable.

Change-Id: If6b7194a199d9ffba5eb1a91f38d3a792f883336
2021-10-12 14:00:01 +03:00
Jeremy Wharton
3b751a35c5 satellite/{payments,satellitedb}: Remove custom coupon implementation
Removes database tables and functionality related to our custom
coupon implementation because it has been superseded by the Stripe
coupon and promo code system. Requires implementations of the
payments Invoices interface to return coupon usages along with
invoices.

Change-Id: Iac52d2ff64afca8cc4dbb2d1f20e6ad4b39ddfde
2021-10-11 19:47:00 +00:00
Egon Elbre
7ebd35337b private/testplanet: document env variables
Change-Id: I122d56b90af3df1f1b17716a31ddca7c115fca7c
2021-10-01 07:18:50 +00:00
Egon Elbre
6511bb91fb private/testplanet: support writing monitoring spans
This allows to set `STORJ_TEST_MONKIT` to either
`svg` or `json` to write individual testplanet test
traces to disk.

It also allows to specify an absolute directory:

  STORJ_TEST_MONKIT=svg,dir:/abs/dir/path

It requires an absolute path, because from the context of
tests, there's no easy way to find the folder where tests
were called.

Change-Id: I6fe008a4d4237d221cf5a5bede798b46399ee197
2021-09-30 09:03:35 +03:00
Vitalii Shpital
1def7b0ec3 web/satellite, testsuite/ui/satellite: added tests for invalid sign up credentials and satellites dropdown
Added tests for signup with invalid email or password.
Added test for signup screen's satellites dropdown.

Change-Id: I76d975039543e315b3e9c9416e3ec1f2a3331a6a
2021-09-29 15:20:24 +03:00
Yaroslav Vorobiov
469ae72c19 satellite/repair: update audit records during repair
Change-Id: I788b2096968f043601aba6502a2e4e784f1f02a0
2021-09-24 00:48:13 +00:00
crawter
e1a4195a35 satellite/metainfo/endpoint: MaxObjectKey length validation for BeginObject and CommitObject added
Change-Id: I6a766415a84b1e3c44ce65052ffa8ed4511d00dd
2021-09-13 15:07:16 +03:00
Egon Elbre
1aec831d98 satellite/audit,storage: increase sleep delay in TestMaxVerifyCount
Currently TextMaxVerifyCount flakes in some tests, try increasing the
sleep time to ensure that things are slow enough to trigger the error
condition.

Also pass ctx to all the funcs so we can handle sleep better.

Change-Id: I605b6ea8b14a0a66d81a605ce3251f57a1669c00
2021-09-10 15:30:37 +00:00
Michał Niewrzał
495e530933 satellite/metainfo: drop metainfo.Service
Drop a service that in fact don't make sense and
its just a wrapper for direct DB methods.

Change-Id: I1cb76f3ecc2d8765964d919c88541179957645c1
2021-09-09 17:30:10 +02:00
Michał Niewrzał
c258f4bbac private/testplanet: move Metabase outside Metainfo for satellite
At some point we moved metabase package outside Metainfo
but we didn't do that for satellite structure. This change
refactors only tests.
When uplink will be adjusted we can remove old entries in
Metainfo struct.

Change-Id: I2b66ed29f539b0ec0f490cad42c72840e0351bcb
2021-09-09 07:15:51 +00:00
Michał Niewrzał
b7555980c4 satellite/metainfo: add zombiedeletion chore
This chore will be deleting zombie objects and it's segments.

Change-Id: Ia5d92dd6c3f52e178d4d43d9603732310761e027
2021-08-24 08:22:06 +00:00
Jeremy Wharton
6a6cc28fc1 satellite/console,private/web: Rate limit coupon code application
Rate limits application of coupon codes by user ID to prevent
brute forcing. Refactors the rate limiter to allow limiting based
on arbitrary criteria and not just by IP.

Change-Id: I99d6749bd5b5e47d7e1aeb0314e363a8e7259dba
2021-08-23 17:00:31 +00:00
Artur M. Wolff
2cd68bf4fb private/lrucache: import from common
Change-Id: Ia1f43d0440fef21122b071b05da59b4cf2689d6c
2021-08-16 10:04:32 +00:00
Yingrong Zhao
077ec96d94 private/server: use quic implementation from storj.io/common
Change-Id: I820cf6444a3ddccee0d7c647dc84c80b2752068c
2021-08-10 13:32:21 +00:00
Yingrong Zhao
ae02f6deda satellite/reputation: return default reputation stats when node is not
found

Change-Id: I587d0ab36ffa0efaf345a6a6e221ae5d2068e1c5
2021-08-04 19:34:54 +00:00
Yingrong Zhao
f8914ccce0 satellite/{repair, overlay}: use reputation store in repair
Change-Id: I48db9e68f48239d48621ccc77d33618ecb83ce1a
2021-07-28 13:22:05 -04:00
Yingrong Zhao
58238d850c satellite/{audit, accounting}: use reputation store in tests
Change-Id: I86a8ccf5dcee8d108196a9f67a476fe0ccbd8257
2021-07-28 13:21:55 -04:00
Yingrong Zhao
55a77d04bc satellite/satellitedb,private: add initial value on testplanet startup
Currently, reputation table is only populated when a node has been
audited. This is ok in production, however a lot of our tests doesn't
upload any data or trigger audits.
This PR adds an initialization step in testplanet to populate reputation
table with zero value for nodes reputation.

Change-Id: I11b381236669db346dc68a48a6d4a27334a0a8b8
2021-07-28 13:20:32 -04:00
Yingrong Zhao
6c7bf357cd satellite/{reputation,audit,overlay}: replace overlay with reputation
package in audit

This PR implements reputation store and replace overlay in audit service
to use such store for storing node's audit stats.

In order to keep the changeset smaller, most of the changes in this PR is for copying audit logic in overlay to
reputation package. In a following PR, the duplicating code will be
removed from overlay.

Change-Id: I16c12494a0970f44c422b26cf603c1dc489e5bc1
2021-07-28 13:10:48 -04:00
Jeremy Wharton
dae6ed7d03 satellite/console: Implement MFA backend
Added MFA passcode and recovery code field for token requests.
Added endpoints for MFA-related activity: enabling MFA,
disabling MFA, generating a new MFA secret key, and
generating new MFA recovery codes.

Change-Id: Ia1443f05d3a2fecaa7f170f56d73c7a4e9b69ad5
2021-07-26 16:37:05 +00:00
Michał Niewrzał
b12d29935a satellite/metabase: remove metaloop package
We moved everything to segment loop so we can now
remove metaloop from code.

Change-Id: I9bd8d2349e5638d7cdad50f2f313f9bd89a8165c
2021-07-22 13:00:45 +00:00
Michał Niewrzał
cbbbfca439 satellite/accounting: move storage node tally to separate service
Current tally is calculating storage both for buckets and
storage nodes. This change is moving nodes storage
calculation to separate service that will be using
segment loop.

Change-Id: I9e68bfa0bc751c82ff738c71ca58d311f257bd8d
2021-07-08 14:02:33 +00:00
Jeremy Wharton
a5f6bb9cc0 satellite/console: Add reCAPTCHA verification step to registration
The user must complete a reCAPTCHA in order to register.
ReCAPTCHA verification failure results in rejection of the
registration attempt.

Change-Id: I34ba7db414d756fd1aaebdc3d19cccbfc7fc1ea3
2021-07-07 21:34:07 +00:00
Yaroslav Vorobiov
cbb4cd3fc3 multinode/reputation: add vetted at timestamp
Change-Id: Id35cb6cfdabf4bf2762e4a162cf3157afb0ff170
2021-07-07 18:11:54 +03:00
Yaroslav Vorobiov
818f6c6ea6 multinode/console: add summary to storage usage API
Change-Id: Ia8a1e598d667f25461f73f1626da22113cb7caeb
2021-07-07 15:00:05 +03:00
Yaroslav Vorobiov
68627e7d80 multinode/console: add reputation satellite api
Change-Id: I7cef6c1c271607f7485f604d5b61587558a31878
2021-07-05 15:32:22 +00:00
Egon Elbre
4031336cbd private/lifecycle: monitor unexpected shutdowns
Change-Id: I9af3a95a1b60c1572cd57bb5e9539d60aa0337bb
2021-06-25 19:26:23 +03:00
Qweder93
b9d7874dc2 multinode: old drpc api returned
drpc api with old names added to prevent breaking
backwards compatibility.

Change-Id: I1b8a41f2c1e0bd11ac83c86f0b1bfbfc1f07378f
2021-06-22 23:02:47 +00:00
Egon Elbre
59e3b586e7 satellite/{gracefulexit,overlay}: enable as of system time queries
Change-Id: I2af5eb0e8a51fca7893ce07b78b5633be71dfef8
2021-06-22 11:50:50 +00:00
Michał Niewrzał
4e645059be satellite: rename Endpoint2 to Endpoint
Change-Id: I91faaa532d5a24f033b9865a89b91bc6821588be
2021-06-22 09:25:05 +00:00
Michał Niewrzał
a93e47514a satellite: remove irreparabledb
This is part of metaloop refactoring. We plan to remove
irreparable at some point but there was not time for it.
Now instead refatoring it for segmentloop its just easier
to drop it.

Later we still need to drop table with migration step.

Change-Id: I270e77f119273d39a1ecdcf5e1c37a5662a29ab4
2021-06-17 07:20:15 +00:00
Egon Elbre
f1a9b45599 mod: update quic for Go 1.17rc1
Change-Id: I4218b695089189d5efac1101559a83079fb812cd
2021-06-14 10:14:36 +03:00
Qweder93
5d70b6abef multinode/bandwidth: added monthly bandwidth summaries
montly bandwidth summaries for single/all nodes, single/all satellites added.

Change-Id: Ic384886c10622df74a4bd0645e2d7f2a85477644
2021-06-11 16:31:59 +00:00
Yaroslav Vorobiov
157d3d980e multinode/console: storage usage and total storage usage
Change-Id: I4970275daf4a4a9c5d02aea6a205891869dd4eff
2021-06-10 16:01:41 +00:00
andriikotko
072dd24696
integration/ui: add UI integration tests
Introduce a new `integration/ui` package for integration tests.

`integration/ui/uitest` contains a helper package for running browser tests.
2021-06-09 14:39:38 +03:00
Egon Elbre
347f5f87e0 satellite/metabase/metaloop: limit as of system time
Currently we did not limit the "as of system time" for iterating over
objects table. Using just an interval would cause problems with the
tests. That could be overcome skipping that interval for tests
altogether, however, we should probably test those more to ensure that
GC stays working as intended.

This is a safer code, however, maybe not as straigthforward as it could
be.

Change-Id: I374f77783b2af42bb6da846735ceea20a7ce5e60
2021-06-07 13:01:06 +00:00
JT Olio
9e2d837473 testplanet: remove brittle config
Change-Id: I032c6a53bf32d1fa3a3e4cd8f46bc38d2e38d73c
2021-06-02 07:14:55 +00:00
JT Olio
da9ca0c650 testplanet/satellite: reduce the number of places default values need to be configured
Satellites set their configuration values to default values using
cfgstruct, however, it turns out our tests don't test these values
at all! Instead, they have a completely separate definition system
that is easy to forget about.

As is to be expected, these values have drifted, and it appears
in a few cases test planet is testing unreasonable values that we
won't see in production, or perhaps worse, features enabled in
production were missed and weren't enabled in testplanet.

This change makes it so all values are configured the same,
systematic way, so it's easy to see when test values are different
than dev values or release values, and it's less hard to forget
to enable features in testplanet.

In terms of reviewing, this change should be actually fairly
easy to review, considering private/testplanet/satellite.go keeps
the current config system and the new one and confirms that they
result in identical configurations, so you can be certain that
nothing was missed and the config is all correct.
You can also check the config lock to see what actual config
values changed.

Change-Id: I6715d0794887f577e21742afcf56fd2b9d12170e
2021-06-01 22:14:17 +00:00
Qweder93
d8c11a79b9 multinode/payouts: paystub extended with disposed
added Disposed to paystub struct to count NetTotal value.

Change-Id: Iaae1f98a69b82166ba5594e8589c7d2a540bd07f
2021-06-01 16:28:20 +00:00
Michał Niewrzał
053e58b683 satellite/metabase: add segmentloop service
We want to move some of current metainfo loop observers to
segment loop. This change adds new service, similar to metainfo
loop but which is iterating only over segments.

Change-Id: I67f7f461781723a4476e2b83377f31736d7c4870
2021-06-01 11:15:07 +00:00
crawter
6d9b91d435 private/multinodepb: drpc operators controller added
Change-Id: Ie9bad9d5dba3e508d4cb8165aaca88d98ef1304d
2021-06-01 11:00:31 +00:00