Commit Graph

293 Commits

Author SHA1 Message Date
Vitalii Shpital
3de7f8d5af satellite/console, web/satellite: feature flag for new project dashboard
Added a feature flag to switch between old and new project dashboard

Change-Id: Ibb1b04ccbddb43e3f41116e3513f917deba95699
2021-11-17 16:02:47 +02:00
andriikotko
09568b3e2b
satellite/console: change default feature flag (#4274)
* satellite/console: change default feature flag
2021-11-16 11:38:42 +02:00
dlamarmorgan
44b1ca6b97 satellite/{console,satellitedb}: move project limits from config file to DB to keep limits on a per user basis
To allow for changing limits for new users, while leaving existing users limits as they are, we must store the project limits for each user. We currently store the limit for the number of projects a user can create in the user DB table. This change would also store the project bandwidth and storage limits in the same table.

Change-Id: If8d79b39de020b969f3445ef2fcc370e51d706c6
2021-11-11 15:10:00 +00:00
Ivan Fraixedes
4e67ea007c satellite/admin: Serve static UI assets
Change the satellite Admin HTTP server for:

* Embedding the UI assets into the Go binary.
* Serve the UI assets from the embedded file system or from a specific
  directory path through a configuration flag, without requiring
  authentication but keeping the authentication verification for the API
  endpoints.
* Add tests to verify that the UI assets are served without
  authentication.

Change-Id: I9003ac96f1ec585a189b67fc1cb315905403d557
2021-11-10 18:47:46 +02:00
Mya
bf51c286d9
satellite/geoip: update node check-in to associate a country code
Resolves https://github.com/storj/storj/issues/4247

Change-Id: Idfd71bf1795d48ca3c686066bbdb95b9c6594f00
2021-11-10 16:44:41 +01:00
Vitalii Shpital
3c683998f5 satellite/console, web/satellite: feature flag for new objects flow
Added a feature flag to switch between old and new objects flow

Change-Id: Id1802f9fb1230928445f16e4d0a62aecdc91c038
2021-11-08 11:45:22 +00:00
Egon Elbre
d043b9facd satellite/console: load index.html from disk when developing
This eases development by not needing to restart satellite after each
change.

Change-Id: I1063b7f431f675fb21e82dcfc2420310a9806422
2021-11-03 18:19:40 +02:00
Egon Elbre
edb8d656de satellite/metainfo: adjust piecedeletion timeouts
Currently slower storagenodes can slow down deletion queue.
To make piece deletion faster reduce the maximum time spent in
either dialing or piece deletion requests.

With this change:
* dial timeout is 3s
* request timeout is 15s
* fail threshold is set to 10min

Similarly, we'll mark storage node as failed when the timeout occurs.
The timeout usually indicates that the storagenode is overwhelmed.
Garbage collection will ensure that the pieces get deleted eventually.

Change-Id: Iec5de699f5917905f5807140e2c3252088c6399b
2021-10-28 13:37:01 +03:00
Yingrong Zhao
d441c8da15 satellite: use segment count for billing
Change-Id: Iafaf41cb6cde4cb2abdac3f75ce0fead838cb758
2021-10-27 19:21:50 +00:00
dlamarmorgan
01cfbde56b satellite/console: increase free tier project bandwidth and storage from 50 GB to 150 GB and reduce free tier max projects from 3 to 1
To make our free tier limits more clear, we will reduce the number of projects allowed from 3 to 1, and increase the storage and bandwidth limit of the free tier from 50 Gb to 150 GB. The total allotments across all projects for a given user are unchanged, just reduced to a single project.

Change-Id: Ic8dddb135f2b83a3f36e2b9fdcb477e351ec137b
2021-10-25 17:34:19 +00:00
Qweder93
50baefa10e satellite/metabase: limit maximum number of parts and size
Multipart upload limits added. Last part has no size limit.
Max number of parts: 10000, min part size: 5 MiB

Change-Id: Ic2262ce25f989b34d92f662bde720d4c4d0dc93d
2021-10-22 10:16:37 +00:00
Monty Anderson
3989107031 web/satellite: add feature flag for new browser
Change-Id: If3773b9df7f79d4c5e094140448d60ec060cdaeb
2021-10-22 03:49:08 +00:00
Vitalii Shpital
0009447773 satellite/console, web/satellite: feature flag for new navigation structure
Change-Id: I6f1fd43ea8154714187c7547ae93de0efe3482e6
2021-10-20 13:51:25 +00:00
littleskunk
016ac75052
satellite/console: Enable new onboarding workflow (#4222) 2021-10-15 21:07:35 +02:00
Jeremy Wharton
3b751a35c5 satellite/{payments,satellitedb}: Remove custom coupon implementation
Removes database tables and functionality related to our custom
coupon implementation because it has been superseded by the Stripe
coupon and promo code system. Requires implementations of the
payments Invoices interface to return coupon usages along with
invoices.

Change-Id: Iac52d2ff64afca8cc4dbb2d1f20e6ad4b39ddfde
2021-10-11 19:47:00 +00:00
Vitalii Shpital
8b91c55ec4 web/satellite, satellite/console: return old onboarding flow with feature flag
Added new feature flag for switching between different flows.
Got back old onboarding flow.

Change-Id: Idbf23ba8ff36900a47e3b4ebeb34b78351202e29
2021-09-21 11:37:30 +00:00
Egon Elbre
6e660cecdd Jenkinsfile: test cross-compile and bump deps
Change-Id: I47091de2e80bd96fcada616f75e1db07a59fb0c1
2021-09-16 18:59:31 +03:00
crawter
e1a4195a35 satellite/metainfo/endpoint: MaxObjectKey length validation for BeginObject and CommitObject added
Change-Id: I6a766415a84b1e3c44ce65052ffa8ed4511d00dd
2021-09-13 15:07:16 +03:00
Michał Niewrzał
b7555980c4 satellite/metainfo: add zombiedeletion chore
This chore will be deleting zombie objects and it's segments.

Change-Id: Ia5d92dd6c3f52e178d4d43d9603732310761e027
2021-08-24 08:22:06 +00:00
Jeremy Wharton
6a6cc28fc1 satellite/console,private/web: Rate limit coupon code application
Rate limits application of coupon codes by user ID to prevent
brute forcing. Refactors the rate limiter to allow limiting based
on arbitrary criteria and not just by IP.

Change-Id: I99d6749bd5b5e47d7e1aeb0314e363a8e7259dba
2021-08-23 17:00:31 +00:00
Moby von Briesen
7f02e1f469 satellite/console: Remove config flag for MFA
MFA is complete and we are good to enable it in production. This change
removes the flag that disables MFA by default.

Change-Id: I2f985ae501171bdab505d664b43c8cfc248bad8d
2021-08-05 18:04:36 +00:00
Yingrong Zhao
646ce5b8cc satellite/overlay: remove reputation logic from overlay
Change-Id: I3492860e4537c7a8e4e824ec4c9c8d179134a0c0
2021-07-28 15:15:28 -04:00
Yingrong Zhao
6c7bf357cd satellite/{reputation,audit,overlay}: replace overlay with reputation
package in audit

This PR implements reputation store and replace overlay in audit service
to use such store for storing node's audit stats.

In order to keep the changeset smaller, most of the changes in this PR is for copying audit logic in overlay to
reputation package. In a following PR, the duplicating code will be
removed from overlay.

Change-Id: I16c12494a0970f44c422b26cf603c1dc489e5bc1
2021-07-28 13:10:48 -04:00
Moby von Briesen
149f6f2626 satellite/payments: Implement coupon codes
Full path: satellite/{payments,console},web/satellite

* Adds the ability to apply coupon codes from the billing page in the
satellite UI.
* Flag for coupon code UI is split into two flags - one for the billing
page and one for the signup page. This commit implements the first, but
not the second.
* Update the Stripe dependency to v72, which is necessary to
use Stripe's promo code functionality.

Change-Id: I19d9815c48205932bef68d87d5cb0b000498fa70
2021-07-26 17:15:55 +00:00
Michał Niewrzał
b12d29935a satellite/metabase: remove metaloop package
We moved everything to segment loop so we can now
remove metaloop from code.

Change-Id: I9bd8d2349e5638d7cdad50f2f313f9bd89a8165c
2021-07-22 13:00:45 +00:00
Michał Niewrzał
27a714e8b0 satellite/accounting/tally: use objects iterator instead metaloop
Bucket tally calculation will be removed from metaloop and will
use metabase objects iterator directly.

At the moment only bucket tally needs objects so it make no sense
to implement separate objects loop.

Change-Id: Iee60059fc8b9a1bf64d01cafe9659b69b0e27eb1
2021-07-20 15:52:18 +00:00
Vitalii Shpital
8855c0dff7 web/satellite: added MFA feature flag, updated client-side api and Vuex store module
Added feature flag for MFA
Added new client-side api call to enable MFA returning secret
Updated users Vuex module to include new API call

Change-Id: Ia9e10f68c4a7da39b4f7c1073e657c2de98fb0db
2021-07-12 10:12:59 +00:00
Jeremy Wharton
a5f6bb9cc0 satellite/console: Add reCAPTCHA verification step to registration
The user must complete a reCAPTCHA in order to register.
ReCAPTCHA verification failure results in rejection of the
registration attempt.

Change-Id: I34ba7db414d756fd1aaebdc3d19cccbfc7fc1ea3
2021-07-07 21:34:07 +00:00
Moby von Briesen
e36001b7cf satellite/console: Implement paid tier
When a user adds a credit card, switch them to the paid tier and update
their projects with new bandwidth/storage limits. New projects for the
paid tier user will also have the updated limits.

The new limits are:
* storage per project - 50 GB free/25 TB paid
* bandwidth per project - 50 GB free/100 TB paid

Change-Id: I7d6467d077e8bb2bbe4bcf88ab8d75490f83165e
2021-07-06 14:30:12 +00:00
Moby von Briesen
4e95d27033 web,satellite: Remove paywall-related functionality
Because of our free/paid tier plan, we do not need a paywall anymore. We
have not used it in a while, but still have leftover code laying around.

Change-Id: Iaea8c39faf042a2f7a6b837727bb135c8bdf2907
2021-06-29 02:47:48 +02:00
Michał Niewrzał
4267a958d4 satellite/satellitedb: use AS OF SYSTEM TIME for GetProjectBandwidth query
Adding AS OF SYSTEM TIME to query that is calculating project bandiwdth.
As an addition method for setting interval is added as test doesn't
work well with default interval.

Change-Id: Id1e15be4f6afff13b9dc2b7f595e2edb6de28db9
2021-06-28 15:19:47 +00:00
Jeremy Wharton
8a070e7c25 satellite/overlay: Ignore unnecessary check-ins
This prevents the database from being contacted unnecessarily,
reducing load.

Change-Id: Ib2420f68a20636ec35eb3dd3df8e02bd5341b419
2021-06-22 09:00:41 +00:00
Egon Elbre
ca64e55281 satellite/gc: remove skip first
We used this to reduce initial load on the core to avoid OOM. However,
this is not a problem anymore with garbage collection running
separately.

Change-Id: Ifd62c822a74974bc21a5913199334469a4bc0130
2021-06-21 18:30:38 +00:00
Egon Elbre
9640cc2c06 satellite/metabase/segmentloop: verify processed count
This adds verification for the processed count and before and after
segment/objects table counts.

This adds new flag:

  metainfo.segment-loop.suspicious-processed-ratio: 0.03

This defaults to 3%, which at 100M segments is 3M segments.

Change-Id: I5ee03e913ddc4e67e94010ced126a2a9ea51f41b
2021-06-17 19:46:55 +03:00
Egon Elbre
341033dda1 satellite/metabase/metaloop: verify processed count
This adds verification for the processed count and before and after
segment/objects table counts.

This adds new flag:

  metainfo.loop.suspicious-processed-ratio: 0.03

This defaults to 3%, which at 100M objects is 3M objects.

Change-Id: Ife5522ecc97bcc5a55667f36868a0f1fc8e4c561
2021-06-17 15:23:53 +03:00
Michał Niewrzał
a93e47514a satellite: remove irreparabledb
This is part of metaloop refactoring. We plan to remove
irreparable at some point but there was not time for it.
Now instead refatoring it for segmentloop its just easier
to drop it.

Later we still need to drop table with migration step.

Change-Id: I270e77f119273d39a1ecdcf5e1c37a5662a29ab4
2021-06-17 07:20:15 +00:00
Egon Elbre
9b2607d6ba satellite: remove garbage collection option from core
We don't run it anywhere in this configuration, so it's not worthwhile
to keep it that way.

Change-Id: I88afb8bb3eb3843801b15454408f10d1353596cb
2021-06-15 21:07:02 +03:00
Egon Elbre
f3a52d1da5 satellite/metabase/segmentloop: limit max interval
Ensure that we don't query too far in the history, which slows things
down.

Change-Id: Ia77aa522f7f4c5d43629d51bb9a51a49fab6fa14
2021-06-10 17:06:33 +00:00
Egon Elbre
347f5f87e0 satellite/metabase/metaloop: limit as of system time
Currently we did not limit the "as of system time" for iterating over
objects table. Using just an interval would cause problems with the
tests. That could be overcome skipping that interval for tests
altogether, however, we should probably test those more to ensure that
GC stays working as intended.

This is a safer code, however, maybe not as straigthforward as it could
be.

Change-Id: I374f77783b2af42bb6da846735ceea20a7ce5e60
2021-06-07 13:01:06 +00:00
JT Olio
da9ca0c650 testplanet/satellite: reduce the number of places default values need to be configured
Satellites set their configuration values to default values using
cfgstruct, however, it turns out our tests don't test these values
at all! Instead, they have a completely separate definition system
that is easy to forget about.

As is to be expected, these values have drifted, and it appears
in a few cases test planet is testing unreasonable values that we
won't see in production, or perhaps worse, features enabled in
production were missed and weren't enabled in testplanet.

This change makes it so all values are configured the same,
systematic way, so it's easy to see when test values are different
than dev values or release values, and it's less hard to forget
to enable features in testplanet.

In terms of reviewing, this change should be actually fairly
easy to review, considering private/testplanet/satellite.go keeps
the current config system and the new one and confirms that they
result in identical configurations, so you can be certain that
nothing was missed and the config is all correct.
You can also check the config lock to see what actual config
values changed.

Change-Id: I6715d0794887f577e21742afcf56fd2b9d12170e
2021-06-01 22:14:17 +00:00
Michał Niewrzał
053e58b683 satellite/metabase: add segmentloop service
We want to move some of current metainfo loop observers to
segment loop. This change adds new service, similar to metainfo
loop but which is iterating only over segments.

Change-Id: I67f7f461781723a4476e2b83377f31736d7c4870
2021-06-01 11:15:07 +00:00
Egon Elbre
0ef537a685 satellite/metabase/metaloop: limit max as of system time
When using a system time too far in the past, the query gets slower.

Change-Id: I7014d13a16b1ec44391549f829e7a57942f1fea7
2021-05-28 12:07:36 +03:00
Moby von Briesen
02fc87e98b satellite/payments: Apply Stripe free tier coupon for new customers
Rather than applying our internal satellite implementation of coupons
when new accounts are created, use a configured Stripe coupon instead.
If no configuration is set, no coupon will be applied.

This change also removes logic for adding coupons to customers who pay
with crypto - they will already have the free tier coupon applied
anyway.

We will be phasing out our internal coupon implementation.

Change-Id: Ieb87ddb3412acbc74986aa9d18a4cbd93c29861a
2021-05-25 17:39:44 +00:00
JT Olio
1852773e3e satellite/contact: rate limit node checkins
Change-Id: Ied386a2350aa073de46443e5259b56d49ec61dbf
2021-05-17 08:15:04 +00:00
Ivan Fraixedes
7fb86617fc satellite/satellitedb: Use CRDB AS OF SYSTEM & batch for GE
Use the 'AS OF SYSTEM TIME' Cockroach DB clause for the Graceful Exit
(a.k.a GE) queries that count the delete the GE queue items of nodes
which have already exited the network.

Split the subquery used for deleting all the transfer queue items of
nodes which has exited when CRDB is used and batch the queries because
CRDB struggles when executing in a single query unlike Postgres.

The new test which has been added to this commit to verify the CRDB
batch logic for deleting all the transfer queue items of the exited
nodes has raised that the Enqueue method has to run in baches when CRDB
is used otherwise CRDB has return the error "driver: bad connection"
when a big a amount of items are passed to be enqueued. This error
didn't happen with the current test implementation it was with an
initial one that it was creating a big amount of exited nodes and
transfer queue items for those nodes.

Change-Id: I6a099cdbc515a240596bc93141fea3182c2e50a9
2021-05-07 13:09:19 -04:00
littleskunk
396cd5a683
satellite/payments: reduce object fee to 0 (#4104) 2021-04-29 22:31:32 +02:00
Moby von Briesen
4c0817bcfb satellite/payments: Populate new coupons during invoice generation
The previously configured never-expiring coupon does not refill every
month. Eventually, even though it never expires, it will run out. This
commit makes several small changes to address this issue for the free
tier:
* Change the config for the promotional coupon to be $1.65 for 1 month
(the change from $10 to $1.65 is due to our recent pricing changes)
* Update PopulatePromotionalCoupons (PPC for brevity) to add promotional
coupons to users with expired and consumed coupons (all users with a
project and no active coupons should get a new coupon when PPC is called)
* Call PPC at the end of the `create-invoice-coupons` stage of invoice
generation - after current coupons are processed and expired/exhausted.
* Remove legacy admin functionality for PPC from satellite/console - we
do not currently use it, but if we did, it should be in satellite/admin
instead.

Change-Id: I77727b97bef972df32ebb23cdc05055827076e2a
2021-04-28 18:07:05 +00:00
Moby von Briesen
b317f28fdb web/satellite: Update registration files
Allows us to remove the following files from satellite branding
repo, with an up-to-date single source of truth now in storj/storj:
* web/satellite/src/common/registrationSuccess.html
* web/satellite/src/common/registrationSuccess.scss
* web/satellite/src/views/register/registerArea.html
* web/satellite/src/views/register/registerArea.scss

The registrationSuccess files have been removed from all satellites in
the branding repository. The registerArea files have been removed only
from production satellites in the branding repository.

Importantly, this change enables the "resend email" functionality on
production satellites - previously, this functionality was available in
storj/storj, but not our branding repository.

Removes the config for VerificationPageURL, which redirected users away
from the satellite app to storj.io after creating an account. In order
for the email resend button to work, we cannot leave the app.

Adds a new config value for partner satellites, which replaces the
partner satellite names config. The new config includes name and
address. It is validated on setup/run to ensure it can be parsed.

Change-Id: I67db0702d9b9641f1a37b599f2929d56f3c33aca
2021-04-28 16:16:16 +00:00
Michał Niewrzał
02460fcc4c satellite/metainfo/expireddeletion: change chore interval to 24h
We want run deletion query more often to avoid long running DB operations.

Change-Id: I72c63bb8ec05cc3ab2e7ab63479ba6cdf830386a
2021-04-27 12:30:23 +00:00
Egon Elbre
dd5eb2616b satellite/metrics: join for monitoring
We don't want metrics to trigger the metaloop on it's own.

Change-Id: I648272a0e3043173b0a377697db1f172712bdda5
2021-04-23 12:21:48 +03:00