Commit Graph

2045 Commits

Author SHA1 Message Date
nadimhq
f16bb4d198
Create a New User Personal Account Test (#4141)
* added signup personal user test & added testDefault:true to OpenRegistrationEnabled in service.go

* added copyright

* fixed import ordering

* fixed comment formatting and gofmt-ed with -s

* gofmt-ed with -s and -w

* fixed fragile elements

* fixed one more fragile element

* fixed nesting

* removed unnecessary timeout

* fixed imports
2021-06-22 07:06:40 -04:00
Michał Niewrzał
4e645059be satellite: rename Endpoint2 to Endpoint
Change-Id: I91faaa532d5a24f033b9865a89b91bc6821588be
2021-06-22 09:25:05 +00:00
Jeremy Wharton
8a070e7c25 satellite/overlay: Ignore unnecessary check-ins
This prevents the database from being contacted unnecessarily,
reducing load.

Change-Id: Ib2420f68a20636ec35eb3dd3df8e02bd5341b419
2021-06-22 09:00:41 +00:00
Egon Elbre
ca64e55281 satellite/gc: remove skip first
We used this to reduce initial load on the core to avoid OOM. However,
this is not a problem anymore with garbage collection running
separately.

Change-Id: Ifd62c822a74974bc21a5913199334469a4bc0130
2021-06-21 18:30:38 +00:00
Egon Elbre
5f901c8066 satellite/metabase/metaloop: add as of system interval to stats
Table stats query should use AS OF SYSTEM INTERVAL in the query to optimize
the behavior.

Change-Id: Ie14f5441062d3d264bb3c626081e77d6edc7b724
2021-06-17 23:43:41 +03:00
Egon Elbre
9640cc2c06 satellite/metabase/segmentloop: verify processed count
This adds verification for the processed count and before and after
segment/objects table counts.

This adds new flag:

  metainfo.segment-loop.suspicious-processed-ratio: 0.03

This defaults to 3%, which at 100M segments is 3M segments.

Change-Id: I5ee03e913ddc4e67e94010ced126a2a9ea51f41b
2021-06-17 19:46:55 +03:00
Egon Elbre
341033dda1 satellite/metabase/metaloop: verify processed count
This adds verification for the processed count and before and after
segment/objects table counts.

This adds new flag:

  metainfo.loop.suspicious-processed-ratio: 0.03

This defaults to 3%, which at 100M objects is 3M objects.

Change-Id: Ife5522ecc97bcc5a55667f36868a0f1fc8e4c561
2021-06-17 15:23:53 +03:00
Michał Niewrzał
a93e47514a satellite: remove irreparabledb
This is part of metaloop refactoring. We plan to remove
irreparable at some point but there was not time for it.
Now instead refatoring it for segmentloop its just easier
to drop it.

Later we still need to drop table with migration step.

Change-Id: I270e77f119273d39a1ecdcf5e1c37a5662a29ab4
2021-06-17 07:20:15 +00:00
Egon Elbre
0df4a27bf7 satellite/metabase: add method to get table statistics
In loop we need to start verifying it's correctness. This allows to
gather these stats.

Change-Id: I146fb50e2b3658b6f3c2682cdc1983e6abd73c29
2021-06-16 18:34:55 +03:00
Egon Elbre
9b2607d6ba satellite: remove garbage collection option from core
We don't run it anywhere in this configuration, so it's not worthwhile
to keep it that way.

Change-Id: I88afb8bb3eb3843801b15454408f10d1353596cb
2021-06-15 21:07:02 +03:00
Egon Elbre
08c0025332 satellite: remove contact service from core
It shouldn't be needed in core.

Change-Id: I9150ee2c43d57dcb5382531c80b079306ce443ca
2021-06-15 21:07:02 +03:00
Michał Niewrzał
1a974c327d satellite/metabase: set Segment.ExpiresAt while getting segment
Change-Id: I6504efd19c3516ffcf97ea3923e7f52e9832613b
2021-06-15 13:57:51 +00:00
JT Olio
6949dc0bac satellite/metaloop: missing monitoring on observers
Change-Id: I630fbb0448c8d08b426486b3e49abfbca03332a6
2021-06-15 13:39:13 +00:00
Michał Niewrzał
9a113da361 satellite/metabase: expose expires_at with loops
Loops are using custom structs to provide
segments while iterating/looping so we need
to expose new field there too.

Change-Id: I12c8f4a01afeac171bf638d278253999fa90a8cb
2021-06-11 16:51:37 +00:00
Egon Elbre
f3a52d1da5 satellite/metabase/segmentloop: limit max interval
Ensure that we don't query too far in the history, which slows things
down.

Change-Id: Ia77aa522f7f4c5d43629d51bb9a51a49fab6fa14
2021-06-10 17:06:33 +00:00
Michał Niewrzał
bc79f01aaa satellite/metabase: set expires_at while committing segment
We added expires_at column to segments table and now
we need to populate this column while committing segment.
We still need to migrate existing segments with
separate tool.

Change-Id: Ibac8c63d97201dd98cc2cb9db385f4cb73bc3f7e
2021-06-10 16:32:28 +00:00
Jeff Wendling
d674bc9c52 satellite/audit: include failing segment info in logs
Change-Id: I972fe19a2479f48bccc8a87a282467345a9dc1ec
2021-06-10 13:47:22 +03:00
Jeff Wendling
944bceabcd satellite/audit: fix reservoir sampling bias
Change-Id: Icc522fd86538b8182a1b7d42c1588c32a257acaf
2021-06-10 13:47:22 +03:00
Fadila Khadar
21731ff8d0 satellite/metabase: add expires_at column to segments
Change-Id: If798198d87edb68cda8859ce2277fdfbd6f8380b
2021-06-07 19:11:20 +00:00
Egon Elbre
347f5f87e0 satellite/metabase/metaloop: limit as of system time
Currently we did not limit the "as of system time" for iterating over
objects table. Using just an interval would cause problems with the
tests. That could be overcome skipping that interval for tests
altogether, however, we should probably test those more to ensure that
GC stays working as intended.

This is a safer code, however, maybe not as straigthforward as it could
be.

Change-Id: I374f77783b2af42bb6da846735ceea20a7ce5e60
2021-06-07 13:01:06 +00:00
Fadila Khadar
3de9655b68 satellite/redis: used bandwidth key depends on day
The redis key associated to bandwidth usage depended on the current month.
This change makes it depends also on the day, so that we update bandwidth usage
daily to take into account changes associated to expired but not used allocated bandwidth.

It also add a test to to check that the allocated but not settled bandwidth is not counted after 2 days.

Change-Id: Iee9dbe3517cc3b9825438360b276a07a43dfbc64
2021-06-05 17:14:52 +00:00
Egon Elbre
e6fe9d209e satellite/metabase: capture iterator errors
It was possible for the iterator to silently ignore scanning, nextQuery
and close errors.

Change-Id: I7e44674d9eae53267a3ed649b7657d932743bf73
2021-06-04 22:46:30 +03:00
Egon Elbre
5044337440 satellite/metabase: add monitoring for objects and segments
Currently it's difficult to gather how many objects and segments are
being inserted. Adding separate monitoring counters make this easier.

Change-Id: I986cd82f03e99d2aa6fc76028255ee1090d1b294
2021-06-04 20:01:10 +03:00
Egon Elbre
cbf5939984 satellite/metabase/metaloop: fix objectsIterated metric
Change-Id: Id4f39e7524d1764cb729ab40895af32666c00204
2021-06-03 20:13:36 +03:00
Fadila Khadar
1d405f45ea satellite/orders: add egress_dead to project_bandwidth_daily_rollups
Migration step for adding a 'egress_dead' column to the project_bandwidth_daily_rollups.
It will be used to track bandwidth allocation that won't be consumned
as the corresponding order has already been processed and has a settled
bandwidth amount lower than the order limit (allocated bandwidth).

Change-Id: Ic07592e69292ae2076e69f6038bb0e0fae79b271
2021-06-03 16:05:56 +00:00
JT Olio
9c46490f6f satellite/accounting/projectbwcleanup: more realistic test value
Change-Id: I2c2dd51f6f73e4e5f24268fe9e9fdb6d25e17ba0
2021-06-03 12:25:22 +00:00
Egon Elbre
4469d229f8 satellite/metabase/{meta,segment}loop: avoid passing config
Currently the iterate is being called in only one location so there's no
benefit in passing them as arguments over using the receiver.

Change-Id: I433a5d8b795b1bcc1f1e9320d87b10820cf537f1
2021-06-02 15:55:59 +00:00
Malcolm Bouzi
136af8e630 web, satellite: allow registering business accounts to ask for contact from sales team
Full prefix: web/satellite, satellite/{console, analytics, satellitedb}

- checkbox added to register view - business tab
- user being saved with new column
- add sales contact choice to Segment calls
- ui fix added to employee count dropdown

Change-Id: Ib976872463b88874ea9714db635d58c79cdbe3a1
2021-06-02 13:10:38 +00:00
Egon Elbre
2cf10a7bf4 satellite/metabase/{segment,meta}loop: avoid no observers error
In a rare case it's possible to start the loop iteration without
observers. The most likely case is that the observer is cancelled and
the coalesce timer trigger asynchronously, although being stopped.

Nevertheless, all the observers may also exit during the iteration, in
either case it should not result in an error.

If there's a probem with the observers, then they can report their own
error as they see fit.

Change-Id: Ie423fec41e6295be05536a4b7b0b6623ffebf2fb
2021-06-02 11:57:59 +00:00
Michał Niewrzał
d987990c15 satellite/satellitedb: removing usage of project_bandwidth_rollups table
We are not using this table so make no sense to put data there.
This change removes only code that is using this table. Before next
release we need to drop table with migration step.

Change-Id: I80f400aa778c717e70324bd00da502b7032c9d9f
2021-06-02 05:58:38 +00:00
JT Olio
da9ca0c650 testplanet/satellite: reduce the number of places default values need to be configured
Satellites set their configuration values to default values using
cfgstruct, however, it turns out our tests don't test these values
at all! Instead, they have a completely separate definition system
that is easy to forget about.

As is to be expected, these values have drifted, and it appears
in a few cases test planet is testing unreasonable values that we
won't see in production, or perhaps worse, features enabled in
production were missed and weren't enabled in testplanet.

This change makes it so all values are configured the same,
systematic way, so it's easy to see when test values are different
than dev values or release values, and it's less hard to forget
to enable features in testplanet.

In terms of reviewing, this change should be actually fairly
easy to review, considering private/testplanet/satellite.go keeps
the current config system and the new one and confirms that they
result in identical configurations, so you can be certain that
nothing was missed and the config is all correct.
You can also check the config lock to see what actual config
values changed.

Change-Id: I6715d0794887f577e21742afcf56fd2b9d12170e
2021-06-01 22:14:17 +00:00
prerna-parashar
75bf2ad586
satellite/analytics: Add analytics for "link shared" in objects view (#4130) 2021-06-01 14:44:37 -07:00
Cameron Ayer
53322bb0a7 satellite/{audit,satellitedb}: release nodes from containment in Reverify rather than (Batch)UpdateStats
Until now, whenever audits were recorded we would try to delete
the node from containment just in case it exists. Since we now
want to treat segment repair downloads as audits, this would
erroneously remove nodes from containment, as repair does not go
through a Reverify step. With this changeset, (Batch)UpdateStats
will not remove nodes from containment. The Reverify method will
remove all necessary nodes from containment.

Change-Id: Iabc9496293076dccba32ddfa028e92580b26167f
2021-06-01 21:02:44 +00:00
Michał Niewrzał
e76cbc9bd5 satellite/gc: move GC to segments loop
This change is refactor to move GC from metainfo loop
(objects/segments)  to segments loop.

Change-Id: I21f1ff7cb0b6f98c41aa8930447b8d9bea227975
2021-06-01 20:36:02 +00:00
JT Olio
b07a39bfea satellite: log check in success node id
This is so we can see what's going on if we get a weird node DoS thing again

Change-Id: I5a14c95277562e496fcefb6d368068a6ec1dbc9f
2021-06-01 19:35:24 +00:00
Michał Niewrzał
053e58b683 satellite/metabase: add segmentloop service
We want to move some of current metainfo loop observers to
segment loop. This change adds new service, similar to metainfo
loop but which is iterating only over segments.

Change-Id: I67f7f461781723a4476e2b83377f31736d7c4870
2021-06-01 11:15:07 +00:00
Egon Elbre
10a0216af5 satellite/metainfo: use range for specifying download limit
Previously the object range was not used for calculating order limit.
This meant that even if you were downloading only a small range it would
account bandwidth based on the full segment.

This doesn't fully address the accounting since the lazy segment
downloads do not send their requested range nor requested limit.

Change-Id: Ic811e570c889be87bac4293547d6537a255078da
2021-06-01 09:36:55 +00:00
Fadila Khadar
aa49c8c44d satellite/orders: fix TestProjectUsageBandwidth
Change-Id: I3e9e81ea1ac2b0d7ef0936e4480a6b13c36bfcc5
2021-06-01 10:41:59 +02:00
Egon Elbre
0ef537a685 satellite/metabase/metaloop: limit max as of system time
When using a system time too far in the past, the query gets slower.

Change-Id: I7014d13a16b1ec44391549f829e7a57942f1fea7
2021-05-28 12:07:36 +03:00
Vitalii Shpital
a5dbc544f4 satellite/console: project member deletion bugs fixed
There was a bug when user tried to get project after removing themselves from it.

Also we made user select firstly created project only if they removed themselves from current selected project.

Change-Id: I4b28ebc1ab4a8c14d05ef702e034f2ab39225cc3
2021-05-27 14:54:54 +00:00
Michał Niewrzał
3af91e7a90 satellite/metabase: add iteration over segments
Method IterateLoopSegments can be used to iterate over all segments in metabase without touching objects.

Change-Id: I3cc0e783884b603b47ef3f8233e357aa8a391250
2021-05-26 09:59:33 +00:00
Moby von Briesen
f536ee3aaf satellite/satellitedb: Migrate non-expiring coupons to expire
Because of recent changes to how coupons for the free tier are handled
(see commit 4c0817bcfb), we no longer want all these $10
non-expiring coupons. After coupons are applied during invoice
generation, if a customer does not have any valid (non expired, non
consumed) coupons, a new promotional coupon is applied.

We could just wait for users to consume all $10 of the non-expiring
coupons, and the new promotional coupon would be applied for the
following billing cycle, but this gets tricky, because if in the final
month, the user is billed for $1 of usage, but only $0.5 of the $10
non-expiring coupon is remaining, the user will be charged for the
remaining $0.5. With the new promotional coupon of $1.65, expiring every
month, this would not be an issue.

So long story short, this commit migrates all non-expiring coupons to
expire within 2 billing periods (all existing non-expiring coupons in
prod were created in early April or later). That way, there is still
enough value in the $10 coupon that we don't have to worry about
customers exceeding it, and the coupons will expire. Then we'll
immediately apply the $1.65 coupon for the next month! And then
hopefully this unfortunate situation will come to a pleasant end.

Change-Id: I8a593948d8876c41a71d886b9a95d4e2c802b4f3
2021-05-25 19:25:02 +00:00
Fadila Khadar
63cfc8fbe0 satellite/orders: use project daily bandwidth rollups
Replace GetProjectAllocatedBandwidth by GetProjectBandwidth which calculates
used bandwidth from allocated and settled bandwidth recorded in the
project_bandwidth_daily_rollups table.
For each day in the month, if the allocated bandwidth is expired, it uses the
settled bandwidth for computing used bandwidth.

Change-Id: Ife723c6d5275338f470619631acb25930d39ac3c
2021-05-25 18:28:58 +00:00
Michał Niewrzał
59eabcca24 satellite/orders: populate
project_bandwidth_daily_rollups table

We want to calculate used bandwidth better so we need to calculate it
from allocated and settled bandwidth. To do this we need first populate
this new table.

https://storjlabs.atlassian.net/browse/PG-56

Change-Id: I308b737bf08ee48ce4e46a3605697ab2095f7257
2021-05-25 18:07:22 +00:00
Moby von Briesen
02fc87e98b satellite/payments: Apply Stripe free tier coupon for new customers
Rather than applying our internal satellite implementation of coupons
when new accounts are created, use a configured Stripe coupon instead.
If no configuration is set, no coupon will be applied.

This change also removes logic for adding coupons to customers who pay
with crypto - they will already have the free tier coupon applied
anyway.

We will be phasing out our internal coupon implementation.

Change-Id: Ieb87ddb3412acbc74986aa9d18a4cbd93c29861a
2021-05-25 17:39:44 +00:00
Fadila Khadar
16022105f1 satellite/orders: create project_bandwidth_daily_rollups table
Change-Id: I8f101c0714b5673450d4f2f01e297a9eddc6caf6
2021-05-25 09:05:47 +00:00
Moby von Briesen
0ff3516f54 satellite/mailservice: Fix bug causing issues with test account creation
The context passed into SendEmail gets canceled before links are clicked
in the test environment. This change passes an un-cancelable context
from SendRenderedAsync so that email sending/link clicking can be
completed even if the parent context is canceled.

Change-Id: I88535d315900d7886877f0e14d1d052745402ac7
2021-05-25 06:19:53 +00:00
Egon Elbre
e7cf369cd8 satellite/metabase: use pgxutil.Conn helper
Clean up underlying pgx.Conn extraction with helper.

Change-Id: Ie43f663a6036c439e675df54fccd3605c4d8d69e
2021-05-25 05:56:35 +00:00
Egon Elbre
cdcc67207c satellite/satellitedb: fix nil panic in UpdateCheckIn
Change-Id: If6ae2c3d9b7c269b0a9d652e68854091f668b5ec
2021-05-25 00:30:36 +03:00
Egon Elbre
10372afbe4 ci: fix lint errors
Change-Id: Ib5893440807811f77175ccd347aa3f8ca9cccbdf
2021-05-17 13:37:31 +00:00