Commit Graph

5512 Commits

Author SHA1 Message Date
JT Olio
da9ca0c650 testplanet/satellite: reduce the number of places default values need to be configured
Satellites set their configuration values to default values using
cfgstruct, however, it turns out our tests don't test these values
at all! Instead, they have a completely separate definition system
that is easy to forget about.

As is to be expected, these values have drifted, and it appears
in a few cases test planet is testing unreasonable values that we
won't see in production, or perhaps worse, features enabled in
production were missed and weren't enabled in testplanet.

This change makes it so all values are configured the same,
systematic way, so it's easy to see when test values are different
than dev values or release values, and it's less hard to forget
to enable features in testplanet.

In terms of reviewing, this change should be actually fairly
easy to review, considering private/testplanet/satellite.go keeps
the current config system and the new one and confirms that they
result in identical configurations, so you can be certain that
nothing was missed and the config is all correct.
You can also check the config lock to see what actual config
values changed.

Change-Id: I6715d0794887f577e21742afcf56fd2b9d12170e
2021-06-01 22:14:17 +00:00
prerna-parashar
75bf2ad586
satellite/analytics: Add analytics for "link shared" in objects view (#4130) 2021-06-01 14:44:37 -07:00
Cameron Ayer
53322bb0a7 satellite/{audit,satellitedb}: release nodes from containment in Reverify rather than (Batch)UpdateStats
Until now, whenever audits were recorded we would try to delete
the node from containment just in case it exists. Since we now
want to treat segment repair downloads as audits, this would
erroneously remove nodes from containment, as repair does not go
through a Reverify step. With this changeset, (Batch)UpdateStats
will not remove nodes from containment. The Reverify method will
remove all necessary nodes from containment.

Change-Id: Iabc9496293076dccba32ddfa028e92580b26167f
2021-06-01 21:02:44 +00:00
Michał Niewrzał
e76cbc9bd5 satellite/gc: move GC to segments loop
This change is refactor to move GC from metainfo loop
(objects/segments)  to segments loop.

Change-Id: I21f1ff7cb0b6f98c41aa8930447b8d9bea227975
2021-06-01 20:36:02 +00:00
JT Olio
b07a39bfea satellite: log check in success node id
This is so we can see what's going on if we get a weird node DoS thing again

Change-Id: I5a14c95277562e496fcefb6d368068a6ec1dbc9f
2021-06-01 19:35:24 +00:00
paul cannon
8675cd178c cmd/uplink: add flags to 'put' subcommand
This enables use of the --progress, --expires, and --metadata flags with
'uplink put'. These flags work similarly to their counterparts in the
'uplink cp' command.

Small difference from cp: "progress" is on by default in 'uplink cp',
but (for backwards compatibility) off by default in 'uplink put'.

Why: Requested by @Toyoo on the forum:
https://forum.storj.io/t/explicit-data-expiration-questions/13854/2

..and it's an easy addition to make.

Change-Id: Id20aedd3a280db43e4883338f92f6beec7a400de
2021-06-01 19:16:22 +00:00
paul cannon
fbd11ba4e9 cmd/uplink: add expanded format for 'uplink ls'
This expanded format shows expiration times for objects and how much
custom metadata each object has.

This commit also organizes output formatting code together in one
section for simpler adjustments in the future.

Change-Id: Ica041c8a1de6ee73c104a0554c5c259e447536c4
2021-06-01 18:50:27 +00:00
Qweder93
d8c11a79b9 multinode/payouts: paystub extended with disposed
added Disposed to paystub struct to count NetTotal value.

Change-Id: Iaae1f98a69b82166ba5594e8589c7d2a540bd07f
2021-06-01 16:28:20 +00:00
crawter
4af0037a67 multinode/console: operators controller added
Change-Id: I170371baec3c6996bd2af3c332620bd6fee3ed63
2021-06-01 19:01:14 +03:00
crawter
0b3ee4bda7 multinode/operators: domain entities and service created
Change-Id: I146229df0a279b95ce4d3ffec658509ccd6ff59a
2021-06-01 16:51:48 +03:00
Michał Niewrzał
053e58b683 satellite/metabase: add segmentloop service
We want to move some of current metainfo loop observers to
segment loop. This change adds new service, similar to metainfo
loop but which is iterating only over segments.

Change-Id: I67f7f461781723a4476e2b83377f31736d7c4870
2021-06-01 11:15:07 +00:00
crawter
6d9b91d435 private/multinodepb: drpc operators controller added
Change-Id: Ie9bad9d5dba3e508d4cb8165aaca88d98ef1304d
2021-06-01 11:00:31 +00:00
crawter
4c704ea9a4 mnd/db/nodes: paginated list added
Change-Id: Ibb049c0ad9b09dd89852234b601d7577735c4d34
2021-06-01 13:19:48 +03:00
Egon Elbre
10a0216af5 satellite/metainfo: use range for specifying download limit
Previously the object range was not used for calculating order limit.
This meant that even if you were downloading only a small range it would
account bandwidth based on the full segment.

This doesn't fully address the accounting since the lazy segment
downloads do not send their requested range nor requested limit.

Change-Id: Ic811e570c889be87bac4293547d6537a255078da
2021-06-01 09:36:55 +00:00
Fadila Khadar
aa49c8c44d satellite/orders: fix TestProjectUsageBandwidth
Change-Id: I3e9e81ea1ac2b0d7ef0936e4480a6b13c36bfcc5
2021-06-01 10:41:59 +02:00
Qweder93
91b7e24d55 multinode/payouts: naming refactoring
Renamed methods, error messages.

Change-Id: I7d7b6b092c05bbc5bf1322855efc5ccb9b312671
2021-05-31 19:47:12 +00:00
NickolaiYurchenko
20b98d31b8 web/multinode: expectations block added
added displaying of undistributed balance and current month expectations

contains changes from https://review.dev.storj.io/c/storj/storj/+/4867 to be workable

Change-Id: I9cb00d2db5b819a71ceeddf91d6863a6b4fa9034
2021-05-31 14:33:03 +00:00
Yaroslav Vorobiov
64e4225721 multinode/console: server reorder payouts API routes
Change-Id: Ia42bb3faae8a8782d841e5cba186dc73478581ac
2021-05-31 16:32:03 +03:00
Yaroslav Vorobiov
23f9beb635 multinode/console: held amount summary
Change-Id: Ia800748343e363d930ce0a0b9ab286b5abdc96af
2021-05-28 20:05:16 +03:00
Egon Elbre
0ef537a685 satellite/metabase/metaloop: limit max as of system time
When using a system time too far in the past, the query gets slower.

Change-Id: I7014d13a16b1ec44391549f829e7a57942f1fea7
2021-05-28 12:07:36 +03:00
Vitalii Shpital
a5dbc544f4 satellite/console: project member deletion bugs fixed
There was a bug when user tried to get project after removing themselves from it.

Also we made user select firstly created project only if they removed themselves from current selected project.

Change-Id: I4b28ebc1ab4a8c14d05ef702e034f2ab39225cc3
2021-05-27 14:54:54 +00:00
Vitalii Shpital
32dd4315af web/satellite: improve AG's selection checkmark
replaced png with svg of AG selection checkmark to improve it's visibility

Change-Id: I868d8c831a33d4cf47dcd40aaa49e1c31fe9cf0b
2021-05-27 16:09:37 +03:00
Michał Niewrzał
3af91e7a90 satellite/metabase: add iteration over segments
Method IterateLoopSegments can be used to iterate over all segments in metabase without touching objects.

Change-Id: I3cc0e783884b603b47ef3f8233e357aa8a391250
2021-05-26 09:59:33 +00:00
Malcolm Bouzi
37bc46c6f6 web/satellite: Fix for Payment Method area buttons on billing page not being aligned
Aligns 'Add STORJ' and 'Add Card' buttons

Change-Id: I6a9a92f59d7e2e903d0e4d683cb09cd8de6da987
2021-05-25 22:01:24 +00:00
Moby von Briesen
f536ee3aaf satellite/satellitedb: Migrate non-expiring coupons to expire
Because of recent changes to how coupons for the free tier are handled
(see commit 4c0817bcfb), we no longer want all these $10
non-expiring coupons. After coupons are applied during invoice
generation, if a customer does not have any valid (non expired, non
consumed) coupons, a new promotional coupon is applied.

We could just wait for users to consume all $10 of the non-expiring
coupons, and the new promotional coupon would be applied for the
following billing cycle, but this gets tricky, because if in the final
month, the user is billed for $1 of usage, but only $0.5 of the $10
non-expiring coupon is remaining, the user will be charged for the
remaining $0.5. With the new promotional coupon of $1.65, expiring every
month, this would not be an issue.

So long story short, this commit migrates all non-expiring coupons to
expire within 2 billing periods (all existing non-expiring coupons in
prod were created in early April or later). That way, there is still
enough value in the $10 coupon that we don't have to worry about
customers exceeding it, and the coupons will expire. Then we'll
immediately apply the $1.65 coupon for the next month! And then
hopefully this unfortunate situation will come to a pleasant end.

Change-Id: I8a593948d8876c41a71d886b9a95d4e2c802b4f3
2021-05-25 19:25:02 +00:00
Fadila Khadar
63cfc8fbe0 satellite/orders: use project daily bandwidth rollups
Replace GetProjectAllocatedBandwidth by GetProjectBandwidth which calculates
used bandwidth from allocated and settled bandwidth recorded in the
project_bandwidth_daily_rollups table.
For each day in the month, if the allocated bandwidth is expired, it uses the
settled bandwidth for computing used bandwidth.

Change-Id: Ife723c6d5275338f470619631acb25930d39ac3c
2021-05-25 18:28:58 +00:00
Michał Niewrzał
59eabcca24 satellite/orders: populate
project_bandwidth_daily_rollups table

We want to calculate used bandwidth better so we need to calculate it
from allocated and settled bandwidth. To do this we need first populate
this new table.

https://storjlabs.atlassian.net/browse/PG-56

Change-Id: I308b737bf08ee48ce4e46a3605697ab2095f7257
2021-05-25 18:07:22 +00:00
Moby von Briesen
02fc87e98b satellite/payments: Apply Stripe free tier coupon for new customers
Rather than applying our internal satellite implementation of coupons
when new accounts are created, use a configured Stripe coupon instead.
If no configuration is set, no coupon will be applied.

This change also removes logic for adding coupons to customers who pay
with crypto - they will already have the free tier coupon applied
anyway.

We will be phasing out our internal coupon implementation.

Change-Id: Ieb87ddb3412acbc74986aa9d18a4cbd93c29861a
2021-05-25 17:39:44 +00:00
Egon Elbre
92226d8ddb storagenode/storagenodedb: fix fillInBlobAccess
fillInBlobAccess was using a non-pointer receiver so the receiver wasn't
being modified. Luckily, this seems to be only being used in tests.

Change-Id: Ice01419933295562d558d48ba314d476660b67bd
2021-05-25 16:41:10 +00:00
Qweder93
099a31c69f multinode/payouts: paystubs for single period added
paystubs for single/all satellites for specific period added.

Change-Id: Ibb822aa11380bc70f4db8c81fa0b9b6f7a739078
2021-05-25 18:59:05 +03:00
Qweder93
1bc4f8fccd multinode/payouts: paystubs for all time added
paystubs for single/all satellites for all time added.

Change-Id: Ia665c69994f769cb0071363a617eaf87ef1a05f2
2021-05-25 18:19:23 +03:00
Egon Elbre
926bcd0f83 mod: bump private
Latest version contains a fix for pgxutil.Conn.

Change-Id: I200b1c11a1e536a57a7384c1a747a7216f8cd9ff
2021-05-25 15:28:16 +03:00
Egon Elbre
6e05b240dc private/lifecycle: log stack on slow shutdown
When services take long to shutdown it's useful to get a stack trace for
diagnosing the underlying problem.

Change-Id: Ic73a45741dfbe8fdddafd56a5b72121da886d133
2021-05-25 09:55:19 +00:00
Fadila Khadar
16022105f1 satellite/orders: create project_bandwidth_daily_rollups table
Change-Id: I8f101c0714b5673450d4f2f01e297a9eddc6caf6
2021-05-25 09:05:47 +00:00
Vitalii Shpital
c410eae177 web/satellite: better error message for duplicated bucket
WHAT:
better error message when trying to create a bucket with a name that already exists

WHY:
better user experience

Change-Id: I1f56439d6dc8503cdca03759b0cc180688aafd87
2021-05-25 08:24:00 +00:00
Vitalii Shpital
b33b0a38d6 web/satellite: remove redundant console errors on duplicated route changes
WHAT:
removed console errors when trying to navigate to current route

WHY:
there shouldn't be errors in console

Change-Id: Ifa6ccf2f336cab58ac5e27b5dae2167a3229f98a
2021-05-25 08:03:15 +00:00
Moby von Briesen
0ff3516f54 satellite/mailservice: Fix bug causing issues with test account creation
The context passed into SendEmail gets canceled before links are clicked
in the test environment. This change passes an un-cancelable context
from SendRenderedAsync so that email sending/link clicking can be
completed even if the parent context is canceled.

Change-Id: I88535d315900d7886877f0e14d1d052745402ac7
2021-05-25 06:19:53 +00:00
Egon Elbre
e7cf369cd8 satellite/metabase: use pgxutil.Conn helper
Clean up underlying pgx.Conn extraction with helper.

Change-Id: Ie43f663a6036c439e675df54fccd3605c4d8d69e
2021-05-25 05:56:35 +00:00
Jeff Wendling
e460dc51f7 cmd/uplinkng: implement object listing
Change-Id: Ib5f6964a0c42718913a680529bb66c6f475aeac9
2021-05-24 23:02:19 +00:00
Egon Elbre
cdcc67207c satellite/satellitedb: fix nil panic in UpdateCheckIn
Change-Id: If6ae2c3d9b7c269b0a9d652e68854091f668b5ec
2021-05-25 00:30:36 +03:00
Qweder93
79172777bd mnd/payouts: estimations replaced with expectations
Added expectations endpoint (estimations and distributed), added
coalesce to db query, so in case of empty payouts db 0 will be returned instead of error.

Change-Id: I535f14ef097876448d8949bc302895b25da2b6e7
2021-05-24 18:11:30 +00:00
Qweder93
83e82eb473 mnd/payouts: estimated payouts per satellite changed to per node
Change-Id: I231e95ece1f52accb17839a00bfc1588b6e2a658
2021-05-24 19:47:57 +03:00
Egon Elbre
515f9b54ff mod: bump monkit
Change-Id: I9be05b91cd9171ebc7d053cd087d6761150d44b6
2021-05-24 16:20:00 +00:00
Qweder93
f0dd7f3739 mnd/payouts: undistributed payouts added
Change-Id: Id61ac58fceca13b9a9e12a7e03c3d9624cce3fea
2021-05-24 18:20:08 +03:00
Vitalii Shpital
2896a2736c web/satellite: remove time caveats for object browser AGs
WHAT:
removed notBefore and notAfter caveats from object browser AGs

WHY:
caused a bug when user's clock is not synced correctly

Change-Id: I1af10418f1668003a09cba313eaed3b0764eabdc
2021-05-24 14:50:17 +00:00
NickolaiYurchenko
f2842a27e2 web/multinode: by node payouts page markup
Change-Id: I4d760a9965d28afc37f4cfa971d95254bfcb1d7e
2021-05-24 17:28:25 +03:00
Michał Niewrzał
c9d4674859 cmd: remove metabase-createdat-migration tool
We finished migration for all satellites and we can now remove this
code.

Change-Id: If92091ba94b51c319528dc95845042c9dfd6d074
2021-05-24 09:37:40 +02:00
Yaroslav Vorobiov
f0d60a6ec2 Jenkins: add multinode UI stage
Change-Id: I77ed791ff9dbb5b046450ab625c1763b34691888
2021-05-21 15:49:31 +03:00
Yaroslav Vorobiov
1ff2e1a474 web/multinode: update test snapshots
Change-Id: I8a7adeba1b48c754dc50e68c479f89903cd98486
2021-05-21 15:46:46 +03:00
Egon Elbre
2d3c40fd71 cmd/internal/asset: delete unused code
Change-Id: I44970a4f5ac42dc3a297263cd64150e94c40f62a
2021-05-20 18:30:19 +03:00