Commit Graph

2274 Commits

Author SHA1 Message Date
Vitalii Shpital
3c683998f5 satellite/console, web/satellite: feature flag for new objects flow
Added a feature flag to switch between old and new objects flow

Change-Id: Id1802f9fb1230928445f16e4d0a62aecdc91c038
2021-11-08 11:45:22 +00:00
Erik van Velzen
91158fd3bf satellite: omit system metadata fields if not requested
To reduce database load, add option to omit as many object properties
as possible when listing objects.

Change-Id: I817633801b00629a4042d1d1bd2389ee581953de
2021-11-08 09:19:44 +01:00
Márton Elek
fb604be460 satellite/gracefulexit: use nodecache with gracefulexit
Change-Id: I700caf6dfd06bd3b3970a8cf7c5a79da4af27b5f
2021-11-05 19:14:14 +00:00
Qweder93
b314559a13 satellite/metainfo: TestBatch params updated
Removed PathCipher and DefaultSegmentSize from CreateBucketParams
since they are unused anymore and breaks Integration on uplink side

Change-Id: I1393a7f1f436940731aa59edd693043336383290
2021-11-05 14:53:55 +02:00
dlamarmorgan
1dd537953e satellite/metainfo: strip the uplink version from the UserAgent and bound its size
The UserAgent should be stored as is, with the exception of removing the trailing version from any libuplink user agents

Change-Id: If17ef2fc4b59480a3477300f2585a07d64cc2bf4
2021-11-04 10:09:54 -07:00
Elek, Márton
27091826ba satellite/{satellitedb|metabase}: add SQL fields for geofencing
Change-Id: Ia698d71db1d3775b908c24bd1c9bc8abfd015ccc
2021-11-04 08:49:56 +00:00
Egon Elbre
d043b9facd satellite/console: load index.html from disk when developing
This eases development by not needing to restart satellite after each
change.

Change-Id: I1063b7f431f675fb21e82dcfc2420310a9806422
2021-11-03 18:19:40 +02:00
Michał Niewrzał
e792727bea satellite/metabase: allow setting metadata with begin object requests
Turns out that S3 protocol is setting object metadata with initial
request, in our case it's BeginObject, so we need to modify metabase
methods to accept metadata at the beginning and at the end.
BeginObject methods will set metadata always (niled or not niled).

One additional improvment for metadata fields was introduced and it's
validating fields as optional if EncryptedMetadata was not set.

This change currently doesn't have any implications and  it's a base for
other changes. Metadata is not set with BeginObject metainfo endpoint
yet.

Change-Id: I1f768407bc3428500b0d30ee188257420d953001
2021-11-03 10:08:10 +00:00
Cameron Ayer
1de8a695e8 satellite/{overlay,satellitedb}: fix stray nodes DQ bug
We had a bug in the stray nodes chore where nodes who had not been seen
in several months were not being DQd. We figured out that this was
happening because we were using two queries: The first to grab
nodes where last_contact_success < some cutoff, the second to DQ them
unless last_contact_success == '0001-01-01 00:00:00+00'. The problem
is that if all of the nodes returned from the first query had
last_contact_success of '0001-01-01 00:00:00+00', we would pass them to
the second query which would not DQ them. This would result in the stray
nodes DQ loop ending since we found a number of nodes to DQ less than the
limit.

The fix: add the "WHERE last_contact_success != '0001-01-01
00:00:00+00'::timestamptz" to the selection query.

Change-Id: I4e60de90b68d8745d641b4467c2b23e0e56f7dff
2021-11-02 17:05:00 +00:00
Yingrong Zhao
774ae017e3 satellite/{satellitedb, web}: display object count in satellite UI
Even though we want to start charging segment fee instead of object fee,
it's hard for users to understand what a segment is. This PR adds the
object count back in the UI alongside with segment count to help address
the issue.

Change-Id: I92eb42c769d350eba68a72443deffec5c278359c
2021-11-01 18:26:13 +00:00
Egon Elbre
edb8d656de satellite/metainfo: adjust piecedeletion timeouts
Currently slower storagenodes can slow down deletion queue.
To make piece deletion faster reduce the maximum time spent in
either dialing or piece deletion requests.

With this change:
* dial timeout is 3s
* request timeout is 15s
* fail threshold is set to 10min

Similarly, we'll mark storage node as failed when the timeout occurs.
The timeout usually indicates that the storagenode is overwhelmed.
Garbage collection will ensure that the pieces get deleted eventually.

Change-Id: Iec5de699f5917905f5807140e2c3252088c6399b
2021-10-28 13:37:01 +03:00
Yingrong Zhao
d441c8da15 satellite: use segment count for billing
Change-Id: Iafaf41cb6cde4cb2abdac3f75ce0fead838cb758
2021-10-27 19:21:50 +00:00
Yingrong Zhao
52f8c8175e satellite/satellitedb: add segments column into invoiceprojectrecords
table and drop not null constraint on objects column

Since, we want to move from charging our customers by object count to
segment count, this PR prepares the database to be able to record segments count
instead of objects count for satellite's billing system

Change-Id: Ie91ef354e78d24a268bc1cdc4327c182f733321e
2021-10-27 14:51:51 -04:00
Márton Elek
20d03bebdb satellite/nodeselection: flexible interface to includes nodes in selection
This commit doesn't change any behavior, just organize the code in
different way to make it easier to implement different Criterias
to include nodes. Today we use NodeID and Subnet based selection
but later Criteria can be extended with different kind of
placement rules (like geofencing).

The change nodeselection is used by segment allocaton (upload) and repair
and excludes nodes from an in-memory selection.

Resolves https://github.com/storj/storj/issues/4240

Change-Id: I0c1955fe16a045e3b76d7e50b2e1f4575a7ff095
2021-10-26 11:01:33 +00:00
Michał Niewrzał
de38e2e7d8 satellite/metainfo: stop using sat StreamID Redundancy
We needed Redundancy insided sat StreamID when uplink was defining RS values. Now it can be removed.

Change-Id: Id37187493eaa00cf29cb0262a050d71add3deb96
2021-10-26 10:23:11 +00:00
paul cannon
eb68dbad41 satellite/satellitedb: fix ordering in listPendingTransitionShim
listPendingTransitionShim is a temporary transition shim intended to
make existing API processes keep working when a future DB schema change
is executed. For more explanation, see the message on commit c053bdbd70.

However, the shim has a small bug: it is missing the ORDER BY clause
that appears in the original ListPending method. This transition shim
code won't ever run until we make the DB schema change, so this bug
hasn't hurt anything yet; it's just important that we fix it before the
DB schema change happens.

Change-Id: I5953651583ee236500c2c07141dfc9d690a95118
2021-10-25 19:33:53 +00:00
Malcolm Bouzi
d90a1b09a2 satellite/console,satellitedb: add signup promo code column to users
This update is to set up users being able to register with a promo code added to their account in place of the free tier coupon.

Change-Id: I7badf87937b12664f145520b6dcc4b26fe750407
2021-10-25 18:08:00 +00:00
dlamarmorgan
01cfbde56b satellite/console: increase free tier project bandwidth and storage from 50 GB to 150 GB and reduce free tier max projects from 3 to 1
To make our free tier limits more clear, we will reduce the number of projects allowed from 3 to 1, and increase the storage and bandwidth limit of the free tier from 50 Gb to 150 GB. The total allotments across all projects for a given user are unchanged, just reduced to a single project.

Change-Id: Ic8dddb135f2b83a3f36e2b9fdcb477e351ec137b
2021-10-25 17:34:19 +00:00
crawter
3c7e289395 satellite/matainfo: handle metabase errors in metainfo endpoint in common way
We should improve the way how we are handling metabase errors in
metainfo endpoint.
https://storjlabs.atlassian.net/browse/PG-316

Change-Id: I1da6f333546cabf34d6eb1de8e94a3ef455d75d5
2021-10-25 16:30:24 +00:00
Ivan Fraixedes
df44e8152d
satellite/admin: Move declaration before first usage
Move a variable declaration just before where is used for making easy to
follow the implementation when reading it.

Change-Id: I3e93597956352037a63c5496b11576fdd2c600a6
2021-10-24 18:38:41 +02:00
Fadila Khadar
f654c3ae4c satellite/gracefulexit: drop unused column in graceful_exit_progress
We don't need column uses_segment_transfer_queue in graceful_exit_progress
as now all exiting nodes are using graceful_exit_segment_transfer_queue and
table graceful_exit_transfer_queue has been dropped.

Change-Id: I4b7c087433f04138cf09bcf8ad3d8de2c185502a
2021-10-22 16:41:27 +00:00
Moby von Briesen
1b22525002 satellite/payments: Fix discount when listing invoices.
The error message Stripe's API was sending:
"This property cannot be expanded (total_discount_amounts).
You may want to try expanding 'data.total_discount_amounts' instead."

Change-Id: I9f8cea4107d826d837755be2c3c04675a36f3c37
2021-10-22 10:16:27 -04:00
Qweder93
50baefa10e satellite/metabase: limit maximum number of parts and size
Multipart upload limits added. Last part has no size limit.
Max number of parts: 10000, min part size: 5 MiB

Change-Id: Ic2262ce25f989b34d92f662bde720d4c4d0dc93d
2021-10-22 10:16:37 +00:00
Egon Elbre
f2d8e97d97 satellite/satellitedb: simplify select nodes query construction
Change-Id: I07009b28762d4485929a2a999e8f4be8179bee51
2021-10-22 07:41:07 +00:00
Monty Anderson
3989107031 web/satellite: add feature flag for new browser
Change-Id: If3773b9df7f79d4c5e094140448d60ec060cdaeb
2021-10-22 03:49:08 +00:00
Michał Niewrzał
b32fbc0f93 satellite/metainfo: speedup deletion tests
Speedup is done by reducing number of testplanet instances
for tests without changing main test logic.

Change-Id: Ic3849485d37b8ca55c013a45b7191dce65b88b04
2021-10-21 15:54:27 +00:00
Egon Elbre
187941ff86 satellite/{metabase,metainfo}: add some missing monitoring
Change-Id: Ie0018f6385bb145d0a514dfe2792f3f299b55b79
2021-10-21 09:15:04 +00:00
Vitalii Shpital
0009447773 satellite/console, web/satellite: feature flag for new navigation structure
Change-Id: I6f1fd43ea8154714187c7547ae93de0efe3482e6
2021-10-20 13:51:25 +00:00
Moby von Briesen
9c8b6c3fc3 testsuite/ui: Add coupon tests
Add some testing around adding/replacing promotional coupons and what
the user sees in the UI depending on their coupon status.

Change-Id: Ice6c0a0644d05af0c30c87a93ba963c0bb09e32d
2021-10-19 19:08:35 -04:00
Brandon Iglesias
8d18033a99
satellite/rewards: adding SeaweedFS to partners list (#4230) 2021-10-19 21:30:31 +02:00
Mya Pitzeruse
d250f9dfee
satellite/core: fix label for orders chore (#4228) 2021-10-19 18:44:15 +02:00
Egon Elbre
51f488fc4b satellite/satellitedb: fix selection query with AOST
Union query for both reputable and new nodes didn't properly work. The
top level query is required to have an `AS OF SYSTEM TIME` statement as
well.

Change-Id: I8ee6dd5b700c2b1ed2aa562962bfa72be7eec30a
2021-10-19 16:59:40 +03:00
Michał Niewrzał
4a146000cc satellite/metainfo: read from DB only needed columns fro bucket
Uplink needs only part of columns we are reading from DB.
To improve performance we should read only those that are
realy needed.

Change-Id: Ib39259318169c46afe5fa4c6ce2184da82e960c8
2021-10-19 11:53:42 +00:00
Michał Niewrzał
8293c9659e satellite/metabase: speedup iterator tests
Main speedup is from reducing number of cases to iterate
in 'boundaries' test case.

Change-Id: I5bbf1311790f3d0e553d59e6a73e1cf3fa601a55
2021-10-19 11:52:07 +00:00
Michał Niewrzał
f5f2398431 satellite/metabase: remove code related to part deletion
It is not used currently.

Change-Id: I32e9fa357814a3159ee8659a607cd6986f361fbf
2021-10-19 10:11:49 +00:00
Moby von Briesen
4ad7056bf7 satellite/payments: Add old invoice list functionality
3b751a35c Removed our old coupon functionality, and slightly reworked
the invoice List() function in the stripecoinpayments package.

It turns out, this is causing some issues when trying to delete users.
This change keeps the new functionality, which is used in the satellite
UI, but under a new name, ListWithDiscounts()

Change-Id: I6a62a1de480e09d005dd22d75aa1e024fd2ed3a0
2021-10-18 18:18:18 -04:00
Yingrong Zhao
35e4a87e60 satellite/repair: ignore expired segments at the beginning of the repair
work

Since we have changed the repair worker to also mark a node as audit
failure if they return a not found error, we should ignore expired
segments when possible

Change-Id: Ie6a677e1d7b234e93965c736d05950440236653c
2021-10-18 18:15:39 +00:00
dlamarmorgan
4bbf667ad1 satellite/{satellitedb,attribution,console}: value attribution changes that add userAgent field to buckets table and all tables that have partner_id
Change-Id: I36a13bb651b86bfc14fe5a0a2258f719e6cd2b48
2021-10-18 13:56:19 +00:00
littleskunk
016ac75052
satellite/console: Enable new onboarding workflow (#4222) 2021-10-15 21:07:35 +02:00
Malcolm Bouzi
d7812a3b17 satellite/console,web/satellite: add slider and unit toggling to project limit updating
Change-Id: Iab7c08baa09d598ff53ddf09ef09dcd2a997a81b
2021-10-15 14:11:50 +00:00
Egon Elbre
52c950e429 testsuite/ui/uitest: add Edge testing
Change-Id: I1e592ac755b582d4fa73e95fd999a35a27f8a116
2021-10-15 13:27:13 +00:00
Michał Niewrzał
96dc43fd49 satellite/metainfo: disable DeletePart method
We can now overwrite segments in pending objects so
we should stop using this method.

Change-Id: Ib24bd53e1a73d8c380b89a994a3cd8d178f4c533
2021-10-15 12:20:50 +02:00
Cameron Ayer
56fe636123 satellite/{reputation/satellitedb}: remove references to contained column in reputations table
We don't use this column for anything. If you want to know if a node is
contained, you can check the pending_audits table.

Change-Id: I5671722a5fc6e1749d3a49e187a56556000ff941
2021-10-14 19:59:03 +00:00
Cameron Ayer
bb21551a9c satellite/satellitedb: remove references to contained column in nodes table
We don't use this column for anything. If you want to know if a node is
contained, you can check the pending_audits table.

Change-Id: I8da1d8e01a2dcaff63c5067a7927b5451424ad04
2021-10-14 19:17:46 +00:00
Jeff Wendling
91ce70e4ed satellite/metabase: allow overwriting segments in pending objects
this is so that multipart upload allows overwriting parts. our
consistency model is more relaxed because part overwrites
won't be atomic if they consist of multiple segments, but at
least this allows overwriting at all.

Change-Id: I21dac4c24046e05efe74e6c6fd189a02c95eb6c8
2021-10-14 10:33:52 +02:00
Moby von Briesen
5b729779a2 satellite/console: Automatically log a user in after verifying email
When an email is verified, insert an auth cookie so that when the user
is redirected after verifying their email, they are immediately taken to
the onboarding flow.

Change-Id: I557d8a2805b24dd8039ada255522bc1b56cc8b53
2021-10-13 13:08:27 +00:00
Michał Niewrzał
b8e8110ca3 mod: bump common and uplink dependency
Contains also adjustments for latest storj/common changes.

Change-Id: I11f70d466b3d9785ee93ac424629b2b868889d02
2021-10-12 19:13:49 +00:00
Michał Niewrzał
1fdb0eaa5b Revert "satellite/metabase: use storj.Nonce instead []byte"
This change introduce problems with server side move so
let's revert it for now. Problem was found when latest
version of storj/storj was used in uplink tests.

This reverts commit 1ef06fae99.

Change-Id: I4d4fad5d1ea04ba15ff9d7bd765f7e078e9187c2
2021-10-12 15:39:54 +02:00
Jeremy Wharton
3b751a35c5 satellite/{payments,satellitedb}: Remove custom coupon implementation
Removes database tables and functionality related to our custom
coupon implementation because it has been superseded by the Stripe
coupon and promo code system. Requires implementations of the
payments Invoices interface to return coupon usages along with
invoices.

Change-Id: Iac52d2ff64afca8cc4dbb2d1f20e6ad4b39ddfde
2021-10-11 19:47:00 +00:00
Michał Niewrzał
1ef06fae99 satellite/metabase: use storj.Nonce instead []byte
We were using mixed types for nonce fields. Protobuf
have storj.Nonce, metabase have []byte. This change
is a refactoring to have everywere its possible only
storj.Nonce.

Change-Id: Id54bd8481f30c721cdaf3df79206d25e7cfdab55
2021-10-11 16:13:34 +00:00