Commit Graph

2459 Commits

Author SHA1 Message Date
Cameron Ayer
251e432e5b satellite/satellitedb: drop contained from nodes and reputations
A few months ago we removed all references to the contained
column in nodes and reputations

bb21551a9c
and
56fe636123

But we never did the migration to drop the columns.
This commit will finally do that.

Change-Id: I82aa2f257b1fb14a2f1c4c4a1589f80895360ae4
2022-01-20 19:46:48 +00:00
Moby von Briesen
464b4b02fa satellite/console/consoleweb: Only replace some kinds of characters in name
The previous change (59648dc272) ends up removing a lot of characters
for valid non-English names. Instead, only replace URL characters such
as slashes, colons, and periods. Since someone may use these characters
to separate two parts of a name, e.g. Name1/Name2, replace these
characters with a hyphen.

Change-Id: I4cc3d1bdb05d525a83970cf1b42479414c9678e7
2022-01-20 11:13:26 -05:00
Clement Sam
2de6feefde satellite/contact: fix connection leak in PingMe endpoint
Fixes connection leak in pingMe endpoint.
includes other minor fixes.

Change-Id: I7c61f620565f46dd113d21a772de7c439be550e3
2022-01-20 14:20:24 +00:00
Moby von Briesen
59648dc272 satellite/console: Filter special characters out of name
When a user is created, but before verification or forgot password email
is sent, remove any special characters in the provided name. This
protects us against certain phishing attacks.

Change-Id: Ieddd3479da20eb80b9f1b56eb86c8f46bca2642c
2022-01-20 00:18:56 +00:00
Clement Sam
9f3c1f9cda satellite/contact: add PingMe endpoint
Change-Id: I832a72fafeacf76ad64a0129bcc6582cc4f9290d
2022-01-19 17:52:33 +00:00
Michał Niewrzał
95750b9d32 satellite/console: verify limits while test
Additional test case to be sure that update operation went well.

Change-Id: I7dbf1e3110f716712ba9747a1915b54112b89801
2022-01-19 16:39:36 +00:00
Qweder93
228f465d45 satellite/metainfo: сombine checks for storage and segment limit
We need to combine methods from accounting.Service (ExceedsStorageUsage and ExceedsSegmentUsage)
to run checks concurrently.

Resolves https://github.com/storj/team-metainfo/issues/73

Change-Id: I47831bca92457f16cfda789da89dbd460738ac97
2022-01-19 10:44:04 +00:00
Michał Niewrzał
270c6e24f2 satellite/console: fix default limit for free-tier segment usage limit
Free-tier segment usage limit was defined as 150k, not 140k. This change
is correcting that.

https://github.com/storj/team-metainfo/issues/8

Change-Id: I71ec0961930b19fd09b2b996e01acd406a8dcf8f
2022-01-19 09:21:13 +00:00
Mya
309ac41bfc satellite/satellitedb/dbx: add schema for oauth tokens
Change-Id: Ia980766a8f86b662545e7d8cae6ecf631e199809
2022-01-18 09:38:12 -06:00
Mya
83f7d3e045 satellite/satellitedb/dbx: add schema for oauth clients
Change-Id: Ib08c7074803805cca8ffaedbd89da54504842af7
2022-01-18 09:38:12 -06:00
Michał Niewrzał
36c07720bb satellite/metainfo: bound user agent metrics
We need to limit number of results for SeriesTag to avoid
overloading our metric system.

Change-Id: I3d9f4f4f6d489bf4ce87c6240f66d48b29b88abb
2022-01-18 12:42:09 +00:00
Mya
cf462fcf92 satellite/admin: update geofence endpoints to follow response conventions
Change-Id: Ieff91c16f91f9720e9c3d8ead0e13c47651d23f5
2022-01-13 12:33:10 -06:00
Michał Niewrzał
5e9643e1b8 satellite/metainfo: cleanup validation
Refactoring to do few things:
* move simple validation before validations with DB calls
* combine validation check/update for storage and segment
limits together

Change-Id: I6c2431ba236d4e388791d2e2d01ca7e0dd4439fc
2022-01-13 13:00:40 +01:00
Michał Niewrzał
18ab464b51 satellite/metainfo: UpdateProjectSegmentUsage enabled only with segment validation
updating project segment usage entry to redis is enabled only if
segment validation is enabled in config value, so no incompatible
changes will be added while deployment.

Change-Id: I1288cb9ff0a8a00f095dc94e20d2f14393e9a613
2022-01-13 12:12:25 +01:00
Ivan Fraixedes
3a987ac00f satellite/admin: Fix client API & docs
The following 2 commits added 2 new query parameters to set the `burst`
and `segments` limits for a project and also to new fields to the
response JSON object body to the "get project limits" endpoint:

* c911360eb5
* b7b010adc9

However, the API documentation and the Typescript client API (used by
the UI) weren't updated with them.

Later, the commit dc6128e9e2 updated the
Typescript client API with the `segments` limit but it didn't update the
documentation to reflect it.

This commit brings all things that were missed in those previous
commits.

Change-Id: Iff12cdd4a0d3c448cd73b57a98d171ba468d2c98
2022-01-13 09:31:01 +00:00
Michał Niewrzał
eb0d08d59b satellite/metainfo: collect versions of user tools
We want to know usage statistics for our main tools
like uplink-cli or rclone. Initially we will collect
only usage stats without relation to specific process
e.g. download or upload.

Change-Id: I203b1a6c07ae014e710368f77163f13fdf10763c
2022-01-12 13:49:15 +00:00
Yaroslav Vorobiov
e0b3c7152b satellite/nodestats: use overlay node for reputation status
Use overlay db node as primary source of truth for reputation status when node
query it's status via Endpoint.Get. If node was was disqualified in overlay reputation
status may be inconsistend with reputation table.

Change-Id: I1bf858a4b020324035847a9215ef29cf5432f6a4
2022-01-11 16:38:27 +00:00
Qweder93
b6625cadea satellite/metainfo: uncomment tests, renamed EncryptedPath into EncryptedObjectKey
moved tests back and renamed params that were renamed on uplink side

Change-Id: I4a9592bd955f2217c5809c235c394a4e78d337ea
2022-01-11 17:18:18 +02:00
Qweder93
332b673a05 satellite/metainfo: comment tests for rename
comment tests that contains fields needs
to rename on uplink side without breaking compatibility.
after rename tests will be moved back from comments.

Change-Id: I3bc4aff6ae7f6711ade956ac389f0d7e1a1ab91a
2022-01-10 17:46:32 +02:00
Qweder93
15c2b2cc10 satellite/metainfo: comment tests for rename
comment tests that contains fields needs
to rename on uplink side without breaking compatibility.
after rename tests will be moved back from comments.

Change-Id: I439783c62678c32805a85aa52bef1d2b767543a1
2022-01-10 14:44:43 +00:00
Qweder93
dc6128e9e2 satellite/admin: add update project segment limit on ui
Added possibility to update project segment limit on
satellite admin panel ui.

Resolves https://github.com/storj/team-metainfo/issues/5

Change-Id: I192d0c9adbab411556aa46673167b3ff7437d04b
2022-01-10 13:26:56 +00:00
Qweder93
8be655c487 satellite/metainfo: add segment limit validation, caching
We want to be able to limit the number of segments per project for users.
To limit this we need to check limit value associated with project
and value of used segments already in BeginMoveObject, BeginMoveSegment
and increment cache segments usage after each CommitSegment call.

Resolves https://github.com/storj/team-metainfo/issues/1

Change-Id: I6290e67c095a174b9d101c4521802d9bfe0453b8
2022-01-10 14:33:26 +02:00
Yingrong Zhao
1f8f7ebf06 satellite/{audit, reputation}: fix potential nodes reputation status
inconsistency

The original design had a flaw which can potentially cause discrepancy
for nodes reputation status between reputations table and nodes table.
In the event of a failure(network issue, db failure, satellite failure, etc.)
happens between update to reputations table and update to nodes table, data
can be out of sync.
This PR tries to fix above issue by passing through node's reputation from
the beginning of an audit/repair(this data is from nodes table) to the next
update in reputation service. If the updated reputation status from the service
is different from the existing node status, the service will try to update nodes
table. In the case of a failure, the service will be able to try update nodes
table again since it can see the discrepancy of the data. This will allow
both tables to be in-sync eventually.

Change-Id: Ic22130b4503a594b7177237b18f7e68305c2f122
2022-01-06 21:05:59 +00:00
littleskunk
b21cbc85f1
satellite/orders: log level warn for remaining "bucketName or projectID not set" (#4326)
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2022-01-06 16:31:26 +01:00
dlamarmorgan
ab37b65cfc satellite/{accounting,orders,satellitedb}: group bucket bandwidth rollups by time window
Batching of the order submissions can lead to combining the allocated
traffic totals for two completely different time windows, resulting
in incorrect customer accounting. This change will group the batched
order submissions by projectID as well as time window, leading to
distinct updates of a buckets bandwidth rollup based on the hour
window in which the order was created.

Change-Id: Ifb4d67923eec8a533b9758379914f17ff7abea32
2022-01-05 20:24:48 +00:00
Egon Elbre
a42b9d1a48 all: fix uses of email.com
email.com is not a domain that should be used for examples
nor tests.

Change-Id: I654d4287d02633d5ed9740e81a79150470eeaf25
2022-01-05 16:29:19 +02:00
Egon Elbre
04d04c0306 web/satellite: remove old object browser
Change-Id: Ief16a3887444df093c644831d86a9b3e865e5867
2022-01-05 16:29:19 +02:00
Malcolm Bouzi
12cb733b16 satellite/satellitedb: add last_verifcation_reminder column to users table
We want to issue a reminder to users when they don't verify their email within 24hrs of registering. This change only adds a column to the users table.

Change-Id: I92e2baeabf179338ffec01574d4752c0ccdba88b
2022-01-04 19:38:04 +00:00
Malcolm Bouzi
6a059b7f9d web/satellite: remove all references to the old onboarding flow
Change-Id: I1f42fabdb407a7eb9c34f0341d332e66c3ce288d
2022-01-04 18:40:08 +00:00
Moby von Briesen
5351c43757 satellite/analytics: track origin header and referer on signup
This allows us to distinguish between accounts created from the signup
page vs. from www.storj.io.
Also set a field `account_created=true` when we send so
that we can see when existing leads have created an account.

Change-Id: Ibef34825a08b6c68b8f2869625e576bb837520e5
2022-01-03 17:47:48 +00:00
Cameron Ayer
2a4ff6f469 satellite/satellitedb: select user_agent in apikeys method GetPagedByProjectID
The user_agent column was missing from the SQL query in
apikeys.GetPagedByProjectID. Add it.

Change-Id: I2201b0ecd1767194a6d2339dba25a0b58e1ee30a
2022-01-03 11:58:26 -05:00
Moby von Briesen
1bd74658df satellite/console: Disable segment limit checking during project update
This is not supported on the client side yet.

Change-Id: Id2df75278efe4b3a48be21d0992ee835e804fc3a
2021-12-23 14:35:45 -05:00
Michał Niewrzał
ff653f6f22 satellite/metainfo: stop using in tests unused method
We would like to remove it from uplink but to do that
we need to stop using it in storj tests.

Change-Id: I8f6af74d6b8b8e447ab8ed3a00517b40840dad58
2021-12-22 15:05:53 +01:00
Michał Niewrzał
db7985ca90 satellite/metainfo: set EncryptedMetadataEncryptedKey while listing objects
At some point we missed to add metadata key to list objects
response. Because of that uplink is taking key from pb.SteamMeta.
We need to clean it up.

Tests will be added on uplink side.

https://github.com/storj/uplink/issues/71

Change-Id: I3328e2f1b86bca15aeaf89f8e59cdca3c8e97742
2021-12-21 13:09:11 +00:00
Michał Niewrzał
87a9292127 satellite/metainfo/zombiedeletion: enable chore by default
We enabled this chore on different satellites
for testing. Works fine so we can enable this
by default for all configurations.

Change-Id: I987639685d8de5c7e5798adca30fe26bdac9e1d1
2021-12-21 13:26:18 +01:00
Stefan Benten
a6139f5a6b satellite/orders: log less decrypt order messages
Currently most of the satellite log is made up by this specific log
message and thus we should increase the log level. We already have
a monkit event tracking this case. In case we need to look into this
more we should just increase the satellite log level.

Change-Id: I27ebed3e6745701c66c83e8c52ddc836ad9d5f4e
2021-12-21 00:39:29 +01:00
Qweder93
b7b010adc9 satellite/admin: add project limit on segment count
Added possibility to set custom value of segments
per project for user/customer.

Change-Id: Ic6e1abe25269755e9e8819fbef065a87de359fce
2021-12-17 12:40:12 +00:00
Ivan Fraixedes
f87ce0cadf satellite/admin/ui: Fix component rendering
I introduced a change in the UI that was a better way to dynamically
render Svelte components
(https://review.dev.storj.io/c/storj/storj/+/5931/4..5) during the
review of the first version of it, however, while it works perfectly
on development mode it doesn't work when the assets are built for
production, failing silently, because the constructor name get renamed
due to the name mangling caused by the minifyer as stated in the
following issue: https://github.com/sveltejs/svelte/issues/6980

This commit use a different alternative not based on the constructor
name and it works fine with the production build.

Change-Id: I643c405f877a9206cf0e51b44d5138e5a9756a79
2021-12-17 11:00:31 +00:00
Ivan Fraixedes
5573ece848 satellite/admin/ui: Migrate to SvelteKit
Migrate the satellite admin UI web app from the Svelte template used to
generate a Svelte App scaffolding to SvelteKit.

There aren't any functional changes in the  application, however, the
commit has a lot because:

1. SvelteKit uses a different directory layout and constraints to it, so
   the files have been moved.
2. The files have changed its formatting due to the new default linter
   configurations that SvelteKit uses.
3. The linter detected some issues with using `object` and `any` types
   in Typescript, so they have been replaced by better general types
   (e.g. Record).

The  migration allows to use the new tooling rather than Rollup
directly, besides that will empower the future of it when it needs more
features (e.g. different routes, etc.).

Change-Id: Ifa6736c13585708337f6c5a59388077b784eaddd
2021-12-17 10:27:13 +00:00
Michał Niewrzał
60d03b9e42 satellite/metainfo: drop GetSegmentByLocation
Method is not used anymore. We need it during transition
from metainfo loop into segment loop.

Change-Id: Id32590ff9b02e06025efce1c32869e2cf9cd0bc5
2021-12-16 16:32:45 +01:00
Michał Niewrzał
3f0babffe8 satellite/{console,satellitedb}: add project segment limit to user
All limits we have for projects have also parent limits stored
with user data. New created project is first taking limits from
owner (user) limits.
This change is extending users table with project_segment_limit
column and adds functionality to get and set value for this
column.

Change-Id: Iff5e36c62b517652390b649fc05992475916ecff
2021-12-16 08:46:01 +00:00
Jeremy Wharton
9d13c649a2 satellite/{console,satellitedb}: Forbid creating users with used email
This change disallows creation of users possessing the same email.
If a user attempts to create an account with an email address
that's already used - whether it belongs to an active account or not -
he will be notified of unsuccessful account creation. If he attempts to
log in using an email address belonging to an inactive account,
he will be presented with a link allowing him to re-send the
verification email. Attempting to register with an email address
belonging to an existing account triggers a password reset email.

Change-Id: Iefd8c3bef00ecb1dd9e8504594607aa0dca7d82e
2021-12-15 17:24:35 +00:00
Michał Niewrzał
911540e76e satellite/orders: avoid logging "bucketName or projectID not set"
When we switched to segments loop we stopped
adding bucket name and project ID to order limit
metadata. This is causing lots of logging that we don't need.

Change-Id: Id3f878a170b7a6b801e8a838ee69165715985d60
2021-12-14 10:31:48 +00:00
dlamarmorgan
9a37399346 satellite/metainfo: remove Gateway-ST library name from UserAgent string
When trimming the UserAgent string, Library names such as Gateway-ST
should be removed.

Change-Id: I3d71fa8514364bf1d4d79b0dc00c85aed337ab45
2021-12-13 18:07:55 +00:00
Michał Niewrzał
c33d2f58cf satellite/metainfo: override StreamMeta only if needed
For backward compatibility we are overriding pb.StreamMeta
values returned as encrypted metadata. It turns out that we
should do it not when target values are missing but when
values to override exists.

This was causing problems after move operation. Details can be
found here https://github.com/storj/uplink/issues/70

Backward compatibility test will be added to storj/uplink testsuite.

Change-Id: I72e7a01226b1dd62902cb0d6ebb1ff91a4693005
2021-12-10 08:41:37 +00:00
prerna-parashar
a41758bba5
Track user events through HubSpot events API (#4300)
Track user events through HubSpot events API
2021-12-09 12:52:51 -08:00
Michał Niewrzał
000957f64a satellite/accounting: add functionality to get and cache project segment count
We want to set maximum number of segments per project. This change will
add functionality to get number of segments currently used by project.
To avoid often DB calls segment cound will be cached and refreshed every
few minutes.

Change-Id: I2ecb6484f5afc3875c0e0dfaea360e8872f9d196
2021-12-09 12:30:52 +00:00
Michał Niewrzał
d94d8d1775 satellite/accounting: expose project segment limit
Exposes functionality to get and update project segment
limit. It will be used to limit number of segments per project
while uploading object.

Change-Id: I971d48eebb4e7db8b01535c3091829e73437f48d
2021-12-08 11:27:21 +00:00
dlamarmorgan
3b51eea312 satellite/metainfo: store empty useragent in bucket attribution instead of throwing error
Previously, only valid partner IDs could be used for bucket level value attribution. Now that any useragent byte slice can be used, we should allow for empty useragent strings to be stored rather than throwing an error or leaving the bucket with no attribution.

Change-Id: I7043f835588dab1c401a27e31afd74b6b5a3e44b
2021-12-07 18:06:27 +00:00
Michał Niewrzał
5013dd8a43 satellite/metainfo: remove error message prefix
Currently detecting if error NotFound is for object
or bucket is very fragile on uplink side. We cannot
change this messages for now.

Change-Id: I6ec4d98116477812f031134e4f1c9e73bdce8b27
2021-12-07 11:59:06 +00:00
Michał Niewrzał
73730b23e7 satellite/satellitedb: add segment_limit colum to projects table
We want to set maximum number of segments per
project. This change adds only column to projects table.
Default value 1M is set to make later migration easier as
we need to set 1M for paid tier users and 140K for free
tier users.

Change-Id: I8e83712e08c5bd91dfa59f652d17e45c14240a36
2021-12-02 12:15:16 +00:00
dlamarmorgan
b3cea3d1b6 satellite/audit: account for piece size during audit reservoir sampling
Treat the piece size as a weight, and perform weighted reservoir sampling as given in Algorithm A-Chao (https://en.wikipedia.org/wiki/Reservoir_sampling#Algorithm_A-Chao)

Change-Id: I299d0026d9e02d03b3d2130b0f32192928e6e326
2021-12-01 18:17:52 +00:00
Vitalii Shpital
b78f65e83b satellite/console, web/satellite: added object and segment count per project info to new project dashboard
Added new query to get project object and segment count.
Added appropriate object and segment count view for new project dashboard.

Change-Id: I69a2e55442f318c51dc365c0c578b964f2f06c7f
2021-12-01 17:46:07 +02:00
Ivan Fraixedes
bde8fb0a59
satellite/admin: Move geofence endpoint to bucket info
Move an endpoint that was classified to return the geofence
configuration of a bucket to return the bucket information, which
also include the geofence configuration, because it's what it was
returning.

Change-Id: I0e0a6aac330296383a50a92d2352df9088df77d5
2021-11-30 18:47:14 +01:00
Ivan Fraixedes
bae8276f3f
satellite/admin/ui: Add missing geofencing endpoints
The satellite admin server was augmented with 3 new endpoints to manage
buckets geofencing configurations, however, they weren't created in the
admin UI API class for making them available in the web interface.

This commit adds these new endpoints to the admin UI.

Change-Id: If060d1f10a3bc6c365e16a891673d4ffc89e4b41
2021-11-30 18:04:22 +01:00
Jeremy Wharton
984792fd1e satellite/satellitedb: Add GetByEmailWithUnverified to users table
Allows us to handle duplicate emails better.

Change-Id: I266057900725e50d1c47977da307714fd32d9081
2021-11-29 20:35:47 +00:00
Cameron Ayer
9c1129b4c4 Revert "satellite/satellitedb: migrate partner_id db column to user_agent db column"
This reverts commit 2c0a360a14.

Avoid big transactions. We'll do it outside of the migration pipeline.

Change-Id: Iade810d81bb2453c9e351149cb84662b207ee527
2021-11-29 10:35:57 -05:00
Igor
4daaa9503e
satellite/console: enable object flow by default (#4298) 2021-11-29 16:43:37 +02:00
Kaloyan Raev
99deec47b6 satellite/metainfo: consistent error message for bucket not found
We should be always returning the name of the bucket that has not been
found.

Change-Id: I7df76e5588c034b83a0702c649becd770858ff32
2021-11-29 07:57:15 +00:00
dlamarmorgan
2c0a360a14 satellite/satellitedb: migrate partner_id db column to user_agent db column
Value attribution codes were converted into UUIDs and stored in the users, projects, api_keys, bucket_metainfos, and value_attributions tables in the partner_id column. This migration will lookup the appropriate partner name associated with each of these UUIDs, and store the partner name directly in the user_agent column within each table. If an error occurs during the partner ID to partner name conversion, the partner ID value will be migrated to user_agent.

A note on the migration test data, postgres.v182.sql:
With one exception, all preexisting rows in the relevant tables had a NULL partner_ID. Therefore, we needed to insert new rows with partner_ID set under the OLD DATA section in order to test that the migration works. For each affected table, we insert one row with a valid partner ID which has a corresponding partner name, and one row with a partner ID which would return an error during the conversion to the partner name.

Change-Id: Iad977d72df0ce95a0c5ca80a065c4276ec1f2354
2021-11-24 22:43:22 +00:00
Michał Niewrzał
de8464e848 satellite/metabase: add GetProjectSegmentCount method
This new method will be used with mechanism to limit number
of segments per project, similar to limiting buckets or bandwidth.
This is only one of multiple changes we will do to implement this
limitation.

Change-Id: Ia516c2a006ad1d7b4431d780679be9d809848f4b
2021-11-24 13:20:02 +00:00
Ivan Fraixedes
1d14e4ef60
satellite/metainfo: Check ctx in download Object & Segment
Check if the context is done at the beginning of the download object and
segment and return right away if it's the case.

Check if Redis returned an error due to context cancellation for not
logging the error.

Change some logging messages of Redis errors to reflect on them if the
error happens while downloading an object or a segment.

Change-Id: I8ed8ff9ff7bb170b560f41356ea06820ce6c4e12
2021-11-24 12:54:11 +01:00
Kaloyan Raev
7d84482381 satellite/metainfo: prevent moving objects between geofenced buckets
Initially, we want to avoid moving objects between buckets with
different geofencing config.

Change-Id: I0cc497873eb5732623ef2d9bc5f78ba1cc48c6b8
2021-11-24 08:34:22 +00:00
Márton Elek
76c2228fbd satellite/metainfo: propagate geofencing between buckets and stream id
Github: https://github.com/storj/storj/issues/4245

Change-Id: I83d34367aab1f3c0d46a044f54980b2d50174b19
2021-11-24 08:05:05 +00:00
Yingrong Zhao
336500c04d satellite/repair: only record audit result if segment can be downloaded
If satellite can't find enough nodes to successfully download a segment,
it probably is not the fault of storage nodes.

Change-Id: I681f66056df0bb940da9edb3a7dbb3658c0a56cb
2021-11-17 15:25:43 +00:00
Vitalii Shpital
3de7f8d5af satellite/console, web/satellite: feature flag for new project dashboard
Added a feature flag to switch between old and new project dashboard

Change-Id: Ibb1b04ccbddb43e3f41116e3513f917deba95699
2021-11-17 16:02:47 +02:00
Artur M. Wolff
89639199fe satellite/repair/repairer: remove unused healthyMap
This change removes unused healthyMap from (*SegmentRepairer).Repair.

Change-Id: Ie80eefdb5b7125bf70986cb13462eee737af214c
2021-11-16 16:49:18 +01:00
Kaloyan Raev
2e363707c5 satellite/buckets: tests for bucket placement config
Tests for checking different scenarios for enabling and disabling
geofence config on empty and non-empty buckets.

Change-Id: I0fe9abb1008d2daee660f22ffe4defe6226b9aa7
2021-11-16 14:58:56 +02:00
Kaloyan Raev
f773bb80c8 mod: bump common to fetch latest placement type changes
Change-Id: I3d0813f05622e706c9be1a578b5e4d4159d16dfc
2021-11-16 12:42:25 +00:00
Mya
a038fc1dc4 satellite/admin: add endpoints for managing a buckets geofence
Change-Id: I407243b8b9c769d968e70478c45230d9fdecaeb2
2021-11-16 12:42:06 +00:00
Mya
814e3126fa satellite/buckets: add new buckets service
The main motivation is to wrap the bucket DB and metainfo DB, so we
could check if a bucket is empty before applying geofencing config.

Change-Id: I8bac21555e01d51a663fb557bc1acfc8106bc2e1
2021-11-16 12:36:17 +02:00
andriikotko
09568b3e2b
satellite/console: change default feature flag (#4274)
* satellite/console: change default feature flag
2021-11-16 11:38:42 +02:00
Ivan Fraixedes
07e07d3a2d satellite/metainfo: Add project ID to error logs
Add the project ID to error logs messages about not being able to
retrieve and track storage or bandwidth usage.

Some error logs related to these errors already contained the project ID
but others didn't, having it in the ones that didn't make them more
consistent and will provide more info which may be helpful for
troubleshooting them.

Change-Id: Ia9fc707a7f3aff0867645bb941badc199c2bf832
2021-11-12 11:28:08 +00:00
Michał Niewrzał
289ae3325a satellite/metabase: always include encryption in listed object
We implemented filtering of system metadata to
reduce DB load in case user will need only list of
objects but we missed that encryption configuration
is needed to decrypt object key.

This change is always including encryption configuration into
list objects results.

Change-Id: Iaf7b3d8441480e3ad787f19a28f1b648584d5f7d
2021-11-12 05:08:01 +00:00
Ivan Fraixedes
6dffbd4173 satellite/metainfo: Log in warn level limits exceeds
Don't log as an Error when users make request that cannot be fulfilled
because they exceed their storage or bandwidth limits.

Those are not system errors and the service is handling them correctly
and showing them in the logs an "ERROR" is misleading.

Change-Id: Iac642b7e8ba92840bb943192ad0694b5f4930258
2021-11-11 15:53:18 +00:00
dlamarmorgan
44b1ca6b97 satellite/{console,satellitedb}: move project limits from config file to DB to keep limits on a per user basis
To allow for changing limits for new users, while leaving existing users limits as they are, we must store the project limits for each user. We currently store the limit for the number of projects a user can create in the user DB table. This change would also store the project bandwidth and storage limits in the same table.

Change-Id: If8d79b39de020b969f3445ef2fcc370e51d706c6
2021-11-11 15:10:00 +00:00
Ivan Fraixedes
4e67ea007c satellite/admin: Serve static UI assets
Change the satellite Admin HTTP server for:

* Embedding the UI assets into the Go binary.
* Serve the UI assets from the embedded file system or from a specific
  directory path through a configuration flag, without requiring
  authentication but keeping the authentication verification for the API
  endpoints.
* Add tests to verify that the UI assets are served without
  authentication.

Change-Id: I9003ac96f1ec585a189b67fc1cb315905403d557
2021-11-10 18:47:46 +02:00
Mya
bf51c286d9
satellite/geoip: update node check-in to associate a country code
Resolves https://github.com/storj/storj/issues/4247

Change-Id: Idfd71bf1795d48ca3c686066bbdb95b9c6594f00
2021-11-10 16:44:41 +01:00
Yaroslav Vorobiov
2ebdc1303f satellite/satellitedb: add disqualification reason to nodes table
Alter satellites DB nodes table to add `disqualification_reason` int column
which contains disqualification type enum of why node has been disqualified.

Change-Id: Ia514557018ca27e1984216dc5004346d59869d16
2021-11-10 14:25:07 +02:00
Michał Niewrzał
12d4dc16a1 satellite/{metainfo,metabase}: make metadata optional for CommitObject
To resolve problem with lack of ability to set metadata while MPU on
gateway we are adding setting metadata with BeginObject. This
change makes also metadata optional while CommitObject. We need
this functionality to not override metadata set with BeginObject in
case when metadata is not set with CommitObject.

Another reson is that we would like to not set metadata at all if user
didn't specify metadata. At the moment we always setting some bytes
for metadata fields e.g. empty EncryptedMetadata field can have key
and nonce set.

Change-Id: Ifee25b7718eb1f919119db9b698b29d8b5ebe2ec
2021-11-10 08:08:46 +00:00
Egon Elbre
1d5b3e508c satellite/admin/ui: fix rollup building
rollup does not handle virtual drives correctly, use preserveSymlinks to
avoid compilation issues.

Change-Id: I7ae9509572ff4e2e74b059db4909f2e612a1331b
2021-11-09 13:16:37 +00:00
Egon Elbre
8eebbf3d7d satellite/audit: fix TestReverify timeouts
Currently the slow db was sleeping for 1s and the timeout for audit was
1s. There's a slight chance that the timeout won't trigger on such a
small difference.

Increase the slow node sleep to 10x of the timeout.

Hopefully fixes #4268

Change-Id: Ifdab45141b3fc7c62bde11813dbc534b3255fe59
2021-11-09 13:16:29 +00:00
Egon Elbre
d5628740fd satellite/overlay/straynodes: prevent disqualification in tests
Currently the test threshold for unneeded node checkins are 30s.
The disqualification threshold was at 30s, which means, it was possible
for all the nodes to get disqualified.

Hopefully fixes #4267

Change-Id: I6b0a10c09b7fd90a9729794885c9e7a593781bad
2021-11-09 12:29:21 +00:00
Ivan Fraixedes
583cdc6e0e satellite/admin/ui: Create Web UI v1
This commit crate the Satellite Admin Web UI v1

* Bootstrap the files installing the project template:
  https://github.com/sveltejs/template
* Add the prettier prettier-plugin-svelte NPM packages as dev
  dependencies.
* Add an empty prettierrc.toml.
* Transform the project template to use Typescript rather than
  Javasript. See: https://github.com/sveltejs/template#using-typescript
* Replace the default favicon by the Storj logo.
* Create components in companion of some Typescript code which allows to
  generate a simple HTML page based on an opinionated definition of API.
* Implement all the Admin API client calls in the format that the UI
  generator components requires for rendering the UI from them.

Change-Id: I58fa586d68dc8998e5d89db169b8e90204f0a96d
2021-11-09 09:59:35 +00:00
Márton Elek
9bdcc415bc satellite/nodeselection: add geofencing constraints to the node selection criteria
Closes https://github.com/storj/storj/issues/4242
Change-Id: Ieda59a4f37c673e4e81abb4c89c09daf3199bbc7
2021-11-08 17:04:31 +00:00
Vitalii Shpital
3c683998f5 satellite/console, web/satellite: feature flag for new objects flow
Added a feature flag to switch between old and new objects flow

Change-Id: Id1802f9fb1230928445f16e4d0a62aecdc91c038
2021-11-08 11:45:22 +00:00
Erik van Velzen
91158fd3bf satellite: omit system metadata fields if not requested
To reduce database load, add option to omit as many object properties
as possible when listing objects.

Change-Id: I817633801b00629a4042d1d1bd2389ee581953de
2021-11-08 09:19:44 +01:00
Márton Elek
fb604be460 satellite/gracefulexit: use nodecache with gracefulexit
Change-Id: I700caf6dfd06bd3b3970a8cf7c5a79da4af27b5f
2021-11-05 19:14:14 +00:00
Qweder93
b314559a13 satellite/metainfo: TestBatch params updated
Removed PathCipher and DefaultSegmentSize from CreateBucketParams
since they are unused anymore and breaks Integration on uplink side

Change-Id: I1393a7f1f436940731aa59edd693043336383290
2021-11-05 14:53:55 +02:00
dlamarmorgan
1dd537953e satellite/metainfo: strip the uplink version from the UserAgent and bound its size
The UserAgent should be stored as is, with the exception of removing the trailing version from any libuplink user agents

Change-Id: If17ef2fc4b59480a3477300f2585a07d64cc2bf4
2021-11-04 10:09:54 -07:00
Elek, Márton
27091826ba satellite/{satellitedb|metabase}: add SQL fields for geofencing
Change-Id: Ia698d71db1d3775b908c24bd1c9bc8abfd015ccc
2021-11-04 08:49:56 +00:00
Egon Elbre
d043b9facd satellite/console: load index.html from disk when developing
This eases development by not needing to restart satellite after each
change.

Change-Id: I1063b7f431f675fb21e82dcfc2420310a9806422
2021-11-03 18:19:40 +02:00
Michał Niewrzał
e792727bea satellite/metabase: allow setting metadata with begin object requests
Turns out that S3 protocol is setting object metadata with initial
request, in our case it's BeginObject, so we need to modify metabase
methods to accept metadata at the beginning and at the end.
BeginObject methods will set metadata always (niled or not niled).

One additional improvment for metadata fields was introduced and it's
validating fields as optional if EncryptedMetadata was not set.

This change currently doesn't have any implications and  it's a base for
other changes. Metadata is not set with BeginObject metainfo endpoint
yet.

Change-Id: I1f768407bc3428500b0d30ee188257420d953001
2021-11-03 10:08:10 +00:00
Cameron Ayer
1de8a695e8 satellite/{overlay,satellitedb}: fix stray nodes DQ bug
We had a bug in the stray nodes chore where nodes who had not been seen
in several months were not being DQd. We figured out that this was
happening because we were using two queries: The first to grab
nodes where last_contact_success < some cutoff, the second to DQ them
unless last_contact_success == '0001-01-01 00:00:00+00'. The problem
is that if all of the nodes returned from the first query had
last_contact_success of '0001-01-01 00:00:00+00', we would pass them to
the second query which would not DQ them. This would result in the stray
nodes DQ loop ending since we found a number of nodes to DQ less than the
limit.

The fix: add the "WHERE last_contact_success != '0001-01-01
00:00:00+00'::timestamptz" to the selection query.

Change-Id: I4e60de90b68d8745d641b4467c2b23e0e56f7dff
2021-11-02 17:05:00 +00:00
Yingrong Zhao
774ae017e3 satellite/{satellitedb, web}: display object count in satellite UI
Even though we want to start charging segment fee instead of object fee,
it's hard for users to understand what a segment is. This PR adds the
object count back in the UI alongside with segment count to help address
the issue.

Change-Id: I92eb42c769d350eba68a72443deffec5c278359c
2021-11-01 18:26:13 +00:00
Egon Elbre
edb8d656de satellite/metainfo: adjust piecedeletion timeouts
Currently slower storagenodes can slow down deletion queue.
To make piece deletion faster reduce the maximum time spent in
either dialing or piece deletion requests.

With this change:
* dial timeout is 3s
* request timeout is 15s
* fail threshold is set to 10min

Similarly, we'll mark storage node as failed when the timeout occurs.
The timeout usually indicates that the storagenode is overwhelmed.
Garbage collection will ensure that the pieces get deleted eventually.

Change-Id: Iec5de699f5917905f5807140e2c3252088c6399b
2021-10-28 13:37:01 +03:00
Yingrong Zhao
d441c8da15 satellite: use segment count for billing
Change-Id: Iafaf41cb6cde4cb2abdac3f75ce0fead838cb758
2021-10-27 19:21:50 +00:00
Yingrong Zhao
52f8c8175e satellite/satellitedb: add segments column into invoiceprojectrecords
table and drop not null constraint on objects column

Since, we want to move from charging our customers by object count to
segment count, this PR prepares the database to be able to record segments count
instead of objects count for satellite's billing system

Change-Id: Ie91ef354e78d24a268bc1cdc4327c182f733321e
2021-10-27 14:51:51 -04:00
Márton Elek
20d03bebdb satellite/nodeselection: flexible interface to includes nodes in selection
This commit doesn't change any behavior, just organize the code in
different way to make it easier to implement different Criterias
to include nodes. Today we use NodeID and Subnet based selection
but later Criteria can be extended with different kind of
placement rules (like geofencing).

The change nodeselection is used by segment allocaton (upload) and repair
and excludes nodes from an in-memory selection.

Resolves https://github.com/storj/storj/issues/4240

Change-Id: I0c1955fe16a045e3b76d7e50b2e1f4575a7ff095
2021-10-26 11:01:33 +00:00
Michał Niewrzał
de38e2e7d8 satellite/metainfo: stop using sat StreamID Redundancy
We needed Redundancy insided sat StreamID when uplink was defining RS values. Now it can be removed.

Change-Id: Id37187493eaa00cf29cb0262a050d71add3deb96
2021-10-26 10:23:11 +00:00
paul cannon
eb68dbad41 satellite/satellitedb: fix ordering in listPendingTransitionShim
listPendingTransitionShim is a temporary transition shim intended to
make existing API processes keep working when a future DB schema change
is executed. For more explanation, see the message on commit c053bdbd70.

However, the shim has a small bug: it is missing the ORDER BY clause
that appears in the original ListPending method. This transition shim
code won't ever run until we make the DB schema change, so this bug
hasn't hurt anything yet; it's just important that we fix it before the
DB schema change happens.

Change-Id: I5953651583ee236500c2c07141dfc9d690a95118
2021-10-25 19:33:53 +00:00
Malcolm Bouzi
d90a1b09a2 satellite/console,satellitedb: add signup promo code column to users
This update is to set up users being able to register with a promo code added to their account in place of the free tier coupon.

Change-Id: I7badf87937b12664f145520b6dcc4b26fe750407
2021-10-25 18:08:00 +00:00
dlamarmorgan
01cfbde56b satellite/console: increase free tier project bandwidth and storage from 50 GB to 150 GB and reduce free tier max projects from 3 to 1
To make our free tier limits more clear, we will reduce the number of projects allowed from 3 to 1, and increase the storage and bandwidth limit of the free tier from 50 Gb to 150 GB. The total allotments across all projects for a given user are unchanged, just reduced to a single project.

Change-Id: Ic8dddb135f2b83a3f36e2b9fdcb477e351ec137b
2021-10-25 17:34:19 +00:00
crawter
3c7e289395 satellite/matainfo: handle metabase errors in metainfo endpoint in common way
We should improve the way how we are handling metabase errors in
metainfo endpoint.
https://storjlabs.atlassian.net/browse/PG-316

Change-Id: I1da6f333546cabf34d6eb1de8e94a3ef455d75d5
2021-10-25 16:30:24 +00:00
Ivan Fraixedes
df44e8152d
satellite/admin: Move declaration before first usage
Move a variable declaration just before where is used for making easy to
follow the implementation when reading it.

Change-Id: I3e93597956352037a63c5496b11576fdd2c600a6
2021-10-24 18:38:41 +02:00
Fadila Khadar
f654c3ae4c satellite/gracefulexit: drop unused column in graceful_exit_progress
We don't need column uses_segment_transfer_queue in graceful_exit_progress
as now all exiting nodes are using graceful_exit_segment_transfer_queue and
table graceful_exit_transfer_queue has been dropped.

Change-Id: I4b7c087433f04138cf09bcf8ad3d8de2c185502a
2021-10-22 16:41:27 +00:00
Moby von Briesen
1b22525002 satellite/payments: Fix discount when listing invoices.
The error message Stripe's API was sending:
"This property cannot be expanded (total_discount_amounts).
You may want to try expanding 'data.total_discount_amounts' instead."

Change-Id: I9f8cea4107d826d837755be2c3c04675a36f3c37
2021-10-22 10:16:27 -04:00
Qweder93
50baefa10e satellite/metabase: limit maximum number of parts and size
Multipart upload limits added. Last part has no size limit.
Max number of parts: 10000, min part size: 5 MiB

Change-Id: Ic2262ce25f989b34d92f662bde720d4c4d0dc93d
2021-10-22 10:16:37 +00:00
Egon Elbre
f2d8e97d97 satellite/satellitedb: simplify select nodes query construction
Change-Id: I07009b28762d4485929a2a999e8f4be8179bee51
2021-10-22 07:41:07 +00:00
Monty Anderson
3989107031 web/satellite: add feature flag for new browser
Change-Id: If3773b9df7f79d4c5e094140448d60ec060cdaeb
2021-10-22 03:49:08 +00:00
Michał Niewrzał
b32fbc0f93 satellite/metainfo: speedup deletion tests
Speedup is done by reducing number of testplanet instances
for tests without changing main test logic.

Change-Id: Ic3849485d37b8ca55c013a45b7191dce65b88b04
2021-10-21 15:54:27 +00:00
Egon Elbre
187941ff86 satellite/{metabase,metainfo}: add some missing monitoring
Change-Id: Ie0018f6385bb145d0a514dfe2792f3f299b55b79
2021-10-21 09:15:04 +00:00
Vitalii Shpital
0009447773 satellite/console, web/satellite: feature flag for new navigation structure
Change-Id: I6f1fd43ea8154714187c7547ae93de0efe3482e6
2021-10-20 13:51:25 +00:00
Moby von Briesen
9c8b6c3fc3 testsuite/ui: Add coupon tests
Add some testing around adding/replacing promotional coupons and what
the user sees in the UI depending on their coupon status.

Change-Id: Ice6c0a0644d05af0c30c87a93ba963c0bb09e32d
2021-10-19 19:08:35 -04:00
Brandon Iglesias
8d18033a99
satellite/rewards: adding SeaweedFS to partners list (#4230) 2021-10-19 21:30:31 +02:00
Mya Pitzeruse
d250f9dfee
satellite/core: fix label for orders chore (#4228) 2021-10-19 18:44:15 +02:00
Egon Elbre
51f488fc4b satellite/satellitedb: fix selection query with AOST
Union query for both reputable and new nodes didn't properly work. The
top level query is required to have an `AS OF SYSTEM TIME` statement as
well.

Change-Id: I8ee6dd5b700c2b1ed2aa562962bfa72be7eec30a
2021-10-19 16:59:40 +03:00
Michał Niewrzał
4a146000cc satellite/metainfo: read from DB only needed columns fro bucket
Uplink needs only part of columns we are reading from DB.
To improve performance we should read only those that are
realy needed.

Change-Id: Ib39259318169c46afe5fa4c6ce2184da82e960c8
2021-10-19 11:53:42 +00:00
Michał Niewrzał
8293c9659e satellite/metabase: speedup iterator tests
Main speedup is from reducing number of cases to iterate
in 'boundaries' test case.

Change-Id: I5bbf1311790f3d0e553d59e6a73e1cf3fa601a55
2021-10-19 11:52:07 +00:00
Michał Niewrzał
f5f2398431 satellite/metabase: remove code related to part deletion
It is not used currently.

Change-Id: I32e9fa357814a3159ee8659a607cd6986f361fbf
2021-10-19 10:11:49 +00:00
Moby von Briesen
4ad7056bf7 satellite/payments: Add old invoice list functionality
3b751a35c Removed our old coupon functionality, and slightly reworked
the invoice List() function in the stripecoinpayments package.

It turns out, this is causing some issues when trying to delete users.
This change keeps the new functionality, which is used in the satellite
UI, but under a new name, ListWithDiscounts()

Change-Id: I6a62a1de480e09d005dd22d75aa1e024fd2ed3a0
2021-10-18 18:18:18 -04:00
Yingrong Zhao
35e4a87e60 satellite/repair: ignore expired segments at the beginning of the repair
work

Since we have changed the repair worker to also mark a node as audit
failure if they return a not found error, we should ignore expired
segments when possible

Change-Id: Ie6a677e1d7b234e93965c736d05950440236653c
2021-10-18 18:15:39 +00:00
dlamarmorgan
4bbf667ad1 satellite/{satellitedb,attribution,console}: value attribution changes that add userAgent field to buckets table and all tables that have partner_id
Change-Id: I36a13bb651b86bfc14fe5a0a2258f719e6cd2b48
2021-10-18 13:56:19 +00:00
littleskunk
016ac75052
satellite/console: Enable new onboarding workflow (#4222) 2021-10-15 21:07:35 +02:00
Malcolm Bouzi
d7812a3b17 satellite/console,web/satellite: add slider and unit toggling to project limit updating
Change-Id: Iab7c08baa09d598ff53ddf09ef09dcd2a997a81b
2021-10-15 14:11:50 +00:00
Egon Elbre
52c950e429 testsuite/ui/uitest: add Edge testing
Change-Id: I1e592ac755b582d4fa73e95fd999a35a27f8a116
2021-10-15 13:27:13 +00:00
Michał Niewrzał
96dc43fd49 satellite/metainfo: disable DeletePart method
We can now overwrite segments in pending objects so
we should stop using this method.

Change-Id: Ib24bd53e1a73d8c380b89a994a3cd8d178f4c533
2021-10-15 12:20:50 +02:00
Cameron Ayer
56fe636123 satellite/{reputation/satellitedb}: remove references to contained column in reputations table
We don't use this column for anything. If you want to know if a node is
contained, you can check the pending_audits table.

Change-Id: I5671722a5fc6e1749d3a49e187a56556000ff941
2021-10-14 19:59:03 +00:00
Cameron Ayer
bb21551a9c satellite/satellitedb: remove references to contained column in nodes table
We don't use this column for anything. If you want to know if a node is
contained, you can check the pending_audits table.

Change-Id: I8da1d8e01a2dcaff63c5067a7927b5451424ad04
2021-10-14 19:17:46 +00:00
Jeff Wendling
91ce70e4ed satellite/metabase: allow overwriting segments in pending objects
this is so that multipart upload allows overwriting parts. our
consistency model is more relaxed because part overwrites
won't be atomic if they consist of multiple segments, but at
least this allows overwriting at all.

Change-Id: I21dac4c24046e05efe74e6c6fd189a02c95eb6c8
2021-10-14 10:33:52 +02:00
Moby von Briesen
5b729779a2 satellite/console: Automatically log a user in after verifying email
When an email is verified, insert an auth cookie so that when the user
is redirected after verifying their email, they are immediately taken to
the onboarding flow.

Change-Id: I557d8a2805b24dd8039ada255522bc1b56cc8b53
2021-10-13 13:08:27 +00:00
Michał Niewrzał
b8e8110ca3 mod: bump common and uplink dependency
Contains also adjustments for latest storj/common changes.

Change-Id: I11f70d466b3d9785ee93ac424629b2b868889d02
2021-10-12 19:13:49 +00:00
Michał Niewrzał
1fdb0eaa5b Revert "satellite/metabase: use storj.Nonce instead []byte"
This change introduce problems with server side move so
let's revert it for now. Problem was found when latest
version of storj/storj was used in uplink tests.

This reverts commit 1ef06fae99.

Change-Id: I4d4fad5d1ea04ba15ff9d7bd765f7e078e9187c2
2021-10-12 15:39:54 +02:00
Jeremy Wharton
3b751a35c5 satellite/{payments,satellitedb}: Remove custom coupon implementation
Removes database tables and functionality related to our custom
coupon implementation because it has been superseded by the Stripe
coupon and promo code system. Requires implementations of the
payments Invoices interface to return coupon usages along with
invoices.

Change-Id: Iac52d2ff64afca8cc4dbb2d1f20e6ad4b39ddfde
2021-10-11 19:47:00 +00:00
Michał Niewrzał
1ef06fae99 satellite/metabase: use storj.Nonce instead []byte
We were using mixed types for nonce fields. Protobuf
have storj.Nonce, metabase have []byte. This change
is a refactoring to have everywere its possible only
storj.Nonce.

Change-Id: Id54bd8481f30c721cdaf3df79206d25e7cfdab55
2021-10-11 16:13:34 +00:00
Egon Elbre
b7405db2a2 satellite/satellitedb: optimize migration test
Cockroach doesn't like concurrent database creation, add a limiter to
avoid overloading the database.

Also limit the cockroach testing to the last 10 migrations.

Change-Id: Ie73c516ac2d362b251f605049e51eb25888432bd
2021-10-11 18:31:20 +03:00
Fadila Khadar
f77f61532a satellite/orders: use egress_dead for calculating allocated bandwidth
Change-Id: I8240d99d0cdaad4c5e059565e88ee9619b62526e
2021-10-11 14:58:26 +02:00
Egon Elbre
9fd091831d ci: optimize benchmarks
We are not using the benchmark results for anything, they are mostly
there to ensure that we don't break the benchmarks. So we can disable
CockroachDB for them.

Similarly add short versions of other tests.

Also try to precompile test/benchmark code.

Change-Id: I60b501789f70c289af68c37a052778dc75ae2b69
2021-10-08 19:42:40 +03:00
Ivan Fraixedes
05960b2cf0
satellite/admin: Response 404 when entity not found
* Add test cases to verify that all the endpoint that target a specific
  entity respond 404 status code when the entity isn't found.
* Fix the endpoints that target a specific entity which responded a 500
  status code response when the entity didn't exist to respond with 404
  status code.

Additionally:

* Simplify some tests using an existing test helper function.
* Rename test functions to start with the entity name (e.g. Project,
  User, etc.) for easing to run a set of test with the `-run` Go test
  flag.

Change-Id: I82aad92e429207b72932ad4b79c08db6b486a19a
2021-10-07 12:42:25 +02:00
Fadila Khadar
fb0d055a41 satellites/orders: populate egress_dead in project_bandwidth_daily_rollups
Populate the egress_dead column for taking into account allocated bandwidth that can be removed because orders have been sent by the storage nodes. The bandwidth not used in these orders can be allocated again.

Change-Id: I78c333a03945cd7330aec052edd3562ec671118e
2021-10-06 16:54:49 +00:00
Michał Niewrzał
8a21a8cbc5 satellite/metainfo: don't enable connection pool if was setup earlier
It's safer to create new connection pool for piece deletion
only if dialer have no existing pool assigned.

Change-Id: I26661683ab7c0198587905478057c01c8f533a7e
2021-10-06 15:25:23 +00:00
Yingrong Zhao
5355928317 satellite/payments/stripecoinpayments: fix nil panic during storj token
deposit

Change-Id: I2294656aad475748b53d786b03c3810b338963bb
2021-10-05 12:13:14 -04:00
Michał Niewrzał
e5bb897364 satellite/internalpb: for StreamID rename EncryptedPath to EncryptedObjectKey
We should be using object naming insted of path.
This is one place where we can easiliy change it.

To regenerate protbuf I had to remove gogo.proto.
Most probably it was confilicting with gogo.proto
from common/pb.

Change-Id: Ia5972f77994765c8f26bf1c3dc8205d2eadd70fa
2021-10-05 15:15:49 +00:00
Michał Niewrzał
f93dc5a166 satellite/metainfo/piecedeletion: enable connection pool
We want to enable connection pool for piece deletion to avoid
doing multiple SSL hanshakes to SN while massive deletion process.

Change-Id: Ic917e4eda304ee16a286926ef046fe9e38bf38ca
2021-10-05 13:30:42 +00:00
Michał Niewrzał
aa80d37053 satellite/metabase: add automatic conversion to DB value for ObjectKey fields
We were manually converting ObjectKey fields to []byte to use it with
SQL query but we can just implement Value method to convert it automatically.

Change-Id: I6d346f4b59718e1e8ef37cd9f95e613b864b42cd
2021-10-05 07:34:00 +00:00
Yaroslav Vorobiov
4b79f5ea86 satellite/repair: test if audit scores increases during repair
Update repair tests to check if audit score increases for nodes
that successfully send pieces during successfull and failed repairs.

Change-Id: Ie6abbde6155ab4697d209366c9fa497e731756e9
2021-10-04 19:39:13 +00:00
Fadila Khadar
38ce0e154c satellite/gracefulexit: drop table graceful_exit_transfer_queue
Drop table graceful_exit_transfer_queue which is not used anymore (replaced by graceful_exit_segment_transfer_queue).

Change-Id: Ie254fe9a54fb0784e350a439ce7a9bc99a3a58b5
2021-10-04 17:26:44 +00:00
Michał Niewrzał
f2f8a9ca1e satellite/metabase: send metric about zombie segment deletion
We would like to verify if zombie object/segment works fine.
We need some metric for that. Figuring out number of deleted
objects is harder so let's for that later.

Change-Id: Ic99e2ce93256130b7c51f514824fddc009655075
2021-10-04 16:25:23 +00:00
paul cannon
5f444c1fee satellite/payments: fix currency rates acquisition
Change-Id: I4667b9d0131efe04f5429e932cbb0f0da8699409
2021-10-04 15:27:34 +00:00
Ivan Fraixedes
bef4f5ec86
satellite/admin: Validate new user data earlier
Validates the new user data, through `Validate` method, on "create user"
endpoint earlier and remove specific checks of data that is already
validated by the `Validate` method.

Change-Id: I6b8f91c0b706f0c8316173c491e25c4858172640
2021-10-04 12:13:05 +02:00