Commit Graph

6106 Commits

Author SHA1 Message Date
Jeff Wendling
baaa96c208 cmd/uplinkng: introduce MultiWriteHandle
Change-Id: I6acf93141ddfa62728164818a322120ed6956b00
2021-12-10 10:23:04 -05:00
Jeff Wendling
34890c9195 cmd/uplinkng: introduce MultiReadHandle
Change-Id: I57b98b5e1406e7b38edf3bc65907d9796a1a663b
2021-12-10 09:30:25 +00:00
Michał Niewrzał
c33d2f58cf satellite/metainfo: override StreamMeta only if needed
For backward compatibility we are overriding pb.StreamMeta
values returned as encrypted metadata. It turns out that we
should do it not when target values are missing but when
values to override exists.

This was causing problems after move operation. Details can be
found here https://github.com/storj/uplink/issues/70

Backward compatibility test will be added to storj/uplink testsuite.

Change-Id: I72e7a01226b1dd62902cb0d6ebb1ff91a4693005
2021-12-10 08:41:37 +00:00
prerna-parashar
a41758bba5
Track user events through HubSpot events API (#4300)
Track user events through HubSpot events API
2021-12-09 12:52:51 -08:00
Ethan Adams
27c6c6aeae
ci: Golangci lint v1.43.0 changes (#4307)
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2021-12-09 20:49:48 +01:00
Michał Niewrzał
000957f64a satellite/accounting: add functionality to get and cache project segment count
We want to set maximum number of segments per project. This change will
add functionality to get number of segments currently used by project.
To avoid often DB calls segment cound will be cached and refreshed every
few minutes.

Change-Id: I2ecb6484f5afc3875c0e0dfaea360e8872f9d196
2021-12-09 12:30:52 +00:00
Stefano Fornari
af3edf2b78
scripts/release.sh: making clear if builing a production or development version (#4304)
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2021-12-09 12:38:23 +01:00
Michał Niewrzał
d94d8d1775 satellite/accounting: expose project segment limit
Exposes functionality to get and update project segment
limit. It will be used to limit number of segments per project
while uploading object.

Change-Id: I971d48eebb4e7db8b01535c3091829e73437f48d
2021-12-08 11:27:21 +00:00
Michał Niewrzał
ace90fd844 mod: bump clingy version
Latest version have a fix for:
https://github.com/storj/storj/issues/4257

Change-Id: I0e3a67fe632cd5c209b24e1ee3080563cafbb626
2021-12-08 11:10:25 +01:00
dlamarmorgan
3b51eea312 satellite/metainfo: store empty useragent in bucket attribution instead of throwing error
Previously, only valid partner IDs could be used for bucket level value attribution. Now that any useragent byte slice can be used, we should allow for empty useragent strings to be stored rather than throwing an error or leaving the bucket with no attribution.

Change-Id: I7043f835588dab1c401a27e31afd74b6b5a3e44b
2021-12-07 18:06:27 +00:00
Michał Niewrzał
5013dd8a43 satellite/metainfo: remove error message prefix
Currently detecting if error NotFound is for object
or bucket is very fragile on uplink side. We cannot
change this messages for now.

Change-Id: I6ec4d98116477812f031134e4f1c9e73bdce8b27
2021-12-07 11:59:06 +00:00
Vitalii Shpital
15a7e7f35c web/satellite: fix delete bucket issue
User could confirm delete bucket action without entering bucket name.

fix for this issue https://github.com/storj/storj/issues/4296

Change-Id: Ic76f59885fd5bd812609f831f8887c07f02360a8
2021-12-06 15:00:54 +00:00
Qweder93
b5194762f2 cmd/uplinkng: share command added
In uplink we have command uplink share and we need
to port it to uplinkng to have command with same functionality.

Command could be executed with parameter or without.
without any flags we share as readonly.

Change-Id: I973b11d00da237358834acf5a863ebab37e684cc
2021-12-02 13:02:14 +00:00
Michał Niewrzał
73730b23e7 satellite/satellitedb: add segment_limit colum to projects table
We want to set maximum number of segments per
project. This change adds only column to projects table.
Default value 1M is set to make later migration easier as
we need to set 1M for paid tier users and 140K for free
tier users.

Change-Id: I8e83712e08c5bd91dfa59f652d17e45c14240a36
2021-12-02 12:15:16 +00:00
dlamarmorgan
b3cea3d1b6 satellite/audit: account for piece size during audit reservoir sampling
Treat the piece size as a weight, and perform weighted reservoir sampling as given in Algorithm A-Chao (https://en.wikipedia.org/wiki/Reservoir_sampling#Algorithm_A-Chao)

Change-Id: I299d0026d9e02d03b3d2130b0f32192928e6e326
2021-12-01 18:17:52 +00:00
Vitalii Shpital
82fb0fce04 web/satellite: added charts date range picker for new project dashboard
Created common VDateRangePicker component to be reused.
Reworked it's styling.
Added date range selection logic for new project dashboard.

Change-Id: Ie9bdd173527514924e437ca5bcc9cfbf7793e4dd
2021-12-01 15:55:36 +00:00
Vitalii Shpital
be10ce84f8 web/satellite: implemented charts UI for new project dashboard
Added bandwidth/storage charts (with test data) to new project dashboard.
Added functional buttons to new project dashboard.
Fixed this issue https://github.com/storj/storj/issues/4262.

Change-Id: Ie87370b8f7b6015bc84022a6086ef1db40e16535
2021-12-01 15:55:20 +00:00
Vitalii Shpital
b78f65e83b satellite/console, web/satellite: added object and segment count per project info to new project dashboard
Added new query to get project object and segment count.
Added appropriate object and segment count view for new project dashboard.

Change-Id: I69a2e55442f318c51dc365c0c578b964f2f06c7f
2021-12-01 17:46:07 +02:00
Vitalii Shpital
16c8a08014 web/satellite: new project dashboard routing updates
Updated routes to handle new project dashboard.

Change-Id: I2e96d4eb3e9acb6bd998e6b9cfb8e7726de7ccdf
2021-12-01 10:34:03 +00:00
Ivan Fraixedes
bde8fb0a59
satellite/admin: Move geofence endpoint to bucket info
Move an endpoint that was classified to return the geofence
configuration of a bucket to return the bucket information, which
also include the geofence configuration, because it's what it was
returning.

Change-Id: I0e0a6aac330296383a50a92d2352df9088df77d5
2021-11-30 18:47:14 +01:00
Ivan Fraixedes
bae8276f3f
satellite/admin/ui: Add missing geofencing endpoints
The satellite admin server was augmented with 3 new endpoints to manage
buckets geofencing configurations, however, they weren't created in the
admin UI API class for making them available in the web interface.

This commit adds these new endpoints to the admin UI.

Change-Id: If060d1f10a3bc6c365e16a891673d4ffc89e4b41
2021-11-30 18:04:22 +01:00
crawter
eec5ad4f5d cmd/uplinkng: access inspect command added
In uplink we have command uplink access inspect and we need
a command with the same functionality for uplinkng.

Command could be executed with parameter or without.
without parameter - we should show default access.
If parameter exists - it could be access name or value.
If access name or value is wrong - we show error.
F.e.
uplinkng access inspect
uplinkng access inspect accessName
uplinkng access inspect accessValue
https://storjlabs.atlassian.net/browse/PG-318

Change-Id: I85fd961283850feb8684db2d126441f6b9bf0270
2021-11-30 12:45:04 +00:00
Jeremy Wharton
984792fd1e satellite/satellitedb: Add GetByEmailWithUnverified to users table
Allows us to handle duplicate emails better.

Change-Id: I266057900725e50d1c47977da307714fd32d9081
2021-11-29 20:35:47 +00:00
Cameron Ayer
9c1129b4c4 Revert "satellite/satellitedb: migrate partner_id db column to user_agent db column"
This reverts commit 2c0a360a14.

Avoid big transactions. We'll do it outside of the migration pipeline.

Change-Id: Iade810d81bb2453c9e351149cb84662b207ee527
2021-11-29 10:35:57 -05:00
Igor
4daaa9503e
satellite/console: enable object flow by default (#4298) 2021-11-29 16:43:37 +02:00
Kaloyan Raev
99deec47b6 satellite/metainfo: consistent error message for bucket not found
We should be always returning the name of the bucket that has not been
found.

Change-Id: I7df76e5588c034b83a0702c649becd770858ff32
2021-11-29 07:57:15 +00:00
dlamarmorgan
2c0a360a14 satellite/satellitedb: migrate partner_id db column to user_agent db column
Value attribution codes were converted into UUIDs and stored in the users, projects, api_keys, bucket_metainfos, and value_attributions tables in the partner_id column. This migration will lookup the appropriate partner name associated with each of these UUIDs, and store the partner name directly in the user_agent column within each table. If an error occurs during the partner ID to partner name conversion, the partner ID value will be migrated to user_agent.

A note on the migration test data, postgres.v182.sql:
With one exception, all preexisting rows in the relevant tables had a NULL partner_ID. Therefore, we needed to insert new rows with partner_ID set under the OLD DATA section in order to test that the migration works. For each affected table, we insert one row with a valid partner ID which has a corresponding partner name, and one row with a partner ID which would return an error during the conversion to the partner name.

Change-Id: Iad977d72df0ce95a0c5ca80a065c4276ec1f2354
2021-11-24 22:43:22 +00:00
Michał Niewrzał
de8464e848 satellite/metabase: add GetProjectSegmentCount method
This new method will be used with mechanism to limit number
of segments per project, similar to limiting buckets or bandwidth.
This is only one of multiple changes we will do to implement this
limitation.

Change-Id: Ia516c2a006ad1d7b4431d780679be9d809848f4b
2021-11-24 13:20:02 +00:00
Ivan Fraixedes
1d14e4ef60
satellite/metainfo: Check ctx in download Object & Segment
Check if the context is done at the beginning of the download object and
segment and return right away if it's the case.

Check if Redis returned an error due to context cancellation for not
logging the error.

Change some logging messages of Redis errors to reflect on them if the
error happens while downloading an object or a segment.

Change-Id: I8ed8ff9ff7bb170b560f41356ea06820ce6c4e12
2021-11-24 12:54:11 +01:00
Kaloyan Raev
7d84482381 satellite/metainfo: prevent moving objects between geofenced buckets
Initially, we want to avoid moving objects between buckets with
different geofencing config.

Change-Id: I0cc497873eb5732623ef2d9bc5f78ba1cc48c6b8
2021-11-24 08:34:22 +00:00
Márton Elek
76c2228fbd satellite/metainfo: propagate geofencing between buckets and stream id
Github: https://github.com/storj/storj/issues/4245

Change-Id: I83d34367aab1f3c0d46a044f54980b2d50174b19
2021-11-24 08:05:05 +00:00
Egon Elbre
1fd14e291f testsuite/ui/uitest: fixes for uitest
Enable leakless for killing processes. Maybe it'll prevent issues with
the jenkins stage stalling.

Unify some of the code.

Set the browser tests non-parallel again, it might be causing some of
the issues.

Change-Id: Idb123bff3640e8adab5db0917ee774f0cf7e2568
2021-11-23 15:22:53 +00:00
Egon Elbre
788d775f6a web/satellite: fix browser progress-bar visibility
When a small value is displayed then the progress wasn't visible.
Add a drop-shadow for a quick-fix and set the overflow property to the
correct value. Currently both need to use !important hack since
the old browser is being imported and the css properties conflict.

Change-Id: I2a2d62f9b16a9ef26aaec91e82f5c05cc33dee23
2021-11-22 14:56:34 +02:00
Egon Elbre
88904301b2 web/satellite/static: fix 404 and 500 page logo link
Change-Id: Ib22b399b142a8cf6622f24e101d9096553e5e25b
2021-11-22 14:17:07 +02:00
Michał Niewrzał
8f07abafe0 cmd/uplinkng/ultest: make possible to test access commands
Test external implementation doesn't support OpenAccess
method. This makes imposible to test output of commands like
inspect or share. This implements only basic functionality.

Change-Id: I127ef0bb45a01634bd5265ed80840f8095c72794
2021-11-22 10:43:02 +00:00
Michał Niewrzał
24cf7e8ea6 cmd/uplinkng: add mv command
Add ability to move files and objects.

Change-Id: I4929da730984c06aa578678b1d8c8e9b4aceade8
2021-11-22 09:07:24 +00:00
littleskunk
07fad75912
storagenode/piecestore: upload and download metrics for Grafana alerts (#4280)
* storagenode/piecestore: Add upload and download metrics for Grafana alerts

* storagenode/piecestore: group download metrics by piece action

Change-Id: Ib2a42b60c56c3f581915d512f4907c8db71e4624

Co-authored-by: Clement Sam <clement@storj.io>
2021-11-18 12:50:39 +00:00
TungHoang
f9b630b0f4
Fix monthly earning estimation (#4282) 2021-11-17 18:26:21 -05:00
Yingrong Zhao
336500c04d satellite/repair: only record audit result if segment can be downloaded
If satellite can't find enough nodes to successfully download a segment,
it probably is not the fault of storage nodes.

Change-Id: I681f66056df0bb940da9edb3a7dbb3658c0a56cb
2021-11-17 15:25:43 +00:00
Vitalii Shpital
3de7f8d5af satellite/console, web/satellite: feature flag for new project dashboard
Added a feature flag to switch between old and new project dashboard

Change-Id: Ibb1b04ccbddb43e3f41116e3513f917deba95699
2021-11-17 16:02:47 +02:00
Vitalii Shpital
b7a5b1f3ab web/satellites: fixes for new objects flow
Fixed issue with project selection for new objects flow.
Fixed issue with navigating from objects browser when upload is in progress.
Fixed command line solution link on Encryption passphrase step.
Renamed Objects nav route to be Buckets nav route.

Change-Id: I4d838854668d6c44491cfd83302ea705ef7c26bd
2021-11-17 11:31:50 +02:00
Artur M. Wolff
89639199fe satellite/repair/repairer: remove unused healthyMap
This change removes unused healthyMap from (*SegmentRepairer).Repair.

Change-Id: Ie80eefdb5b7125bf70986cb13462eee737af214c
2021-11-16 16:49:18 +01:00
Kaloyan Raev
2e363707c5 satellite/buckets: tests for bucket placement config
Tests for checking different scenarios for enabling and disabling
geofence config on empty and non-empty buckets.

Change-Id: I0fe9abb1008d2daee660f22ffe4defe6226b9aa7
2021-11-16 14:58:56 +02:00
Kaloyan Raev
f773bb80c8 mod: bump common to fetch latest placement type changes
Change-Id: I3d0813f05622e706c9be1a578b5e4d4159d16dfc
2021-11-16 12:42:25 +00:00
Mya
a038fc1dc4 satellite/admin: add endpoints for managing a buckets geofence
Change-Id: I407243b8b9c769d968e70478c45230d9fdecaeb2
2021-11-16 12:42:06 +00:00
Mya
814e3126fa satellite/buckets: add new buckets service
The main motivation is to wrap the bucket DB and metainfo DB, so we
could check if a bucket is empty before applying geofencing config.

Change-Id: I8bac21555e01d51a663fb557bc1acfc8106bc2e1
2021-11-16 12:36:17 +02:00
andriikotko
09568b3e2b
satellite/console: change default feature flag (#4274)
* satellite/console: change default feature flag
2021-11-16 11:38:42 +02:00
Egon Elbre
dfd2977a01 testsuite/ui: fixes to setup
* Remove "enable-logging", because it ends up spawning consoles on
  Windows.
* Remove "disable-gpu", if we have a GPU, then let's use it.
* Create custom client, so we can attach logging to CDP.
* Ignore potential context.Canceled.
* Fix onboarding wizard test for new objects browser.
* Return an error on a context cancellation.
* Wire all loggers to planet.

Change-Id: I67eb138ba31252f55ac5b383679d033bcf71f1b2
2021-11-15 15:56:35 +02:00
Elek, Márton
debb82f596
Geofencing and advanced placement-constraint support (#4227) 2021-11-15 15:23:41 +02:00
Ivan Fraixedes
cb499cb7db
Makefile: Fix target builds the satellite admin UI
Fix the Makefile target which build the satellite admin UI adding a
missed npm command which must run before the npm run build command.

Change-Id: I96c9b0bd2ad2d909fc680db2b22a6cf13931111a
2021-11-12 13:29:55 +01:00