Commit Graph

6094 Commits

Author SHA1 Message Date
Qweder93
b5194762f2 cmd/uplinkng: share command added
In uplink we have command uplink share and we need
to port it to uplinkng to have command with same functionality.

Command could be executed with parameter or without.
without any flags we share as readonly.

Change-Id: I973b11d00da237358834acf5a863ebab37e684cc
2021-12-02 13:02:14 +00:00
Michał Niewrzał
73730b23e7 satellite/satellitedb: add segment_limit colum to projects table
We want to set maximum number of segments per
project. This change adds only column to projects table.
Default value 1M is set to make later migration easier as
we need to set 1M for paid tier users and 140K for free
tier users.

Change-Id: I8e83712e08c5bd91dfa59f652d17e45c14240a36
2021-12-02 12:15:16 +00:00
dlamarmorgan
b3cea3d1b6 satellite/audit: account for piece size during audit reservoir sampling
Treat the piece size as a weight, and perform weighted reservoir sampling as given in Algorithm A-Chao (https://en.wikipedia.org/wiki/Reservoir_sampling#Algorithm_A-Chao)

Change-Id: I299d0026d9e02d03b3d2130b0f32192928e6e326
2021-12-01 18:17:52 +00:00
Vitalii Shpital
82fb0fce04 web/satellite: added charts date range picker for new project dashboard
Created common VDateRangePicker component to be reused.
Reworked it's styling.
Added date range selection logic for new project dashboard.

Change-Id: Ie9bdd173527514924e437ca5bcc9cfbf7793e4dd
2021-12-01 15:55:36 +00:00
Vitalii Shpital
be10ce84f8 web/satellite: implemented charts UI for new project dashboard
Added bandwidth/storage charts (with test data) to new project dashboard.
Added functional buttons to new project dashboard.
Fixed this issue https://github.com/storj/storj/issues/4262.

Change-Id: Ie87370b8f7b6015bc84022a6086ef1db40e16535
2021-12-01 15:55:20 +00:00
Vitalii Shpital
b78f65e83b satellite/console, web/satellite: added object and segment count per project info to new project dashboard
Added new query to get project object and segment count.
Added appropriate object and segment count view for new project dashboard.

Change-Id: I69a2e55442f318c51dc365c0c578b964f2f06c7f
2021-12-01 17:46:07 +02:00
Vitalii Shpital
16c8a08014 web/satellite: new project dashboard routing updates
Updated routes to handle new project dashboard.

Change-Id: I2e96d4eb3e9acb6bd998e6b9cfb8e7726de7ccdf
2021-12-01 10:34:03 +00:00
Ivan Fraixedes
bde8fb0a59
satellite/admin: Move geofence endpoint to bucket info
Move an endpoint that was classified to return the geofence
configuration of a bucket to return the bucket information, which
also include the geofence configuration, because it's what it was
returning.

Change-Id: I0e0a6aac330296383a50a92d2352df9088df77d5
2021-11-30 18:47:14 +01:00
Ivan Fraixedes
bae8276f3f
satellite/admin/ui: Add missing geofencing endpoints
The satellite admin server was augmented with 3 new endpoints to manage
buckets geofencing configurations, however, they weren't created in the
admin UI API class for making them available in the web interface.

This commit adds these new endpoints to the admin UI.

Change-Id: If060d1f10a3bc6c365e16a891673d4ffc89e4b41
2021-11-30 18:04:22 +01:00
crawter
eec5ad4f5d cmd/uplinkng: access inspect command added
In uplink we have command uplink access inspect and we need
a command with the same functionality for uplinkng.

Command could be executed with parameter or without.
without parameter - we should show default access.
If parameter exists - it could be access name or value.
If access name or value is wrong - we show error.
F.e.
uplinkng access inspect
uplinkng access inspect accessName
uplinkng access inspect accessValue
https://storjlabs.atlassian.net/browse/PG-318

Change-Id: I85fd961283850feb8684db2d126441f6b9bf0270
2021-11-30 12:45:04 +00:00
Jeremy Wharton
984792fd1e satellite/satellitedb: Add GetByEmailWithUnverified to users table
Allows us to handle duplicate emails better.

Change-Id: I266057900725e50d1c47977da307714fd32d9081
2021-11-29 20:35:47 +00:00
Cameron Ayer
9c1129b4c4 Revert "satellite/satellitedb: migrate partner_id db column to user_agent db column"
This reverts commit 2c0a360a14.

Avoid big transactions. We'll do it outside of the migration pipeline.

Change-Id: Iade810d81bb2453c9e351149cb84662b207ee527
2021-11-29 10:35:57 -05:00
Igor
4daaa9503e
satellite/console: enable object flow by default (#4298) 2021-11-29 16:43:37 +02:00
Kaloyan Raev
99deec47b6 satellite/metainfo: consistent error message for bucket not found
We should be always returning the name of the bucket that has not been
found.

Change-Id: I7df76e5588c034b83a0702c649becd770858ff32
2021-11-29 07:57:15 +00:00
dlamarmorgan
2c0a360a14 satellite/satellitedb: migrate partner_id db column to user_agent db column
Value attribution codes were converted into UUIDs and stored in the users, projects, api_keys, bucket_metainfos, and value_attributions tables in the partner_id column. This migration will lookup the appropriate partner name associated with each of these UUIDs, and store the partner name directly in the user_agent column within each table. If an error occurs during the partner ID to partner name conversion, the partner ID value will be migrated to user_agent.

A note on the migration test data, postgres.v182.sql:
With one exception, all preexisting rows in the relevant tables had a NULL partner_ID. Therefore, we needed to insert new rows with partner_ID set under the OLD DATA section in order to test that the migration works. For each affected table, we insert one row with a valid partner ID which has a corresponding partner name, and one row with a partner ID which would return an error during the conversion to the partner name.

Change-Id: Iad977d72df0ce95a0c5ca80a065c4276ec1f2354
2021-11-24 22:43:22 +00:00
Michał Niewrzał
de8464e848 satellite/metabase: add GetProjectSegmentCount method
This new method will be used with mechanism to limit number
of segments per project, similar to limiting buckets or bandwidth.
This is only one of multiple changes we will do to implement this
limitation.

Change-Id: Ia516c2a006ad1d7b4431d780679be9d809848f4b
2021-11-24 13:20:02 +00:00
Ivan Fraixedes
1d14e4ef60
satellite/metainfo: Check ctx in download Object & Segment
Check if the context is done at the beginning of the download object and
segment and return right away if it's the case.

Check if Redis returned an error due to context cancellation for not
logging the error.

Change some logging messages of Redis errors to reflect on them if the
error happens while downloading an object or a segment.

Change-Id: I8ed8ff9ff7bb170b560f41356ea06820ce6c4e12
2021-11-24 12:54:11 +01:00
Kaloyan Raev
7d84482381 satellite/metainfo: prevent moving objects between geofenced buckets
Initially, we want to avoid moving objects between buckets with
different geofencing config.

Change-Id: I0cc497873eb5732623ef2d9bc5f78ba1cc48c6b8
2021-11-24 08:34:22 +00:00
Márton Elek
76c2228fbd satellite/metainfo: propagate geofencing between buckets and stream id
Github: https://github.com/storj/storj/issues/4245

Change-Id: I83d34367aab1f3c0d46a044f54980b2d50174b19
2021-11-24 08:05:05 +00:00
Egon Elbre
1fd14e291f testsuite/ui/uitest: fixes for uitest
Enable leakless for killing processes. Maybe it'll prevent issues with
the jenkins stage stalling.

Unify some of the code.

Set the browser tests non-parallel again, it might be causing some of
the issues.

Change-Id: Idb123bff3640e8adab5db0917ee774f0cf7e2568
2021-11-23 15:22:53 +00:00
Egon Elbre
788d775f6a web/satellite: fix browser progress-bar visibility
When a small value is displayed then the progress wasn't visible.
Add a drop-shadow for a quick-fix and set the overflow property to the
correct value. Currently both need to use !important hack since
the old browser is being imported and the css properties conflict.

Change-Id: I2a2d62f9b16a9ef26aaec91e82f5c05cc33dee23
2021-11-22 14:56:34 +02:00
Egon Elbre
88904301b2 web/satellite/static: fix 404 and 500 page logo link
Change-Id: Ib22b399b142a8cf6622f24e101d9096553e5e25b
2021-11-22 14:17:07 +02:00
Michał Niewrzał
8f07abafe0 cmd/uplinkng/ultest: make possible to test access commands
Test external implementation doesn't support OpenAccess
method. This makes imposible to test output of commands like
inspect or share. This implements only basic functionality.

Change-Id: I127ef0bb45a01634bd5265ed80840f8095c72794
2021-11-22 10:43:02 +00:00
Michał Niewrzał
24cf7e8ea6 cmd/uplinkng: add mv command
Add ability to move files and objects.

Change-Id: I4929da730984c06aa578678b1d8c8e9b4aceade8
2021-11-22 09:07:24 +00:00
littleskunk
07fad75912
storagenode/piecestore: upload and download metrics for Grafana alerts (#4280)
* storagenode/piecestore: Add upload and download metrics for Grafana alerts

* storagenode/piecestore: group download metrics by piece action

Change-Id: Ib2a42b60c56c3f581915d512f4907c8db71e4624

Co-authored-by: Clement Sam <clement@storj.io>
2021-11-18 12:50:39 +00:00
TungHoang
f9b630b0f4
Fix monthly earning estimation (#4282) 2021-11-17 18:26:21 -05:00
Yingrong Zhao
336500c04d satellite/repair: only record audit result if segment can be downloaded
If satellite can't find enough nodes to successfully download a segment,
it probably is not the fault of storage nodes.

Change-Id: I681f66056df0bb940da9edb3a7dbb3658c0a56cb
2021-11-17 15:25:43 +00:00
Vitalii Shpital
3de7f8d5af satellite/console, web/satellite: feature flag for new project dashboard
Added a feature flag to switch between old and new project dashboard

Change-Id: Ibb1b04ccbddb43e3f41116e3513f917deba95699
2021-11-17 16:02:47 +02:00
Vitalii Shpital
b7a5b1f3ab web/satellites: fixes for new objects flow
Fixed issue with project selection for new objects flow.
Fixed issue with navigating from objects browser when upload is in progress.
Fixed command line solution link on Encryption passphrase step.
Renamed Objects nav route to be Buckets nav route.

Change-Id: I4d838854668d6c44491cfd83302ea705ef7c26bd
2021-11-17 11:31:50 +02:00
Artur M. Wolff
89639199fe satellite/repair/repairer: remove unused healthyMap
This change removes unused healthyMap from (*SegmentRepairer).Repair.

Change-Id: Ie80eefdb5b7125bf70986cb13462eee737af214c
2021-11-16 16:49:18 +01:00
Kaloyan Raev
2e363707c5 satellite/buckets: tests for bucket placement config
Tests for checking different scenarios for enabling and disabling
geofence config on empty and non-empty buckets.

Change-Id: I0fe9abb1008d2daee660f22ffe4defe6226b9aa7
2021-11-16 14:58:56 +02:00
Kaloyan Raev
f773bb80c8 mod: bump common to fetch latest placement type changes
Change-Id: I3d0813f05622e706c9be1a578b5e4d4159d16dfc
2021-11-16 12:42:25 +00:00
Mya
a038fc1dc4 satellite/admin: add endpoints for managing a buckets geofence
Change-Id: I407243b8b9c769d968e70478c45230d9fdecaeb2
2021-11-16 12:42:06 +00:00
Mya
814e3126fa satellite/buckets: add new buckets service
The main motivation is to wrap the bucket DB and metainfo DB, so we
could check if a bucket is empty before applying geofencing config.

Change-Id: I8bac21555e01d51a663fb557bc1acfc8106bc2e1
2021-11-16 12:36:17 +02:00
andriikotko
09568b3e2b
satellite/console: change default feature flag (#4274)
* satellite/console: change default feature flag
2021-11-16 11:38:42 +02:00
Egon Elbre
dfd2977a01 testsuite/ui: fixes to setup
* Remove "enable-logging", because it ends up spawning consoles on
  Windows.
* Remove "disable-gpu", if we have a GPU, then let's use it.
* Create custom client, so we can attach logging to CDP.
* Ignore potential context.Canceled.
* Fix onboarding wizard test for new objects browser.
* Return an error on a context cancellation.
* Wire all loggers to planet.

Change-Id: I67eb138ba31252f55ac5b383679d033bcf71f1b2
2021-11-15 15:56:35 +02:00
Elek, Márton
debb82f596
Geofencing and advanced placement-constraint support (#4227) 2021-11-15 15:23:41 +02:00
Ivan Fraixedes
cb499cb7db
Makefile: Fix target builds the satellite admin UI
Fix the Makefile target which build the satellite admin UI adding a
missed npm command which must run before the npm run build command.

Change-Id: I96c9b0bd2ad2d909fc680db2b22a6cf13931111a
2021-11-12 13:29:55 +01:00
Ivan Fraixedes
07e07d3a2d satellite/metainfo: Add project ID to error logs
Add the project ID to error logs messages about not being able to
retrieve and track storage or bandwidth usage.

Some error logs related to these errors already contained the project ID
but others didn't, having it in the ones that didn't make them more
consistent and will provide more info which may be helpful for
troubleshooting them.

Change-Id: Ia9fc707a7f3aff0867645bb941badc199c2bf832
2021-11-12 11:28:08 +00:00
Ivan Fraixedes
ab425d4248 build: Consider the Admin UI assets
Add into the Makefile and Jenkins the operations to build the Admin UI
assets and lint them.

Change-Id: I687c1adec796fd591664e34eb73a465e53096b58
2021-11-12 08:48:28 +00:00
Michał Niewrzał
289ae3325a satellite/metabase: always include encryption in listed object
We implemented filtering of system metadata to
reduce DB load in case user will need only list of
objects but we missed that encryption configuration
is needed to decrypt object key.

This change is always including encryption configuration into
list objects results.

Change-Id: Iaf7b3d8441480e3ad787f19a28f1b648584d5f7d
2021-11-12 05:08:01 +00:00
Gabriel De Almeida
5ed3846e16 testsuite/ui/satellite: add browser tests
Also, we have multiple tests that contain the same sign up
and login logic and refactoring them would make tests
cleaner and more readable.

Change-Id: I25bf95ca2fffb7f494a3c93a4a4afe429ece70d3
2021-11-11 15:03:07 -05:00
Ivan Fraixedes
6dffbd4173 satellite/metainfo: Log in warn level limits exceeds
Don't log as an Error when users make request that cannot be fulfilled
because they exceed their storage or bandwidth limits.

Those are not system errors and the service is handling them correctly
and showing them in the logs an "ERROR" is misleading.

Change-Id: Iac642b7e8ba92840bb943192ad0694b5f4930258
2021-11-11 15:53:18 +00:00
dlamarmorgan
44b1ca6b97 satellite/{console,satellitedb}: move project limits from config file to DB to keep limits on a per user basis
To allow for changing limits for new users, while leaving existing users limits as they are, we must store the project limits for each user. We currently store the limit for the number of projects a user can create in the user DB table. This change would also store the project bandwidth and storage limits in the same table.

Change-Id: If8d79b39de020b969f3445ef2fcc370e51d706c6
2021-11-11 15:10:00 +00:00
Vitalii Shpital
b41e140b26 web/satellite: fixed dropdowns weird behavior
Reworked click-outside behavior.
Fixed bucket names selection and date picker dropdowns.

Change-Id: I1d798139e6bf713650b7071e5a0547d5d576c7a6
2021-11-11 15:45:18 +02:00
Egon Elbre
96058be1a7 storagenode/storagenodedb: fix error message
Change-Id: Ibc98aa6539199f8dc8117f21a6ae645cea419403
2021-11-10 17:46:30 +00:00
Egon Elbre
58916d8ba9 ci: don't trigger UI tests when go.mod changes
Currently testsuite/go.mod is being updated automatically and
we don't want to trigger UI tests, since they are a bit flaky.

Change-Id: Ie1c847e262ad723a006071f7a8f5cd7f26f4244e
2021-11-10 17:16:08 +00:00
Ivan Fraixedes
4e67ea007c satellite/admin: Serve static UI assets
Change the satellite Admin HTTP server for:

* Embedding the UI assets into the Go binary.
* Serve the UI assets from the embedded file system or from a specific
  directory path through a configuration flag, without requiring
  authentication but keeping the authentication verification for the API
  endpoints.
* Add tests to verify that the UI assets are served without
  authentication.

Change-Id: I9003ac96f1ec585a189b67fc1cb315905403d557
2021-11-10 18:47:46 +02:00
Egon Elbre
af5b90ed32 storagenode/storagenodedb: include dbname in error
When something happens during opening or closing then it wasn't clear
which database had the issue.

Fixes storj/storj#4271

Change-Id: I34c8bae79a5b41ccd9b40aa8d836805f8c1a573c
2021-11-10 16:25:34 +00:00
Mya
bf51c286d9
satellite/geoip: update node check-in to associate a country code
Resolves https://github.com/storj/storj/issues/4247

Change-Id: Idfd71bf1795d48ca3c686066bbdb95b9c6594f00
2021-11-10 16:44:41 +01:00