Commit Graph

6068 Commits

Author SHA1 Message Date
Yingrong Zhao
336500c04d satellite/repair: only record audit result if segment can be downloaded
If satellite can't find enough nodes to successfully download a segment,
it probably is not the fault of storage nodes.

Change-Id: I681f66056df0bb940da9edb3a7dbb3658c0a56cb
2021-11-17 15:25:43 +00:00
Vitalii Shpital
3de7f8d5af satellite/console, web/satellite: feature flag for new project dashboard
Added a feature flag to switch between old and new project dashboard

Change-Id: Ibb1b04ccbddb43e3f41116e3513f917deba95699
2021-11-17 16:02:47 +02:00
Vitalii Shpital
b7a5b1f3ab web/satellites: fixes for new objects flow
Fixed issue with project selection for new objects flow.
Fixed issue with navigating from objects browser when upload is in progress.
Fixed command line solution link on Encryption passphrase step.
Renamed Objects nav route to be Buckets nav route.

Change-Id: I4d838854668d6c44491cfd83302ea705ef7c26bd
2021-11-17 11:31:50 +02:00
Artur M. Wolff
89639199fe satellite/repair/repairer: remove unused healthyMap
This change removes unused healthyMap from (*SegmentRepairer).Repair.

Change-Id: Ie80eefdb5b7125bf70986cb13462eee737af214c
2021-11-16 16:49:18 +01:00
Kaloyan Raev
2e363707c5 satellite/buckets: tests for bucket placement config
Tests for checking different scenarios for enabling and disabling
geofence config on empty and non-empty buckets.

Change-Id: I0fe9abb1008d2daee660f22ffe4defe6226b9aa7
2021-11-16 14:58:56 +02:00
Kaloyan Raev
f773bb80c8 mod: bump common to fetch latest placement type changes
Change-Id: I3d0813f05622e706c9be1a578b5e4d4159d16dfc
2021-11-16 12:42:25 +00:00
Mya
a038fc1dc4 satellite/admin: add endpoints for managing a buckets geofence
Change-Id: I407243b8b9c769d968e70478c45230d9fdecaeb2
2021-11-16 12:42:06 +00:00
Mya
814e3126fa satellite/buckets: add new buckets service
The main motivation is to wrap the bucket DB and metainfo DB, so we
could check if a bucket is empty before applying geofencing config.

Change-Id: I8bac21555e01d51a663fb557bc1acfc8106bc2e1
2021-11-16 12:36:17 +02:00
andriikotko
09568b3e2b
satellite/console: change default feature flag (#4274)
* satellite/console: change default feature flag
2021-11-16 11:38:42 +02:00
Egon Elbre
dfd2977a01 testsuite/ui: fixes to setup
* Remove "enable-logging", because it ends up spawning consoles on
  Windows.
* Remove "disable-gpu", if we have a GPU, then let's use it.
* Create custom client, so we can attach logging to CDP.
* Ignore potential context.Canceled.
* Fix onboarding wizard test for new objects browser.
* Return an error on a context cancellation.
* Wire all loggers to planet.

Change-Id: I67eb138ba31252f55ac5b383679d033bcf71f1b2
2021-11-15 15:56:35 +02:00
Elek, Márton
debb82f596
Geofencing and advanced placement-constraint support (#4227) 2021-11-15 15:23:41 +02:00
Ivan Fraixedes
cb499cb7db
Makefile: Fix target builds the satellite admin UI
Fix the Makefile target which build the satellite admin UI adding a
missed npm command which must run before the npm run build command.

Change-Id: I96c9b0bd2ad2d909fc680db2b22a6cf13931111a
2021-11-12 13:29:55 +01:00
Ivan Fraixedes
07e07d3a2d satellite/metainfo: Add project ID to error logs
Add the project ID to error logs messages about not being able to
retrieve and track storage or bandwidth usage.

Some error logs related to these errors already contained the project ID
but others didn't, having it in the ones that didn't make them more
consistent and will provide more info which may be helpful for
troubleshooting them.

Change-Id: Ia9fc707a7f3aff0867645bb941badc199c2bf832
2021-11-12 11:28:08 +00:00
Ivan Fraixedes
ab425d4248 build: Consider the Admin UI assets
Add into the Makefile and Jenkins the operations to build the Admin UI
assets and lint them.

Change-Id: I687c1adec796fd591664e34eb73a465e53096b58
2021-11-12 08:48:28 +00:00
Michał Niewrzał
289ae3325a satellite/metabase: always include encryption in listed object
We implemented filtering of system metadata to
reduce DB load in case user will need only list of
objects but we missed that encryption configuration
is needed to decrypt object key.

This change is always including encryption configuration into
list objects results.

Change-Id: Iaf7b3d8441480e3ad787f19a28f1b648584d5f7d
2021-11-12 05:08:01 +00:00
Gabriel De Almeida
5ed3846e16 testsuite/ui/satellite: add browser tests
Also, we have multiple tests that contain the same sign up
and login logic and refactoring them would make tests
cleaner and more readable.

Change-Id: I25bf95ca2fffb7f494a3c93a4a4afe429ece70d3
2021-11-11 15:03:07 -05:00
Ivan Fraixedes
6dffbd4173 satellite/metainfo: Log in warn level limits exceeds
Don't log as an Error when users make request that cannot be fulfilled
because they exceed their storage or bandwidth limits.

Those are not system errors and the service is handling them correctly
and showing them in the logs an "ERROR" is misleading.

Change-Id: Iac642b7e8ba92840bb943192ad0694b5f4930258
2021-11-11 15:53:18 +00:00
dlamarmorgan
44b1ca6b97 satellite/{console,satellitedb}: move project limits from config file to DB to keep limits on a per user basis
To allow for changing limits for new users, while leaving existing users limits as they are, we must store the project limits for each user. We currently store the limit for the number of projects a user can create in the user DB table. This change would also store the project bandwidth and storage limits in the same table.

Change-Id: If8d79b39de020b969f3445ef2fcc370e51d706c6
2021-11-11 15:10:00 +00:00
Vitalii Shpital
b41e140b26 web/satellite: fixed dropdowns weird behavior
Reworked click-outside behavior.
Fixed bucket names selection and date picker dropdowns.

Change-Id: I1d798139e6bf713650b7071e5a0547d5d576c7a6
2021-11-11 15:45:18 +02:00
Egon Elbre
96058be1a7 storagenode/storagenodedb: fix error message
Change-Id: Ibc98aa6539199f8dc8117f21a6ae645cea419403
2021-11-10 17:46:30 +00:00
Egon Elbre
58916d8ba9 ci: don't trigger UI tests when go.mod changes
Currently testsuite/go.mod is being updated automatically and
we don't want to trigger UI tests, since they are a bit flaky.

Change-Id: Ie1c847e262ad723a006071f7a8f5cd7f26f4244e
2021-11-10 17:16:08 +00:00
Ivan Fraixedes
4e67ea007c satellite/admin: Serve static UI assets
Change the satellite Admin HTTP server for:

* Embedding the UI assets into the Go binary.
* Serve the UI assets from the embedded file system or from a specific
  directory path through a configuration flag, without requiring
  authentication but keeping the authentication verification for the API
  endpoints.
* Add tests to verify that the UI assets are served without
  authentication.

Change-Id: I9003ac96f1ec585a189b67fc1cb315905403d557
2021-11-10 18:47:46 +02:00
Egon Elbre
af5b90ed32 storagenode/storagenodedb: include dbname in error
When something happens during opening or closing then it wasn't clear
which database had the issue.

Fixes storj/storj#4271

Change-Id: I34c8bae79a5b41ccd9b40aa8d836805f8c1a573c
2021-11-10 16:25:34 +00:00
Mya
bf51c286d9
satellite/geoip: update node check-in to associate a country code
Resolves https://github.com/storj/storj/issues/4247

Change-Id: Idfd71bf1795d48ca3c686066bbdb95b9c6594f00
2021-11-10 16:44:41 +01:00
Vitalii Shpital
dda6720dd6 web/satellite: added 2 new steps for onb CLI flow
Added 2 new steps to the beginning of onb CLI flow if it is visited from new navigation side bar.
Steps are: Access grant name and Access grant permissions.
Fixed wording for success screen.

Change-Id: If1a7605e0920d1262d13142843df95b438ce7717
2021-11-10 14:36:11 +00:00
Yaroslav Vorobiov
2ebdc1303f satellite/satellitedb: add disqualification reason to nodes table
Alter satellites DB nodes table to add `disqualification_reason` int column
which contains disqualification type enum of why node has been disqualified.

Change-Id: Ia514557018ca27e1984216dc5004346d59869d16
2021-11-10 14:25:07 +02:00
Michał Niewrzał
12d4dc16a1 satellite/{metainfo,metabase}: make metadata optional for CommitObject
To resolve problem with lack of ability to set metadata while MPU on
gateway we are adding setting metadata with BeginObject. This
change makes also metadata optional while CommitObject. We need
this functionality to not override metadata set with BeginObject in
case when metadata is not set with CommitObject.

Another reson is that we would like to not set metadata at all if user
didn't specify metadata. At the moment we always setting some bytes
for metadata fields e.g. empty EncryptedMetadata field can have key
and nonce set.

Change-Id: Ifee25b7718eb1f919119db9b698b29d8b5ebe2ec
2021-11-10 08:08:46 +00:00
Egon Elbre
fe3856ff91 web/satellite: remove ramda dependency
The package was barely used and the amount of code needed to remove it
is minor.

Also remove top-level package-lock, because it's not being used.

Change-Id: Ifb4ec4fe7f4fa36ce6e707a33d47f04c583cbe6c
2021-11-09 13:19:58 +00:00
Egon Elbre
1d5b3e508c satellite/admin/ui: fix rollup building
rollup does not handle virtual drives correctly, use preserveSymlinks to
avoid compilation issues.

Change-Id: I7ae9509572ff4e2e74b059db4909f2e612a1331b
2021-11-09 13:16:37 +00:00
Egon Elbre
8eebbf3d7d satellite/audit: fix TestReverify timeouts
Currently the slow db was sleeping for 1s and the timeout for audit was
1s. There's a slight chance that the timeout won't trigger on such a
small difference.

Increase the slow node sleep to 10x of the timeout.

Hopefully fixes #4268

Change-Id: Ifdab45141b3fc7c62bde11813dbc534b3255fe59
2021-11-09 13:16:29 +00:00
Egon Elbre
d5628740fd satellite/overlay/straynodes: prevent disqualification in tests
Currently the test threshold for unneeded node checkins are 30s.
The disqualification threshold was at 30s, which means, it was possible
for all the nodes to get disqualified.

Hopefully fixes #4267

Change-Id: I6b0a10c09b7fd90a9729794885c9e7a593781bad
2021-11-09 12:29:21 +00:00
Ivan Fraixedes
583cdc6e0e satellite/admin/ui: Create Web UI v1
This commit crate the Satellite Admin Web UI v1

* Bootstrap the files installing the project template:
  https://github.com/sveltejs/template
* Add the prettier prettier-plugin-svelte NPM packages as dev
  dependencies.
* Add an empty prettierrc.toml.
* Transform the project template to use Typescript rather than
  Javasript. See: https://github.com/sveltejs/template#using-typescript
* Replace the default favicon by the Storj logo.
* Create components in companion of some Typescript code which allows to
  generate a simple HTML page based on an opinionated definition of API.
* Implement all the Admin API client calls in the format that the UI
  generator components requires for rendering the UI from them.

Change-Id: I58fa586d68dc8998e5d89db169b8e90204f0a96d
2021-11-09 09:59:35 +00:00
Egon Elbre
3122de4564 web/satellite/browser: fix formatting
For some reason these files were skipped in the automatic format PR.

Change-Id: I2bc59b6f47437670e99dfd9ea6c4f44be2604cfc
2021-11-08 17:52:44 +00:00
Márton Elek
9bdcc415bc satellite/nodeselection: add geofencing constraints to the node selection criteria
Closes https://github.com/storj/storj/issues/4242
Change-Id: Ieda59a4f37c673e4e81abb4c89c09daf3199bbc7
2021-11-08 17:04:31 +00:00
Vitalii Shpital
696b8f0d8e web/satellite: moved object's passphrase step to be after buckets
Moved passphrase step to be after buckets screen.
Seems like it accidently enabled bucket force delete feature because buckets AG is being generated with empty passphrase.

Change-Id: I7ca0daf4e49045bf4d49b996eb5e2406132caeea
2021-11-08 11:45:35 +00:00
Vitalii Shpital
3c683998f5 satellite/console, web/satellite: feature flag for new objects flow
Added a feature flag to switch between old and new objects flow

Change-Id: Id1802f9fb1230928445f16e4d0a62aecdc91c038
2021-11-08 11:45:22 +00:00
Erik van Velzen
91158fd3bf satellite: omit system metadata fields if not requested
To reduce database load, add option to omit as many object properties
as possible when listing objects.

Change-Id: I817633801b00629a4042d1d1bd2389ee581953de
2021-11-08 09:19:44 +01:00
Márton Elek
fb604be460 satellite/gracefulexit: use nodecache with gracefulexit
Change-Id: I700caf6dfd06bd3b3970a8cf7c5a79da4af27b5f
2021-11-05 19:14:14 +00:00
Vitalii Shpital
98b59fe305 web/satellite: reworked Encrypt your data component
Reworked component to have all needed warnings and info.
Removed Warning route from objects flow.
Updated tests.

Change-Id: Iced68aca7fbf4e1f7b7e5302147a51fd0e9115c8
2021-11-05 17:36:01 +00:00
Egon Elbre
4a530ccffd cmd/storagenode: simplify windows service loop
Remove some testing code that is not necessary for the service.

Change-Id: Idd09d62bf022d6e66943983f98642fc3c9aa72f7
2021-11-05 15:56:18 +00:00
Vitalii Shpital
609526f6aa web/satellite: reworked new navigation sidebar
Reworked projects selection.
Added billing route to sidebar.
Reworked accounts container.
Fixed navigation console errors.
Fixed Quick start -> CLI flow back route bug.

Change-Id: Ia926146172598b9abde9a0ab5ed38cd7f9719ddd
2021-11-05 15:27:02 +00:00
Monty Anderson
804304a75d web/satellite: integrate browser
* create `isInitalized` property to check S3 client exists
* hide FileBrowser until `isInitialized === true`
* change broken reference from `buttonUpload` to `buttonFileUpload`
* add feature flag `new-browser` to switch between external and internal browser
* fix handling of `webkitRelativePath` for folder upload feature
* only load external styles when using external browser via contingent `file-browser` class

Change-Id: Ibdbfa45d8dbe3c6d828473c16e4cdb8694562c1a
2021-11-05 16:30:07 +02:00
Qweder93
b314559a13 satellite/metainfo: TestBatch params updated
Removed PathCipher and DefaultSegmentSize from CreateBucketParams
since they are unused anymore and breaks Integration on uplink side

Change-Id: I1393a7f1f436940731aa59edd693043336383290
2021-11-05 14:53:55 +02:00
Egon Elbre
e721081354 cmd/storagenode: fix service initialization code
func init() code isn't that well defined and reordering of them
could cause problems when starting the whole process from it.

Change-Id: I4088a0db156ece15354877011a481f6f91c9b332
2021-11-05 01:09:41 +02:00
dlamarmorgan
1dd537953e satellite/metainfo: strip the uplink version from the UserAgent and bound its size
The UserAgent should be stored as is, with the exception of removing the trailing version from any libuplink user agents

Change-Id: If17ef2fc4b59480a3477300f2585a07d64cc2bf4
2021-11-04 10:09:54 -07:00
Elek, Márton
27091826ba satellite/{satellitedb|metabase}: add SQL fields for geofencing
Change-Id: Ia698d71db1d3775b908c24bd1c9bc8abfd015ccc
2021-11-04 08:49:56 +00:00
Clement Sam
16a334020f cmd/uplinkng: add ranged download
This change adds the ability to download byte ranges
to uplinkng.

Extended the uplinkng Filesystem interface with Stat
method and an OpenOptions struct as parameter for the
Open method.

Also added a few tests for the ranged download

Change-Id: I89a7276a75c51a4b22d7a450f15b3eb18ba838d4
2021-11-03 19:51:25 +00:00
Egon Elbre
6b7473e8b9 web/satellite/images: reduce svg size
Currently uploadObjectStep.svg was a large file that was embedded into
the js bundle. This reduces the gzipped bundle size by 80KB.

Change-Id: Iff673ca44330b345eb6b317175cd8bf4e1efd29b
2021-11-03 18:21:27 +00:00
Egon Elbre
d043b9facd satellite/console: load index.html from disk when developing
This eases development by not needing to restart satellite after each
change.

Change-Id: I1063b7f431f675fb21e82dcfc2420310a9806422
2021-11-03 18:19:40 +02:00
Egon Elbre
6a69c66477 cmd/storj-sim: support printing node url
Add support for node url-s with PEER_URL to help configuring
authservice and gateway.

   storj-sim network env SATELLITE_0_URL

Change-Id: I8b595b398007730662f100a4e6b3529cc0a7512a
2021-11-03 16:11:41 +02:00