Commit Graph

6400 Commits

Author SHA1 Message Date
Vitalii Shpital
b41e140b26 web/satellite: fixed dropdowns weird behavior
Reworked click-outside behavior.
Fixed bucket names selection and date picker dropdowns.

Change-Id: I1d798139e6bf713650b7071e5a0547d5d576c7a6
2021-11-11 15:45:18 +02:00
Egon Elbre
96058be1a7 storagenode/storagenodedb: fix error message
Change-Id: Ibc98aa6539199f8dc8117f21a6ae645cea419403
2021-11-10 17:46:30 +00:00
Egon Elbre
58916d8ba9 ci: don't trigger UI tests when go.mod changes
Currently testsuite/go.mod is being updated automatically and
we don't want to trigger UI tests, since they are a bit flaky.

Change-Id: Ie1c847e262ad723a006071f7a8f5cd7f26f4244e
2021-11-10 17:16:08 +00:00
Ivan Fraixedes
4e67ea007c satellite/admin: Serve static UI assets
Change the satellite Admin HTTP server for:

* Embedding the UI assets into the Go binary.
* Serve the UI assets from the embedded file system or from a specific
  directory path through a configuration flag, without requiring
  authentication but keeping the authentication verification for the API
  endpoints.
* Add tests to verify that the UI assets are served without
  authentication.

Change-Id: I9003ac96f1ec585a189b67fc1cb315905403d557
2021-11-10 18:47:46 +02:00
Egon Elbre
af5b90ed32 storagenode/storagenodedb: include dbname in error
When something happens during opening or closing then it wasn't clear
which database had the issue.

Fixes storj/storj#4271

Change-Id: I34c8bae79a5b41ccd9b40aa8d836805f8c1a573c
2021-11-10 16:25:34 +00:00
Mya
bf51c286d9
satellite/geoip: update node check-in to associate a country code
Resolves https://github.com/storj/storj/issues/4247

Change-Id: Idfd71bf1795d48ca3c686066bbdb95b9c6594f00
2021-11-10 16:44:41 +01:00
Vitalii Shpital
dda6720dd6 web/satellite: added 2 new steps for onb CLI flow
Added 2 new steps to the beginning of onb CLI flow if it is visited from new navigation side bar.
Steps are: Access grant name and Access grant permissions.
Fixed wording for success screen.

Change-Id: If1a7605e0920d1262d13142843df95b438ce7717
2021-11-10 14:36:11 +00:00
Yaroslav Vorobiov
2ebdc1303f satellite/satellitedb: add disqualification reason to nodes table
Alter satellites DB nodes table to add `disqualification_reason` int column
which contains disqualification type enum of why node has been disqualified.

Change-Id: Ia514557018ca27e1984216dc5004346d59869d16
2021-11-10 14:25:07 +02:00
Michał Niewrzał
12d4dc16a1 satellite/{metainfo,metabase}: make metadata optional for CommitObject
To resolve problem with lack of ability to set metadata while MPU on
gateway we are adding setting metadata with BeginObject. This
change makes also metadata optional while CommitObject. We need
this functionality to not override metadata set with BeginObject in
case when metadata is not set with CommitObject.

Another reson is that we would like to not set metadata at all if user
didn't specify metadata. At the moment we always setting some bytes
for metadata fields e.g. empty EncryptedMetadata field can have key
and nonce set.

Change-Id: Ifee25b7718eb1f919119db9b698b29d8b5ebe2ec
2021-11-10 08:08:46 +00:00
Egon Elbre
fe3856ff91 web/satellite: remove ramda dependency
The package was barely used and the amount of code needed to remove it
is minor.

Also remove top-level package-lock, because it's not being used.

Change-Id: Ifb4ec4fe7f4fa36ce6e707a33d47f04c583cbe6c
2021-11-09 13:19:58 +00:00
Egon Elbre
1d5b3e508c satellite/admin/ui: fix rollup building
rollup does not handle virtual drives correctly, use preserveSymlinks to
avoid compilation issues.

Change-Id: I7ae9509572ff4e2e74b059db4909f2e612a1331b
2021-11-09 13:16:37 +00:00
Egon Elbre
8eebbf3d7d satellite/audit: fix TestReverify timeouts
Currently the slow db was sleeping for 1s and the timeout for audit was
1s. There's a slight chance that the timeout won't trigger on such a
small difference.

Increase the slow node sleep to 10x of the timeout.

Hopefully fixes #4268

Change-Id: Ifdab45141b3fc7c62bde11813dbc534b3255fe59
2021-11-09 13:16:29 +00:00
Egon Elbre
d5628740fd satellite/overlay/straynodes: prevent disqualification in tests
Currently the test threshold for unneeded node checkins are 30s.
The disqualification threshold was at 30s, which means, it was possible
for all the nodes to get disqualified.

Hopefully fixes #4267

Change-Id: I6b0a10c09b7fd90a9729794885c9e7a593781bad
2021-11-09 12:29:21 +00:00
Ivan Fraixedes
583cdc6e0e satellite/admin/ui: Create Web UI v1
This commit crate the Satellite Admin Web UI v1

* Bootstrap the files installing the project template:
  https://github.com/sveltejs/template
* Add the prettier prettier-plugin-svelte NPM packages as dev
  dependencies.
* Add an empty prettierrc.toml.
* Transform the project template to use Typescript rather than
  Javasript. See: https://github.com/sveltejs/template#using-typescript
* Replace the default favicon by the Storj logo.
* Create components in companion of some Typescript code which allows to
  generate a simple HTML page based on an opinionated definition of API.
* Implement all the Admin API client calls in the format that the UI
  generator components requires for rendering the UI from them.

Change-Id: I58fa586d68dc8998e5d89db169b8e90204f0a96d
2021-11-09 09:59:35 +00:00
Egon Elbre
3122de4564 web/satellite/browser: fix formatting
For some reason these files were skipped in the automatic format PR.

Change-Id: I2bc59b6f47437670e99dfd9ea6c4f44be2604cfc
2021-11-08 17:52:44 +00:00
Márton Elek
9bdcc415bc satellite/nodeselection: add geofencing constraints to the node selection criteria
Closes https://github.com/storj/storj/issues/4242
Change-Id: Ieda59a4f37c673e4e81abb4c89c09daf3199bbc7
2021-11-08 17:04:31 +00:00
Vitalii Shpital
696b8f0d8e web/satellite: moved object's passphrase step to be after buckets
Moved passphrase step to be after buckets screen.
Seems like it accidently enabled bucket force delete feature because buckets AG is being generated with empty passphrase.

Change-Id: I7ca0daf4e49045bf4d49b996eb5e2406132caeea
2021-11-08 11:45:35 +00:00
Vitalii Shpital
3c683998f5 satellite/console, web/satellite: feature flag for new objects flow
Added a feature flag to switch between old and new objects flow

Change-Id: Id1802f9fb1230928445f16e4d0a62aecdc91c038
2021-11-08 11:45:22 +00:00
Erik van Velzen
91158fd3bf satellite: omit system metadata fields if not requested
To reduce database load, add option to omit as many object properties
as possible when listing objects.

Change-Id: I817633801b00629a4042d1d1bd2389ee581953de
2021-11-08 09:19:44 +01:00
Márton Elek
fb604be460 satellite/gracefulexit: use nodecache with gracefulexit
Change-Id: I700caf6dfd06bd3b3970a8cf7c5a79da4af27b5f
2021-11-05 19:14:14 +00:00
Vitalii Shpital
98b59fe305 web/satellite: reworked Encrypt your data component
Reworked component to have all needed warnings and info.
Removed Warning route from objects flow.
Updated tests.

Change-Id: Iced68aca7fbf4e1f7b7e5302147a51fd0e9115c8
2021-11-05 17:36:01 +00:00
Egon Elbre
4a530ccffd cmd/storagenode: simplify windows service loop
Remove some testing code that is not necessary for the service.

Change-Id: Idd09d62bf022d6e66943983f98642fc3c9aa72f7
2021-11-05 15:56:18 +00:00
Vitalii Shpital
609526f6aa web/satellite: reworked new navigation sidebar
Reworked projects selection.
Added billing route to sidebar.
Reworked accounts container.
Fixed navigation console errors.
Fixed Quick start -> CLI flow back route bug.

Change-Id: Ia926146172598b9abde9a0ab5ed38cd7f9719ddd
2021-11-05 15:27:02 +00:00
Monty Anderson
804304a75d web/satellite: integrate browser
* create `isInitalized` property to check S3 client exists
* hide FileBrowser until `isInitialized === true`
* change broken reference from `buttonUpload` to `buttonFileUpload`
* add feature flag `new-browser` to switch between external and internal browser
* fix handling of `webkitRelativePath` for folder upload feature
* only load external styles when using external browser via contingent `file-browser` class

Change-Id: Ibdbfa45d8dbe3c6d828473c16e4cdb8694562c1a
2021-11-05 16:30:07 +02:00
Qweder93
b314559a13 satellite/metainfo: TestBatch params updated
Removed PathCipher and DefaultSegmentSize from CreateBucketParams
since they are unused anymore and breaks Integration on uplink side

Change-Id: I1393a7f1f436940731aa59edd693043336383290
2021-11-05 14:53:55 +02:00
Egon Elbre
e721081354 cmd/storagenode: fix service initialization code
func init() code isn't that well defined and reordering of them
could cause problems when starting the whole process from it.

Change-Id: I4088a0db156ece15354877011a481f6f91c9b332
2021-11-05 01:09:41 +02:00
dlamarmorgan
1dd537953e satellite/metainfo: strip the uplink version from the UserAgent and bound its size
The UserAgent should be stored as is, with the exception of removing the trailing version from any libuplink user agents

Change-Id: If17ef2fc4b59480a3477300f2585a07d64cc2bf4
2021-11-04 10:09:54 -07:00
Elek, Márton
27091826ba satellite/{satellitedb|metabase}: add SQL fields for geofencing
Change-Id: Ia698d71db1d3775b908c24bd1c9bc8abfd015ccc
2021-11-04 08:49:56 +00:00
Clement Sam
16a334020f cmd/uplinkng: add ranged download
This change adds the ability to download byte ranges
to uplinkng.

Extended the uplinkng Filesystem interface with Stat
method and an OpenOptions struct as parameter for the
Open method.

Also added a few tests for the ranged download

Change-Id: I89a7276a75c51a4b22d7a450f15b3eb18ba838d4
2021-11-03 19:51:25 +00:00
Egon Elbre
6b7473e8b9 web/satellite/images: reduce svg size
Currently uploadObjectStep.svg was a large file that was embedded into
the js bundle. This reduces the gzipped bundle size by 80KB.

Change-Id: Iff673ca44330b345eb6b317175cd8bf4e1efd29b
2021-11-03 18:21:27 +00:00
Egon Elbre
d043b9facd satellite/console: load index.html from disk when developing
This eases development by not needing to restart satellite after each
change.

Change-Id: I1063b7f431f675fb21e82dcfc2420310a9806422
2021-11-03 18:19:40 +02:00
Egon Elbre
6a69c66477 cmd/storj-sim: support printing node url
Add support for node url-s with PEER_URL to help configuring
authservice and gateway.

   storj-sim network env SATELLITE_0_URL

Change-Id: I8b595b398007730662f100a4e6b3529cc0a7512a
2021-11-03 16:11:41 +02:00
Michał Niewrzał
e792727bea satellite/metabase: allow setting metadata with begin object requests
Turns out that S3 protocol is setting object metadata with initial
request, in our case it's BeginObject, so we need to modify metabase
methods to accept metadata at the beginning and at the end.
BeginObject methods will set metadata always (niled or not niled).

One additional improvment for metadata fields was introduced and it's
validating fields as optional if EncryptedMetadata was not set.

This change currently doesn't have any implications and  it's a base for
other changes. Metadata is not set with BeginObject metainfo endpoint
yet.

Change-Id: I1f768407bc3428500b0d30ee188257420d953001
2021-11-03 10:08:10 +00:00
Jeff Wendling
bd2448bc4d cmd/uplinkng: expand version information
This includes the information that used to exist in uplink
as well as all of the module versions.

Change-Id: Ie6d6fa12da9c04b50611e5885e05ab0c24a2ec0d
2021-11-03 09:26:56 +00:00
Cameron Ayer
1de8a695e8 satellite/{overlay,satellitedb}: fix stray nodes DQ bug
We had a bug in the stray nodes chore where nodes who had not been seen
in several months were not being DQd. We figured out that this was
happening because we were using two queries: The first to grab
nodes where last_contact_success < some cutoff, the second to DQ them
unless last_contact_success == '0001-01-01 00:00:00+00'. The problem
is that if all of the nodes returned from the first query had
last_contact_success of '0001-01-01 00:00:00+00', we would pass them to
the second query which would not DQ them. This would result in the stray
nodes DQ loop ending since we found a number of nodes to DQ less than the
limit.

The fix: add the "WHERE last_contact_success != '0001-01-01
00:00:00+00'::timestamptz" to the selection query.

Change-Id: I4e60de90b68d8745d641b4467c2b23e0e56f7dff
2021-11-02 17:05:00 +00:00
Yingrong Zhao
774ae017e3 satellite/{satellitedb, web}: display object count in satellite UI
Even though we want to start charging segment fee instead of object fee,
it's hard for users to understand what a segment is. This PR adds the
object count back in the UI alongside with segment count to help address
the issue.

Change-Id: I92eb42c769d350eba68a72443deffec5c278359c
2021-11-01 18:26:13 +00:00
Egon Elbre
187e508364 web/satellite/browser: auto-format code
Change-Id: I74504758be935ec5580713bcf8d566c444cfd51e
2021-11-01 17:07:09 +02:00
Monty Anderson
ea8782fcfa web/satellite: move browser from storj/browser
Change-Id: I1ee976ca117307dd1464dd78295b17c790fe8483
2021-11-01 16:48:06 +02:00
Vitalii Shpital
460f8fcdd4 web/satellite: added satellite name to MFA label
Changed MFA label from "STORJ DCS" to "STORJ <sat_name>" for the users who use the same email for different satellites.

Change-Id: I1ff68a08fee1da57ab3c32173ce8d3b52cac39d9
2021-11-01 13:55:47 +00:00
Vitalii Shpital
4a955b8a76 web/satellite: account area stylings and status fixes
Fixed user's paid tier status to be set by amount of CCs instead of db field.
Fixed some minor styling issues.

Change-Id: Ie1989289a1bf3ce964de7fe19ca8dd3e11dffc38
2021-11-01 13:22:10 +00:00
Artur M. Wolff
785eb93cd7 ci: install gateway@latest instead of @main
Gateway-ST frequent release cycle has been resurrected, which means it's
safer to use the latest release tag in the storj repository's CI now.

Change-Id: I9df1c789a9b9418ba7cceaec9cfec3cc6c448284
2021-11-01 12:16:38 +00:00
Egon Elbre
431f55d53b ci: cross-compile storagenode-updater
Change-Id: I1db757b64e61f185ba710bf669f2e62441b9ab71
2021-11-01 11:21:06 +00:00
Jeremy Wharton
136899eeea web/satellite: Trim whitespace around login credentials
Trims whitespace around the email address, MFA passcode,
and MFA recovery code before sending a login request to the satellite.

Change-Id: I145e6a5991270b0e49ac3f09ab3c9d1c9f970699
2021-10-30 19:39:13 +00:00
Jeff Wendling
4d023f773f cmd/storagenode-updater: update to correct version
Currently if the nodes is below the minimum version it will immediately
update to the suggested version, regardless if its eligible from the seed
or not. This change corrects the behaviour to update to the minimum
version only and then properly respect/wait for the rollout to include it.

Updates based on logic here: https://review.dev.storj.io/c/storj/private/+/6187

Change-Id: Ic6c91c48ae9b8a116378b2573fbfca7e7bd5cc32
2021-10-29 20:38:30 +00:00
Jeff Wendling
f94d8a200c cmd/uplinkng: mkdir config file folder
If the directory doesn't exist, then the first run in a non-migration
setting will error because it cannot create the config file. This
change creates the directory.

Change-Id: I439159c00047e9ae20e139318dad5a047c853253
2021-10-29 11:07:55 +00:00
Stefan Benten
8c1a149de2 build: bump private dependency
Enables proper version printing.

Change-Id: I37e12d6823f86f9275ac9f64abf324003019bd18
2021-10-29 12:25:33 +13:00
mya
9749f9756c
cmd/uplinkng: port expanded option to show additional metadata (#4229) 2021-10-28 11:51:33 -05:00
Egon Elbre
edb8d656de satellite/metainfo: adjust piecedeletion timeouts
Currently slower storagenodes can slow down deletion queue.
To make piece deletion faster reduce the maximum time spent in
either dialing or piece deletion requests.

With this change:
* dial timeout is 3s
* request timeout is 15s
* fail threshold is set to 10min

Similarly, we'll mark storage node as failed when the timeout occurs.
The timeout usually indicates that the storagenode is overwhelmed.
Garbage collection will ensure that the pieces get deleted eventually.

Change-Id: Iec5de699f5917905f5807140e2c3252088c6399b
2021-10-28 13:37:01 +03:00
Michał Niewrzał
d3a0364f21 Makefile: add uplinkng to build binaries
We want uplinkng binaries easily accessible.

Change-Id: I83e1d90f9413ebd97e1a51d0435548bc0cac8ad8
2021-10-28 09:47:36 +02:00
Yingrong Zhao
fcde92d38d web/satellite: display segment count in billing
Change-Id: I472d65967c656889b8ffeb799cb46bc18b2a9b15
2021-10-27 19:51:45 +00:00