Commit Graph

6294 Commits

Author SHA1 Message Date
Michał Niewrzał
ff653f6f22 satellite/metainfo: stop using in tests unused method
We would like to remove it from uplink but to do that
we need to stop using it in storj tests.

Change-Id: I8f6af74d6b8b8e447ab8ed3a00517b40840dad58
2021-12-22 15:05:53 +01:00
Jeff Wendling
4426470173 cmd/uplinkng: rename parallelism flags, small fixes
this makes the flags match rclone nomenclature

fixes test-uplinkng to use the temporary config dir
instead of the machine default, and clean up some.

bumps clingy so that the command errors when an unknown
command is specified.

also fixes some printfs in share to use clingy stdout.
it still does some external actions that should be
passed through a ulext.External for mocking, but
that's ok for now.

Change-Id: Icc231e7e26393541c312396fec907b640b97718e
2021-12-22 13:13:30 +00:00
Michal Niewrzal
da9e0188f8
Update README.md (#4320)
* remove alpha badge
* remove information about "beta phase" and "network under construction" from License section

Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2021-12-22 14:12:58 +01:00
Stefan Benten
0334465306
Makefile: fix variable used to trigger release drafting (#4319)
Small fix to properly trigger the script.
2021-12-22 13:33:07 +01:00
Egon Elbre
5f39f20766 web/: add check for change to eslint import
Change-Id: Idf94c8e0c92d34e840db549e9e704eb3373762ac
2021-12-21 15:59:23 +00:00
Stefan Benten
e2481fb510 scripts,Jenkinsfile,Makefile: upload binaries to drafted github release
This change adds a script and the needed build logic to create a draft github release and then upload the created binaries to it in order to make the actual publishing a lot less error prone.
The logic is currently that it only does this for all github tags, but not for every main build/push. This is handled by the checks in the script itself.

Change-Id: Ie172a8e4a97200de901a26a055aa5a8a54b60a2a
2021-12-21 13:55:38 +00:00
Michał Niewrzał
db7985ca90 satellite/metainfo: set EncryptedMetadataEncryptedKey while listing objects
At some point we missed to add metadata key to list objects
response. Because of that uplink is taking key from pb.SteamMeta.
We need to clean it up.

Tests will be added on uplink side.

https://github.com/storj/uplink/issues/71

Change-Id: I3328e2f1b86bca15aeaf89f8e59cdca3c8e97742
2021-12-21 13:09:11 +00:00
Michał Niewrzał
87a9292127 satellite/metainfo/zombiedeletion: enable chore by default
We enabled this chore on different satellites
for testing. Works fine so we can enable this
by default for all configurations.

Change-Id: I987639685d8de5c7e5798adca30fe26bdac9e1d1
2021-12-21 13:26:18 +01:00
Stefan Benten
a6139f5a6b satellite/orders: log less decrypt order messages
Currently most of the satellite log is made up by this specific log
message and thus we should increase the log level. We already have
a monkit event tracking this case. In case we need to look into this
more we should just increase the satellite log level.

Change-Id: I27ebed3e6745701c66c83e8c52ddc836ad9d5f4e
2021-12-21 00:39:29 +01:00
nadimhq
a04b9ce372
testsuite/ui/multinode: adding more UI tests (#4301)
added tests for add node (check modal, cancel with separate buttons, add new node, add node that already exists) and more options (change display name, delete node(&cancel delete), copy node ID).
2021-12-20 14:30:48 -05:00
Ivan Fraixedes
c10cc302b5
Makefile: Fix target for building Admin UI
Fix the Makefile target for building the Satellite Admin UI.

Change-Id: Id9087c313c451fa68796c9a495704ea3b3728a5c
2021-12-20 12:11:10 +01:00
Ethan
185617326c satellite/email: Move MIME-Version header to the last element before Content-Type
Some MTAs are marking our emails as SPAM because of the current location of the header.

Change-Id: I3f12c36b184f69b643a8299f5ccbc609f4599ef9
2021-12-17 11:26:25 -06:00
Qweder93
b7b010adc9 satellite/admin: add project limit on segment count
Added possibility to set custom value of segments
per project for user/customer.

Change-Id: Ic6e1abe25269755e9e8819fbef065a87de359fce
2021-12-17 12:40:12 +00:00
Ivan Fraixedes
f87ce0cadf satellite/admin/ui: Fix component rendering
I introduced a change in the UI that was a better way to dynamically
render Svelte components
(https://review.dev.storj.io/c/storj/storj/+/5931/4..5) during the
review of the first version of it, however, while it works perfectly
on development mode it doesn't work when the assets are built for
production, failing silently, because the constructor name get renamed
due to the name mangling caused by the minifyer as stated in the
following issue: https://github.com/sveltejs/svelte/issues/6980

This commit use a different alternative not based on the constructor
name and it works fine with the production build.

Change-Id: I643c405f877a9206cf0e51b44d5138e5a9756a79
2021-12-17 11:00:31 +00:00
Ivan Fraixedes
5573ece848 satellite/admin/ui: Migrate to SvelteKit
Migrate the satellite admin UI web app from the Svelte template used to
generate a Svelte App scaffolding to SvelteKit.

There aren't any functional changes in the  application, however, the
commit has a lot because:

1. SvelteKit uses a different directory layout and constraints to it, so
   the files have been moved.
2. The files have changed its formatting due to the new default linter
   configurations that SvelteKit uses.
3. The linter detected some issues with using `object` and `any` types
   in Typescript, so they have been replaced by better general types
   (e.g. Record).

The  migration allows to use the new tooling rather than Rollup
directly, besides that will empower the future of it when it needs more
features (e.g. different routes, etc.).

Change-Id: Ifa6736c13585708337f6c5a59388077b784eaddd
2021-12-17 10:27:13 +00:00
Malcolm Bouzi
db0bd38d95 web/satellite: Add new user events to segment for link clicks
We'd like to have a better understading of what pages users are clicking on the most

Change-Id: Ib3983cd250a95106acb162c2284fc1d11f2a3152
2021-12-16 18:11:11 +00:00
Michał Niewrzał
60d03b9e42 satellite/metainfo: drop GetSegmentByLocation
Method is not used anymore. We need it during transition
from metainfo loop into segment loop.

Change-Id: Id32590ff9b02e06025efce1c32869e2cf9cd0bc5
2021-12-16 16:32:45 +01:00
Michał Niewrzał
3f0babffe8 satellite/{console,satellitedb}: add project segment limit to user
All limits we have for projects have also parent limits stored
with user data. New created project is first taking limits from
owner (user) limits.
This change is extending users table with project_segment_limit
column and adds functionality to get and set value for this
column.

Change-Id: Iff5e36c62b517652390b649fc05992475916ecff
2021-12-16 08:46:01 +00:00
Jeremy Wharton
9d13c649a2 satellite/{console,satellitedb}: Forbid creating users with used email
This change disallows creation of users possessing the same email.
If a user attempts to create an account with an email address
that's already used - whether it belongs to an active account or not -
he will be notified of unsuccessful account creation. If he attempts to
log in using an email address belonging to an inactive account,
he will be presented with a link allowing him to re-send the
verification email. Attempting to register with an email address
belonging to an existing account triggers a password reset email.

Change-Id: Iefd8c3bef00ecb1dd9e8504594607aa0dca7d82e
2021-12-15 17:24:35 +00:00
Clement Sam
137641f090 multinode/nodes: pass Node entity to Add method instead of set of a parameters
The Add method on the multinode DB interface accepts
a set of parameters which are already fields in the nodes.Node struct
excluding the name field.

When adding a new node, you're forced to call UpdateName() method
after calling the Add() method in order to save a node and update
the name.

This change allows passing the nodes.Node entity which includes
the name field. With this, a new node can be added together with
the name without invoking the UpdateName() method.

Change-Id: I281ec628dffaade35d6db4479a84f39636200072
2021-12-15 12:39:24 +00:00
nadimhq
596124f0f6
docs: Adding a testplan folder with a README and TEMPLATE for testplan (#4306)
Added a testplan folder in docs, with a README and TEMPLATE for testplan

We want community involvement with testplans, since we believe that testplans written as early as possible before implementation starts work like a checklist and as soon as we finish the implementation we can compare it to the test plan to check for any bugs. Also even before implementation is finished, if a developer can read a test plan beforehand they can prevent most bugs, hence why we believe that the earlier we write a test plan the more bugs we can prevent.

Co-authored-by: Igor <38665104+ihaid@users.noreply.github.com>
2021-12-15 07:32:36 -05:00
Clement Sam
ed9bde8f69 cmd/storagenode: add info command
The info command prints the details of the storagenode
to stdout.

It returns the storagenode info in JSON format
if --json flag is specified which can be piped
to the multinode add command.

Change-Id: I0163db8e02c4ec7346bfa69274d1772669357c6c
2021-12-14 17:05:52 +00:00
Clement Sam
60c8280565 cmd/multinode: add add command to multinode
This change adds an add command to the multinode CLI.

The add command takes a json <file> as argument.
If dash (-) is specified, it reads data from stdin.

The <file> specified can be json file containing array of
nodes data or a single node object.

Change-Id: I44d68486dc9aea0bd0311a40e84d3262a0303aef
2021-12-14 17:05:52 +00:00
Michał Niewrzał
7fba79be3f cmd/uplinkng: add register command
Change-Id: I9add258249c618e804e3b006dc1afc5325019009
2021-12-14 10:03:55 -06:00
Michał Niewrzał
75acd6109d cmd/{uplink,uplinkng}: set user agent for uplink binaries
We want to monitor traffic and tools that are used
to interact with our network so we need to append
its user agent.
The same user agent is appended for current uplink
and uplinkng as eventually we will remove first.

Change-Id: I116080d6c2c6c85d591771facf01356de02a9392
2021-12-14 13:00:01 +00:00
nerdatwork
f83a801eff
web/satellite: Fix subtitle in disable 2FA (#4285)
Co-authored-by: littleskunk <jens.heimbuerge@googlemail.com>
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2021-12-14 13:37:17 +01:00
Michał Niewrzał
911540e76e satellite/orders: avoid logging "bucketName or projectID not set"
When we switched to segments loop we stopped
adding bucket name and project ID to order limit
metadata. This is causing lots of logging that we don't need.

Change-Id: Id3f878a170b7a6b801e8a838ee69165715985d60
2021-12-14 10:31:48 +00:00
Jeff Wendling
9615bd191f cmd/uplinkng: update clingy for better boolean support
this allows commands like

    uplinkng cp -r sj://foo sj://bar

to work correctly, rather than complain that sj://foo is
not a boolean.

Change-Id: I003e47aabb85566bc2b454851cf55043b17ee7ea
2021-12-14 09:41:38 +00:00
Jeff Wendling
e6aa52cd6b cmd/uplinkng: reorganize iterator helpers
Change-Id: I8cf27c8fcbc35bf75000669cb5c02e30b5663a39
2021-12-14 09:41:18 +00:00
Z
4762493e0f
storagenode/reputation: fix missing space in suspension message (#4309)
Fix missing space in the suspended message

Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2021-12-13 19:30:33 +00:00
dlamarmorgan
9a37399346 satellite/metainfo: remove Gateway-ST library name from UserAgent string
When trimming the UserAgent string, Library names such as Gateway-ST
should be removed.

Change-Id: I3d71fa8514364bf1d4d79b0dc00c85aed337ab45
2021-12-13 18:07:55 +00:00
Jeff Wendling
c8c4e8835e cmd/uplinkng: file level parallel copy
Change-Id: I074abca67ea6f4a2fd0983fe452c74f76ee62c6a
2021-12-13 12:14:39 -05:00
Gabriel De Almeida
b6f1efbbdc web/satellite: fixing linting errors in the FileShareModal
Transitioned all functions into typescript code and changed the vue component to be a class component.

Change-Id: I5ffcca5abb1595d53e880d4ccdfa26e23b1a214d
2021-12-13 17:40:13 +02:00
Gabriel De Almeida
cd0d61f525 web/satellite: fixing linting errors in the FileEntry component
Transitioned all functions into typescript code and changed the vue component to be a class component.

Change-Id: If804c8c58399c304c416f2e53672fa6f9c8bbffe
2021-12-13 17:37:25 +02:00
Gabriel De Almeida
e67b905b6d web/satellite: fixing linting errors in the FileBrowser component
Transitioned all functions into typescript code and changed the vue component to be a class component.

Change-Id: Iffd8e8b368ee69f15c8f4ad9b34dad9c36a1f931
2021-12-13 15:33:57 +00:00
Gabriel De Almeida
2be56e335c web/satellite: fixing linting errors in the file browser
Transitioning all browser components and the browser files vuex store into typescript code and fixing all linting errors.

Change-Id: I91d166153fcba82a06cdccb04ded963141855239
2021-12-13 15:33:17 +00:00
Stefan Benten
5c84243d99 Jenkinsfile: downgrade containers for version tests
Version Tests currently fail with go 1.17.5 containers due to
this dependency in the chain:
github.com/marten-seemann/qtls-go1-16@v0.1.3

Change-Id: Ie1e6595d5a8d35ee07b066c3bd3e3583892ad578
2021-12-13 11:16:25 +00:00
Stefan Benten
6799354748 ci: bump to go 1.17.5
Change-Id: I072bebaac8c12b31bc68ab4058b1991c4f6d5e12
2021-12-13 11:25:22 +01:00
Jeff Wendling
baaa96c208 cmd/uplinkng: introduce MultiWriteHandle
Change-Id: I6acf93141ddfa62728164818a322120ed6956b00
2021-12-10 10:23:04 -05:00
Jeff Wendling
34890c9195 cmd/uplinkng: introduce MultiReadHandle
Change-Id: I57b98b5e1406e7b38edf3bc65907d9796a1a663b
2021-12-10 09:30:25 +00:00
Michał Niewrzał
c33d2f58cf satellite/metainfo: override StreamMeta only if needed
For backward compatibility we are overriding pb.StreamMeta
values returned as encrypted metadata. It turns out that we
should do it not when target values are missing but when
values to override exists.

This was causing problems after move operation. Details can be
found here https://github.com/storj/uplink/issues/70

Backward compatibility test will be added to storj/uplink testsuite.

Change-Id: I72e7a01226b1dd62902cb0d6ebb1ff91a4693005
2021-12-10 08:41:37 +00:00
prerna-parashar
a41758bba5
Track user events through HubSpot events API (#4300)
Track user events through HubSpot events API
2021-12-09 12:52:51 -08:00
Ethan Adams
27c6c6aeae
ci: Golangci lint v1.43.0 changes (#4307)
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2021-12-09 20:49:48 +01:00
Michał Niewrzał
000957f64a satellite/accounting: add functionality to get and cache project segment count
We want to set maximum number of segments per project. This change will
add functionality to get number of segments currently used by project.
To avoid often DB calls segment cound will be cached and refreshed every
few minutes.

Change-Id: I2ecb6484f5afc3875c0e0dfaea360e8872f9d196
2021-12-09 12:30:52 +00:00
Stefano Fornari
af3edf2b78
scripts/release.sh: making clear if builing a production or development version (#4304)
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2021-12-09 12:38:23 +01:00
Michał Niewrzał
d94d8d1775 satellite/accounting: expose project segment limit
Exposes functionality to get and update project segment
limit. It will be used to limit number of segments per project
while uploading object.

Change-Id: I971d48eebb4e7db8b01535c3091829e73437f48d
2021-12-08 11:27:21 +00:00
Michał Niewrzał
ace90fd844 mod: bump clingy version
Latest version have a fix for:
https://github.com/storj/storj/issues/4257

Change-Id: I0e3a67fe632cd5c209b24e1ee3080563cafbb626
2021-12-08 11:10:25 +01:00
dlamarmorgan
3b51eea312 satellite/metainfo: store empty useragent in bucket attribution instead of throwing error
Previously, only valid partner IDs could be used for bucket level value attribution. Now that any useragent byte slice can be used, we should allow for empty useragent strings to be stored rather than throwing an error or leaving the bucket with no attribution.

Change-Id: I7043f835588dab1c401a27e31afd74b6b5a3e44b
2021-12-07 18:06:27 +00:00
Michał Niewrzał
5013dd8a43 satellite/metainfo: remove error message prefix
Currently detecting if error NotFound is for object
or bucket is very fragile on uplink side. We cannot
change this messages for now.

Change-Id: I6ec4d98116477812f031134e4f1c9e73bdce8b27
2021-12-07 11:59:06 +00:00
Vitalii Shpital
15a7e7f35c web/satellite: fix delete bucket issue
User could confirm delete bucket action without entering bucket name.

fix for this issue https://github.com/storj/storj/issues/4296

Change-Id: Ic76f59885fd5bd812609f831f8887c07f02360a8
2021-12-06 15:00:54 +00:00