Commit Graph

6628 Commits

Author SHA1 Message Date
Moby von Briesen
24bbb7f236 satellite/satellitedb: Add webapp sessions table
Enables us to better manage sessions for satellite UI users.

Change-Id: I6695f8f1174d2e5fb970efddb0a8b14725e2b57e
2022-05-04 20:37:56 +00:00
Clement Sam
b471a5d8e3 cmd/storagenode/Dockerfile: bump storagenode-base image version
The new storagenode base image version contains the fix for the
failing "processes" supervisord event listener.

Resolves https://github.com/storj/storj/issues/4772

Change-Id: I6d67aa6f85ee33cd9abe6a663e4f9a84ea57fdbf
2022-05-04 16:00:26 +00:00
Egon Elbre
8620532a05 web/satellite: bump dependencies
This contains also multiple fixes to make it work.
The following is a non-exhaustive list.

When @Prop default value is a callback, then it is called
instead of set verbatim. This means, when you want a default
value to be a callback, then it needs to be `default: () => () => X`.

jest does not yet properly support WebWorkers, hence the code introduces
an indirection to provide the worker URL.

This in turn required removing the global "store" dependency from
the tests. As a consequence the new NotificatorPlugin takes store
as a dependency. And many of the tests are adjusted to not import
store directly.

Moved StoreModule definition to avoid initializing the global store.

Some of the router code was moved into store. We can later figure out
how to structure it better and move it back.

bip39 needs explicit fallbacks for some of the dependencies.

Fixes to timer mocking. jest supports it natively.

Remove sinon dependency. jest provides all the functionality we need.

Change-Id: I7af3599390c63ce9f99dbd0b1e0870e9f8ca994d
2022-05-04 15:02:01 +00:00
Vitalii
640c74a5e0 web/satellite: prevent access grant creation
In case user requests account deletion access grant is being created when user visits buckets screen.
With this fix access grant won't be created if user deletes all the buckets first and then deletes all the access grants.

Change-Id: I9a348bcf35a050838dbf6e1e7c682499d2f0a278
2022-05-04 13:05:45 +00:00
Fadila Khadar
af1f0aa943 satellite/repair: lighten tests covering excluded countries
TestSegmentInExcludedCountriesRepair and TestSegmentInExcludedCountriesRepairIrreparable are using 20 storage nodes.
This change make them use 7 by adjusting the test redundancy scheme.

Change-Id: I1a44aa8b997d6edcc9a3305fdd0dac57e4d525b5
2022-05-04 07:51:59 +00:00
Clement Sam
9751ac9de2 cmd/storagenode: fix failing supervisord "processes" eventlistener
/bin/stop-supervisor fails in posix shell since the standard read utility
takes at least one variable's name as argument.

Changing the header #!bin/sh to #!/bin/bash fixes this issue.
`read` with no variable's name works in bash.

Looks like the shell in alpine isn't POSIX-compliant so we didn't
encounter this issue on alpine.

Also, I changed the name from "processes" to "processes-exit-eventlistener"
to make it clearer in the logs since supervisord spawns event listeners as
separate processes.

Change-Id: Ife9378c2013e2eb54f2adcd52a163d64eaacbbab
2022-05-04 02:44:23 +00:00
Clement Sam
87cb2e92b9 cmd/storagenode: bump storagenode-base image version
Change-Id: I8df391ee12c70cc599b54115b54eb987a2114b74
2022-05-04 01:59:00 +00:00
Clement Sam
a9faf756b0 cmd/storagenode: get rid of CRITICAL log when running supervisord as root
Closes https://github.com/storj/storj/issues/4729

Change-Id: Iea9a60ff60d753af375977f9bd90557574f80de0
2022-05-03 21:18:11 +00:00
Qweder93
384118ee1e satellite/metabase: use pgutil.NullTimestampTZArray
This code cleanup to use helper DB method.

Change-Id: Idc438234cb4f7da23d6ade2d69f5d889f15e9009
2022-05-03 19:30:21 +00:00
Egon Elbre
580cfb6d4b testsuite/ui/uitest: do not use localhost for testing
We must always use 127.0.0.1 for testing and optionally support
customizing it for different scenarios.

Also few minor fixes to testsuite/ui/satellite.

Change-Id: I46ef151c6aa92768cb24fc2b8c768f10d5dea75d
2022-05-03 15:09:15 +00:00
Clement Sam
4a46d41c1c cmd/storagenode: change supervisord pidfile location
When running the docker auto-updater image as non-root user,
supervisord logs a "CRIT could not write pidfile /run/supervisord.pid"
since the user does not have permission to the /run directory.

Changing the location to /etc/supervisor fixes it because permissions
are set for non-root access of the /etc/supervisor directory.

Closes https://github.com/storj/storj/issues/4730

Change-Id: Id463f3a08db44dd9283921ece4575abdad9bd7f2
2022-05-03 12:54:40 +00:00
Egon Elbre
37c18bf790 web/multinode: components must not have constructor
Per https://class-component.vuejs.org/guide/caveats.html we always
must use lifecycle hooks intead of constructor. Otherwise some
of the props might not be initialized.

Change-Id: Ic6242cae61cd0775c3b0295850c267ee875cb0e0
2022-05-02 09:06:36 +00:00
hovex023
58f957bd1d
web/satellite: Add new access grants flow (#4765)
* Added new feature Flag for new Access Grant Flow. 

* Added 3 cards to access grant view for S3, CLI and Access grant to replace old header

* Added new formatting, text and Icon for Access Grant Delete Popup modal
2022-04-29 10:31:52 -05:00
Stefan Benten
345ab87b5c cmd/uplink: adding output flag for ls command
With this change users can use the uplink cli in
scripts (ie. bash) more easily, since the output
can be switched to an easier processable json format.
It keeps the default of tabbed output.


Change-Id: I37e2c55f75c2250c3119fd8df8b66a766ff9096b
2022-04-29 10:32:04 +00:00
Vitalii
96411ba56a rest-api: endpoint reworkings
Added documentation.
Replaced PUT request with POST request.
Added inline param support for PATCH request.
Replaced unix timestamps handling with RFC-3339 timestampts handling.
Added 'Bearer' method requirement for Authorization header.

Change-Id: I4faa3864051dd18826c2c583ada53666d4aaec44
2022-04-28 18:17:54 +00:00
prerna-parashar
894b7b1cf3
satellite/analytics: Track event "Credit Card Added" (#4749)
satellite/analytics: Track event "Credit Card Added"
2022-04-28 11:12:42 -07:00
Qweder93
04b22f24eb satellite/metabase: copy expires_at to segments while FinishCopyObject
Resolves: https://github.com/storj/storj/issues/4726

Change-Id: I0bb222b7d5d3c66fe96c78a8aa1ea8f3b0a33f2b
2022-04-27 18:39:54 +00:00
Paul Willoughby
8efed4b270 cmd/uplink: raise fd limits
Change-Id: I507c92c38d45d3d2be3edee08b6b85552ae21c43
2022-04-27 18:11:44 +00:00
Mya
5cebbdee03 web/satellite: add consent screen for oauth
When an application wants to interact with resources on behalf of
an end-user, it needs to be granted access. In OAuth, this is done
when a user submits the consent screen.

Change-Id: Id838772f76999f63f5c9dbdda0995697b41c123a
2022-04-27 14:33:07 +00:00
Jeff Wendling
4f196dd39e cmd/uplink: bump clingy
this causes an error code to be returned for missing commands

Change-Id: I1ac33ec7c0de506e964fb823e0ed80832cb80d3b
2022-04-27 10:25:44 +00:00
Jeff Wendling
fead0db981 cmd/uplink: return nicer error message during import
Fixes #4750

Change-Id: Icef3340dd04d1c502b1651dcc70362895f258563
2022-04-27 09:30:57 +00:00
Jeff Wendling
f25ead5f98 cmd/uplink: set default parallelism to 1
Change-Id: Ic4198131c9958cc864fd861f983e32776bf56595
2022-04-26 22:55:11 +00:00
Cameron
dd6ab25cbd {cmd/satellite/reports, satellite/attribution}: type and variable name adjustments
Change-Id: I553b7ed9cb702479911baaf54d77d8f2662faaa2
2022-04-26 20:12:38 +00:00
Egon Elbre
175d1e694c mod: bump storj.io/private
This updates the database drivers and sqlite implementation.

Change-Id: I384d81d7beca7267f7c1d6115b3e5e33629f8dc7
2022-04-26 19:18:52 +00:00
nadimhq
dd6aac9b39
satellite/metainfo: enable server-side copy in production (#4708)
these changes enable server-side copy in production
2022-04-26 09:58:16 -04:00
Michał Niewrzał
db5d781735 satellite/metainfo: simplify version collector logging
Version collector previously returned errors and logged them in the
calling code.  It is cleaner to log inside version collector.

Change-Id: I52cb49a1ef53f3f1f51692ddb26ec095cfd0f100
2022-04-26 10:16:20 +00:00
Michał Niewrzał
e66beb2429 satellite/metainfo: set metadata with BeginObject
This change makes possible to set metadata at the beginning
of upload.

https://github.com/storj/team-metainfo/issues/105

Change-Id: If5b6b90243d9c9318a8b72c2af3fbde400c5d8e7
2022-04-26 08:04:27 +00:00
Michał Niewrzał
6e5a94698e satellite/metabase: add option to override metadata with CommitObject
We were already able to override (or not) metadata with this method
but to be explicit we are introducting new option to control storing
metadata with object. Separate option should be less error prone.

https://github.com/storj/team-metainfo/issues/105

Change-Id: I4c5bce953a633a0009b05c5ca84266ca6ceefc26
2022-04-26 08:03:52 +00:00
Moby von Briesen
ed5ebb2527 satellite: Rename "acct mgmt api" to "rest api"
"REST API" is a more accurate descriptor of the generated API in the
console package than "account management API". The generated API is very
flexible and will allow us to implement many more endpoints outside the
scope of "account management", and "account management" is not very well
defined to begin with.

Change-Id: Ie87faeaa3c743ef4371eaf0edd2826303d592da7
2022-04-25 18:51:46 +00:00
Egon Elbre
1ed36e9fea cmd/uplink: make clearer ctx cancellation path in copy
When ctx is cancelled limiter won't start a new goroutine.
The code didn't immediately return an error in that case.

The dst.Commit(ctx) would fail anyways due to a cancelled ctx.
However, we can make the behavior clearer by returning immediately.

Change-Id: I65df7ca85de55813f3200a50db2eaaa7a297ba2c
2022-04-25 18:16:46 +03:00
Egon Elbre
c2bdd4effa cmd/uplink/ulfs: disallow writes after first failure
It was possible for the a previous write / part to fail or be aborted
and the next part write still happened. This causes a data ordering
corruption.

The whole write to parallel stdout fails, so there shouldn't be
confusion with regards to the output acceptability. However, it would
be clearer, if we avoided writing out-of-order data... mainly to be
clear that we didn't corrupt the data, just that it's incomplete.

Change-Id: I97b0d14404f29e8615e7d29b10cbd61ccb861e40
2022-04-25 18:16:46 +03:00
Vitalii
318f4dc688 satellite/admin: extend user update query
Extended user update query so prod owner can change user's paid tier status, bandwidth, storage and segment limits.

Change-Id: I82768afd1e50f653a50f7020310ce1e91578d746
2022-04-25 13:44:25 +00:00
JT Olio
abc9e2493f satellite/compensation: add a code that crypthopper-go now uses
it's a bit weird that these code definitions are in storj/storj
instead of storj/crypthopper-go, but as it stands, we should make
sure this package knows about all the codes in use.

Change-Id: I8df4666a015098e2d2e536d2f6c8ca5317a4369c
2022-04-25 10:46:51 +00:00
Egon Elbre
0513d4cf3a docs/blueprints: add async packing
Describe a process for packing small files.

Change-Id: I9f6ad9a85ccf0a92705a47e2ff8efa0072e65230
2022-04-23 06:34:17 +00:00
Egon Elbre
847ddaaab0 cmd/uplink: cancel on failed copy
Also ensure that abort is given at least 5 seconds to clear up any
pending uploads on cancellation.

Change-Id: I814aa407ee5783f2609a76b54de2879dcd5f89bb
2022-04-22 14:57:24 +03:00
Kaloyan Raev
978e0f1a26 cmd/uplink: cp sets connection pool capacity based on parallelism
If the cp command is executed with higher level of parallelism, it would
open more connections to storage nodes at the same time. Therefore, the
connection pool capacity should be expanded accordingly.

The pool capacity is set to 100 * parallelism.

Change-Id: Ia8b3ab6a99340d8cbb87a7b80c3354b2b21c1958
2022-04-21 14:10:08 +00:00
Clement Sam
b24e5cbc44 cmd/storagenode: silent authentication checks CRITICAL logs by supervisord
set dummy user/password in the [unix_http_server] and [supervisorctl] sections of the config file to remove the CRIT log entries as suggested here: https://github.com/Supervisor/supervisor/issues/717

Closes https://github.com/storj/storj/issues/4731

Change-Id: Iec10be39416f3aaa2d7febb414e15a63a4b23a4c
2022-04-21 13:10:42 +00:00
Qweder93
cb1bf52794 satellite/metainfo: add expireddeletion test for copied objects
We implemented server-side copy feature and we would like to
confirm that it is not affecting expired deletion service.

Resolves: https://github.com/storj/storj/issues/4698

Change-Id: Ia8ca27a7ab7764a48a0c85dc7be80a58bfc83729
2022-04-21 12:07:00 +00:00
Qweder93
7e6618cfad satellite/accounting: test tally on copied object flaky fix
Resolves: https://github.com/storj/storj/issues/4732

Change-Id: I2abf50aafc62dd97be12b0c0573db2f95d560796
2022-04-21 10:30:17 +00:00
Yaroslav Vorobiov
553ef9d640 satellite/satellitedb: suspended column removal migration
Last part of backwards compatible db migration to remove "suspended" column.
Removes exeption which removes "suspended" column in tests from `migrate_test.go`.
Adds DB migration to remove "suspended" column from 'nodes' and 'reputations' tables.

Change-Id: I02051279f6f4181e966c567919af0e774583f165
2022-04-20 17:59:11 +00:00
Michał Niewrzał
671923cc60 satellite/admin: add mising segment limit to top menu
We missed menu entry for segment limits.

Change-Id: I977515acb282582b0730e581662ca262cc75f0b5
2022-04-20 16:36:14 +00:00
Yaroslav Vorobiov
3f47d19aa6 satellite/overlay: add disqualification reason
Add disqualification reason to NodeDossier.
Extend DB.DisqualifyNode with disqualification reason.
Extend reputation Service.TestDisqualifyNode with disqualification reason.

Change-Id: I8611b6340c7f42ac1bb8bd0fd7f0648ad650ab2d
2022-04-20 13:29:31 +00:00
Yaroslav Vorobiov
4223fa01f8 satellite/reputation: add disqualification reason for status update
Set disqualification reason when reputations stats are updated on DB.Update.
Added tests for DisqualifyNode and for disqualification cases which happens during Update.

Change-Id: I00130ab5d9722422805159ad2f183c205de60f7e
2022-04-20 13:29:10 +00:00
paul cannon
1422a1ff19 cmd/uplink: use 64 MiB for parallel chunk size, not 64 MB
I don't think it should matter for correctness whether this matches the
segment size or not, so I think there is something else wrong. However,
making this change seems to eliminate the "corruption when ulimit -n is
too low" problem we're seeing right now.

Change-Id: I232fe0d0a371b86ddf902e8c2d4778e140b2f1fc
2022-04-19 12:08:08 -05:00
Vitalii
d506e9c07b web/satellite: store selected os tab during onboarding flow
Store selected os tab during onboarding flow.
Use stored value to show correct tab during all the CLI steps.

Change-Id: I82e9af7e5dd3a7eaf503ed4f38789b7b0fb4aa84
2022-04-19 14:39:25 +00:00
paul cannon
3f3f028c88 satellite/gracefulexit: don't mark GE done when it's not done
When an api server is processing a graceful exit (node is connected and
getting lists of pieces to transfer), and the api server is shut down,
it was incorrectly marking all pending graceful exits as complete. The
GE then either passed or failed depending on the ratio of successfully
transferred pieces to unsuccessful pieces. In at least one case, _no_
pieces were transferred at all before the GE was marked a success.

Change-Id: I62cfab54a2296572c2e654eb460b62f772b7a60b
2022-04-19 14:00:29 +00:00
Cameron
48fb3e947c cmd/satellite/reports: sum attribution data by user agent
Attribution is attached to bucket usage, but that's more granular than
necessary for the attribution report. This change iterates over the
bucket attributions, parses the user agent, converts the first entry
to lower case, and uses that as the key to a map which holds the
attribution totals for each unique user agent.

Change-Id: Ib2962ba0f57daa8a7298f11fcb1ac44a8bb97875
2022-04-18 13:23:37 +00:00
Egon Elbre
d7cd3a0ff8 Jenkinsfile: mod-tidy doesn't need the initial file
Change-Id: I7f35157b5f5306a84817019e3b9613e40ac45f3f
2022-04-17 17:39:03 +00:00
NickolaiYurchenko
c32ca6e67f apigen: endpint to update project
Implemented new endpoint for project update using apigen.
Implemented new service method compatible with new generated api.

Change-Id: Ic0a7e0bbf3ea942275bd927d6e30cfb7e721e9c1
2022-04-14 22:21:08 +00:00
NickolaiYurchenko
c944c3f289 web/satellite: error while adding invalid card handled
Change-Id: I9773503baef91b176a080c3081f8ac63e5ca27e8
2022-04-14 21:52:58 +00:00