Commit Graph

6564 Commits

Author SHA1 Message Date
Vitalii
96411ba56a rest-api: endpoint reworkings
Added documentation.
Replaced PUT request with POST request.
Added inline param support for PATCH request.
Replaced unix timestamps handling with RFC-3339 timestampts handling.
Added 'Bearer' method requirement for Authorization header.

Change-Id: I4faa3864051dd18826c2c583ada53666d4aaec44
2022-04-28 18:17:54 +00:00
prerna-parashar
894b7b1cf3
satellite/analytics: Track event "Credit Card Added" (#4749)
satellite/analytics: Track event "Credit Card Added"
2022-04-28 11:12:42 -07:00
Qweder93
04b22f24eb satellite/metabase: copy expires_at to segments while FinishCopyObject
Resolves: https://github.com/storj/storj/issues/4726

Change-Id: I0bb222b7d5d3c66fe96c78a8aa1ea8f3b0a33f2b
2022-04-27 18:39:54 +00:00
Paul Willoughby
8efed4b270 cmd/uplink: raise fd limits
Change-Id: I507c92c38d45d3d2be3edee08b6b85552ae21c43
2022-04-27 18:11:44 +00:00
Mya
5cebbdee03 web/satellite: add consent screen for oauth
When an application wants to interact with resources on behalf of
an end-user, it needs to be granted access. In OAuth, this is done
when a user submits the consent screen.

Change-Id: Id838772f76999f63f5c9dbdda0995697b41c123a
2022-04-27 14:33:07 +00:00
Jeff Wendling
4f196dd39e cmd/uplink: bump clingy
this causes an error code to be returned for missing commands

Change-Id: I1ac33ec7c0de506e964fb823e0ed80832cb80d3b
2022-04-27 10:25:44 +00:00
Jeff Wendling
fead0db981 cmd/uplink: return nicer error message during import
Fixes #4750

Change-Id: Icef3340dd04d1c502b1651dcc70362895f258563
2022-04-27 09:30:57 +00:00
Jeff Wendling
f25ead5f98 cmd/uplink: set default parallelism to 1
Change-Id: Ic4198131c9958cc864fd861f983e32776bf56595
2022-04-26 22:55:11 +00:00
Cameron
dd6ab25cbd {cmd/satellite/reports, satellite/attribution}: type and variable name adjustments
Change-Id: I553b7ed9cb702479911baaf54d77d8f2662faaa2
2022-04-26 20:12:38 +00:00
Egon Elbre
175d1e694c mod: bump storj.io/private
This updates the database drivers and sqlite implementation.

Change-Id: I384d81d7beca7267f7c1d6115b3e5e33629f8dc7
2022-04-26 19:18:52 +00:00
nadimhq
dd6aac9b39
satellite/metainfo: enable server-side copy in production (#4708)
these changes enable server-side copy in production
2022-04-26 09:58:16 -04:00
Michał Niewrzał
db5d781735 satellite/metainfo: simplify version collector logging
Version collector previously returned errors and logged them in the
calling code.  It is cleaner to log inside version collector.

Change-Id: I52cb49a1ef53f3f1f51692ddb26ec095cfd0f100
2022-04-26 10:16:20 +00:00
Michał Niewrzał
e66beb2429 satellite/metainfo: set metadata with BeginObject
This change makes possible to set metadata at the beginning
of upload.

https://github.com/storj/team-metainfo/issues/105

Change-Id: If5b6b90243d9c9318a8b72c2af3fbde400c5d8e7
2022-04-26 08:04:27 +00:00
Michał Niewrzał
6e5a94698e satellite/metabase: add option to override metadata with CommitObject
We were already able to override (or not) metadata with this method
but to be explicit we are introducting new option to control storing
metadata with object. Separate option should be less error prone.

https://github.com/storj/team-metainfo/issues/105

Change-Id: I4c5bce953a633a0009b05c5ca84266ca6ceefc26
2022-04-26 08:03:52 +00:00
Moby von Briesen
ed5ebb2527 satellite: Rename "acct mgmt api" to "rest api"
"REST API" is a more accurate descriptor of the generated API in the
console package than "account management API". The generated API is very
flexible and will allow us to implement many more endpoints outside the
scope of "account management", and "account management" is not very well
defined to begin with.

Change-Id: Ie87faeaa3c743ef4371eaf0edd2826303d592da7
2022-04-25 18:51:46 +00:00
Egon Elbre
1ed36e9fea cmd/uplink: make clearer ctx cancellation path in copy
When ctx is cancelled limiter won't start a new goroutine.
The code didn't immediately return an error in that case.

The dst.Commit(ctx) would fail anyways due to a cancelled ctx.
However, we can make the behavior clearer by returning immediately.

Change-Id: I65df7ca85de55813f3200a50db2eaaa7a297ba2c
2022-04-25 18:16:46 +03:00
Egon Elbre
c2bdd4effa cmd/uplink/ulfs: disallow writes after first failure
It was possible for the a previous write / part to fail or be aborted
and the next part write still happened. This causes a data ordering
corruption.

The whole write to parallel stdout fails, so there shouldn't be
confusion with regards to the output acceptability. However, it would
be clearer, if we avoided writing out-of-order data... mainly to be
clear that we didn't corrupt the data, just that it's incomplete.

Change-Id: I97b0d14404f29e8615e7d29b10cbd61ccb861e40
2022-04-25 18:16:46 +03:00
Vitalii
318f4dc688 satellite/admin: extend user update query
Extended user update query so prod owner can change user's paid tier status, bandwidth, storage and segment limits.

Change-Id: I82768afd1e50f653a50f7020310ce1e91578d746
2022-04-25 13:44:25 +00:00
JT Olio
abc9e2493f satellite/compensation: add a code that crypthopper-go now uses
it's a bit weird that these code definitions are in storj/storj
instead of storj/crypthopper-go, but as it stands, we should make
sure this package knows about all the codes in use.

Change-Id: I8df4666a015098e2d2e536d2f6c8ca5317a4369c
2022-04-25 10:46:51 +00:00
Egon Elbre
0513d4cf3a docs/blueprints: add async packing
Describe a process for packing small files.

Change-Id: I9f6ad9a85ccf0a92705a47e2ff8efa0072e65230
2022-04-23 06:34:17 +00:00
Egon Elbre
847ddaaab0 cmd/uplink: cancel on failed copy
Also ensure that abort is given at least 5 seconds to clear up any
pending uploads on cancellation.

Change-Id: I814aa407ee5783f2609a76b54de2879dcd5f89bb
2022-04-22 14:57:24 +03:00
Kaloyan Raev
978e0f1a26 cmd/uplink: cp sets connection pool capacity based on parallelism
If the cp command is executed with higher level of parallelism, it would
open more connections to storage nodes at the same time. Therefore, the
connection pool capacity should be expanded accordingly.

The pool capacity is set to 100 * parallelism.

Change-Id: Ia8b3ab6a99340d8cbb87a7b80c3354b2b21c1958
2022-04-21 14:10:08 +00:00
Clement Sam
b24e5cbc44 cmd/storagenode: silent authentication checks CRITICAL logs by supervisord
set dummy user/password in the [unix_http_server] and [supervisorctl] sections of the config file to remove the CRIT log entries as suggested here: https://github.com/Supervisor/supervisor/issues/717

Closes https://github.com/storj/storj/issues/4731

Change-Id: Iec10be39416f3aaa2d7febb414e15a63a4b23a4c
2022-04-21 13:10:42 +00:00
Qweder93
cb1bf52794 satellite/metainfo: add expireddeletion test for copied objects
We implemented server-side copy feature and we would like to
confirm that it is not affecting expired deletion service.

Resolves: https://github.com/storj/storj/issues/4698

Change-Id: Ia8ca27a7ab7764a48a0c85dc7be80a58bfc83729
2022-04-21 12:07:00 +00:00
Qweder93
7e6618cfad satellite/accounting: test tally on copied object flaky fix
Resolves: https://github.com/storj/storj/issues/4732

Change-Id: I2abf50aafc62dd97be12b0c0573db2f95d560796
2022-04-21 10:30:17 +00:00
Yaroslav Vorobiov
553ef9d640 satellite/satellitedb: suspended column removal migration
Last part of backwards compatible db migration to remove "suspended" column.
Removes exeption which removes "suspended" column in tests from `migrate_test.go`.
Adds DB migration to remove "suspended" column from 'nodes' and 'reputations' tables.

Change-Id: I02051279f6f4181e966c567919af0e774583f165
2022-04-20 17:59:11 +00:00
Michał Niewrzał
671923cc60 satellite/admin: add mising segment limit to top menu
We missed menu entry for segment limits.

Change-Id: I977515acb282582b0730e581662ca262cc75f0b5
2022-04-20 16:36:14 +00:00
Yaroslav Vorobiov
3f47d19aa6 satellite/overlay: add disqualification reason
Add disqualification reason to NodeDossier.
Extend DB.DisqualifyNode with disqualification reason.
Extend reputation Service.TestDisqualifyNode with disqualification reason.

Change-Id: I8611b6340c7f42ac1bb8bd0fd7f0648ad650ab2d
2022-04-20 13:29:31 +00:00
Yaroslav Vorobiov
4223fa01f8 satellite/reputation: add disqualification reason for status update
Set disqualification reason when reputations stats are updated on DB.Update.
Added tests for DisqualifyNode and for disqualification cases which happens during Update.

Change-Id: I00130ab5d9722422805159ad2f183c205de60f7e
2022-04-20 13:29:10 +00:00
paul cannon
1422a1ff19 cmd/uplink: use 64 MiB for parallel chunk size, not 64 MB
I don't think it should matter for correctness whether this matches the
segment size or not, so I think there is something else wrong. However,
making this change seems to eliminate the "corruption when ulimit -n is
too low" problem we're seeing right now.

Change-Id: I232fe0d0a371b86ddf902e8c2d4778e140b2f1fc
2022-04-19 12:08:08 -05:00
Vitalii
d506e9c07b web/satellite: store selected os tab during onboarding flow
Store selected os tab during onboarding flow.
Use stored value to show correct tab during all the CLI steps.

Change-Id: I82e9af7e5dd3a7eaf503ed4f38789b7b0fb4aa84
2022-04-19 14:39:25 +00:00
paul cannon
3f3f028c88 satellite/gracefulexit: don't mark GE done when it's not done
When an api server is processing a graceful exit (node is connected and
getting lists of pieces to transfer), and the api server is shut down,
it was incorrectly marking all pending graceful exits as complete. The
GE then either passed or failed depending on the ratio of successfully
transferred pieces to unsuccessful pieces. In at least one case, _no_
pieces were transferred at all before the GE was marked a success.

Change-Id: I62cfab54a2296572c2e654eb460b62f772b7a60b
2022-04-19 14:00:29 +00:00
Cameron
48fb3e947c cmd/satellite/reports: sum attribution data by user agent
Attribution is attached to bucket usage, but that's more granular than
necessary for the attribution report. This change iterates over the
bucket attributions, parses the user agent, converts the first entry
to lower case, and uses that as the key to a map which holds the
attribution totals for each unique user agent.

Change-Id: Ib2962ba0f57daa8a7298f11fcb1ac44a8bb97875
2022-04-18 13:23:37 +00:00
Egon Elbre
d7cd3a0ff8 Jenkinsfile: mod-tidy doesn't need the initial file
Change-Id: I7f35157b5f5306a84817019e3b9613e40ac45f3f
2022-04-17 17:39:03 +00:00
NickolaiYurchenko
c32ca6e67f apigen: endpint to update project
Implemented new endpoint for project update using apigen.
Implemented new service method compatible with new generated api.

Change-Id: Ic0a7e0bbf3ea942275bd927d6e30cfb7e721e9c1
2022-04-14 22:21:08 +00:00
NickolaiYurchenko
c944c3f289 web/satellite: error while adding invalid card handled
Change-Id: I9773503baef91b176a080c3081f8ac63e5ca27e8
2022-04-14 21:52:58 +00:00
Michał Niewrzał
6188c4c0dd satellite/metainfo: enable segment limit by default
Change-Id: I26cc0bbd97d6cc2ffc96213788f261efb749d028
2022-04-14 06:12:09 +00:00
Vitalii
513ad99756 satellite/satellitedb: migrate wrongly affected users to have default limits
Migrate free tier users to have default limits if their limits were set to 0.
They were affected by incorrect working of Update user query.

Change-Id: I4c49c8d99b12dba2b9b0ab61b2175085976dcc95
2022-04-13 13:33:03 +00:00
Qweder93
8e0196a6cc satellite/accounting: test accounting/tally with object copies
We implemented server-side copy feature and we would like to
confirm that it is not affecting accounting/tally service.

Resolves https://github.com/storj/storj/issues/4697

Change-Id: I3944ea52c0acc68107ec15c1911750dc7d947501
2022-04-13 09:16:43 +00:00
Yaroslav Vorobiov
a401450e3d cmd/storagenode-updater: add should-update cmd
This change add Separate CMD to check if a binary needs to be updated

Change-Id: I87f4214f84105464d283f1622dcb4ca9e796e771
2022-04-12 17:55:54 +00:00
Clement Sam
e9611801ad cmd/{storagenode,storagenode-updater}: add Process name to logs
Now that we have both the storagenode and updater processes running
in a single docker container, we need a way to know which log entry
is logged by any of the processes.

This change includes a Process field in the log entries.

Resolves https://github.com/storj/storj/issues/4648

Change-Id: I167b9ab65728a41136d264b5fe2c41bb64ed1785
2022-04-12 16:49:56 +00:00
Michał Niewrzał
623cb16b6e satellite/gracefulexit: test GE with copies
Test case to verify if server-side copy doesn't affect
GE in any negative way.

Fixes https://github.com/storj/storj/issues/4699

Change-Id: I8c385767cca61499d46d9cb8de7318c56e5d7397
2022-04-12 15:59:14 +00:00
Erik van Velzen
86d742f7c6 satellite/audit: verify auditing of copies
Check that audit works in the face of copies.

Closes https://github.com/storj/storj/issues/4695

Change-Id: I1ee79a73c28e3f4842eebe8c4e4cd9ecf2e51e57
2022-04-12 15:24:54 +00:00
Michał Niewrzał
61a76f35ef Jenkinsfile: enable 'Cockroach Backwards Compatibility' tests
We had an issue with CRDB where newer version forbid type
of SQL query we were using. We disabled CRDB bc tests until we
will have release without problematic query. Now we can enable
tests.

Change-Id: I275cbecebdcbfef587281f2daaf677d01860b23d
2022-04-12 13:15:43 +00:00
Clement Sam
583d48c432 mod: bump storj/private
Change-Id: Ia8299a082347b8f4ecc2f36988387b3a48701da5
2022-04-12 11:09:10 +00:00
Vitalii Shpital
b4cec4fd0b satellite/satellitedb: add columns to users table to control failed login attempts
Added failed_login_count and login_lockout_expiration columns to users table to control users failed login attempts.
We want to prevent brute forcing of user login so this is the first step.

Change-Id: I06b0b9f5415a1922e08cd9908893b2fd3c26bca0
2022-04-12 08:37:07 +00:00
Vitalii
36bef42e0a web/satellite: add missing copy buttons for onboarding flow
Added missing copy buttons for commands on Install Uplink CLI screen of onboarding flow.

Change-Id: I34c4e7b56e955d8571aea2d570e31cb7bfb011c5
2022-04-12 08:09:42 +00:00
Vitalii
fc074b693b web/satellite: fix demo-bucket creation issue
Store demo-bucket creation status in browser local store so that it's not created every time when user enters buckets screen.
It will start working after first usage in new browser.
So if user removes bucket in Chrome and opens Buckets screen in Firefox demo-bucket will be created but only once.

Change-Id: I9f5811d97ab6208c5f757ededcd7c36cd864795c
2022-04-12 07:42:43 +00:00
Erik van Velzen
16480c97da satellite/metabase: deduplicate delete query
Use the same query when deleting a single object or multiple.
I have chosen not to deduplicate the row "scan" logic because
it is less complicated code and this change would expand to other
parts of the codebase.

Part of https://github.com/storj/storj/issues/4700

Change-Id: I7a958c78c903b2bddd72ca217971f7e8e02a0d0c
2022-04-11 14:12:18 +00:00
Kaloyan Raev
de6852510f private/testplanet: configurable timeout
Testplanet limits the execution of a single test case to 3 minutes.

This change adds a Timeout field to the Testplanet's config, so test
cases can configure their timeout. This is helpful when executing larger
3rd party test suite on top of Testplanet.

Change-Id: Ibbf7c5ffdc0a9e723e7e28b885eac084f04c6ca1
2022-04-11 13:19:08 +00:00