Commit Graph

7190 Commits

Author SHA1 Message Date
Wilfred Asomani
05e57edb20 web/satellite: see token payments history
This change lists token payment histories from storjscan/coinpayments on the "Payment Methods > Storj Tokens" page

see: https://github.com/storj/storj/issues/4941

Change-Id: I178ba7940c4cb132f05673607030725a4c4b3e1c
2022-09-30 16:44:47 +00:00
Jeremy Wharton
fe3aedebe8 web/satellite: move mobile Billing nav item into My Account dropdown
This change moves the top-level Billing navigation item into the
My Account dropdown menu in the mobile view to match the behavior of
the desktop view.

Change-Id: Ibb192ec0d34e98851193efb28280bea90fa58302
2022-09-29 19:39:38 +00:00
Jennifer Johnson
47f9915a27 satellite/payments/storjscan: add info level log statement when a user successfully claims a wallet address.
If we need to restore the satelliteDB from a backup, we must preserve the user - storj token wallet address association. This commit adds a log statement of this information after a user successfully claims a wallet. We can perform a SQL update to reassign the wallet address to the user if needed.

Change-Id: Ia5c25d7ac57e59b35865d74068196e42bc4ffe87
2022-09-29 18:20:06 +00:00
Vitalii
92b25c8c96 web/satellite: added server-side encryption step for create S3 creds flow
Added server-side encryption step to create s3 credentials flow.
Reworked create access grant modal to use new VModal common component.

Issue:
https://github.com/storj/storj/issues/5119

Change-Id: Ie13fef80ced7a0330d87afe757813f35dab1c072
2022-09-29 13:46:46 +00:00
Igor
96d4968041
use python2 for draft release (#5216) 2022-09-29 16:22:08 +03:00
Michal Niewrzal
6bb3ea3b6e satellite/metabase: add tests for ListVerifySegments
Change-Id: Idc97111ab61ac3b23303371a30e3a66ea702f96e
2022-09-29 11:52:51 +00:00
prerna-parashar
94c5aaf70e
satellite/analytics: Added analytics for user behavior on Billing page (#5205)
* Added analytics for Billing page

* Fixed Jenkins errors

* Fixed lint errors

* Fixed Lint errors

* Fixed Lint errors

Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
2022-09-29 09:20:24 +03:00
Jennifer Johnson
182b94aab2 docs/blueprints: design doc for changes needed to allow for scaling audit workers
Change-Id: I3e214c900a02ba15c2fb47558f0bc75310db2a7c
2022-09-28 19:47:41 +00:00
Fadila Khadar
02924d0ded satellite/metainfo: temporary feature flag for listing query testing
Fixes: https://github.com/storj/storj/issues/5144

Change-Id: I7650f4d5dd0378e2246339e79710a695996a845c
2022-09-28 17:46:52 +00:00
paul cannon
802ff18bd8 satellite/audit: better handling of piece fetch errors
We have an alert on `not_enough_shares_for_audit` which fires too
frequently. Every time so far, it has been because of a network blip of
some nature on the satellite side.

Satellite operators are expected to have other means in place for
alerting on network problems and fixing them, so it's not necessary for
the audit framework to act in that way.

Instead, in this change, we add three new metrics,
`audit_not_enough_nodes_online`, `audit_not_enough_shares_acquired`, and
`audit_suspected_network_problem`. When an audit fails, and emits
`not_enough_shares_for_audit`, we will now determine whether it looks
like we are having network problems (most errors are connection
failures, possibly also some successful connections which subsequently
time out) or whether something else has happened.

After this is deployed, we can remove the alert on
`not_enough_shares_for_audit` and add new alerts on
`audit_not_enough_nodes_online` and `audit_not_enough_shares_acquired`.
`audit_suspected_network_problem` does not need an alert.

Refs: https://github.com/storj/storj/issues/4669

Change-Id: Ibb256bc19d2578904f71f5229111ac98e5212fcb
2022-09-28 17:02:06 +00:00
Igor
352e937813
scripts: add changelog to release draft (#4791)
* scripts: add changelog to release draft

* scripts: fix run command

* rename variable

* rename variable

* fix command

* add comment

* rename variable

Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2022-09-28 19:17:40 +03:00
Ethan
77357f9877 satellite/{accounting,redis}: Use ARGV in redis scripts
While investigating high memory consumption on a Redis instance, it was found that much of the memory consumed was from cached redis scripts.

Redis caches scripts based on the hash of the script itself.  This change uses ARGV to reduce the number of cached scripts.

Change-Id: Ia878fe81552a3067f09e60c44bb4ace25c6b5f9a
2022-09-28 14:43:00 +00:00
Egon Elbre
2b2b6d2288 satellite/admin/ui: use a valid go.mod stub
Some tools check all the go.mod files and fail when it's incorrect.

Change-Id: I1be5bd1b502e15c90b2f69caedbf5545d366a87d
2022-09-28 12:41:30 +00:00
Fadila Khadar
f58129786b satellite/metabase: copy handles version when overwriting target location
When doing server-side copy, deletes the committed version of the target location if it already exists. It does not touch pending versions. The version of the copy is set to the highest already existing version + 1.

Fixes: https://github.com/storj/storj/issues/5071

Change-Id: I1d91ac17054834b1f4f0970a9fa5d58198c58a37
2022-09-28 11:39:24 +00:00
Michal Niewrzal
a97cd97789 satellite/orders: remove unused service dependency
Orders service doesn't need buckets service anymore.

Change-Id: I27853cda87e82b528f53667e4b4866801f7bfb62
2022-09-28 08:56:36 +00:00
dlamarmorgan
88328ab240 satellite/payments/billing/billing.go: change default loop interval
Change the default loop interval for querying for new payments and adding them into the billing table from 1 minute to 15 seconds.

Change-Id: I26cf4a764cbe1de4c9b839ad60352374d8231522
2022-09-27 11:42:28 -07:00
Jeff Wendling
929642b675 cmd/uplink: remove io/ioutil
it has been deprecated since go1.16.

Change-Id: I8c15b24176e5a4687fe297d569505e84fc5affda
2022-09-27 15:53:07 +00:00
Cameron
d8fb082f89 satellite/console: create project salt endpoint on satellite web server
Introduces a new endpoint on the satellite web server to get the
project's salt. The endpoint utilizes a new console service method
GetSalt which in turn calls the project DB GetSalt method if the
user is authorized. It returns the project salt bytes as a base64
encoded string in the response.

Change-Id: Ia13b5a4b8580e7bdad0dbb98014a276b1c74b46d
2022-09-27 12:19:28 +00:00
Jeff Wendling
fa4af92392 cmd/uplink: improve cp behavior
allow multiple source paths and a single destination path.
this makes commands like `uplink cp foo* sj://bucket` work
as expected.

require at least one remote path when copying. this ensures
that users don't accidentally overwrite their local files
with other local files, which is almost never what they wanted
because they would just use cp.

Change-Id: I28948f4ff735d29db06de81fc8c2a15b9f4ee3f5
2022-09-27 10:23:41 +00:00
Erik van Velzen
d2a67fb8f7 satellite/gc/sender: concurrency
Restore functionality where retain filters can be sent out to multiple
storage nodes simultaneously.

Fixes https://github.com/storj/team-metainfo/issues/121

Change-Id: I2bf86a166b09c6a277c1cb455cdca0165ce6b8af
2022-09-27 08:10:10 +00:00
Egon Elbre
8069973dee cmd/tools/segment-verify: add failure tests
* Fix an invalid slice index calculation.

Change-Id: I7f1b85edc46df362697aa132b967d5d23f9d5522
2022-09-26 19:38:16 +00:00
Jeff Wendling
57743997b5 cmd/uplink: bump clingy
Change-Id: I464825907f19315293f944780d2e5d5ee52bf3fa
2022-09-26 17:00:45 +00:00
Wilfred Asomani
9b172cbb3a web/satellite: allow multiple create of multiple access types
This change allows the user to select multiple access grants types to create.

see: https://github.com/storj/storj/issues/4999

Change-Id: I6fa11c340dd0f2c4923d27d67fa19b92aee4061f
2022-09-26 19:18:57 +03:00
Egon Elbre
a8edeb7814 web: create valid go.mod stubs
Some tools like check-downgrades doesn't like invalid go.mod files.

Change-Id: I5a773e98f821adcbaa373a474d9420ba7447f5f7
2022-09-26 17:11:10 +03:00
Cameron
41c2bca096 satellite/{console,metainfo,satellitedb}: add projectdb method GetSalt
Add new project db method, GetSalt, to get project salt. If salt
column is empty, return the sha-256 hash of the project ID. This
new method is used in metainfo endpoint ProjectInfo to return the
project salt to the client. This is backwards compatible because
the salt column is not populated yet. The updated endpoint will
do the same thing as the current endpoint.

Change-Id: I7eba376c865e10995a5a916302feca7cd7c7efa2
2022-09-26 13:12:21 +00:00
Michal Niewrzal
1aecca1e76 satellite/repair/checker: tiny cleanup
* unused slice removed
* variable moved closer to place of use

Change-Id: I86126b8337225d4b31cabf89bc9640add7409398
2022-09-26 11:20:10 +00:00
Erik van Velzen
7c809e2955 satellite/metabase: test list objects with version >1
Assert that listing works with our new object consistency approach.

Half of https://github.com/storj/storj/issues/4868

Change-Id: I5e92f86122b50103cec7bf6d3b2c8ed103caceec
2022-09-26 08:07:11 +00:00
Erik van Velzen
6ee3993f6c satellite/gc: e2e gc test
Restore previously existing end-to-end garbage collection test using
the new separate services for bloom filter generation and storage node
communication.

Original tests can be found under:
https://github.com/storj/storj/blob/v1.63.1/satellite/gc/gc_test.go

Change-Id: I42d1ab0f9981dfe183140da4d08087f4a6cd9296
2022-09-26 07:56:35 +00:00
dlamarmorgan
3c4b2cb09a satellite/console/service.go: return billing balance as USD value
Return the balance as currency object with a value and currency. The values are returned in USDollarsMicro (6 digits after the decimal).

Change-Id: I88c87faf3311b72dedd293d4e754c2fd5c03c128
2022-09-23 09:35:26 -07:00
Michal Niewrzal
a32619c6e3 private/testplanet: benchmark with testplanet
Currently, its not strightforward how to benchmark with testplanet.
This change add Bench method to make it easy.

Change-Id: I212dfe71a18bb6ddd7a127e5b6d313b1b0c1f824
2022-09-23 16:06:43 +00:00
dlamarmorgan
0b468aca56 satellite/payments/storjscan/storjscan.go: change default block confirmations
Change the default number of required block confirmations for a payment to be confirmed from 12 to 15.

Change-Id: I44c258134c293e7691623bc00c504130aa69a96a
2022-09-23 14:30:05 +00:00
Erik van Velzen
9e2a57130a cmd/uplink: restore explicit share expiry
Due to a programming error it was possible to "share" without an expiry
implicitly. This pollutes the auth service database.

fixes https://github.com/storj/storj/issues/5188

Change-Id: I04a345662c26948c6be6c1ae6bee3b5a583bebc4
2022-09-23 11:57:08 +00:00
Artur M. Wolff
fe85fefcc5 ci: add check-downgrades checks
Change-Id: Idfe81897dc701c24c0f81b0021cb1d640e8224ac
2022-09-23 11:15:27 +00:00
paul cannon
7f1cad6faf satellite/repair: better handling of piece fetch errors
We have an alert on `repair_too_many_nodes_failed` which fires too
frequently. Every time so far, it has been because of a network blip of
some nature on the satellite side.

Satellite operators are expected to have other means in place for
alerting on network problems and fixing them, so it's not necessary for
the repair framework to act in that way.

Instead, in this change, we change the way that
`repair_too_many_nodes_failed` works. When a repair fails, we collect
piece fetch errors by type and determine from them whether it looks like
we are having network problems (most errors are connection failures,
possibly also some successful connections which subsequently time out)
or whether something else has happened.

We will now only emit `repair_too_many_nodes_failed` when the outcome
does not look like a network failure. In the network failure case, we
will instead emit `repair_suspected_network_problem`.

Refs: https://github.com/storj/storj/issues/4669

Change-Id: I49df98da5df9c606b95ad08a2bdfec8092fba926
2022-09-23 09:35:06 +00:00
Michal Niewrzal
289daf6787 satellite/metainfo: close project while testing
Project was not closed in test cases.

Change-Id: I700f671121433167fce7f350f84aa0559233c43c
2022-09-23 07:45:38 +00:00
Egon Elbre
f98d551c9b cmd/tools/segment-verify: test service
Change-Id: Ibd83960c18123e8f29e22089007dc32c8d532240
2022-09-22 17:23:02 +00:00
paul cannon
7d0885bbaa satellite/repair: move over audit.Pieces
This structure is entirely unused within the audit module, and is only
used by repair code. Accordingly, this change moves the structure from
audit code to repair code.

Also, we take the opportunity here to rename the structure to something
less generic.

Refs: https://github.com/storj/storj/issues/4669

Change-Id: If85b37e08620cda1fde2afe98206293e02b5c36e
2022-09-22 16:43:03 +00:00
Michal Niewrzal
70eda67bc1 satellite/metabase: don't count objects for loop
Before and after segments loop we are collecting stats about number of entries in segments and objects table. We are using number of segments
to validate loop execution but currently we are not using number of
objects anywhere. This change drops SQL query to count objects count as
it's not use anywhere at the moment.

Change-Id: I25ce77758870beb0daa5c0e21084a4c633a26f15
2022-09-22 16:02:08 +00:00
Jeremy Wharton
b7ae52fb35 web/satellite: add captcha verification to forgot password form
This change adds the option of requiring users to pass a captcha
verification before password recovery emails can be sent to them.
The captcha for the Forgot Password form shares a configuration with
the one for the login form.

Change-Id: I27fdb4ed42cd9da1b4b8b1e80357892dc4091765
2022-09-22 15:20:04 +00:00
Vitalii
f83652ee1d web/satellite: slight refactoring of new billing screen
New billing screen implementation is a total disaster and needs much bigger refactoring effort.
Tried to rework overall components to look like figma designs.
Fixed applying of coupon code flow.

Issue:
https://github.com/storj/storj/issues/5103

Change-Id: I42663208a003ec0abc390366a0e52c21a62751e1
2022-09-22 14:26:46 +00:00
Egon Elbre
0bfaadcc6c cmd/tools/segment-verify: fixes and more tests
* Disallow too large listing limit, which would cause a lot of memory to
  be consumed.
* Fix throttling logic and add a test.
* Fix read error handling; depending on the concurrency it can return
  the NotFound status either in the Read or Close.

Change-Id: I778f04a5961988b2480df5c7faaa22393fc5d760
2022-09-22 10:32:30 +00:00
Michal Niewrzal
a22e6bdf67 satellite/gc/bloomfilter: use int64 to count pieces
Pieces count in DB are stored as int64 and we would like to align bloom
filter processing with this type.

Change-Id: Iaec767e609a40d802077ae057520541805a7c44f
2022-09-22 09:39:53 +00:00
Qweder93
c210776a36 satellite/{metabase, metainfo}: merge logic for BeginCopyObject and BeginMoveObject
Closes https://github.com/storj/storj/issues/4514

Change-Id: I4d59fe9e3d30f67ce642d6266b878956e38f5db1
2022-09-22 09:02:20 +00:00
Malcolm Bouzi
01735c82d8 web/satellite: the date picker is currently excluding the end date.
This change makes sure that any activity that occured on end date is included.

Change-Id: I2dd3f79fc16d8b097a5ba345f0f14375165a0947
2022-09-21 17:23:46 +00:00
Wilfred Asomani
d6c7aa9290 web/satellite: fix billing history table styling
This change fixes an issue where the billing history will not display properly

see: https://github.com/storj/storj/issues/5097

Change-Id: Iac9a1f3038d4fdebf38448cccd1084ddc830402a
2022-09-21 15:49:30 +00:00
Egon Elbre
0e99f7a8cf cmd/tools/segment-verify: add loading of priority nodes
Change-Id: Idcc41469ea5f71eab1b9dccbe0f14da537386a17
2022-09-21 14:56:13 +00:00
Egon Elbre
8b527f2d12 cmd/tools/segment-verify: add throttling
Change-Id: Ia0b4ec255adc90d874f4366b80799414a1a94700
2022-09-21 14:52:51 +00:00
Jeremy Wharton
07c337f5c4 web/satellite: resolve issues with registration form resubmission
This change fixes an issue where the account registration form could
not be resubmitted if any of the following occured in the previous
submission attempt:
- The Terms of Service checkbox was unchecked
- The beta satellite terms checkbox was unchecked
- The captcha modal was closed by clicking outside its boundaries

Resolves #5167

Change-Id: If00443dc715ccd37332559af019a1c4f6c6cb61f
2022-09-21 14:18:57 +00:00
Michal Niewrzal
4e71dd302a satellite/{metainfo,metabase}: prepare feature flag MultipleVersions
We will introduce new logic for creating new objects (BeginObject).
Instead of using single version internally (1) we will be selecting first
available version during object creation. Because we need to be sure
that everything is wired up correctly we need a feature flag to be
able to control if new feature is enabled.

Change-Id: If0f8496397130811f43bf9db9fdcc2b30cd2e4ca
2022-09-21 13:07:13 +00:00
Erik van Velzen
cc39727f69 satellite/metabase: test list pending versioned objects
Assert that listing pending objects works with our new object consistency
approach.

Half of https://github.com/storj/storj/issues/4868

Change-Id: Ic7bf3b20db57e64853d0464d7dc0da5441efd56f
2022-09-21 13:00:25 +02:00