Commit Graph

7023 Commits

Author SHA1 Message Date
Michal Niewrzal
11d1e623b5 satellite/metabase/segmentloop: don't do rate limiting if disabled
We have a code to  limit segments loop in case it will hit DB to hard
but so far we didn't use this loop feature  in production. This is a
simple change to avoid logic responsible for rate limiting and its
monitoring if limiting is disabled (RateLimit = 0)

Change-Id: I43e07b407c6e65cf252303159d052eef250d1bea
2022-10-11 10:55:30 +00:00
Egon Elbre
142a04f208 cmd/tools/segment-verify: add connection pool
Change-Id: If0f85edbf99438ac41c23fc7107fdab926288cc2
2022-10-11 09:06:44 +03:00
Egon Elbre
8916f2ee92 cmd/tools/segment-verify: allow ignoring specific nodes
This adds a new flag that allows to ignore some nodes completely.

Change-Id: I203d25f931262c809037e25e9c37e9a89bf47026
2022-10-10 20:14:38 +03:00
Egon Elbre
9e50d837e3 cmd/tools/segment-verify: add tool for summarizing log
Change-Id: I3177ab71dfd25e11adfedce32a530d83dda63bd6
2022-10-10 20:02:50 +03:00
Egon Elbre
5f01dad3a3 cmd/tools/segment-verify: add total progress indicator
Change-Id: Ib729abf6adbeba8d94e08c7e11497c6d5ddd5ec2
2022-10-10 20:02:30 +03:00
Michal Niewrzal
db1409eea6 satellite/metabase: use SUBSTRING with objects iterator
Until this change we were stripping prefix from object key on satellite side. Because of that we were transferring over network unnecessary data
from DB. This change adjusts iterator SQL queries to use SUBSTRING to
remove prefix on DB side and avoid sending it to satellite.

Benchmark against 'main':
unfortunately "time/op" is very unstable while doing local bench in this
case and sometimes  there is no difference in time and sometimes its up to 18%. I never saw results when old solution is faster then new one. Results for "alloc/op" and "allocs/op" are rather consistent.

name                                                 old time/op    new time/op    delta
NonRecursiveListing/Cockroach/listing_no_prefix-8      1.98ms ± 6%    2.05ms ±23%     ~     (p=1.000 n=9+10)
NonRecursiveListing/Cockroach/listing_with_prefix-8    3.97ms ± 8%    3.42ms ±20%  -13.86%  (p=0.005 n=10+10)
NonRecursiveListing/Cockroach/listing_only_prefix-8    8.42ms ±16%    7.58ms ± 5%   -9.91%  (p=0.002 n=10+10)

name                                                 old alloc/op   new alloc/op   delta
NonRecursiveListing/Cockroach/listing_no_prefix-8      16.7kB ± 0%    16.9kB ± 0%   +1.16%  (p=0.000 n=10+10)
NonRecursiveListing/Cockroach/listing_with_prefix-8    27.3kB ± 0%    28.2kB ± 0%   +3.31%  (p=0.000 n=10+10)
NonRecursiveListing/Cockroach/listing_only_prefix-8    60.0kB ± 0%    62.4kB ± 0%   +3.93%  (p=0.000 n=10+8)

name                                                 old allocs/op  new allocs/op  delta
NonRecursiveListing/Cockroach/listing_no_prefix-8         312 ± 0%       315 ± 0%   +0.96%  (p=0.000 n=10+10)
NonRecursiveListing/Cockroach/listing_with_prefix-8       526 ± 0%       541 ± 0%   +2.85%  (p=0.000 n=10+10)
NonRecursiveListing/Cockroach/listing_only_prefix-8     1.16k ± 0%     1.23k ± 0%   +5.24%  (p=0.000 n=10+10)

Change-Id: I23e501494ededafb2dd5ea903e8e4e313b42e956
2022-10-10 14:27:26 +00:00
nadimhq
c921cd5ccf
docs/testplan: Adding a testplan for Token Payment Processor (#4672)
* docs/testplan: Adding a testplan for Token Payment Processor

This testplan is going to cover the token payment processor, it will go over must haves and additional features that we can add on later.

* docs/testplan: Adding a testplan for Token Payment Processor

Updated final testplan for review

* docs/testplan: Adding a testplan for Token Payment Processor

Removed unnecessary tests related to mempool scanning

* Update token-payment-processor-testplan.md

Co-authored-by: littleskunk <jens.heimbuerge@googlemail.com>
2022-10-10 15:01:14 +02:00
Malcolm Bouzi
ff99392fd7 web/satellite: update bucket share modals to match designs
Change-Id: Ie4f5dc10c94c6b44c99380582741bf8d07b4ce22
2022-10-07 17:33:40 +00:00
Wilfred Asomani
903ea38c86 web/satellite: increment login failed_login_count in sql
This change increments users' failed_login_count in the database layer to avoid potential data race.
It also updates the login_lockout_expiration as well in one operation.

see: https://github.com/storj/storj/issues/4986

Change-Id: I74624f1bee31667b269cb205d74d16e79daabcb6
2022-10-07 16:46:29 +00:00
JT Olio
d632f23950 cmd/uplink: add eventkit
Change-Id: If109c8f7de257b77794e45599487ad2c46f2c3ec
2022-10-07 12:08:08 -04:00
dlamarmorgan
d39caf7674 cmd/satellite: change invoicing flags to iso8601
Update input parameters for invoicing commands to use iso8601 standard formatting.

Change-Id: I3b4a7df268e8f1a5cd51b0dab03ed38e13a76554
2022-10-07 07:58:23 -07:00
Lizzy Thomson
ef04eb8bea satellite/satellitedb: add columns in projects table
add user_specified_usage_limit and user_specified_bandwidth_limit columns in projects table

github issue https://github.com/storj/storj/issues/5185

Change-Id: Ia8c9b190a2bdb684ac9d43af8f2159027d65aa88
2022-10-06 18:20:39 -06:00
NickolaiYurchenko
6884b587e3 web/satellite: limit warning banner for free tier users added
Shown only for free tier users.
Only appears at > 80% used storage or bandwidth (or both)
Different banner and modal styles for 100% used
Added resizable listener for banner

Change-Id: I29ac1777127155dab83b4eb113a1f033ea926524
2022-10-06 15:53:53 +00:00
Michal Niewrzal
4d9c9138ce satellite/metainfo: use multiple object versions internally
With this change we are switching methods to begin object, from
BeginObjectExactVersion to BeginObjectNextVersion. Main implication
is that from now it will be possible to have object with version
different than 1. New object will always get first available version.

Main reason to do this it to avoid deleting existing object during
reuploading object. Now we can create multiple pending objects but
only last committed will be available to the user. Any previous
committed object will be deleted.Because of that we moved logic to
delete existing object from BeginObject to CommitoObject request.

New logic is behind feature flat to be able to test it well first
before enablng on production.

Fixes https://github.com/storj/storj/issues/4871

Change-Id: I2dd9c7364fd93796a05ef607bda9c39a741e6a89
2022-10-06 15:19:02 +00:00
Egon Elbre
ea4b3023d9 cmd/tools/segment-verify: fix piece id derivation
Change-Id: Ib27fd8630e1e5a90060dff2a09c51f488960177f
2022-10-06 13:43:08 +00:00
Egon Elbre
c8506cdda3 satellite/metabase,cmd/tools/segment-verify: simplify interface
Change-Id: Icdd445b1713bc26cee3b3a125b68b0cde0739837
2022-10-06 13:42:00 +00:00
Márton Elek
ac6bb1e187 storage: experimental flag to turn off file sync
Full file sync before saving files to piecestore seems to be very expensive.

Bwfore we do any step to eliminate them we are planning to do more measurement which requires a temporary flag to turn it off. (not for production).

Change-Id: I5cb8f8cb348ca3590fb5eae14d02edb3f0424617
2022-10-06 13:10:34 +00:00
Márton Elek
f61507a125 satellite: include 'unsafe' in the new migration options
Change-Id: I9d30c6ece430aad420332df159d0d41c6688f9d6
2022-10-06 12:05:14 +00:00
Egon Elbre
c1817ab743 cmd/tools/segment-verify: a few fixes
The flags weren't properly loading from config.

The code assumed that every node that's online for downloading also have
data uploaded to them -- which is not true.

Change-Id: Ifd65a47b9eca5b4841231928244fab17acbde6fb
2022-10-05 15:51:38 +00:00
Michal Niewrzal
bbed10d7d4 satellite/{accounting/tally,metabase}: remove monkit for fast methods
This is another change to remove monkit calls from fast methods. Those
calls are visible in CPU profiles.

Change-Id: Ib3beba0dca6a6d93c3342b0994c580f78bbdd50b
2022-10-05 08:38:48 +00:00
Yaroslav Vorobiov
14d0b0ee20 testsuite/storjscan: add backfill payments test
Add backfill payments test to testsuite/storjscan.
Test ensures that even when something happens with
storjscan and it's unvailable from satellite perspective
that all payments made during storjscan downtime will be
picked up during next StorjscanChore cycle.

Change-Id: I4db699a29061023e159b3191a1e4dcfae6d0175e
2022-10-04 18:56:48 +00:00
Márton Elek
9ddd20b72e satellite/projectaccounting: fix project usage right after the genesis
After you create a brand new cluster (with storj-up, for example) the project usage fails during the first 5 minutes.

The problem is the usage of `AS OF SYSTEM TIME` which points to a time where the master database didn't exist.

In this specific case the database not found error can be ignored to avoid such messages. (if the database is really missing, we will have problems way more earlier, eg. at the login)

Change-Id: I51ee78994d91fc2a14b56646402faaaa8154c934
2022-10-04 14:41:30 +00:00
Márton Elek
7f6f7e1246
satellite: optional migration for integration tests
This patch addresses the following issues:

 1. Running full migration in cockroachdb is quite slow. We already have an approach for unit tests to start from the latest snapshot. This patch makes it possible to use it for integrations tests.
 2. Migration requires executing a separated command which makes it hard to run application in containerized test environments (like storj-up) or from IDE. This patch introduces a hidden flag to run migration.
 3. Test user creation is painful. We do it with calling GraphQL + admin API. Providing an option with testuser makes the integration tests significant more simple (especially as the projectID -> access grant can be predictable)

Change-Id: I61010728727b490ea6aac32620f2da0484966727
2022-10-04 11:35:03 +02:00
dlamarmorgan
85f9dad225 satellite/payments/stripecoinpayments/service.go: add date to pay invoices command
Add an extra parameter to the pay-invoices command that can be used to restrict which invoices will have a payment attempted in stripe. The parameter should be of the form MM/DD/YYY and any invoices created on or after the date will have token balances applied and be processed for payment according to stripe subscriptions settings.

Change-Id: I5da5070d3ac97f45c05c02f2849254bdc44413c3
2022-10-03 21:28:37 -07:00
Jeremy Wharton
3d6c3e31f1 satellite/payments/stripecoinpayments: auto advance $0 invoices
This change causes new invoices to be scheduled for automatic
advancement through Stripe if their amount due is zero. Invoices
marked for automatic advancement are exempt from the manual invoice
finalization procedure.

Change-Id: Ic583db4c86ec5243d7506d380ca3faee5e9a58d3
2022-10-03 21:54:39 -05:00
Jeremy Wharton
4ab8031171 cmd/satellite: combine draft invoice generation commands
This change introduces the generate-invoices satellite billing
command whose functionality is equivalent to running
apply-free-coupons, prepare-invoice-records,
create-project-invoice-items, and create-invoices in order.
Invoice finalization must still be performed separately.

Change-Id: Ia3d80b95eef1f2776c38bd730ed731e42ec4c35e
2022-10-03 21:19:44 -05:00
Jeremy Wharton
480715e3b8 web/satellite: remove prefix from downloaded access grant contents
This change removes the "access grant:" prefix from the file contents
of access grants downloaded from the satellite UI. The prefix prevented
access grant files from being imported into uplink.

Resolves #5066

Change-Id: I879a3d2d6009f8ccd50351de79dc0a05df7b6810
2022-10-03 22:05:34 +00:00
Egon Elbre
4c374a2357 cmd/tools/segment-verify: add tiny readme
Change-Id: Ia314c615f8b7fdb13e2b5f81c1be82ec686ca819
2022-10-03 16:01:24 +00:00
Vitalii
509bb1aa01 web/satellite: optimized project selection
Optimized project selection API requests.
We make parallel requests where possible.
We fetch needed data depending on the current navigation route instead of fetching all the dashboard data.

Change-Id: Ideb5de52cccdb5a73d53422b8f982111e211a714
2022-10-03 15:06:58 +00:00
Michal Niewrzal
6fcc5c5cb8 satellite/metabase/segmentloop: remove unused monkit calls
Monkit calls for fast methods which are executed very frequently can
slowdown whole process. This change removes monkit calls which are not
used.

See https://review.dev.storj.io/c/storj/storj/+/8498 as an example of
speed improvement after removing monkit calls.

Change-Id: If6567d80e05b748e6393b58a5142e43013107c61
2022-10-03 14:07:18 +00:00
Michal Niewrzal
91c3a45874 satellite/accounting/nodetally: remove monkit call from observer
Benchmark against 'main':
name                                         old time/op    new time/op    delta
RemoteSegment/Cockroach/multiple_segments-8    5.56µs ± 5%    0.69µs ±12%   -87.57%  (p=0.008 n=5+5)

name                                         old alloc/op   new alloc/op   delta
RemoteSegment/Cockroach/multiple_segments-8    2.72kB ± 0%    0.00kB           ~     (p=0.079 n=4+5)

name                                         old allocs/op  new allocs/op  delta
RemoteSegment/Cockroach/multiple_segments-8      50.0 ± 0%       0.0       -100.00%  (p=0.008 n=5+5)

Change-Id: I20527fb576cd81db667a81929fa95b810ee11b14
2022-10-03 13:32:20 +00:00
Michal Niewrzal
5dc5f076c9 satellite/repair/checker: remove monitoring from fast methods
It looks that monikt monitoring can give high CPU overhead for
segments loop observer. With this code we are changing how monitoring
is initialized for observer methods. This optimization affects mainly
path where segment is healthy and doesn't require repair. Benchmark
is also added to show difference between old and new approach.

Benchmark against 'main':
name                                       old time/op    new time/op    delta
RemoteSegment/Cockroach/healthy_segment-8    8.55µs ± 4%    1.37µs ± 6%  -84.03%  (p=0.008 n=5+5)

name                                       old alloc/op   new alloc/op   delta
RemoteSegment/Cockroach/healthy_segment-8    2.63kB ± 0%    0.17kB ± 0%  -93.62%  (p=0.008 n=5+5)

name                                       old allocs/op  new allocs/op  delta
RemoteSegment/Cockroach/healthy_segment-8      54.0 ± 0%       8.0 ± 0%  -85.19%  (p=0.008 n=5+5)

Change-Id: Ie138eab0d59e436395b13f57bdfb11f9871d4c18
2022-10-03 12:15:03 +00:00
Michal Niewrzal
e37435602f satellite/audit: optimize loop observer
Two things were done to optimize audit observer:
* monik call was removed as we have different way to track it
* no new allocation for audit.Segment struct inside observer

Benchmark against 'main':
name                                         old time/op    new time/op    delta
RemoteSegment/Cockroach/multiple_segments-8    5.85µs ± 1%    0.74µs ± 4%   -87.28%  (p=0.008 n=5+5)

name                                         old alloc/op   new alloc/op   delta
RemoteSegment/Cockroach/multiple_segments-8    2.72kB ± 0%    0.00kB           ~     (p=0.079 n=4+5)

name                                         old allocs/op  new allocs/op  delta
RemoteSegment/Cockroach/multiple_segments-8      50.0 ± 0%       0.0       -100.00%  (p=0.008 n=5+5)

Change-Id: Ib973e48782bad4346eee1cd5aee77f0a50f69258
2022-10-02 22:24:37 +00:00
Wilfred Asomani
05e57edb20 web/satellite: see token payments history
This change lists token payment histories from storjscan/coinpayments on the "Payment Methods > Storj Tokens" page

see: https://github.com/storj/storj/issues/4941

Change-Id: I178ba7940c4cb132f05673607030725a4c4b3e1c
2022-09-30 16:44:47 +00:00
Jeremy Wharton
fe3aedebe8 web/satellite: move mobile Billing nav item into My Account dropdown
This change moves the top-level Billing navigation item into the
My Account dropdown menu in the mobile view to match the behavior of
the desktop view.

Change-Id: Ibb192ec0d34e98851193efb28280bea90fa58302
2022-09-29 19:39:38 +00:00
Jennifer Johnson
47f9915a27 satellite/payments/storjscan: add info level log statement when a user successfully claims a wallet address.
If we need to restore the satelliteDB from a backup, we must preserve the user - storj token wallet address association. This commit adds a log statement of this information after a user successfully claims a wallet. We can perform a SQL update to reassign the wallet address to the user if needed.

Change-Id: Ia5c25d7ac57e59b35865d74068196e42bc4ffe87
2022-09-29 18:20:06 +00:00
Vitalii
92b25c8c96 web/satellite: added server-side encryption step for create S3 creds flow
Added server-side encryption step to create s3 credentials flow.
Reworked create access grant modal to use new VModal common component.

Issue:
https://github.com/storj/storj/issues/5119

Change-Id: Ie13fef80ced7a0330d87afe757813f35dab1c072
2022-09-29 13:46:46 +00:00
Igor
96d4968041
use python2 for draft release (#5216) 2022-09-29 16:22:08 +03:00
Michal Niewrzal
6bb3ea3b6e satellite/metabase: add tests for ListVerifySegments
Change-Id: Idc97111ab61ac3b23303371a30e3a66ea702f96e
2022-09-29 11:52:51 +00:00
prerna-parashar
94c5aaf70e
satellite/analytics: Added analytics for user behavior on Billing page (#5205)
* Added analytics for Billing page

* Fixed Jenkins errors

* Fixed lint errors

* Fixed Lint errors

* Fixed Lint errors

Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
2022-09-29 09:20:24 +03:00
Jennifer Johnson
182b94aab2 docs/blueprints: design doc for changes needed to allow for scaling audit workers
Change-Id: I3e214c900a02ba15c2fb47558f0bc75310db2a7c
2022-09-28 19:47:41 +00:00
Fadila Khadar
02924d0ded satellite/metainfo: temporary feature flag for listing query testing
Fixes: https://github.com/storj/storj/issues/5144

Change-Id: I7650f4d5dd0378e2246339e79710a695996a845c
2022-09-28 17:46:52 +00:00
paul cannon
802ff18bd8 satellite/audit: better handling of piece fetch errors
We have an alert on `not_enough_shares_for_audit` which fires too
frequently. Every time so far, it has been because of a network blip of
some nature on the satellite side.

Satellite operators are expected to have other means in place for
alerting on network problems and fixing them, so it's not necessary for
the audit framework to act in that way.

Instead, in this change, we add three new metrics,
`audit_not_enough_nodes_online`, `audit_not_enough_shares_acquired`, and
`audit_suspected_network_problem`. When an audit fails, and emits
`not_enough_shares_for_audit`, we will now determine whether it looks
like we are having network problems (most errors are connection
failures, possibly also some successful connections which subsequently
time out) or whether something else has happened.

After this is deployed, we can remove the alert on
`not_enough_shares_for_audit` and add new alerts on
`audit_not_enough_nodes_online` and `audit_not_enough_shares_acquired`.
`audit_suspected_network_problem` does not need an alert.

Refs: https://github.com/storj/storj/issues/4669

Change-Id: Ibb256bc19d2578904f71f5229111ac98e5212fcb
2022-09-28 17:02:06 +00:00
Igor
352e937813
scripts: add changelog to release draft (#4791)
* scripts: add changelog to release draft

* scripts: fix run command

* rename variable

* rename variable

* fix command

* add comment

* rename variable

Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2022-09-28 19:17:40 +03:00
Ethan
77357f9877 satellite/{accounting,redis}: Use ARGV in redis scripts
While investigating high memory consumption on a Redis instance, it was found that much of the memory consumed was from cached redis scripts.

Redis caches scripts based on the hash of the script itself.  This change uses ARGV to reduce the number of cached scripts.

Change-Id: Ia878fe81552a3067f09e60c44bb4ace25c6b5f9a
2022-09-28 14:43:00 +00:00
Egon Elbre
2b2b6d2288 satellite/admin/ui: use a valid go.mod stub
Some tools check all the go.mod files and fail when it's incorrect.

Change-Id: I1be5bd1b502e15c90b2f69caedbf5545d366a87d
2022-09-28 12:41:30 +00:00
Fadila Khadar
f58129786b satellite/metabase: copy handles version when overwriting target location
When doing server-side copy, deletes the committed version of the target location if it already exists. It does not touch pending versions. The version of the copy is set to the highest already existing version + 1.

Fixes: https://github.com/storj/storj/issues/5071

Change-Id: I1d91ac17054834b1f4f0970a9fa5d58198c58a37
2022-09-28 11:39:24 +00:00
Michal Niewrzal
a97cd97789 satellite/orders: remove unused service dependency
Orders service doesn't need buckets service anymore.

Change-Id: I27853cda87e82b528f53667e4b4866801f7bfb62
2022-09-28 08:56:36 +00:00
dlamarmorgan
88328ab240 satellite/payments/billing/billing.go: change default loop interval
Change the default loop interval for querying for new payments and adding them into the billing table from 1 minute to 15 seconds.

Change-Id: I26cf4a764cbe1de4c9b839ad60352374d8231522
2022-09-27 11:42:28 -07:00
Jeff Wendling
929642b675 cmd/uplink: remove io/ioutil
it has been deprecated since go1.16.

Change-Id: I8c15b24176e5a4687fe297d569505e84fc5affda
2022-09-27 15:53:07 +00:00