Commit Graph

7038 Commits

Author SHA1 Message Date
Wilfred Asomani
b23b3e02e6 web/satellite: rewrite VButton with composition API
This change updates VButton with composition API, updates .eslintrc.js to ignore resulting import errors, and fix resulting test failures.

see: https://github.com/storj/storj/issues/5114

Change-Id: Iaf9e8323333c797936d3acb38dd32f6e27d1ed08
2022-10-13 12:54:39 +00:00
Erik van Velzen
0cfbd34da2 docs/blueprints: byte range multipart copy
Design proposal for approval to support S3's UploadPartCopy. Major
revision 2.

Change-Id: I9cbe934a773d3496101f32afb19492ea75f48061
2022-10-13 12:03:17 +00:00
Márton Elek
ea106f8894 go.mod: bump storj/private
Change-Id: I087b39c267c6492c4a5c85f470631b2f206cdf39
2022-10-13 11:23:57 +00:00
Cameron
98fed4bc30 {satellite/console,web/satellite}: get project salt from satellite
Add getSalt to projects api. Add action, GET_SALT, on Store
Projects module to make the api request and return the salt
string everywhere in the web app that generates an access grant.
The Wasm code which is used to create the access grant has been
changed to decode the salt as a base64 encoded string. The names
of the function calls in the changed Wasm code have also been
changed to ensure that access grant creation fails if JS access
grant worker code and Wasm code are not the same version.

https://github.com/storj/storj-private/issues/64

Change-Id: Ia2bc4cbadad84b066ca1882b042a3f0bb13c783a
2022-10-12 19:06:27 +00:00
Jeremy Wharton
1887660678 web/satellite: update custom HTML styling for registration page
Add styling for strong text in `customHtmlDescription` for
alternate signup landing pages in `registrationViewConfig`.

Change-Id: Ie688815713b84edb46532bc000152625ccb38ca7
2022-10-12 16:15:31 +00:00
Michal Niewrzal
e5ac8430c3 satellite/metainfo: delete pieces from nodes on object commit
We have new flow where existing object is deleted not on begin
object but on commit object. Deletion on commit object is still
missing deletion from storage nodes. This change adds this part
to the code.

Fixes https://github.com/storj/storj/issues/5222

Change-Id: Ibfd34665b2a055ec6c0d6e260c1a57e8a4c62b0e
2022-10-12 15:02:24 +00:00
Egon Elbre
dd60318147 cmd/tools/segment-verify: use resolved ip
Change-Id: I3662aaea3ff8721c415c038b2b5324d165b60975
2022-10-12 12:43:11 +00:00
Clement Sam
d800b51dd9 cmd/multinode: generate identity when not provided
Closes https://github.com/storj/storj/issues/4974

Change-Id: I7a07d846be7ac8f8f7d7e9ad61511ff84d2ab400
2022-10-12 12:01:14 +00:00
JT Olio
9c09a8dbd5 cmd/uplink: don't send TODO as the instance id
there's not really anything better to send. uplinks have
rotating node ids on each startup, so that's not right
here. i don't think anyone will use instance id for
uplinks so let's just fold and send nothing.

Change-Id: I2511605e95eba1816d662d385b28d5feab8c4eb0
2022-10-12 06:56:19 +00:00
Yaroslav Vorobiov
5cfd64579e private/blockchain: fix Hex to add 0x prefix
Change-Id: I47da9cea160fe9bbdf452956836e3d4b5b44646f
2022-10-11 19:15:24 +00:00
Egon Elbre
8b70f969b6 all: fix nolint directives
Change-Id: I261c8b12e4961e6401cc4024fa5abc35b1a5efa6
2022-10-11 18:31:20 +00:00
Malcolm Bouzi
773683221d web/satellite: add right border to desktop navbar to match designs.
Change-Id: I62e473fd86e16eacec2f367d232fc9f99b2f5e66
2022-10-11 17:42:03 +00:00
Vitalii
2e6b473aba web/satellite: improved modal experience
Made modals to close on background click.
Refactored new folder and object details modals to use VModal component.

Change-Id: Ib7c6ecb576507f2088c0c90fab262113ac7c0760
2022-10-11 16:49:11 +00:00
Michal Niewrzal
1c9f4d1824 satellite/metainfo: more tests for CopyObject
Change-Id: Ibb29f348a15f4afe5d75f9a81987cff4633ceb56
2022-10-11 15:57:23 +00:00
Egon Elbre
ff22fc7ddd all: fix deprecated ioutil commands
Change-Id: I59db35116ec7215a1b8e2ae7dbd319fa099adfac
2022-10-11 15:27:29 +00:00
Michal Niewrzal
11d1e623b5 satellite/metabase/segmentloop: don't do rate limiting if disabled
We have a code to  limit segments loop in case it will hit DB to hard
but so far we didn't use this loop feature  in production. This is a
simple change to avoid logic responsible for rate limiting and its
monitoring if limiting is disabled (RateLimit = 0)

Change-Id: I43e07b407c6e65cf252303159d052eef250d1bea
2022-10-11 10:55:30 +00:00
Egon Elbre
142a04f208 cmd/tools/segment-verify: add connection pool
Change-Id: If0f85edbf99438ac41c23fc7107fdab926288cc2
2022-10-11 09:06:44 +03:00
Egon Elbre
8916f2ee92 cmd/tools/segment-verify: allow ignoring specific nodes
This adds a new flag that allows to ignore some nodes completely.

Change-Id: I203d25f931262c809037e25e9c37e9a89bf47026
2022-10-10 20:14:38 +03:00
Egon Elbre
9e50d837e3 cmd/tools/segment-verify: add tool for summarizing log
Change-Id: I3177ab71dfd25e11adfedce32a530d83dda63bd6
2022-10-10 20:02:50 +03:00
Egon Elbre
5f01dad3a3 cmd/tools/segment-verify: add total progress indicator
Change-Id: Ib729abf6adbeba8d94e08c7e11497c6d5ddd5ec2
2022-10-10 20:02:30 +03:00
Michal Niewrzal
db1409eea6 satellite/metabase: use SUBSTRING with objects iterator
Until this change we were stripping prefix from object key on satellite side. Because of that we were transferring over network unnecessary data
from DB. This change adjusts iterator SQL queries to use SUBSTRING to
remove prefix on DB side and avoid sending it to satellite.

Benchmark against 'main':
unfortunately "time/op" is very unstable while doing local bench in this
case and sometimes  there is no difference in time and sometimes its up to 18%. I never saw results when old solution is faster then new one. Results for "alloc/op" and "allocs/op" are rather consistent.

name                                                 old time/op    new time/op    delta
NonRecursiveListing/Cockroach/listing_no_prefix-8      1.98ms ± 6%    2.05ms ±23%     ~     (p=1.000 n=9+10)
NonRecursiveListing/Cockroach/listing_with_prefix-8    3.97ms ± 8%    3.42ms ±20%  -13.86%  (p=0.005 n=10+10)
NonRecursiveListing/Cockroach/listing_only_prefix-8    8.42ms ±16%    7.58ms ± 5%   -9.91%  (p=0.002 n=10+10)

name                                                 old alloc/op   new alloc/op   delta
NonRecursiveListing/Cockroach/listing_no_prefix-8      16.7kB ± 0%    16.9kB ± 0%   +1.16%  (p=0.000 n=10+10)
NonRecursiveListing/Cockroach/listing_with_prefix-8    27.3kB ± 0%    28.2kB ± 0%   +3.31%  (p=0.000 n=10+10)
NonRecursiveListing/Cockroach/listing_only_prefix-8    60.0kB ± 0%    62.4kB ± 0%   +3.93%  (p=0.000 n=10+8)

name                                                 old allocs/op  new allocs/op  delta
NonRecursiveListing/Cockroach/listing_no_prefix-8         312 ± 0%       315 ± 0%   +0.96%  (p=0.000 n=10+10)
NonRecursiveListing/Cockroach/listing_with_prefix-8       526 ± 0%       541 ± 0%   +2.85%  (p=0.000 n=10+10)
NonRecursiveListing/Cockroach/listing_only_prefix-8     1.16k ± 0%     1.23k ± 0%   +5.24%  (p=0.000 n=10+10)

Change-Id: I23e501494ededafb2dd5ea903e8e4e313b42e956
2022-10-10 14:27:26 +00:00
nadimhq
c921cd5ccf
docs/testplan: Adding a testplan for Token Payment Processor (#4672)
* docs/testplan: Adding a testplan for Token Payment Processor

This testplan is going to cover the token payment processor, it will go over must haves and additional features that we can add on later.

* docs/testplan: Adding a testplan for Token Payment Processor

Updated final testplan for review

* docs/testplan: Adding a testplan for Token Payment Processor

Removed unnecessary tests related to mempool scanning

* Update token-payment-processor-testplan.md

Co-authored-by: littleskunk <jens.heimbuerge@googlemail.com>
2022-10-10 15:01:14 +02:00
Malcolm Bouzi
ff99392fd7 web/satellite: update bucket share modals to match designs
Change-Id: Ie4f5dc10c94c6b44c99380582741bf8d07b4ce22
2022-10-07 17:33:40 +00:00
Wilfred Asomani
903ea38c86 web/satellite: increment login failed_login_count in sql
This change increments users' failed_login_count in the database layer to avoid potential data race.
It also updates the login_lockout_expiration as well in one operation.

see: https://github.com/storj/storj/issues/4986

Change-Id: I74624f1bee31667b269cb205d74d16e79daabcb6
2022-10-07 16:46:29 +00:00
JT Olio
d632f23950 cmd/uplink: add eventkit
Change-Id: If109c8f7de257b77794e45599487ad2c46f2c3ec
2022-10-07 12:08:08 -04:00
dlamarmorgan
d39caf7674 cmd/satellite: change invoicing flags to iso8601
Update input parameters for invoicing commands to use iso8601 standard formatting.

Change-Id: I3b4a7df268e8f1a5cd51b0dab03ed38e13a76554
2022-10-07 07:58:23 -07:00
Lizzy Thomson
ef04eb8bea satellite/satellitedb: add columns in projects table
add user_specified_usage_limit and user_specified_bandwidth_limit columns in projects table

github issue https://github.com/storj/storj/issues/5185

Change-Id: Ia8c9b190a2bdb684ac9d43af8f2159027d65aa88
2022-10-06 18:20:39 -06:00
NickolaiYurchenko
6884b587e3 web/satellite: limit warning banner for free tier users added
Shown only for free tier users.
Only appears at > 80% used storage or bandwidth (or both)
Different banner and modal styles for 100% used
Added resizable listener for banner

Change-Id: I29ac1777127155dab83b4eb113a1f033ea926524
2022-10-06 15:53:53 +00:00
Michal Niewrzal
4d9c9138ce satellite/metainfo: use multiple object versions internally
With this change we are switching methods to begin object, from
BeginObjectExactVersion to BeginObjectNextVersion. Main implication
is that from now it will be possible to have object with version
different than 1. New object will always get first available version.

Main reason to do this it to avoid deleting existing object during
reuploading object. Now we can create multiple pending objects but
only last committed will be available to the user. Any previous
committed object will be deleted.Because of that we moved logic to
delete existing object from BeginObject to CommitoObject request.

New logic is behind feature flat to be able to test it well first
before enablng on production.

Fixes https://github.com/storj/storj/issues/4871

Change-Id: I2dd9c7364fd93796a05ef607bda9c39a741e6a89
2022-10-06 15:19:02 +00:00
Egon Elbre
ea4b3023d9 cmd/tools/segment-verify: fix piece id derivation
Change-Id: Ib27fd8630e1e5a90060dff2a09c51f488960177f
2022-10-06 13:43:08 +00:00
Egon Elbre
c8506cdda3 satellite/metabase,cmd/tools/segment-verify: simplify interface
Change-Id: Icdd445b1713bc26cee3b3a125b68b0cde0739837
2022-10-06 13:42:00 +00:00
Márton Elek
ac6bb1e187 storage: experimental flag to turn off file sync
Full file sync before saving files to piecestore seems to be very expensive.

Bwfore we do any step to eliminate them we are planning to do more measurement which requires a temporary flag to turn it off. (not for production).

Change-Id: I5cb8f8cb348ca3590fb5eae14d02edb3f0424617
2022-10-06 13:10:34 +00:00
Márton Elek
f61507a125 satellite: include 'unsafe' in the new migration options
Change-Id: I9d30c6ece430aad420332df159d0d41c6688f9d6
2022-10-06 12:05:14 +00:00
Egon Elbre
c1817ab743 cmd/tools/segment-verify: a few fixes
The flags weren't properly loading from config.

The code assumed that every node that's online for downloading also have
data uploaded to them -- which is not true.

Change-Id: Ifd65a47b9eca5b4841231928244fab17acbde6fb
2022-10-05 15:51:38 +00:00
Michal Niewrzal
bbed10d7d4 satellite/{accounting/tally,metabase}: remove monkit for fast methods
This is another change to remove monkit calls from fast methods. Those
calls are visible in CPU profiles.

Change-Id: Ib3beba0dca6a6d93c3342b0994c580f78bbdd50b
2022-10-05 08:38:48 +00:00
Yaroslav Vorobiov
14d0b0ee20 testsuite/storjscan: add backfill payments test
Add backfill payments test to testsuite/storjscan.
Test ensures that even when something happens with
storjscan and it's unvailable from satellite perspective
that all payments made during storjscan downtime will be
picked up during next StorjscanChore cycle.

Change-Id: I4db699a29061023e159b3191a1e4dcfae6d0175e
2022-10-04 18:56:48 +00:00
Márton Elek
9ddd20b72e satellite/projectaccounting: fix project usage right after the genesis
After you create a brand new cluster (with storj-up, for example) the project usage fails during the first 5 minutes.

The problem is the usage of `AS OF SYSTEM TIME` which points to a time where the master database didn't exist.

In this specific case the database not found error can be ignored to avoid such messages. (if the database is really missing, we will have problems way more earlier, eg. at the login)

Change-Id: I51ee78994d91fc2a14b56646402faaaa8154c934
2022-10-04 14:41:30 +00:00
Márton Elek
7f6f7e1246
satellite: optional migration for integration tests
This patch addresses the following issues:

 1. Running full migration in cockroachdb is quite slow. We already have an approach for unit tests to start from the latest snapshot. This patch makes it possible to use it for integrations tests.
 2. Migration requires executing a separated command which makes it hard to run application in containerized test environments (like storj-up) or from IDE. This patch introduces a hidden flag to run migration.
 3. Test user creation is painful. We do it with calling GraphQL + admin API. Providing an option with testuser makes the integration tests significant more simple (especially as the projectID -> access grant can be predictable)

Change-Id: I61010728727b490ea6aac32620f2da0484966727
2022-10-04 11:35:03 +02:00
dlamarmorgan
85f9dad225 satellite/payments/stripecoinpayments/service.go: add date to pay invoices command
Add an extra parameter to the pay-invoices command that can be used to restrict which invoices will have a payment attempted in stripe. The parameter should be of the form MM/DD/YYY and any invoices created on or after the date will have token balances applied and be processed for payment according to stripe subscriptions settings.

Change-Id: I5da5070d3ac97f45c05c02f2849254bdc44413c3
2022-10-03 21:28:37 -07:00
Jeremy Wharton
3d6c3e31f1 satellite/payments/stripecoinpayments: auto advance $0 invoices
This change causes new invoices to be scheduled for automatic
advancement through Stripe if their amount due is zero. Invoices
marked for automatic advancement are exempt from the manual invoice
finalization procedure.

Change-Id: Ic583db4c86ec5243d7506d380ca3faee5e9a58d3
2022-10-03 21:54:39 -05:00
Jeremy Wharton
4ab8031171 cmd/satellite: combine draft invoice generation commands
This change introduces the generate-invoices satellite billing
command whose functionality is equivalent to running
apply-free-coupons, prepare-invoice-records,
create-project-invoice-items, and create-invoices in order.
Invoice finalization must still be performed separately.

Change-Id: Ia3d80b95eef1f2776c38bd730ed731e42ec4c35e
2022-10-03 21:19:44 -05:00
Jeremy Wharton
480715e3b8 web/satellite: remove prefix from downloaded access grant contents
This change removes the "access grant:" prefix from the file contents
of access grants downloaded from the satellite UI. The prefix prevented
access grant files from being imported into uplink.

Resolves #5066

Change-Id: I879a3d2d6009f8ccd50351de79dc0a05df7b6810
2022-10-03 22:05:34 +00:00
Egon Elbre
4c374a2357 cmd/tools/segment-verify: add tiny readme
Change-Id: Ia314c615f8b7fdb13e2b5f81c1be82ec686ca819
2022-10-03 16:01:24 +00:00
Vitalii
509bb1aa01 web/satellite: optimized project selection
Optimized project selection API requests.
We make parallel requests where possible.
We fetch needed data depending on the current navigation route instead of fetching all the dashboard data.

Change-Id: Ideb5de52cccdb5a73d53422b8f982111e211a714
2022-10-03 15:06:58 +00:00
Michal Niewrzal
6fcc5c5cb8 satellite/metabase/segmentloop: remove unused monkit calls
Monkit calls for fast methods which are executed very frequently can
slowdown whole process. This change removes monkit calls which are not
used.

See https://review.dev.storj.io/c/storj/storj/+/8498 as an example of
speed improvement after removing monkit calls.

Change-Id: If6567d80e05b748e6393b58a5142e43013107c61
2022-10-03 14:07:18 +00:00
Michal Niewrzal
91c3a45874 satellite/accounting/nodetally: remove monkit call from observer
Benchmark against 'main':
name                                         old time/op    new time/op    delta
RemoteSegment/Cockroach/multiple_segments-8    5.56µs ± 5%    0.69µs ±12%   -87.57%  (p=0.008 n=5+5)

name                                         old alloc/op   new alloc/op   delta
RemoteSegment/Cockroach/multiple_segments-8    2.72kB ± 0%    0.00kB           ~     (p=0.079 n=4+5)

name                                         old allocs/op  new allocs/op  delta
RemoteSegment/Cockroach/multiple_segments-8      50.0 ± 0%       0.0       -100.00%  (p=0.008 n=5+5)

Change-Id: I20527fb576cd81db667a81929fa95b810ee11b14
2022-10-03 13:32:20 +00:00
Michal Niewrzal
5dc5f076c9 satellite/repair/checker: remove monitoring from fast methods
It looks that monikt monitoring can give high CPU overhead for
segments loop observer. With this code we are changing how monitoring
is initialized for observer methods. This optimization affects mainly
path where segment is healthy and doesn't require repair. Benchmark
is also added to show difference between old and new approach.

Benchmark against 'main':
name                                       old time/op    new time/op    delta
RemoteSegment/Cockroach/healthy_segment-8    8.55µs ± 4%    1.37µs ± 6%  -84.03%  (p=0.008 n=5+5)

name                                       old alloc/op   new alloc/op   delta
RemoteSegment/Cockroach/healthy_segment-8    2.63kB ± 0%    0.17kB ± 0%  -93.62%  (p=0.008 n=5+5)

name                                       old allocs/op  new allocs/op  delta
RemoteSegment/Cockroach/healthy_segment-8      54.0 ± 0%       8.0 ± 0%  -85.19%  (p=0.008 n=5+5)

Change-Id: Ie138eab0d59e436395b13f57bdfb11f9871d4c18
2022-10-03 12:15:03 +00:00
Michal Niewrzal
e37435602f satellite/audit: optimize loop observer
Two things were done to optimize audit observer:
* monik call was removed as we have different way to track it
* no new allocation for audit.Segment struct inside observer

Benchmark against 'main':
name                                         old time/op    new time/op    delta
RemoteSegment/Cockroach/multiple_segments-8    5.85µs ± 1%    0.74µs ± 4%   -87.28%  (p=0.008 n=5+5)

name                                         old alloc/op   new alloc/op   delta
RemoteSegment/Cockroach/multiple_segments-8    2.72kB ± 0%    0.00kB           ~     (p=0.079 n=4+5)

name                                         old allocs/op  new allocs/op  delta
RemoteSegment/Cockroach/multiple_segments-8      50.0 ± 0%       0.0       -100.00%  (p=0.008 n=5+5)

Change-Id: Ib973e48782bad4346eee1cd5aee77f0a50f69258
2022-10-02 22:24:37 +00:00
Wilfred Asomani
05e57edb20 web/satellite: see token payments history
This change lists token payment histories from storjscan/coinpayments on the "Payment Methods > Storj Tokens" page

see: https://github.com/storj/storj/issues/4941

Change-Id: I178ba7940c4cb132f05673607030725a4c4b3e1c
2022-09-30 16:44:47 +00:00
Jeremy Wharton
fe3aedebe8 web/satellite: move mobile Billing nav item into My Account dropdown
This change moves the top-level Billing navigation item into the
My Account dropdown menu in the mobile view to match the behavior of
the desktop view.

Change-Id: Ibb192ec0d34e98851193efb28280bea90fa58302
2022-09-29 19:39:38 +00:00