Commit Graph

6847 Commits

Author SHA1 Message Date
Michal Niewrzal
d1d98af098 satellite/metabase: jump after prefix at the end of iteration page
For non-recursive listing if cursor to the next iteration page
is a prefix then jump after this prefix to avoid listing objects from
this prefix.

Benchmark against 'main' - 7e5025cac0
name                                                 old time/op    new time/op    delta
NonRecursiveListing/Cockroach/listing_no_prefix-8      11.7ms ±29%     1.7ms ± 7%  -85.52%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8    4.60ms ± 6%    3.23ms ± 3%  -29.72%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_only_prefix-8    7.38ms ± 7%    8.47ms ±34%     ~     (p=0.421 n=5+5)

name                                                 old alloc/op   new alloc/op   delta
NonRecursiveListing/Cockroach/listing_no_prefix-8      92.3kB ± 0%    16.9kB ± 0%  -81.72%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8    38.0kB ± 0%    27.4kB ± 0%  -28.02%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_only_prefix-8    59.8kB ± 0%    60.1kB ± 0%   +0.38%  (p=0.008 n=5+5)

name                                                 old allocs/op  new allocs/op  delta
NonRecursiveListing/Cockroach/listing_no_prefix-8       1.78k ± 0%     0.31k ± 0%  -82.48%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8       734 ± 0%       526 ± 0%  -28.34%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_only_prefix-8     1.15k ± 0%     1.16k ± 0%   +0.87%  (p=0.008 n=5+5)

Discussed here https://github.com/storj/team-metainfo/issues/116

Change-Id: Iff671f062d9af83ec419334089163dd204d0cac0
2022-08-23 13:32:30 +00:00
Clement Sam
bac0155664 storagenode/storagenodedb: fix null at_rest_total values for storage usage
Wrapping a COALESCE around the computed at_rest_total value to fallback
to the original at_rest_total value when the computed value is null.

https://forum.storj.io/t/release-preparation-v1-62/19444/5?u=clement

Change-Id: Ifa268ccbe35a63e3b68f07464194fa034ad261b5
2022-08-23 12:56:28 +00:00
NickolaiYurchenko
f35b4163f9 web/satellite: multiple passphrase notification added
after redirecting from bucket page to upload page if we have no files/folders
related to this passphrase but have other objects in bucket we show closable
reminder that we support multiple passphrases per bucket

Change-Id: I6420aedd5605100e4aa35b598771e5298e251f91
2022-08-23 12:14:56 +00:00
Clement Sam
7e5025cac0 {storagenode,multinode/nodes}: use multinodeauth.Secret instead of []byte for APISecret
When enconding structs into JSON, byte slices are marshalled as base64
encoded string using the base64.StdEncoding.Encode():
ea9c3fd42d/src/encoding/json/encode.go (L833-L861)

We, however, expect API Secrets to be encoded as base64URL, so when
an marshalled secret (with byte slice type) is added to the multinode
dashboard, it fails with `illegal base64 data at input byte XX`.

This change changes the type of APISecret field in the
multinode/nodes.Nodes struct to use multinodeauth.Secret type instead
of []byte.
multinodeauth.Secret is extended with custom MarshalJSON and
UnmarshalJSON methods which implement the json.Marshaler and
json.Unmarshaler interfaces, respectively.

Resolves https://github.com/storj/storj/issues/4949

Change-Id: Ib14b5f49ceaac109620c25d7ff83be865c698343
2022-08-23 11:04:04 +00:00
Wilfred Asomani
a4192acabb console/satellite: track signup captcha scores
This change tracks signup captcha scores in the signup_captcha column in the users table.
It slightly modifies the captcha verify method to return both the score and success.

see: https://github.com/storj/storj/issues/5067

Change-Id: I7b3993e44958cfcf179806c7df19d6887fe3eda9
2022-08-23 10:13:49 +00:00
dlamarmorgan
cd89e1e557 satellite/satellitedb: remove direct import of pgxerrcode
Use the provided ConstraintViolation method of the pgutil package rather than importing jackc pgxerrcode directly.

Change-Id: I4e86713000b3f5f0aadd54beee8ee239f0c8df8e
2022-08-22 11:31:48 -07:00
Vitalii
998835a0c3 web/satellite: make upgrade account modal responsive
Made upgrade account modal responsive.

Change-Id: I1232cde049d91414c4361836d4ac31c1d5dbdea4
2022-08-22 14:37:58 +00:00
Márton Elek
0cc054e931 build: turn off postgres unit tests (build time test)
Change-Id: Iecb018791d9533c354375f9b7865c83ba32092f5
2022-08-22 13:18:59 +00:00
Artur M. Wolff
36ae8d4bb9 satellite/metabase: revert non-recursive listing optimization
This change reverts satellite/metabase/iterator.go of 7390f389c to the
previous version (without optimization) but leaves added benchmarks in
place.

We noticed that this optimization doesn't work and actually elevates
listing times for most buckets, hence the revert until we come up with a
better idea.

Benchmarks:

name                                                old time/op new time/op delta
NonRecursiveListing/Postgres/listing_no_prefix-8    1.30ms ± 4% 4.52ms ± 4% +246.92% (p=0.008 n=5+5)
NonRecursiveListing/Postgres/listing_with_prefix-8  3.26ms ± 3% 4.44ms ± 2%  +36.19% (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_no_prefix-8    618µs ± 3% 2225µs ± 2% +259.94% (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8 1.81ms ± 5% 2.60ms ± 5%  +43.96% (p=0.008 n=5+5)

Updates storj/team-metainfo#115

Change-Id: I96e4e7a563b188df478f8489027dc0042469b839
2022-08-22 14:14:10 +02:00
Clement Sam
31242f3820 satellite: use table aliases for query in QueryStorageNodeUsage endpoint
Following the changes made to fix the storage usage graph
on the storagenode dashboard, we added a new
interval_end_time column to the accounting_rollups table.

We noticed a two-day delay in the graph, turns out the sub-query
was wrong due to the conflicting interval_end_time column
in both tables so we have to explicitly state which table
column we are referring to.

Also, set the default for interval_end_time in the accounting
rollups to the start_time if the interval_end_time is null
which will be removed once we backfill the column and alter
it to be a non-nullable column.

Updates https://github.com/storj/storj/issues/4178

Change-Id: Iff32b261d07b6ee219d2b6b6542377f0c54633a1
2022-08-22 11:39:45 +00:00
Michał Niewrzał
71c1d0fff2 satellite/metabase: reduce queries for iterator boundaries tests
This is longest metabase at the moment and would nice to speed
up is a bit to improve overal tests execution time.

Change-Id: I86da8e0e593d20024b3ec778cbeab34a4613151f
2022-08-22 10:47:28 +00:00
Márton Elek
6b27c64833 testplanet: support snapshot based migration for storagenode
Similar to the existing snapshot based tests of satellite/metabase db we make a migration here which is:
 * dedicated to unit tests
 * faster (with less steps)
 * but safe: additional unit test ensures that the snapshot based migration and normal prod migration have the same results.

Change-Id: Ie324b09f64b4553df02247a9461ece305a6cf832
2022-08-22 09:46:27 +00:00
Egon Elbre
8ddd5557f5 satellite/satellitedb: reduce migration tests
Database migration tests are rather slow, reduce them to
only the last 10 migrations, which should be sufficient.

Change-Id: Ib9d964fe6ec86ddeeef26c66b6ea9207b7868855
2022-08-22 09:04:49 +00:00
Malcolm Bouzi
6b08b0a656 web/satellite: Replace coinpayments functionality in card on new billing screen with a message to contact support.
Change-Id: Ic91f9aa5dcd629982c4ed60702c5bfd8d8a4c3d7
2022-08-22 07:47:48 +00:00
Ivan Fraixedes
aa1f50197a satellite/metainfo: Don't log error due to ctx cancellation
Context cancellation that aborts a non-essential Redis operation must
not be logged as an error because the operation is intentionally
canceled.

We are actually considering them not to be an error in following
operation because of the same reason and we return a RPC canceled status
code.

On the other hand it doesn't make sense to continue if the context is
canceled because although this is a non-essential operation if this one
is canceled due to the context the next one will be canceled for the
same reason, hence, we return earlier.

Change-Id: Ib3331975adeb06367d1ea0a578263ef50ae3f079
2022-08-19 18:41:54 +00:00
Yaroslav Vorobiov
9c77c762cb satellite/payments/storjscan: change USD currency to USDollarsMicro
Changes all usd amounts to use monetary.USDollarsMicro to have better prescision.

Change-Id: I391e2680f7fa11a0493e5af77488acff6dbf4999
2022-08-19 16:05:17 +00:00
Sean Harvey
38d2d0f2fc
mod: bump storj/private and storj/common dependencies
Change-Id: I73ce29b1a6c28d6eec1570dc6568052b16502c61
2022-08-19 15:08:50 +12:00
Yaroslav Vorobiov
7a9b2a707b satellite/payments/monetary: add USDMicro and json marshaling
Adds USDMicro currency which support fraction of a cent with decimal places
for better billing amounts accuracy.
Adds JSON marshaling and unmarshaling for monetary.Amount, so that it
can be converted to/from JSON.

Change-Id: I034eba120ed23b6ba00b2d81a4f1b9db5f9a203f
2022-08-18 18:38:01 +00:00
JT Olio
e0d3e48b66 satellite/consoleweb: monitor visits without blowing up influx namespace
Change-Id: I43de5409e66b9ef34b6c35f211b9a252d3cc3caa
2022-08-18 17:48:28 +00:00
Cameron
fa23e55654 {private,satellite}: add Typescript generation to REST API generation
github issue: https://github.com/storj/storj/issues/4984

Change-Id: Id3473a6642f7a4af80edda25a6242559efaf69e9
2022-08-18 16:37:48 +00:00
cl-mitch
c255d6949f web/satellite: Minor access grant updates
This change is a copy of the stale Github PR at https://github.com/storj/storj/pull/4972
It fixes some of the remaining minor issues outlined in https://github.com/storj/storj/issues/4966

Change-Id: Ide973b1b23440bf37a19af8aa6ead39b470ceb25
2022-08-18 15:23:44 +00:00
Malcolm Bouzi
cf4bfbcf2d web/satellite: make login and registration satellite dropdowns accessible through tab usage.
Change-Id: I279ac065eb00e6e9412c78ff8c3eb9f2e4a308ec
2022-08-18 06:22:07 +00:00
Wilfred Asomani
874e28e283 web/satellite: fix inactive buttons on billing page
This change fixes makes the "See Payment" and "Edit payment method" on the billing page
navigate to the correct corresponding pages.

Change-Id: Ia2045394643d11171cc8ad5ded6654c4293875d6
2022-08-17 20:32:27 +00:00
Wilfred Asomani
c59d0d415c satellitedb: add signup_captcha column to users table
The signup_captcha column will hold the captcha scores of new users.

see: https://github.com/storj/storj/issues/5067

Change-Id: Ia322af29a3b5b019b417843272506a3dbd1397e4
2022-08-17 14:31:52 -05:00
paul cannon
0dcc0a9ee0 satellite/reputation: reconfigure lambda and alpha
This is in response to community feedback that our existing reputation
calculation is too likely to disqualify storage nodes unfairly with
extreme swings up and down.

For details and analysis, please see the data_loss_vs_dq_chance_sim.py
tool, the "tuning reputation further.ipynb" Jupyter notebook in the
storj/datascience repository, and the discussion at

    https://forum.storj.io/t/tuning-audit-scoring/14084

In brief: changing the lambda and initial-alpha parameters in this way
causes the swings in reputation to be smaller and less likely to put a
node past the disqualification threshold unfairly.

Note: this change will cause a one-time reset of all (non-disqualified)
node reputations, because the new initial alpha value of 1000 is
dramatically different, and the disqualification threshold is going to
be much higher.

Change-Id: Id6dc4ba8fde1be3db4255b72282207bab5491ca3
2022-08-17 18:52:53 +00:00
Wilfred Asomani
99dee2daa3 web/satellite: fix bucket page guide positioning
This change fixes an issue where the bucket guide goes off screen

see: https://github.com/storj/storj/issues/5086

Change-Id: Idf0718df2ea49b195b649445f60568c35ca856d9
2022-08-17 17:57:47 +00:00
Vitalii
418372621c web/satellite: improve access grants empty state
Improved access grants empty state to show correct label depending on search query.

Issue: https://github.com/storj/storj/issues/5074

Change-Id: Ibc5e8857ed213bcea28774151472e8750400beee
2022-08-17 17:06:12 +00:00
Vitalii
9dcef77639 web/satellite: fix for billing screen
Fixed old billing layout to show new billing layout.
There was a routing issue when navigating from account dropdown.

Change-Id: I65d91681b2891ec347a6729ae3db4cd0e6073161
2022-08-17 16:28:30 +00:00
Wilfred Asomani
286f68a0bd web/satellite: fix page content bottom padding
This change removes the errant bottom padding on all page contents.

Change-Id: Ife4d86853fffda06ad34b08999d5c52a3b5754d7
2022-08-17 15:42:00 +00:00
Cameron
c82424c21d satellite/admin: add Burst field to Update Project Limits params
Since 6 fields were sent, but only 5 fields were used as input (no burst),
the Segments UI input field was being set to the burst field in the
object, leaving segments empty.

Change-Id: I4e58ed8a36e2022d1d0844e9f9001562db3dcbb8
2022-08-17 10:40:28 -04:00
Wilfred Asomani
eb547740f4 web/satellite: fix mobile signup page
This change fixes an issue where description texts will not show on the partner signup page on small screens.

Change-Id: I59868b5c598499e3fc6dba7991453ec5c4e4d998
2022-08-17 14:00:19 +00:00
Jeremy Wharton
ffd35ecfdc web/satellite: move billing nav item into My Account dropdown
This change moves the top-level Billing navigation item into the
My Account dropdown menu. A notification has been added that
informs users of this change.

Change-Id: I641b0e5ad0a6de0206f6cde9d3022033c419215a
2022-08-17 06:42:00 +00:00
dlamarmorgan
8eec61d315 satellitedb: add timestamp index to billing table
Adding an index to the timestamp field of the billing transaction table to improve query performance. This should prevent having to do a full table scan when we query for the last billing transaction of a particular source and/or type.

Change-Id: I581f09494cc8662a12efba4302022a07121ba309
2022-08-16 22:50:02 +00:00
dlamarmorgan
f596f72f44 satellite/payments/storjscan: add billing update to chore
Update chore to add confirmed transactions to the billing table.

Change-Id: I3c9a98c2ddc96f8a8905250376a1f5490d810277
2022-08-16 11:18:15 -07:00
Ivan Fraixedes
7299e9f92e
satellite/console: Add missing monkit calls
Add monkit monitoring calls to a couple of the service's methods because
they were missed.

Change-Id: I327a79634b7cdcaeb0e2701a737bb714a6478c4a
2022-08-16 13:20:18 +02:00
dlamarmorgan
ac241501ca satellitedb: add wallet address index to storjscan_wallet table
Adding an index to the wallet address field of the storjscan wallet table to improve query performance. This should prevent having to do a full table scan when we query for one or more wallet addresses for a given user in our queries.

Change-Id: Ic1b5d06c2258489e5464d186fed5270172f8cba5
2022-08-15 18:05:06 +00:00
Jeremy Wharton
1f0638719e private/apigen,cmd/apigentest: add tests for generated API code
This change implements a unit test for ensuring proper
processing of requests and responses by generated API code.
Additionally, this change requires API handlers to explicitly receive
Monkit scopes rather than assuming that `mon` will always exist in the
generated API code's namespace.

Change-Id: Iea56f139f9dad0050b7d09ea765189280c3466f2
2022-08-15 16:48:41 +00:00
wilfredasomani
26bed35f33 web/satellite: update access grants table
This change updates the access grants table to use the new table component.

see: https://github.com/storj/storj/issues/4994

Change-Id: I6fa8c3feb2023c2dacc7ea5536ba9e16652f8c26
2022-08-15 11:55:53 +00:00
Vitalii
0ec28ca623 web/satellite: add token funds modal
Created new modal which shows user their native STORJ token wallet address.
There are QR and copy buttons.
It will be used only in new billing screen.

Change-Id: Icef3c8668c548b779c07fe2b85eb5761cd1221a3
2022-08-15 14:17:30 +03:00
cl-mitch
d910074097
web/satellite Billing screen bug fixes (#5050)
Changed the Billing History tab to only display Stripe payments, updated the coupons date to change based on the duration, and addressed the following tickets:
https://github.com/storj/storj/issues/5028
https://github.com/storj/storj/issues/5027
https://github.com/storj/storj/issues/5026
https://github.com/storj/storj/issues/5021
2022-08-12 14:01:13 -05:00
Qweder93
2abe709b6e satellite/metabase: add metabase.GetObjectLatestVersion back
Restored GetObjectLatestVersion and renamed it to GetObjectLastCommitted
Add test cases to cover server-side copy

Closes https://github.com/storj/storj/issues/4866

Change-Id: I343b339a60152b8fb92fda97baf80bd8fe60d631
2022-08-11 22:14:38 +00:00
Vitalii
c552c343f9 web/satellite: make modals responsive
Made next modals reponsive:
AddTeamMemberModal
ChangePasswordModal
CreateProjectModal
CreateProjectPromptModal
DisableMFAModal
EditProfileModal
EnableMFAModal
MFARecoveryCodesModal

Issue:
https://github.com/storj/storj/issues/5031

Change-Id: I9c80137febd1d16678f3de523634efc5010c8a9d
2022-08-11 20:00:12 +00:00
hovex023
4be219c320
Web/satellite: Add responsiveness to onboarding flow (#5065)
* added media queries for onboarding flow

* responsive changes to create bucket steps

Change-Id: I2799a75db4e5029fa8a1ce11afac04333cca09c7

Co-authored-by: Moby von Briesen <mobyvb@gmail.com>
2022-08-11 13:45:11 -05:00
kimbotsao
c3e7b1409d satellitedb: add salt to projects table
Change-Id: I53691849efc169c495566648c12ae3d639614c3d
2022-08-11 12:16:53 -04:00
wilfredasomani
8b338fb655 web/satellite: show guide on bucket page
This change shows a new "no bucket" page and a
tooltip to guide users to open a bucket for upload.

see: https://github.com/storj/storj/issues/4969

Change-Id: I99fd855fe39ba7ad308b1c90f79703eeee1b1627
2022-08-11 15:36:51 +00:00
Márton Elek
96b88e286e build: fix conditional run of web builds
Jenkins doesn't do a very good job with identifying what has been changed.

While it has a syntax to defined patterns, it compares the current build with the previous build (in case of git-verify it can be a totally different branch) instead of checking the HEAD commit.

This patch introduces shell scripts to do this better:
 * It doesn't depend on Jenkins any more
 * It can be executed locally
 * It can detect web changes properly (see the relation change as an example).

Change-Id: I9d37775e3818c08c4aa96ffb78f84d57f28a2c95
2022-08-11 14:34:44 +00:00
wilfredasomani
cff8158054 satellite/console: add audit logs for failed logins
This change adds login failure logs for specific userID/email.

see: https://github.com/storj/storj/issues/4987

Change-Id: I58529145d7bd65abe47e002f34ec88018f641268
2022-08-11 12:11:21 +00:00
Márton Elek
f507de67f9 satellite: in-memory 'top'-like counter for project/partner
As a reminder
 * This counters are for data with high-cardinality
 * We have strong upper bound for memory limits
 * They can be accessed from /top monitoring interface

Example:

```
curl 172.20.0.10:11111/top
since ~ 2022-08-09T07:45:58Z
auth_request_count project=9094cff8-104e-4956-a367-97ea134b7e06 11.000000
auth_request_buckets  1.000000
auth_request_discarded  0.000000
auth_request_count partner=00000000-0000-0000-0000-000000000000 11.000000
auth_request_buckets  1.000000
auth_request_discarded  0.000000
```

Note: discarded 0 --> we didn't hit the memory limit.

Change-Id: I8db09b4aa61bade55cb324b84b7fbcb8f068c179
2022-08-11 10:21:54 +00:00
Márton Elek
0698b1ef88 go.mod: bump storj/private version
Change-Id: I72ff0a6d8b8d929f0753988e447bf2b41552199c
2022-08-11 09:08:32 +00:00
hovex023
4418216b4c
Web/satellite: Create Access Flow Modal Refactor (#5015)
Extrapolated each stage in the Access Grant flow into its own component and replaced the code on CreateAccessModal with the new components.
2022-08-10 17:36:41 -05:00