Commit Graph

7010 Commits

Author SHA1 Message Date
Wilfred Asomani
0c4e45875f web/satellite: change segment learn-more link
This change uses the correct docs link for segments.

see: https://github.com/storj/storj/issues/5098

Change-Id: I49c0a171fa29064a495f6aa50624c71d9bfbc703
2022-08-25 15:27:57 +00:00
Wilfred Asomani
0bfdc9da30 web/satellite: fix billing duplicate navigation error
This change fixes the duplicate navigation error on the new billing screen.

see: https://github.com/storj/storj/issues/5096

Change-Id: Ie0512de806835acfc29e86182e4c49ced6355383
2022-08-25 14:46:29 +00:00
Vitalii
3019ad3c35 web/satellite: apply passphrase to newly created access grants
Apply passphrase when creating access grants or S3 credentials.

Change-Id: I59fc391ffdca59e8cd6bdce5632b565e9e9237ea
2022-08-25 15:43:56 +03:00
Clement Sam
ea0124a183 cmd/multinode: handle JSON input with BOM to add command
Decode the JSON input string to its corresponding unicode
decoding if the special byte order mark is present.

Fixes https://github.com/storj/storj/issues/4950

Change-Id: If91bac22590c89b35c58bf54f6d3bdb8a67d7a4f
2022-08-25 03:05:49 +00:00
Michal Niewrzal
d6054214c3 satellite/metainfo: move metadata validation to a proper place
Metadata validation for CommitObject request was placed in a wrong
place. There is a case (old uplink) where encrypted key is bundled
inside encrypted metadata bytes and we need to extract it before we
can validate it. This change moves metadata validation to a place where
we are sure we have encrypted metadata and encrypted metadata encrypted
key ready to be checked.

"Run Versions Test" is covering this case and it was failing without
this change.

Change-Id: Ib709ad901fbb3fa4865a393195b7b3f4c0d87e7a
2022-08-24 10:50:43 +00:00
Michał Niewrzał
ea38860aab satellite/metainfo: align satellite StreamID protobuf version with metabase
For object Version in different places we are using different types.
Satellite StreamID is using int32 but metabase accepts int64. Metabase
type is correct one and we should align other places with it.

As a small addition this change is also passing version correctly
between requests instead of using hardcoded value.

Change-Id: I63634d73c0a48c009e4db5f203ff18b7f3218b02
2022-08-24 10:17:34 +00:00
Qweder93
2139d3966b satellite/metabase: update UpdateObjectMetadata to work with latest commited object
Updated metabase.UpdateObjectMetadata method to update set metdata always for last committed object

Closes https://github.com/storj/storj/issues/4870

Change-Id: I060683e31efcaf3e2531fea143cf0567e5ff5f73
2022-08-24 10:28:57 +03:00
Márton Elek
3d79c5f3a1 storagenode/docker: allow to configure updater with separated env variables
We couldn't use environment variables safely to configure storagenode, since we introduced the embedded updater.

For example STORJ_DEBUG_ADDR=localhost:11111 would try to set debug port 11111 for both the storagenode and storagenode-updated, causing port conflict.

This small change enables to configure storagenode-updater with STORJUPDATER_... environment variables.

Tested with creating custom image and installing to my own storage node.

Change-Id: I6b0a601a4dc63d2d1ff3c191ae89981434e55c30
2022-08-24 06:50:30 +00:00
Márton Elek
4b1be6bf8e storagenode/satellite: support different piece hash algorithms
Change-Id: I3db321e79f12f3ebaa249e6c32fa37fd9615687e
2022-08-23 18:15:06 +00:00
Márton Elek
d23cfa6445 go.mod: bump storj.io/common and storj.io/uplink
Change-Id: Ie78b69f2ee855e9e45ef241d4b8f8c225502b8a7
2022-08-23 17:47:08 +00:00
Jeremy Wharton
3f26cc599f satellite/console,web/satellite: invalidate sessions after inactivity
Sessions now expire after a much shorter amount of time, requiring
clients to issue API requests for session extension. This is handled
behind the scenes as the user interacts with the page, but once session
expiration is imminent, a modal appears which informs the user of his
inactivity and presents him with the choice of loging out or preserving
his session.

Change-Id: I68008d45859c814a835d65d882ad5ad2199d618e
2022-08-23 15:51:05 +00:00
Michał Niewrzał
b722c29e77 satellite/metainfo: add missing metadata validation
We are missing metadata validation for UpdateObjectMetadata and
FinishCopyOject requests.

Change-Id: Idca6a4d1fe108e1593405fd3913442f5b69d09e7
2022-08-23 16:56:45 +02:00
NickolaiYurchenko
3c0fc3a530 web/satellite: table responsiveness added
added Resizable wrapper to track screen dimensions changes
items to render computable added
TODO:
implement file browser table resposiveness

Change-Id: I913d5cdb5285d6f0fb7135183ff299c37f01ad4a
2022-08-23 14:08:01 +00:00
Michal Niewrzal
d1d98af098 satellite/metabase: jump after prefix at the end of iteration page
For non-recursive listing if cursor to the next iteration page
is a prefix then jump after this prefix to avoid listing objects from
this prefix.

Benchmark against 'main' - 7e5025cac0
name                                                 old time/op    new time/op    delta
NonRecursiveListing/Cockroach/listing_no_prefix-8      11.7ms ±29%     1.7ms ± 7%  -85.52%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8    4.60ms ± 6%    3.23ms ± 3%  -29.72%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_only_prefix-8    7.38ms ± 7%    8.47ms ±34%     ~     (p=0.421 n=5+5)

name                                                 old alloc/op   new alloc/op   delta
NonRecursiveListing/Cockroach/listing_no_prefix-8      92.3kB ± 0%    16.9kB ± 0%  -81.72%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8    38.0kB ± 0%    27.4kB ± 0%  -28.02%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_only_prefix-8    59.8kB ± 0%    60.1kB ± 0%   +0.38%  (p=0.008 n=5+5)

name                                                 old allocs/op  new allocs/op  delta
NonRecursiveListing/Cockroach/listing_no_prefix-8       1.78k ± 0%     0.31k ± 0%  -82.48%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8       734 ± 0%       526 ± 0%  -28.34%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_only_prefix-8     1.15k ± 0%     1.16k ± 0%   +0.87%  (p=0.008 n=5+5)

Discussed here https://github.com/storj/team-metainfo/issues/116

Change-Id: Iff671f062d9af83ec419334089163dd204d0cac0
2022-08-23 13:32:30 +00:00
Clement Sam
bac0155664 storagenode/storagenodedb: fix null at_rest_total values for storage usage
Wrapping a COALESCE around the computed at_rest_total value to fallback
to the original at_rest_total value when the computed value is null.

https://forum.storj.io/t/release-preparation-v1-62/19444/5?u=clement

Change-Id: Ifa268ccbe35a63e3b68f07464194fa034ad261b5
2022-08-23 12:56:28 +00:00
NickolaiYurchenko
f35b4163f9 web/satellite: multiple passphrase notification added
after redirecting from bucket page to upload page if we have no files/folders
related to this passphrase but have other objects in bucket we show closable
reminder that we support multiple passphrases per bucket

Change-Id: I6420aedd5605100e4aa35b598771e5298e251f91
2022-08-23 12:14:56 +00:00
Clement Sam
7e5025cac0 {storagenode,multinode/nodes}: use multinodeauth.Secret instead of []byte for APISecret
When enconding structs into JSON, byte slices are marshalled as base64
encoded string using the base64.StdEncoding.Encode():
ea9c3fd42d/src/encoding/json/encode.go (L833-L861)

We, however, expect API Secrets to be encoded as base64URL, so when
an marshalled secret (with byte slice type) is added to the multinode
dashboard, it fails with `illegal base64 data at input byte XX`.

This change changes the type of APISecret field in the
multinode/nodes.Nodes struct to use multinodeauth.Secret type instead
of []byte.
multinodeauth.Secret is extended with custom MarshalJSON and
UnmarshalJSON methods which implement the json.Marshaler and
json.Unmarshaler interfaces, respectively.

Resolves https://github.com/storj/storj/issues/4949

Change-Id: Ib14b5f49ceaac109620c25d7ff83be865c698343
2022-08-23 11:04:04 +00:00
Wilfred Asomani
a4192acabb console/satellite: track signup captcha scores
This change tracks signup captcha scores in the signup_captcha column in the users table.
It slightly modifies the captcha verify method to return both the score and success.

see: https://github.com/storj/storj/issues/5067

Change-Id: I7b3993e44958cfcf179806c7df19d6887fe3eda9
2022-08-23 10:13:49 +00:00
dlamarmorgan
cd89e1e557 satellite/satellitedb: remove direct import of pgxerrcode
Use the provided ConstraintViolation method of the pgutil package rather than importing jackc pgxerrcode directly.

Change-Id: I4e86713000b3f5f0aadd54beee8ee239f0c8df8e
2022-08-22 11:31:48 -07:00
Vitalii
998835a0c3 web/satellite: make upgrade account modal responsive
Made upgrade account modal responsive.

Change-Id: I1232cde049d91414c4361836d4ac31c1d5dbdea4
2022-08-22 14:37:58 +00:00
Márton Elek
0cc054e931 build: turn off postgres unit tests (build time test)
Change-Id: Iecb018791d9533c354375f9b7865c83ba32092f5
2022-08-22 13:18:59 +00:00
Artur M. Wolff
36ae8d4bb9 satellite/metabase: revert non-recursive listing optimization
This change reverts satellite/metabase/iterator.go of 7390f389c to the
previous version (without optimization) but leaves added benchmarks in
place.

We noticed that this optimization doesn't work and actually elevates
listing times for most buckets, hence the revert until we come up with a
better idea.

Benchmarks:

name                                                old time/op new time/op delta
NonRecursiveListing/Postgres/listing_no_prefix-8    1.30ms ± 4% 4.52ms ± 4% +246.92% (p=0.008 n=5+5)
NonRecursiveListing/Postgres/listing_with_prefix-8  3.26ms ± 3% 4.44ms ± 2%  +36.19% (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_no_prefix-8    618µs ± 3% 2225µs ± 2% +259.94% (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8 1.81ms ± 5% 2.60ms ± 5%  +43.96% (p=0.008 n=5+5)

Updates storj/team-metainfo#115

Change-Id: I96e4e7a563b188df478f8489027dc0042469b839
2022-08-22 14:14:10 +02:00
Clement Sam
31242f3820 satellite: use table aliases for query in QueryStorageNodeUsage endpoint
Following the changes made to fix the storage usage graph
on the storagenode dashboard, we added a new
interval_end_time column to the accounting_rollups table.

We noticed a two-day delay in the graph, turns out the sub-query
was wrong due to the conflicting interval_end_time column
in both tables so we have to explicitly state which table
column we are referring to.

Also, set the default for interval_end_time in the accounting
rollups to the start_time if the interval_end_time is null
which will be removed once we backfill the column and alter
it to be a non-nullable column.

Updates https://github.com/storj/storj/issues/4178

Change-Id: Iff32b261d07b6ee219d2b6b6542377f0c54633a1
2022-08-22 11:39:45 +00:00
Michał Niewrzał
71c1d0fff2 satellite/metabase: reduce queries for iterator boundaries tests
This is longest metabase at the moment and would nice to speed
up is a bit to improve overal tests execution time.

Change-Id: I86da8e0e593d20024b3ec778cbeab34a4613151f
2022-08-22 10:47:28 +00:00
Márton Elek
6b27c64833 testplanet: support snapshot based migration for storagenode
Similar to the existing snapshot based tests of satellite/metabase db we make a migration here which is:
 * dedicated to unit tests
 * faster (with less steps)
 * but safe: additional unit test ensures that the snapshot based migration and normal prod migration have the same results.

Change-Id: Ie324b09f64b4553df02247a9461ece305a6cf832
2022-08-22 09:46:27 +00:00
Egon Elbre
8ddd5557f5 satellite/satellitedb: reduce migration tests
Database migration tests are rather slow, reduce them to
only the last 10 migrations, which should be sufficient.

Change-Id: Ib9d964fe6ec86ddeeef26c66b6ea9207b7868855
2022-08-22 09:04:49 +00:00
Malcolm Bouzi
6b08b0a656 web/satellite: Replace coinpayments functionality in card on new billing screen with a message to contact support.
Change-Id: Ic91f9aa5dcd629982c4ed60702c5bfd8d8a4c3d7
2022-08-22 07:47:48 +00:00
Ivan Fraixedes
aa1f50197a satellite/metainfo: Don't log error due to ctx cancellation
Context cancellation that aborts a non-essential Redis operation must
not be logged as an error because the operation is intentionally
canceled.

We are actually considering them not to be an error in following
operation because of the same reason and we return a RPC canceled status
code.

On the other hand it doesn't make sense to continue if the context is
canceled because although this is a non-essential operation if this one
is canceled due to the context the next one will be canceled for the
same reason, hence, we return earlier.

Change-Id: Ib3331975adeb06367d1ea0a578263ef50ae3f079
2022-08-19 18:41:54 +00:00
Yaroslav Vorobiov
9c77c762cb satellite/payments/storjscan: change USD currency to USDollarsMicro
Changes all usd amounts to use monetary.USDollarsMicro to have better prescision.

Change-Id: I391e2680f7fa11a0493e5af77488acff6dbf4999
2022-08-19 16:05:17 +00:00
Sean Harvey
38d2d0f2fc
mod: bump storj/private and storj/common dependencies
Change-Id: I73ce29b1a6c28d6eec1570dc6568052b16502c61
2022-08-19 15:08:50 +12:00
Yaroslav Vorobiov
7a9b2a707b satellite/payments/monetary: add USDMicro and json marshaling
Adds USDMicro currency which support fraction of a cent with decimal places
for better billing amounts accuracy.
Adds JSON marshaling and unmarshaling for monetary.Amount, so that it
can be converted to/from JSON.

Change-Id: I034eba120ed23b6ba00b2d81a4f1b9db5f9a203f
2022-08-18 18:38:01 +00:00
JT Olio
e0d3e48b66 satellite/consoleweb: monitor visits without blowing up influx namespace
Change-Id: I43de5409e66b9ef34b6c35f211b9a252d3cc3caa
2022-08-18 17:48:28 +00:00
Cameron
fa23e55654 {private,satellite}: add Typescript generation to REST API generation
github issue: https://github.com/storj/storj/issues/4984

Change-Id: Id3473a6642f7a4af80edda25a6242559efaf69e9
2022-08-18 16:37:48 +00:00
cl-mitch
c255d6949f web/satellite: Minor access grant updates
This change is a copy of the stale Github PR at https://github.com/storj/storj/pull/4972
It fixes some of the remaining minor issues outlined in https://github.com/storj/storj/issues/4966

Change-Id: Ide973b1b23440bf37a19af8aa6ead39b470ceb25
2022-08-18 15:23:44 +00:00
Malcolm Bouzi
cf4bfbcf2d web/satellite: make login and registration satellite dropdowns accessible through tab usage.
Change-Id: I279ac065eb00e6e9412c78ff8c3eb9f2e4a308ec
2022-08-18 06:22:07 +00:00
Wilfred Asomani
874e28e283 web/satellite: fix inactive buttons on billing page
This change fixes makes the "See Payment" and "Edit payment method" on the billing page
navigate to the correct corresponding pages.

Change-Id: Ia2045394643d11171cc8ad5ded6654c4293875d6
2022-08-17 20:32:27 +00:00
Wilfred Asomani
c59d0d415c satellitedb: add signup_captcha column to users table
The signup_captcha column will hold the captcha scores of new users.

see: https://github.com/storj/storj/issues/5067

Change-Id: Ia322af29a3b5b019b417843272506a3dbd1397e4
2022-08-17 14:31:52 -05:00
paul cannon
0dcc0a9ee0 satellite/reputation: reconfigure lambda and alpha
This is in response to community feedback that our existing reputation
calculation is too likely to disqualify storage nodes unfairly with
extreme swings up and down.

For details and analysis, please see the data_loss_vs_dq_chance_sim.py
tool, the "tuning reputation further.ipynb" Jupyter notebook in the
storj/datascience repository, and the discussion at

    https://forum.storj.io/t/tuning-audit-scoring/14084

In brief: changing the lambda and initial-alpha parameters in this way
causes the swings in reputation to be smaller and less likely to put a
node past the disqualification threshold unfairly.

Note: this change will cause a one-time reset of all (non-disqualified)
node reputations, because the new initial alpha value of 1000 is
dramatically different, and the disqualification threshold is going to
be much higher.

Change-Id: Id6dc4ba8fde1be3db4255b72282207bab5491ca3
2022-08-17 18:52:53 +00:00
Wilfred Asomani
99dee2daa3 web/satellite: fix bucket page guide positioning
This change fixes an issue where the bucket guide goes off screen

see: https://github.com/storj/storj/issues/5086

Change-Id: Idf0718df2ea49b195b649445f60568c35ca856d9
2022-08-17 17:57:47 +00:00
Vitalii
418372621c web/satellite: improve access grants empty state
Improved access grants empty state to show correct label depending on search query.

Issue: https://github.com/storj/storj/issues/5074

Change-Id: Ibc5e8857ed213bcea28774151472e8750400beee
2022-08-17 17:06:12 +00:00
Vitalii
9dcef77639 web/satellite: fix for billing screen
Fixed old billing layout to show new billing layout.
There was a routing issue when navigating from account dropdown.

Change-Id: I65d91681b2891ec347a6729ae3db4cd0e6073161
2022-08-17 16:28:30 +00:00
Wilfred Asomani
286f68a0bd web/satellite: fix page content bottom padding
This change removes the errant bottom padding on all page contents.

Change-Id: Ife4d86853fffda06ad34b08999d5c52a3b5754d7
2022-08-17 15:42:00 +00:00
Cameron
c82424c21d satellite/admin: add Burst field to Update Project Limits params
Since 6 fields were sent, but only 5 fields were used as input (no burst),
the Segments UI input field was being set to the burst field in the
object, leaving segments empty.

Change-Id: I4e58ed8a36e2022d1d0844e9f9001562db3dcbb8
2022-08-17 10:40:28 -04:00
Wilfred Asomani
eb547740f4 web/satellite: fix mobile signup page
This change fixes an issue where description texts will not show on the partner signup page on small screens.

Change-Id: I59868b5c598499e3fc6dba7991453ec5c4e4d998
2022-08-17 14:00:19 +00:00
Jeremy Wharton
ffd35ecfdc web/satellite: move billing nav item into My Account dropdown
This change moves the top-level Billing navigation item into the
My Account dropdown menu. A notification has been added that
informs users of this change.

Change-Id: I641b0e5ad0a6de0206f6cde9d3022033c419215a
2022-08-17 06:42:00 +00:00
dlamarmorgan
8eec61d315 satellitedb: add timestamp index to billing table
Adding an index to the timestamp field of the billing transaction table to improve query performance. This should prevent having to do a full table scan when we query for the last billing transaction of a particular source and/or type.

Change-Id: I581f09494cc8662a12efba4302022a07121ba309
2022-08-16 22:50:02 +00:00
dlamarmorgan
f596f72f44 satellite/payments/storjscan: add billing update to chore
Update chore to add confirmed transactions to the billing table.

Change-Id: I3c9a98c2ddc96f8a8905250376a1f5490d810277
2022-08-16 11:18:15 -07:00
Ivan Fraixedes
7299e9f92e
satellite/console: Add missing monkit calls
Add monkit monitoring calls to a couple of the service's methods because
they were missed.

Change-Id: I327a79634b7cdcaeb0e2701a737bb714a6478c4a
2022-08-16 13:20:18 +02:00
dlamarmorgan
ac241501ca satellitedb: add wallet address index to storjscan_wallet table
Adding an index to the wallet address field of the storjscan wallet table to improve query performance. This should prevent having to do a full table scan when we query for one or more wallet addresses for a given user in our queries.

Change-Id: Ic1b5d06c2258489e5464d186fed5270172f8cba5
2022-08-15 18:05:06 +00:00
Jeremy Wharton
1f0638719e private/apigen,cmd/apigentest: add tests for generated API code
This change implements a unit test for ensuring proper
processing of requests and responses by generated API code.
Additionally, this change requires API handlers to explicitly receive
Monkit scopes rather than assuming that `mon` will always exist in the
generated API code's namespace.

Change-Id: Iea56f139f9dad0050b7d09ea765189280c3466f2
2022-08-15 16:48:41 +00:00