Commit Graph

6889 Commits

Author SHA1 Message Date
Vitalii
ddc850dc21 web/satellite: added server-side encryption banner to buckets page
Added server-side encryption warning button to buckets page.
It can be dismissed and will show up again only on next user login.

Issue:
https://github.com/storj/storj/issues/5118

Change-Id: Iec5de91974ad5177a322b205f6b361c86f06923e
2022-09-02 09:36:18 +00:00
Moby von Briesen
61e18e8361 web/satellite: Update docs.storj.io anchor links
Reverts two of the changes in commit 119e61fcb because the timing of the
new docs deployment is changing.

Change-Id: Iaf804a41073d50700f67b153ef088631edefe6e5
2022-09-02 08:44:51 +00:00
Michal Niewrzal
68f6d93f29 satellite/gc/bloomfilter: add service to collect bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket. This this initial change to add such
service. Added service is joining segment loop and collects all
bloom filters.

Sending bloom filters to the bucket will be added as a subsequent
change.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I2551723605afa41bec84826b0c647cd1f61f3b14
2022-09-02 08:10:46 +00:00
Jeremy Wharton
1613d37466 satellite/console: add endpoints for listing and revoking api keys
This change adds the following endpoints:
- projects/apikeys/{id}: returns a paged list of API keys for the
	project specified by the given ID
- apikeys/delete/{id}: deletes the API key specified by the given ID

Additionally, the API Go code generator has been given the ability to
process unsigned integer parameters.

Change-Id: I5ff24e012da24a3f06bea1ebb62bae6ff62f951a
2022-09-01 14:24:00 -05:00
Malcolm Bouzi
8e3aa4d25b web/satellite: Creating a new folder on the bucket page is moved from an input field in the table list to a modal.
Change-Id: Ie15d4ec63b54c8be42f3c66e100e919b94adc345
2022-09-01 15:56:38 +00:00
Moby von Briesen
0ab0cef06c web/satellite: Upgrade to Vue 2.7
2.7 natively supports the Composition API, which makes transitioning to
Vue 3 easier to do step-by-step.

Converted ::v-deep selectors to :deep() to fix new build warnings.

Change-Id: I21b34e10f55eaf9dcdad87729f02f3b0d723b6b1
2022-09-01 10:41:59 -04:00
Vitalii
6f76b7e282 web/satellite: fix access grants used for sharing objects and buckets
Fix access grants used for linksharing.
Allow only List and Download.

Change-Id: I6123eb63af1ebefa0d771505632250c1c098a3b3
2022-09-01 16:30:22 +03:00
dlamarmorgan
335e11dacd satellitedb: add token balance to API endpoint
Add the users current wallet balance to the endpoints for claiming and listing storjscan wallets. Also prevent a user with a claimed wallet address from claiming a new wallet.

Change-Id: I0dbf1303699f924d05c8c52359038dc5ef6c42a1
2022-08-31 22:55:16 +00:00
Moby von Briesen
119e61fcb0 web/satellite: Update dashboard links to docs.storj.io
The anchor tag for these links is changing, so we need to update the UI
for them to work. Also remove trailing slashes from docs.storj.io links.

Change-Id: Ic4682dae76c44cf803b16ebfe7352405520c4307
2022-08-31 15:42:12 -04:00
Cameron
7933e0c4c7 cmd: add command to migrate project public_ids
This is to update all projects to have a public_id if they do not have
one.

github issue: https://github.com/storj/storj/issues/4861

Change-Id: Icfa42b62e15ca75d3c04a0aab48a3c3b0f3a9d6e
2022-08-31 13:14:17 +00:00
Qweder93
df5b0d0044 satellite/metabase: added method DeleteObjectLastCommitted
method similar to metabase.DeleteObjectExactVersion which will delete last committed object

Closes 	https://github.com/storj/storj/issues/4872

Change-Id: Ia9f8c227dc59575bf8ed297886b35536097028b4
2022-08-31 12:26:56 +00:00
Michal Niewrzal
2ae1db53b6 cmd/satellite: add command for collecting GC bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket. This change is extending satellite binary
with appropriete command.

New GC service for collecting bloom filter will be a subsequent
change.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: Ibc03e119c340919cf468fc1f5a4f3d187bb3a5a1
2022-08-31 11:40:36 +00:00
Márton Elek
ea1408f7a8 go.mod: bump clingy dependency
As a reminder: latest clingy removed the requirement of having custom context (which made the usage of context.WithValue harder) and uses simple context instead.

Clingy saves the stdin/stdout/stderr to the context (earlier to separated context type) to make it available for unit testing.

Change-Id: I8896574f4670721de43a577cd4b35952e3b5d00e
2022-08-31 10:24:27 +00:00
dlamarmorgan
47301e5718 billing: add USDollarsMicro to billing DB
Adds USDollarsMicro currency to the billing DB which support fraction of a cent with decimal places for better billing amounts accuracy.

Change-Id: Id07dfae104d94e27c7b22ab8f5781010e16c4c8e
2022-08-30 14:48:51 -07:00
dlamarmorgan
afe58323f9 satellitedb: remove use of batch payment insert
Removed batch insert of payments since they do not guarantee order. Order of payments sent to the payments DB is important, because the billing chore will request new payments based on the last received payment. If the last payment inserted is not the last payment received, duplicate payments will be inserted into the billing table.

Change-Id: Ic3335c89fa8031f7bc16f417ca23ed83301ef8f6
2022-08-30 14:45:55 -07:00
Moby von Briesen
8830d1eb7f web/satellite: Styling updates to registration page
Improve styling for lists in `customHtmlDescription` for alternate
signup landing pages in `registrationViewConfig`.

Change-Id: If20831044099a1183e5dc8b41668bf2f68c272a2
2022-08-30 20:33:22 +00:00
Michal Niewrzal
00c594362c satellite: skeleton for new GC peer
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket. This change is just initial code with peer
which will be used to build this new process.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I10a52b74865ce8ec4c29b7c6a2836f9232620422
2022-08-30 15:41:49 +00:00
Erik van Velzen
4393b8a5ce satellite/internalpb: new protobuf for retain filters
Add new protobuff message to store retain filters before sending them
out to storagenodes.

This allows the retain filters to be generated by a separate command on
a backup of the database.

Change-Id: I8a2892d9d34bed1dbd6e5a7ded000dcdea7ec061
2022-08-30 12:37:40 +02:00
Michal Niewrzal
fa76dfe577 satellite/metabase: use GetObjectLastCommitted with move/copy
We are preparing to use object versions internally and to do
that we need to prepare different parts of the system to handle
object versions different than '1'. This change adjust code
responsible for server-side move and copy.

What was done:
* begin methods for move and copy are now using GetObjectLastCommitted
to find object
* results from begin move and copy operation contains now version to
be able to map object correctly with finish operation
* begin methods are putting version into satellite stream id and
finish methods are using this version as parameter instead hardcoded
value

Fixes https://github.com/storj/storj/issues/4867

Change-Id: I1380911279c21e10a3fff0342793efd2e73eafad
2022-08-30 06:36:02 +00:00
Vitalii
d585f2550d web/satellite: share bucket feature
Implemented share bucket feature.
Refactored share object modal a bit (has to be refactored entirely).

Issue:
https://github.com/storj/storj/issues/4945

Change-Id: Icefd4bfe3eef9173ae824eea44d30450acde8044
2022-08-29 18:41:06 +03:00
paul cannon
a57cbc4e0b satellite/satellitedb: drop _gob columns
This concludes the saga that began with commit c053bdbd, migrating away
from using gob-encoded big.Float values in the database to using
integers with an implied number of decimal places.

Nothing is using or accessing or expecting the _gob columns to exist
anymore. The transition code and the migration chore are gone. All that
remains is to drop the columns.

Change-Id: I9b15ee52f7781510a6dc91cf7c54f7f9022b1210
2022-08-26 19:45:22 +00:00
JT Olio
e621190b99 satellitedb: randomize nodes for restore from trash
Change-Id: I030bd9093ec13a882edc7e0d325b9b19c201e135
2022-08-26 17:19:41 +00:00
Lizzy Thomson
8f1c0925dc web/satellite: update responsiveness in billing pages
update responsiveness in billing pages to match figma designs including added billing header scrolling for smaller windows

Fixes https://github.com/storj/storj/issues/5033

Change-Id: I94fc836f95e3985d6b675d3495961ec703e43044
2022-08-26 16:50:05 +00:00
Wilfred Asomani
72086e6dea console/satellite: make file browser responsive
see: https://github.com/storj/storj/issues/5037

Change-Id: Icb62ed7c2cdb5db7c64659e70952cae5fa90ae70
2022-08-26 16:19:01 +00:00
Michal Niewrzal
2163d60037 satellite/metabase/segmentloop: add missing monitoring
Currently we are tracking loop time with RunOnce method
metrics but it may give inaccurate values if loop if waiting
for observer for a ver long time. We had such case with GC
which is only loop observer in GC peer and its joning loop
every 5 days.

Change-Id: I08546c912e00c3641488de6a5d75948fe75c8e99
2022-08-26 14:01:22 +02:00
Márton Elek
7e71986493 storagenode: accept HTTP calls on public port, listening for monitoring requests
Today each storagenode should have a port which is opened for the internet, and handles DRPC protocol calls.

When we do a HTTP call on the DRPC endpoint, it hangs until a timeout.

This patch changes the behavior: the main DRPC port of the storagenodes can accept HTTP requests and can be used to monitor the status of the node:

 * if returns with HTTP 200 only if the storagnode is healthy (not suspended / disqualified + online score > 0.9)
 * it CAN include information about the current status (per satellite). It's opt-in, you should configure it so.

In this way it becomes extremely easy to monitor storagenodes with external uptime services.

Note: this patch exposes some information which was not easily available before (especially the node status, and used satellites). I think it should be acceptable:

 * Until having more community satellites, all storagenodes are connected to the main Storj satellites.
 * With community satellites, it's good thing to have more transparency (easy way to check who is connected to which satellites)

The implementation is based on this line:

```
http.Serve(NewPrefixedListener([]byte("GET / HT"), publicMux.Route("GET / HT")), p.public.http)
```

This line answers to the TCP requests with `GET / HT...` (GET HTTP request to the route), but puts back the removed prefix.

Change-Id: I3700c7e24524850825ecdf75a4bcc3b4afcb3a74
2022-08-26 09:38:09 +00:00
paul cannon
07bbe7d340 satellite/overlay: don't insert new nodes if contact check failed
Currently, the satellite tracks connectivity information about all nodes
that have contacted it, even if we have never successfully contacted
the node back.

This behavior was leveraged during a security audit to create hundreds
of thousands of "junk nodes" in the nodes table on one satellite, which
affected performance of queries such as node selection.

With this change, we should no longer track information about nodes that
have never been successfully contacted.

Note that it will still be possible to cause the creation of "junk
node" entries in the db; the attacker just has to set up individual
publicly-routable IP+port pairs for each node as it is created, so it
can respond to a PingBack.

Change-Id: Ibb6da6cc908fd4fc85aae1ba00313ba2738409ab
2022-08-26 09:05:04 +00:00
Sean Harvey
5a5f609022
satellite/metainfo: add copy and move log and metrics
Change-Id: I579c03c389dea7f8e7e92ade5bcac53740ce69c8
2022-08-26 18:55:29 +12:00
JT Olio
12ef68bdf4 satellitedb: restore-trash shouldn't bother with nodes we've never talked to
Change-Id: I3c631e92a9a2670c52c9fa23b2b1baf67d3c9a9c
2022-08-25 16:14:29 +00:00
Wilfred Asomani
0c4e45875f web/satellite: change segment learn-more link
This change uses the correct docs link for segments.

see: https://github.com/storj/storj/issues/5098

Change-Id: I49c0a171fa29064a495f6aa50624c71d9bfbc703
2022-08-25 15:27:57 +00:00
Wilfred Asomani
0bfdc9da30 web/satellite: fix billing duplicate navigation error
This change fixes the duplicate navigation error on the new billing screen.

see: https://github.com/storj/storj/issues/5096

Change-Id: Ie0512de806835acfc29e86182e4c49ced6355383
2022-08-25 14:46:29 +00:00
Vitalii
3019ad3c35 web/satellite: apply passphrase to newly created access grants
Apply passphrase when creating access grants or S3 credentials.

Change-Id: I59fc391ffdca59e8cd6bdce5632b565e9e9237ea
2022-08-25 15:43:56 +03:00
Clement Sam
ea0124a183 cmd/multinode: handle JSON input with BOM to add command
Decode the JSON input string to its corresponding unicode
decoding if the special byte order mark is present.

Fixes https://github.com/storj/storj/issues/4950

Change-Id: If91bac22590c89b35c58bf54f6d3bdb8a67d7a4f
2022-08-25 03:05:49 +00:00
Michal Niewrzal
d6054214c3 satellite/metainfo: move metadata validation to a proper place
Metadata validation for CommitObject request was placed in a wrong
place. There is a case (old uplink) where encrypted key is bundled
inside encrypted metadata bytes and we need to extract it before we
can validate it. This change moves metadata validation to a place where
we are sure we have encrypted metadata and encrypted metadata encrypted
key ready to be checked.

"Run Versions Test" is covering this case and it was failing without
this change.

Change-Id: Ib709ad901fbb3fa4865a393195b7b3f4c0d87e7a
2022-08-24 10:50:43 +00:00
Michał Niewrzał
ea38860aab satellite/metainfo: align satellite StreamID protobuf version with metabase
For object Version in different places we are using different types.
Satellite StreamID is using int32 but metabase accepts int64. Metabase
type is correct one and we should align other places with it.

As a small addition this change is also passing version correctly
between requests instead of using hardcoded value.

Change-Id: I63634d73c0a48c009e4db5f203ff18b7f3218b02
2022-08-24 10:17:34 +00:00
Qweder93
2139d3966b satellite/metabase: update UpdateObjectMetadata to work with latest commited object
Updated metabase.UpdateObjectMetadata method to update set metdata always for last committed object

Closes https://github.com/storj/storj/issues/4870

Change-Id: I060683e31efcaf3e2531fea143cf0567e5ff5f73
2022-08-24 10:28:57 +03:00
Márton Elek
3d79c5f3a1 storagenode/docker: allow to configure updater with separated env variables
We couldn't use environment variables safely to configure storagenode, since we introduced the embedded updater.

For example STORJ_DEBUG_ADDR=localhost:11111 would try to set debug port 11111 for both the storagenode and storagenode-updated, causing port conflict.

This small change enables to configure storagenode-updater with STORJUPDATER_... environment variables.

Tested with creating custom image and installing to my own storage node.

Change-Id: I6b0a601a4dc63d2d1ff3c191ae89981434e55c30
2022-08-24 06:50:30 +00:00
Márton Elek
4b1be6bf8e storagenode/satellite: support different piece hash algorithms
Change-Id: I3db321e79f12f3ebaa249e6c32fa37fd9615687e
2022-08-23 18:15:06 +00:00
Márton Elek
d23cfa6445 go.mod: bump storj.io/common and storj.io/uplink
Change-Id: Ie78b69f2ee855e9e45ef241d4b8f8c225502b8a7
2022-08-23 17:47:08 +00:00
Jeremy Wharton
3f26cc599f satellite/console,web/satellite: invalidate sessions after inactivity
Sessions now expire after a much shorter amount of time, requiring
clients to issue API requests for session extension. This is handled
behind the scenes as the user interacts with the page, but once session
expiration is imminent, a modal appears which informs the user of his
inactivity and presents him with the choice of loging out or preserving
his session.

Change-Id: I68008d45859c814a835d65d882ad5ad2199d618e
2022-08-23 15:51:05 +00:00
Michał Niewrzał
b722c29e77 satellite/metainfo: add missing metadata validation
We are missing metadata validation for UpdateObjectMetadata and
FinishCopyOject requests.

Change-Id: Idca6a4d1fe108e1593405fd3913442f5b69d09e7
2022-08-23 16:56:45 +02:00
NickolaiYurchenko
3c0fc3a530 web/satellite: table responsiveness added
added Resizable wrapper to track screen dimensions changes
items to render computable added
TODO:
implement file browser table resposiveness

Change-Id: I913d5cdb5285d6f0fb7135183ff299c37f01ad4a
2022-08-23 14:08:01 +00:00
Michal Niewrzal
d1d98af098 satellite/metabase: jump after prefix at the end of iteration page
For non-recursive listing if cursor to the next iteration page
is a prefix then jump after this prefix to avoid listing objects from
this prefix.

Benchmark against 'main' - 7e5025cac0
name                                                 old time/op    new time/op    delta
NonRecursiveListing/Cockroach/listing_no_prefix-8      11.7ms ±29%     1.7ms ± 7%  -85.52%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8    4.60ms ± 6%    3.23ms ± 3%  -29.72%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_only_prefix-8    7.38ms ± 7%    8.47ms ±34%     ~     (p=0.421 n=5+5)

name                                                 old alloc/op   new alloc/op   delta
NonRecursiveListing/Cockroach/listing_no_prefix-8      92.3kB ± 0%    16.9kB ± 0%  -81.72%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8    38.0kB ± 0%    27.4kB ± 0%  -28.02%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_only_prefix-8    59.8kB ± 0%    60.1kB ± 0%   +0.38%  (p=0.008 n=5+5)

name                                                 old allocs/op  new allocs/op  delta
NonRecursiveListing/Cockroach/listing_no_prefix-8       1.78k ± 0%     0.31k ± 0%  -82.48%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_with_prefix-8       734 ± 0%       526 ± 0%  -28.34%  (p=0.008 n=5+5)
NonRecursiveListing/Cockroach/listing_only_prefix-8     1.15k ± 0%     1.16k ± 0%   +0.87%  (p=0.008 n=5+5)

Discussed here https://github.com/storj/team-metainfo/issues/116

Change-Id: Iff671f062d9af83ec419334089163dd204d0cac0
2022-08-23 13:32:30 +00:00
Clement Sam
bac0155664 storagenode/storagenodedb: fix null at_rest_total values for storage usage
Wrapping a COALESCE around the computed at_rest_total value to fallback
to the original at_rest_total value when the computed value is null.

https://forum.storj.io/t/release-preparation-v1-62/19444/5?u=clement

Change-Id: Ifa268ccbe35a63e3b68f07464194fa034ad261b5
2022-08-23 12:56:28 +00:00
NickolaiYurchenko
f35b4163f9 web/satellite: multiple passphrase notification added
after redirecting from bucket page to upload page if we have no files/folders
related to this passphrase but have other objects in bucket we show closable
reminder that we support multiple passphrases per bucket

Change-Id: I6420aedd5605100e4aa35b598771e5298e251f91
2022-08-23 12:14:56 +00:00
Clement Sam
7e5025cac0 {storagenode,multinode/nodes}: use multinodeauth.Secret instead of []byte for APISecret
When enconding structs into JSON, byte slices are marshalled as base64
encoded string using the base64.StdEncoding.Encode():
ea9c3fd42d/src/encoding/json/encode.go (L833-L861)

We, however, expect API Secrets to be encoded as base64URL, so when
an marshalled secret (with byte slice type) is added to the multinode
dashboard, it fails with `illegal base64 data at input byte XX`.

This change changes the type of APISecret field in the
multinode/nodes.Nodes struct to use multinodeauth.Secret type instead
of []byte.
multinodeauth.Secret is extended with custom MarshalJSON and
UnmarshalJSON methods which implement the json.Marshaler and
json.Unmarshaler interfaces, respectively.

Resolves https://github.com/storj/storj/issues/4949

Change-Id: Ib14b5f49ceaac109620c25d7ff83be865c698343
2022-08-23 11:04:04 +00:00
Wilfred Asomani
a4192acabb console/satellite: track signup captcha scores
This change tracks signup captcha scores in the signup_captcha column in the users table.
It slightly modifies the captcha verify method to return both the score and success.

see: https://github.com/storj/storj/issues/5067

Change-Id: I7b3993e44958cfcf179806c7df19d6887fe3eda9
2022-08-23 10:13:49 +00:00
dlamarmorgan
cd89e1e557 satellite/satellitedb: remove direct import of pgxerrcode
Use the provided ConstraintViolation method of the pgutil package rather than importing jackc pgxerrcode directly.

Change-Id: I4e86713000b3f5f0aadd54beee8ee239f0c8df8e
2022-08-22 11:31:48 -07:00
Vitalii
998835a0c3 web/satellite: make upgrade account modal responsive
Made upgrade account modal responsive.

Change-Id: I1232cde049d91414c4361836d4ac31c1d5dbdea4
2022-08-22 14:37:58 +00:00
Márton Elek
0cc054e931 build: turn off postgres unit tests (build time test)
Change-Id: Iecb018791d9533c354375f9b7865c83ba32092f5
2022-08-22 13:18:59 +00:00