Commit Graph

6899 Commits

Author SHA1 Message Date
Egon Elbre
731916ffa4 cmd/inspector: move command back from cmd/tools/
Some of the makefiles require it to be in cmd/inspector.
This was initially moved in 381c1e1257.

Change-Id: I850f3bfa1d66e66fbd1d9069e353984a27a7c58c
2022-09-07 09:35:12 +00:00
Jeremy Wharton
3342fbb918 web/satellite: make project dashboard/overview responsive
This change modifies the new project dashboard to accommodate
small screen sizes.

Resolves https://github.com/storj/storj/issues/5036

Change-Id: I27a8acd144a24f21f0fa43808640ba5bca6b49a8
2022-09-07 08:45:40 +00:00
NickolaiYurchenko
c619ff45d8 web/satellite: users page made responsive
v-table visual bug with name fixed
v-header search position fixed
v-search expands 100% on smaller screens

Change-Id: Iadd6ecea4fba3d39af5b4ea48705f2e199810080
2022-09-07 07:19:51 +00:00
prerna-parashar
54eaf79b5f
satellite/analytics: Re-added user click events in Access Management flow (#5151)
* Re-added user click events in Access Management flow
2022-09-06 09:55:33 -07:00
Egon Elbre
381c1e1257 cmd/tools: move tooling to a separate folder
This helps to cleanup the cmd folder a bit.

Change-Id: I24025c3dbfd35966325d7d5aaa95cd9a1176a8b7
2022-09-02 18:25:36 +03:00
Wilfred Asomani
b60c3ea0a1 web/satellite: make hiding bucket guide user friendly
This change makes actions to hide the bucket guide much clearer by using only one "OK" action button.

see: https://github.com/storj/storj/issues/5094

Change-Id: I3de8c1e21a5ec5dbcd91e69c1579fac7f46d39ca
2022-09-02 14:13:44 +00:00
Egon Elbre
794a4cc365 private/apigen: make code compile with 1.16
Our Test Versions still requires 1.16 to be compatible with our oldest
uplink versions. These changes make the code compile with 1.16.

Also, it makes go generate work in private/apigen/example.

Change-Id: Ib2f7493941a16f361328fe01d2be293f26123719
2022-09-02 16:34:46 +03:00
Egon Elbre
1343528a43 satellite/console/.../consoleapi/gen: ensure go generate works
Currently the paths were set relative to the root of the module,
however the code did not ensure that we are running relative to the
module directory.

Also, ensure typescript output corresponds to our styling.

Change-Id: I2b3cbd4ea8f2615e35c7b58c6fb8851669c47885
2022-09-02 16:34:46 +03:00
Clement Sam
64e5fb7772 storagenode/collector: fix error check when file does not exist
The collector calls the Delete() method on the pieces
which returns an error which is wrapped by many error classes.
Delete() method is using Stat() from
1aec831d98/storage/filestore/dir.go (L328)
under the hood.
os.IsNotExist(errors.Unwrap(err) will always be false unless
errors.Unwrap(err) is called multiple times till it gets to
the core os.ErrNotExist. Here is a test case to explain better:

    func TestABC(t *testing.T) {
	classA := errs.Class("A")
	classB := errs.Class("B")

	wrappedError := classB.Wrap(classA.Wrap(os.ErrNotExist))

	require.True(t, os.IsNotExist(errs.Unwrap(wrappedError)))
	require.True(t, os.IsNotExist(errors.Unwrap(wrappedError)))
    }

Using errs.Is() seems to resolve this even without unwrapping the error:

    func TestABC(t *testing.T) {
	classA := errs.Class("A")
	classB := errs.Class("B")

	wrappedError := classB.Wrap(classA.Wrap(os.ErrNotExist))

	require.True(t, errs.Is(wrappedError, os.ErrNotExist))
	require.False(t, errs.Is(wrappedError, os.ErrExist))
	require.False(t, os.IsNotExist(wrappedError))
    }

Does not resolve the collector issue here but enhances it:
https://github.com/storj/storj/issues/4192

Change-Id: Ifb75dd15b54c1e1a5e23f6eba2d621d64874a5cc
2022-09-02 12:26:33 +00:00
Michal Niewrzal
d905931ed9 private/testplanet: integrate GC bloom filter service
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket.
This change adds integration with testplanet which makes writing
unit tests possible.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I7b335c5dafa8cffe265c56b75d8c8f8567580893
2022-09-02 11:52:45 +00:00
Vitalii
ddc850dc21 web/satellite: added server-side encryption banner to buckets page
Added server-side encryption warning button to buckets page.
It can be dismissed and will show up again only on next user login.

Issue:
https://github.com/storj/storj/issues/5118

Change-Id: Iec5de91974ad5177a322b205f6b361c86f06923e
2022-09-02 09:36:18 +00:00
Moby von Briesen
61e18e8361 web/satellite: Update docs.storj.io anchor links
Reverts two of the changes in commit 119e61fcb because the timing of the
new docs deployment is changing.

Change-Id: Iaf804a41073d50700f67b153ef088631edefe6e5
2022-09-02 08:44:51 +00:00
Michal Niewrzal
68f6d93f29 satellite/gc/bloomfilter: add service to collect bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket. This this initial change to add such
service. Added service is joining segment loop and collects all
bloom filters.

Sending bloom filters to the bucket will be added as a subsequent
change.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I2551723605afa41bec84826b0c647cd1f61f3b14
2022-09-02 08:10:46 +00:00
Jeremy Wharton
1613d37466 satellite/console: add endpoints for listing and revoking api keys
This change adds the following endpoints:
- projects/apikeys/{id}: returns a paged list of API keys for the
	project specified by the given ID
- apikeys/delete/{id}: deletes the API key specified by the given ID

Additionally, the API Go code generator has been given the ability to
process unsigned integer parameters.

Change-Id: I5ff24e012da24a3f06bea1ebb62bae6ff62f951a
2022-09-01 14:24:00 -05:00
Malcolm Bouzi
8e3aa4d25b web/satellite: Creating a new folder on the bucket page is moved from an input field in the table list to a modal.
Change-Id: Ie15d4ec63b54c8be42f3c66e100e919b94adc345
2022-09-01 15:56:38 +00:00
Moby von Briesen
0ab0cef06c web/satellite: Upgrade to Vue 2.7
2.7 natively supports the Composition API, which makes transitioning to
Vue 3 easier to do step-by-step.

Converted ::v-deep selectors to :deep() to fix new build warnings.

Change-Id: I21b34e10f55eaf9dcdad87729f02f3b0d723b6b1
2022-09-01 10:41:59 -04:00
Vitalii
6f76b7e282 web/satellite: fix access grants used for sharing objects and buckets
Fix access grants used for linksharing.
Allow only List and Download.

Change-Id: I6123eb63af1ebefa0d771505632250c1c098a3b3
2022-09-01 16:30:22 +03:00
dlamarmorgan
335e11dacd satellitedb: add token balance to API endpoint
Add the users current wallet balance to the endpoints for claiming and listing storjscan wallets. Also prevent a user with a claimed wallet address from claiming a new wallet.

Change-Id: I0dbf1303699f924d05c8c52359038dc5ef6c42a1
2022-08-31 22:55:16 +00:00
Moby von Briesen
119e61fcb0 web/satellite: Update dashboard links to docs.storj.io
The anchor tag for these links is changing, so we need to update the UI
for them to work. Also remove trailing slashes from docs.storj.io links.

Change-Id: Ic4682dae76c44cf803b16ebfe7352405520c4307
2022-08-31 15:42:12 -04:00
Cameron
7933e0c4c7 cmd: add command to migrate project public_ids
This is to update all projects to have a public_id if they do not have
one.

github issue: https://github.com/storj/storj/issues/4861

Change-Id: Icfa42b62e15ca75d3c04a0aab48a3c3b0f3a9d6e
2022-08-31 13:14:17 +00:00
Qweder93
df5b0d0044 satellite/metabase: added method DeleteObjectLastCommitted
method similar to metabase.DeleteObjectExactVersion which will delete last committed object

Closes 	https://github.com/storj/storj/issues/4872

Change-Id: Ia9f8c227dc59575bf8ed297886b35536097028b4
2022-08-31 12:26:56 +00:00
Michal Niewrzal
2ae1db53b6 cmd/satellite: add command for collecting GC bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket. This change is extending satellite binary
with appropriete command.

New GC service for collecting bloom filter will be a subsequent
change.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: Ibc03e119c340919cf468fc1f5a4f3d187bb3a5a1
2022-08-31 11:40:36 +00:00
Márton Elek
ea1408f7a8 go.mod: bump clingy dependency
As a reminder: latest clingy removed the requirement of having custom context (which made the usage of context.WithValue harder) and uses simple context instead.

Clingy saves the stdin/stdout/stderr to the context (earlier to separated context type) to make it available for unit testing.

Change-Id: I8896574f4670721de43a577cd4b35952e3b5d00e
2022-08-31 10:24:27 +00:00
dlamarmorgan
47301e5718 billing: add USDollarsMicro to billing DB
Adds USDollarsMicro currency to the billing DB which support fraction of a cent with decimal places for better billing amounts accuracy.

Change-Id: Id07dfae104d94e27c7b22ab8f5781010e16c4c8e
2022-08-30 14:48:51 -07:00
dlamarmorgan
afe58323f9 satellitedb: remove use of batch payment insert
Removed batch insert of payments since they do not guarantee order. Order of payments sent to the payments DB is important, because the billing chore will request new payments based on the last received payment. If the last payment inserted is not the last payment received, duplicate payments will be inserted into the billing table.

Change-Id: Ic3335c89fa8031f7bc16f417ca23ed83301ef8f6
2022-08-30 14:45:55 -07:00
Moby von Briesen
8830d1eb7f web/satellite: Styling updates to registration page
Improve styling for lists in `customHtmlDescription` for alternate
signup landing pages in `registrationViewConfig`.

Change-Id: If20831044099a1183e5dc8b41668bf2f68c272a2
2022-08-30 20:33:22 +00:00
Michal Niewrzal
00c594362c satellite: skeleton for new GC peer
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket. This change is just initial code with peer
which will be used to build this new process.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I10a52b74865ce8ec4c29b7c6a2836f9232620422
2022-08-30 15:41:49 +00:00
Erik van Velzen
4393b8a5ce satellite/internalpb: new protobuf for retain filters
Add new protobuff message to store retain filters before sending them
out to storagenodes.

This allows the retain filters to be generated by a separate command on
a backup of the database.

Change-Id: I8a2892d9d34bed1dbd6e5a7ded000dcdea7ec061
2022-08-30 12:37:40 +02:00
Michal Niewrzal
fa76dfe577 satellite/metabase: use GetObjectLastCommitted with move/copy
We are preparing to use object versions internally and to do
that we need to prepare different parts of the system to handle
object versions different than '1'. This change adjust code
responsible for server-side move and copy.

What was done:
* begin methods for move and copy are now using GetObjectLastCommitted
to find object
* results from begin move and copy operation contains now version to
be able to map object correctly with finish operation
* begin methods are putting version into satellite stream id and
finish methods are using this version as parameter instead hardcoded
value

Fixes https://github.com/storj/storj/issues/4867

Change-Id: I1380911279c21e10a3fff0342793efd2e73eafad
2022-08-30 06:36:02 +00:00
Vitalii
d585f2550d web/satellite: share bucket feature
Implemented share bucket feature.
Refactored share object modal a bit (has to be refactored entirely).

Issue:
https://github.com/storj/storj/issues/4945

Change-Id: Icefd4bfe3eef9173ae824eea44d30450acde8044
2022-08-29 18:41:06 +03:00
paul cannon
a57cbc4e0b satellite/satellitedb: drop _gob columns
This concludes the saga that began with commit c053bdbd, migrating away
from using gob-encoded big.Float values in the database to using
integers with an implied number of decimal places.

Nothing is using or accessing or expecting the _gob columns to exist
anymore. The transition code and the migration chore are gone. All that
remains is to drop the columns.

Change-Id: I9b15ee52f7781510a6dc91cf7c54f7f9022b1210
2022-08-26 19:45:22 +00:00
JT Olio
e621190b99 satellitedb: randomize nodes for restore from trash
Change-Id: I030bd9093ec13a882edc7e0d325b9b19c201e135
2022-08-26 17:19:41 +00:00
Lizzy Thomson
8f1c0925dc web/satellite: update responsiveness in billing pages
update responsiveness in billing pages to match figma designs including added billing header scrolling for smaller windows

Fixes https://github.com/storj/storj/issues/5033

Change-Id: I94fc836f95e3985d6b675d3495961ec703e43044
2022-08-26 16:50:05 +00:00
Wilfred Asomani
72086e6dea console/satellite: make file browser responsive
see: https://github.com/storj/storj/issues/5037

Change-Id: Icb62ed7c2cdb5db7c64659e70952cae5fa90ae70
2022-08-26 16:19:01 +00:00
Michal Niewrzal
2163d60037 satellite/metabase/segmentloop: add missing monitoring
Currently we are tracking loop time with RunOnce method
metrics but it may give inaccurate values if loop if waiting
for observer for a ver long time. We had such case with GC
which is only loop observer in GC peer and its joning loop
every 5 days.

Change-Id: I08546c912e00c3641488de6a5d75948fe75c8e99
2022-08-26 14:01:22 +02:00
Márton Elek
7e71986493 storagenode: accept HTTP calls on public port, listening for monitoring requests
Today each storagenode should have a port which is opened for the internet, and handles DRPC protocol calls.

When we do a HTTP call on the DRPC endpoint, it hangs until a timeout.

This patch changes the behavior: the main DRPC port of the storagenodes can accept HTTP requests and can be used to monitor the status of the node:

 * if returns with HTTP 200 only if the storagnode is healthy (not suspended / disqualified + online score > 0.9)
 * it CAN include information about the current status (per satellite). It's opt-in, you should configure it so.

In this way it becomes extremely easy to monitor storagenodes with external uptime services.

Note: this patch exposes some information which was not easily available before (especially the node status, and used satellites). I think it should be acceptable:

 * Until having more community satellites, all storagenodes are connected to the main Storj satellites.
 * With community satellites, it's good thing to have more transparency (easy way to check who is connected to which satellites)

The implementation is based on this line:

```
http.Serve(NewPrefixedListener([]byte("GET / HT"), publicMux.Route("GET / HT")), p.public.http)
```

This line answers to the TCP requests with `GET / HT...` (GET HTTP request to the route), but puts back the removed prefix.

Change-Id: I3700c7e24524850825ecdf75a4bcc3b4afcb3a74
2022-08-26 09:38:09 +00:00
paul cannon
07bbe7d340 satellite/overlay: don't insert new nodes if contact check failed
Currently, the satellite tracks connectivity information about all nodes
that have contacted it, even if we have never successfully contacted
the node back.

This behavior was leveraged during a security audit to create hundreds
of thousands of "junk nodes" in the nodes table on one satellite, which
affected performance of queries such as node selection.

With this change, we should no longer track information about nodes that
have never been successfully contacted.

Note that it will still be possible to cause the creation of "junk
node" entries in the db; the attacker just has to set up individual
publicly-routable IP+port pairs for each node as it is created, so it
can respond to a PingBack.

Change-Id: Ibb6da6cc908fd4fc85aae1ba00313ba2738409ab
2022-08-26 09:05:04 +00:00
Sean Harvey
5a5f609022
satellite/metainfo: add copy and move log and metrics
Change-Id: I579c03c389dea7f8e7e92ade5bcac53740ce69c8
2022-08-26 18:55:29 +12:00
JT Olio
12ef68bdf4 satellitedb: restore-trash shouldn't bother with nodes we've never talked to
Change-Id: I3c631e92a9a2670c52c9fa23b2b1baf67d3c9a9c
2022-08-25 16:14:29 +00:00
Wilfred Asomani
0c4e45875f web/satellite: change segment learn-more link
This change uses the correct docs link for segments.

see: https://github.com/storj/storj/issues/5098

Change-Id: I49c0a171fa29064a495f6aa50624c71d9bfbc703
2022-08-25 15:27:57 +00:00
Wilfred Asomani
0bfdc9da30 web/satellite: fix billing duplicate navigation error
This change fixes the duplicate navigation error on the new billing screen.

see: https://github.com/storj/storj/issues/5096

Change-Id: Ie0512de806835acfc29e86182e4c49ced6355383
2022-08-25 14:46:29 +00:00
Vitalii
3019ad3c35 web/satellite: apply passphrase to newly created access grants
Apply passphrase when creating access grants or S3 credentials.

Change-Id: I59fc391ffdca59e8cd6bdce5632b565e9e9237ea
2022-08-25 15:43:56 +03:00
Clement Sam
ea0124a183 cmd/multinode: handle JSON input with BOM to add command
Decode the JSON input string to its corresponding unicode
decoding if the special byte order mark is present.

Fixes https://github.com/storj/storj/issues/4950

Change-Id: If91bac22590c89b35c58bf54f6d3bdb8a67d7a4f
2022-08-25 03:05:49 +00:00
Michal Niewrzal
d6054214c3 satellite/metainfo: move metadata validation to a proper place
Metadata validation for CommitObject request was placed in a wrong
place. There is a case (old uplink) where encrypted key is bundled
inside encrypted metadata bytes and we need to extract it before we
can validate it. This change moves metadata validation to a place where
we are sure we have encrypted metadata and encrypted metadata encrypted
key ready to be checked.

"Run Versions Test" is covering this case and it was failing without
this change.

Change-Id: Ib709ad901fbb3fa4865a393195b7b3f4c0d87e7a
2022-08-24 10:50:43 +00:00
Michał Niewrzał
ea38860aab satellite/metainfo: align satellite StreamID protobuf version with metabase
For object Version in different places we are using different types.
Satellite StreamID is using int32 but metabase accepts int64. Metabase
type is correct one and we should align other places with it.

As a small addition this change is also passing version correctly
between requests instead of using hardcoded value.

Change-Id: I63634d73c0a48c009e4db5f203ff18b7f3218b02
2022-08-24 10:17:34 +00:00
Qweder93
2139d3966b satellite/metabase: update UpdateObjectMetadata to work with latest commited object
Updated metabase.UpdateObjectMetadata method to update set metdata always for last committed object

Closes https://github.com/storj/storj/issues/4870

Change-Id: I060683e31efcaf3e2531fea143cf0567e5ff5f73
2022-08-24 10:28:57 +03:00
Márton Elek
3d79c5f3a1 storagenode/docker: allow to configure updater with separated env variables
We couldn't use environment variables safely to configure storagenode, since we introduced the embedded updater.

For example STORJ_DEBUG_ADDR=localhost:11111 would try to set debug port 11111 for both the storagenode and storagenode-updated, causing port conflict.

This small change enables to configure storagenode-updater with STORJUPDATER_... environment variables.

Tested with creating custom image and installing to my own storage node.

Change-Id: I6b0a601a4dc63d2d1ff3c191ae89981434e55c30
2022-08-24 06:50:30 +00:00
Márton Elek
4b1be6bf8e storagenode/satellite: support different piece hash algorithms
Change-Id: I3db321e79f12f3ebaa249e6c32fa37fd9615687e
2022-08-23 18:15:06 +00:00
Márton Elek
d23cfa6445 go.mod: bump storj.io/common and storj.io/uplink
Change-Id: Ie78b69f2ee855e9e45ef241d4b8f8c225502b8a7
2022-08-23 17:47:08 +00:00
Jeremy Wharton
3f26cc599f satellite/console,web/satellite: invalidate sessions after inactivity
Sessions now expire after a much shorter amount of time, requiring
clients to issue API requests for session extension. This is handled
behind the scenes as the user interacts with the page, but once session
expiration is imminent, a modal appears which informs the user of his
inactivity and presents him with the choice of loging out or preserving
his session.

Change-Id: I68008d45859c814a835d65d882ad5ad2199d618e
2022-08-23 15:51:05 +00:00