Commit Graph

6969 Commits

Author SHA1 Message Date
JT Olio
3b7d8e8798 cmd/tools/convert-node-id: tool for node id conversions
Change-Id: I3d18a051131135dfdbfd9c54002efbc853e4004d
2022-09-14 11:13:50 +03:00
Qweder93
04411e6871 satellite/accounting: use single call to get project limits
segment and usage limits now taken by single call

Closes https://github.com/storj/team-metainfo/issues/124

Change-Id: I55e05e199d2575cb8d985f4d75ba061002c97f0f
2022-09-13 21:50:52 +00:00
Fadila Khadar
601cadb92c satellite/metabase: copy when source and destination are the same
If source and destination are the same, do nothing and return the original object.
Later, we will have to handle the case when metadata is changed, but for now it's not possible through libuplink.

An issue has been created for this (https://github.com/storj/storj/issues/5168)

Change-Id: I91cf48afeec498d3b2c219fa0d3baf2163cff384
2022-09-13 19:21:13 +00:00
Márton Elek
26444194c2 uplink: fix distributed tracing reporter
This patch fix the beavior of the distributed tracing reporter.

1. For developer build we don't append the date
 * We don't need to separate service instances in jaeger (search by trace ID)
 * It's usually 0000-00-000 anyway as release.sh is not used for dev builds

2. Tracing ID MUST be unique
 * Instead of trusting the user to set a unique value (how can they do it?), we generate a random number

3. To make it possible to find the trace, there is a new flag to print out the generated tracing ID

4. Monkit `remoteTrace` call is replaced with normal monkit Task.
 * remoteTrace call assumes that we have a parent span in an other service (which is already sent to the server)
 * Here we must send out the parent span, as this is the beginning of the trace

5. We properly close the Jaeger UDP collector, and we wait until remaining messages are sent out

Change-Id: Iabf5abf25f4f20881188f88edcbadca95ac74927
2022-09-13 14:36:41 +00:00
Mya
6d017a1a92 satellite/oidc: include nodeURL in oidc configuration
By including NodeURL in the OIDC well-known configuration, we're
able to discover the NodeURL for a given HTTP address without
any client side assumptions or needing to make a drpc to discover
this. Instead, it's included in a call that is already made by an
OIDC/OAuth enabled client.

Change-Id: If00f31665ca69b1f522e26fec825b29ad03fe7f9
2022-09-13 13:46:30 +00:00
Vitalii
ad83fd893b web/satellite: fix add card container's close icon
Fixed X-icon positioning for add card container on new billing screen.

Issue:
https://github.com/storj/storj/issues/5120

Change-Id: I9c2a62a8ff9a268671d798d1568b07d54c6282a6
2022-09-12 20:01:17 +03:00
Egon Elbre
8459a3faa2 cmd/tools/segment-verify: add processing of a collection of segments
This adds logic to put segments into batches and retry them.

Change-Id: I86450e2f5ab13edb3ae9a5d5b9417e0cced2892b
2022-09-12 18:29:04 +03:00
Egon Elbre
3467fd4b7f cmd/tools/segment-verify: implement batch creation
Implements creating roughly load-balanced set of batched
that can be used to make multiple requests.

Change-Id: I349b276176dcb8ba9163e7e06a94509d73fa5ddc
2022-09-12 18:29:01 +03:00
Egon Elbre
546aa315fa satellite/metabase: add ListVerifySegments
We need a method for getting a list of segments from the metabase,
without converting the aliases and omitting all inline segments.

Change-Id: I26d919c675fc285ab03a35b327edd9b5c8bbe4b0
2022-09-12 17:32:29 +03:00
Michal Niewrzal
464ab43558 satellite/metainfo: fix deletion tests
Main issue with those tests was that for case where all objects were
uploaded at once (case "some nodes down" and "all nodes down").
Because all objects had the same name while upload each new object
was overwriting existing object. Because of that instead had several
objects to delete by test explicitly we had just 1. Test were not
failing because while overwriting existing object we were deleting it
but it was not what this test should do.

Change-Id: I602116f00be66589c7c0e68fe28c25e5c03e6b5d
2022-09-12 11:23:53 +00:00
Michal Niewrzal
158eb2381e satellite/gc/bloomfilter: uploading bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket.

This change add main logic to new service. After collecting all bloom
filters with segment loop and piece tracker all filters are marshaled
and packed into zip files. Each zip contains up to "ZipBatchSize" bloom
filters and it's uploaded to specified in configuration bucket.

All uploaded objects have specified expiration time to not delete them
manually.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I2b6bc02a7dd7c3a639e75810fd013ae4afdc80a2
2022-09-12 08:33:53 +00:00
Michal Niewrzal
eea3fac0d3 satellite/metabase: BeginObjectNextVersion returns object
We plan to replace metabase.BeginObjectExactVersion usage in
metainfo.BeginObject with metabase.BeginObjectNextVersion. To make this
switch as simple a possible would be nice to have the same results for
both methods. This change is extending return value for
BeginObjectNextVersion to whole object struct. Tests were also adjusted
to be more like metabase.BeginObjectExactVersion tests.

Part of https://github.com/storj/storj/issues/4871

Change-Id: I4db99d74af07e5a73757b55233e0bbdc7b99d565
2022-09-12 07:54:14 +00:00
NickolaiYurchenko
fc9bd515fd web/satellite: user settings page made responsive
existing adaptations fixed
added new adaptation media queries
modals related to setting page fixed on larger screens
cursor changed to pointer for passphrase type selection

Change-Id: Iee3dd141c2beea6951ff43b31506e709fb1d6b9d
2022-09-09 15:15:33 +00:00
Vitalii
c5bca894fd web/satellite: fix linter
Added imports linting, trailing commas, trailing semicolons, single quotes and spaces between curly braces.

Change-Id: I5de5d3eea48753dfe2737983b230bafaffe898c8
2022-09-09 11:02:04 +00:00
NickolaiYurchenko
e4412fa480 web/satellite: bucket creation flow made responsive
added media queries to fix view for tablet and mobile screens

Change-Id: Ia6dfdbc5e3a663a91313a902a4e81829fdc00c4c
2022-09-08 17:04:43 +03:00
Jeremy Wharton
92d81db183 satellite/console,web/satellite: limit new password length to 128
This change implements a requirement that all new passwords must be
no longer than 128 characters.

Change-Id: I7a8f23c87190d465eed59b3e0627ccd32d465bb9
2022-09-08 13:29:47 +00:00
Márton Elek
1646a0ac9e cmd/uplink: experimental environment variable to change piece hash algorithm
Change-Id: I7420919146c64d29ff1023f7498867d3d30892d0
2022-09-08 10:41:04 +00:00
Qweder93
98fb4c4f32 satellite/metainfo: added test for object TTL validation while upload
Closes https://github.com/storj/team-metainfo/issues/113

Change-Id: I6ff2a5f2cdab6fd42d94fbfc335084b927d86bea
2022-09-08 11:39:41 +03:00
Qweder93
2a1bcfed81 {satellite/metabase, satellite/metainfo, satellite/inspector} : Use metabase.GetObjectLastCommitted instead metabase.GetObjectExactVersion
adjusted metainfo, metabase and inspector methods to use GetObjectLastCommitted,
added test cases to confirm that this method will work correctly with version higher then 1

Closes https://github.com/storj/storj/issues/4869
Closes https://github.com/storj/storj/issues/4873

Change-Id: I8c338dcd1db82c141383f41339a295d54e2fa039
2022-09-08 07:27:22 +00:00
Yaroslav Vorobiov
ad7c5b1483 satellite/payments/monetary: remove pkg and all its references
Remove pkg satellite/payments/monetary as it moved to storj.io/common.
Update all code pkg references from monetary to common/currency.

Change-Id: If2519f4c80cf315a9299e6521a6b9bbc6c399156
2022-09-07 12:58:00 +00:00
Egon Elbre
731916ffa4 cmd/inspector: move command back from cmd/tools/
Some of the makefiles require it to be in cmd/inspector.
This was initially moved in 381c1e1257.

Change-Id: I850f3bfa1d66e66fbd1d9069e353984a27a7c58c
2022-09-07 09:35:12 +00:00
Jeremy Wharton
3342fbb918 web/satellite: make project dashboard/overview responsive
This change modifies the new project dashboard to accommodate
small screen sizes.

Resolves https://github.com/storj/storj/issues/5036

Change-Id: I27a8acd144a24f21f0fa43808640ba5bca6b49a8
2022-09-07 08:45:40 +00:00
NickolaiYurchenko
c619ff45d8 web/satellite: users page made responsive
v-table visual bug with name fixed
v-header search position fixed
v-search expands 100% on smaller screens

Change-Id: Iadd6ecea4fba3d39af5b4ea48705f2e199810080
2022-09-07 07:19:51 +00:00
prerna-parashar
54eaf79b5f
satellite/analytics: Re-added user click events in Access Management flow (#5151)
* Re-added user click events in Access Management flow
2022-09-06 09:55:33 -07:00
Egon Elbre
381c1e1257 cmd/tools: move tooling to a separate folder
This helps to cleanup the cmd folder a bit.

Change-Id: I24025c3dbfd35966325d7d5aaa95cd9a1176a8b7
2022-09-02 18:25:36 +03:00
Wilfred Asomani
b60c3ea0a1 web/satellite: make hiding bucket guide user friendly
This change makes actions to hide the bucket guide much clearer by using only one "OK" action button.

see: https://github.com/storj/storj/issues/5094

Change-Id: I3de8c1e21a5ec5dbcd91e69c1579fac7f46d39ca
2022-09-02 14:13:44 +00:00
Egon Elbre
794a4cc365 private/apigen: make code compile with 1.16
Our Test Versions still requires 1.16 to be compatible with our oldest
uplink versions. These changes make the code compile with 1.16.

Also, it makes go generate work in private/apigen/example.

Change-Id: Ib2f7493941a16f361328fe01d2be293f26123719
2022-09-02 16:34:46 +03:00
Egon Elbre
1343528a43 satellite/console/.../consoleapi/gen: ensure go generate works
Currently the paths were set relative to the root of the module,
however the code did not ensure that we are running relative to the
module directory.

Also, ensure typescript output corresponds to our styling.

Change-Id: I2b3cbd4ea8f2615e35c7b58c6fb8851669c47885
2022-09-02 16:34:46 +03:00
Clement Sam
64e5fb7772 storagenode/collector: fix error check when file does not exist
The collector calls the Delete() method on the pieces
which returns an error which is wrapped by many error classes.
Delete() method is using Stat() from
1aec831d98/storage/filestore/dir.go (L328)
under the hood.
os.IsNotExist(errors.Unwrap(err) will always be false unless
errors.Unwrap(err) is called multiple times till it gets to
the core os.ErrNotExist. Here is a test case to explain better:

    func TestABC(t *testing.T) {
	classA := errs.Class("A")
	classB := errs.Class("B")

	wrappedError := classB.Wrap(classA.Wrap(os.ErrNotExist))

	require.True(t, os.IsNotExist(errs.Unwrap(wrappedError)))
	require.True(t, os.IsNotExist(errors.Unwrap(wrappedError)))
    }

Using errs.Is() seems to resolve this even without unwrapping the error:

    func TestABC(t *testing.T) {
	classA := errs.Class("A")
	classB := errs.Class("B")

	wrappedError := classB.Wrap(classA.Wrap(os.ErrNotExist))

	require.True(t, errs.Is(wrappedError, os.ErrNotExist))
	require.False(t, errs.Is(wrappedError, os.ErrExist))
	require.False(t, os.IsNotExist(wrappedError))
    }

Does not resolve the collector issue here but enhances it:
https://github.com/storj/storj/issues/4192

Change-Id: Ifb75dd15b54c1e1a5e23f6eba2d621d64874a5cc
2022-09-02 12:26:33 +00:00
Michal Niewrzal
d905931ed9 private/testplanet: integrate GC bloom filter service
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket.
This change adds integration with testplanet which makes writing
unit tests possible.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I7b335c5dafa8cffe265c56b75d8c8f8567580893
2022-09-02 11:52:45 +00:00
Vitalii
ddc850dc21 web/satellite: added server-side encryption banner to buckets page
Added server-side encryption warning button to buckets page.
It can be dismissed and will show up again only on next user login.

Issue:
https://github.com/storj/storj/issues/5118

Change-Id: Iec5de91974ad5177a322b205f6b361c86f06923e
2022-09-02 09:36:18 +00:00
Moby von Briesen
61e18e8361 web/satellite: Update docs.storj.io anchor links
Reverts two of the changes in commit 119e61fcb because the timing of the
new docs deployment is changing.

Change-Id: Iaf804a41073d50700f67b153ef088631edefe6e5
2022-09-02 08:44:51 +00:00
Michal Niewrzal
68f6d93f29 satellite/gc/bloomfilter: add service to collect bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket. This this initial change to add such
service. Added service is joining segment loop and collects all
bloom filters.

Sending bloom filters to the bucket will be added as a subsequent
change.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I2551723605afa41bec84826b0c647cd1f61f3b14
2022-09-02 08:10:46 +00:00
Jeremy Wharton
1613d37466 satellite/console: add endpoints for listing and revoking api keys
This change adds the following endpoints:
- projects/apikeys/{id}: returns a paged list of API keys for the
	project specified by the given ID
- apikeys/delete/{id}: deletes the API key specified by the given ID

Additionally, the API Go code generator has been given the ability to
process unsigned integer parameters.

Change-Id: I5ff24e012da24a3f06bea1ebb62bae6ff62f951a
2022-09-01 14:24:00 -05:00
Malcolm Bouzi
8e3aa4d25b web/satellite: Creating a new folder on the bucket page is moved from an input field in the table list to a modal.
Change-Id: Ie15d4ec63b54c8be42f3c66e100e919b94adc345
2022-09-01 15:56:38 +00:00
Moby von Briesen
0ab0cef06c web/satellite: Upgrade to Vue 2.7
2.7 natively supports the Composition API, which makes transitioning to
Vue 3 easier to do step-by-step.

Converted ::v-deep selectors to :deep() to fix new build warnings.

Change-Id: I21b34e10f55eaf9dcdad87729f02f3b0d723b6b1
2022-09-01 10:41:59 -04:00
Vitalii
6f76b7e282 web/satellite: fix access grants used for sharing objects and buckets
Fix access grants used for linksharing.
Allow only List and Download.

Change-Id: I6123eb63af1ebefa0d771505632250c1c098a3b3
2022-09-01 16:30:22 +03:00
dlamarmorgan
335e11dacd satellitedb: add token balance to API endpoint
Add the users current wallet balance to the endpoints for claiming and listing storjscan wallets. Also prevent a user with a claimed wallet address from claiming a new wallet.

Change-Id: I0dbf1303699f924d05c8c52359038dc5ef6c42a1
2022-08-31 22:55:16 +00:00
Moby von Briesen
119e61fcb0 web/satellite: Update dashboard links to docs.storj.io
The anchor tag for these links is changing, so we need to update the UI
for them to work. Also remove trailing slashes from docs.storj.io links.

Change-Id: Ic4682dae76c44cf803b16ebfe7352405520c4307
2022-08-31 15:42:12 -04:00
Cameron
7933e0c4c7 cmd: add command to migrate project public_ids
This is to update all projects to have a public_id if they do not have
one.

github issue: https://github.com/storj/storj/issues/4861

Change-Id: Icfa42b62e15ca75d3c04a0aab48a3c3b0f3a9d6e
2022-08-31 13:14:17 +00:00
Qweder93
df5b0d0044 satellite/metabase: added method DeleteObjectLastCommitted
method similar to metabase.DeleteObjectExactVersion which will delete last committed object

Closes 	https://github.com/storj/storj/issues/4872

Change-Id: Ia9f8c227dc59575bf8ed297886b35536097028b4
2022-08-31 12:26:56 +00:00
Michal Niewrzal
2ae1db53b6 cmd/satellite: add command for collecting GC bloom filters
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket. This change is extending satellite binary
with appropriete command.

New GC service for collecting bloom filter will be a subsequent
change.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: Ibc03e119c340919cf468fc1f5a4f3d187bb3a5a1
2022-08-31 11:40:36 +00:00
Márton Elek
ea1408f7a8 go.mod: bump clingy dependency
As a reminder: latest clingy removed the requirement of having custom context (which made the usage of context.WithValue harder) and uses simple context instead.

Clingy saves the stdin/stdout/stderr to the context (earlier to separated context type) to make it available for unit testing.

Change-Id: I8896574f4670721de43a577cd4b35952e3b5d00e
2022-08-31 10:24:27 +00:00
dlamarmorgan
47301e5718 billing: add USDollarsMicro to billing DB
Adds USDollarsMicro currency to the billing DB which support fraction of a cent with decimal places for better billing amounts accuracy.

Change-Id: Id07dfae104d94e27c7b22ab8f5781010e16c4c8e
2022-08-30 14:48:51 -07:00
dlamarmorgan
afe58323f9 satellitedb: remove use of batch payment insert
Removed batch insert of payments since they do not guarantee order. Order of payments sent to the payments DB is important, because the billing chore will request new payments based on the last received payment. If the last payment inserted is not the last payment received, duplicate payments will be inserted into the billing table.

Change-Id: Ic3335c89fa8031f7bc16f417ca23ed83301ef8f6
2022-08-30 14:45:55 -07:00
Moby von Briesen
8830d1eb7f web/satellite: Styling updates to registration page
Improve styling for lists in `customHtmlDescription` for alternate
signup landing pages in `registrationViewConfig`.

Change-Id: If20831044099a1183e5dc8b41668bf2f68c272a2
2022-08-30 20:33:22 +00:00
Michal Niewrzal
00c594362c satellite: skeleton for new GC peer
We would like to have separate process/command to collect bloom
filters from source different than production DBs. Such process will
use segment loop to build bloom filters for all storage nodes and
will send it to Storj bucket. This change is just initial code with peer
which will be used to build this new process.

Updates https://github.com/storj/team-metainfo/issues/120

Change-Id: I10a52b74865ce8ec4c29b7c6a2836f9232620422
2022-08-30 15:41:49 +00:00
Erik van Velzen
4393b8a5ce satellite/internalpb: new protobuf for retain filters
Add new protobuff message to store retain filters before sending them
out to storagenodes.

This allows the retain filters to be generated by a separate command on
a backup of the database.

Change-Id: I8a2892d9d34bed1dbd6e5a7ded000dcdea7ec061
2022-08-30 12:37:40 +02:00
Michal Niewrzal
fa76dfe577 satellite/metabase: use GetObjectLastCommitted with move/copy
We are preparing to use object versions internally and to do
that we need to prepare different parts of the system to handle
object versions different than '1'. This change adjust code
responsible for server-side move and copy.

What was done:
* begin methods for move and copy are now using GetObjectLastCommitted
to find object
* results from begin move and copy operation contains now version to
be able to map object correctly with finish operation
* begin methods are putting version into satellite stream id and
finish methods are using this version as parameter instead hardcoded
value

Fixes https://github.com/storj/storj/issues/4867

Change-Id: I1380911279c21e10a3fff0342793efd2e73eafad
2022-08-30 06:36:02 +00:00
Vitalii
d585f2550d web/satellite: share bucket feature
Implemented share bucket feature.
Refactored share object modal a bit (has to be refactored entirely).

Issue:
https://github.com/storj/storj/issues/4945

Change-Id: Icefd4bfe3eef9173ae824eea44d30450acde8044
2022-08-29 18:41:06 +03:00