Commit Graph

6463 Commits

Author SHA1 Message Date
Erik van Velzen
53e851bfb2 satellite/metabase/delete: adjust delete for copy feature
Adjust deletion of committed exact version so that it takes into account
copied objects.

Change-Id: I5daaa94ff8ad87417d52c79f305bc3275b683efa
2022-03-04 13:50:58 +00:00
Michał Niewrzał
39ac90835f satellite/satellitedb: change default segment limit to 100M for paying users
We decided that we want to have segment limit for paying users high
enough to not have to change it too often.

Fixes https://github.com/storj/storj/issues/4590

Change-Id: Ic1c38bf3e2fcc000548ff4c7e7004647b39fbecf
2022-03-04 13:18:14 +00:00
Igor
4af0bc08bd
Maintainers: update release instructions (#4456)
Co-authored-by: littleskunk <jens.heimbuerge@googlemail.com>
2022-03-04 14:16:36 +01:00
Moby von Briesen
ee5f897339 satellite/analytics: Add client-side analytics events
There are two events in
web/satellite/src/utils/constants/analyticsEventNames.ts which did not
have corresponding entries in the backend analytics service.

Change-Id: If0f67cef2ed312953e580d855d63366e7c12786a
2022-03-03 17:55:35 +00:00
Jeremy Wharton
66e6a75e2a satellite/console,web/satellite: Add MFA to password reset
Users will be required to enter a MFA passcode or recovery code
upon attempting a password reset for an account with MFA enabled.

Change-Id: I08d07597035d5a25849dbc70f7fd686753530610
2022-03-03 17:27:04 +00:00
Moby von Briesen
b2d342aa9b satellite/overlay: Add ability to exclude country codes on upload
Create global config to specify a list of country codes that should be
excluded from node selection during uploads.

This exclusion is not implemented when the upload selection cache is
disabled.

Change-Id: Ic41e8b4f18857a11045668eac23107da99668a72
2022-03-03 16:58:48 +00:00
Erik van Velzen
b4e42ceb23 cmd/uplinkng: fix linkshare bugs
Fix various bugs by remove superflous steps and letting libuplink
build the url.

$ uplinkng share --not-after +1h --url sj://mybucket/myprefix/

before: https://link.us1.storjshare.io/s/jxcvcme2xkb44xsec235xb2ccmbq//myprefix//
after: https://link.us1.storjshare.io/s/jxcvcme2xkb44xsec235xb2ccmbq/mybucket/myprefix/

$ uplinkng share --not-after +1h --url $(printf "sj://waterbear/aa\x03bb")

before: https://link.us1.storjshare.io/s/jwbcxjiv4ept7t2g3qmxfb73ljga//aabb
after: https://link.us1.storjshare.io/s/jvod7pixige62yrp5tubn4ct47yq/waterbear/aa%03bb

$ uplinkng share --not-after +1h --url

before: no url generated, no error
after: error "Need at least a bucket to create a working linkshare URL"
Change-Id: Ibca6847f8fc5d5fad9ee4642d11d3081d75d77ca
2022-03-03 16:00:10 +00:00
NickolaiYurchenko
64176aaca4 web/satellite: registration success redirect from config
This change allows us to send newly registered users to a configured URL
to help us track user conversions for marketing campaigns.
Brave conversions continue to be tracked using the /signup-success page
within the satellite app.

Change-Id: I9b451947ce0f39d3c99b233cb4b806d361151823
2022-03-03 10:13:21 -05:00
Erik van Velzen
56d1b8bbca satellite/metabasetest: improve test copy creation
Return the newly created segments separately so we can make better
assertion to test deletion.

Change-Id: I33414df4455f87db62d486957425d4984b66c0b2
2022-03-03 13:36:52 +01:00
Vitalii Shpital
9b5904cd49 satellite/{projectaccounting, console}:query to get single bucket rollup
Added new projectaccounting query to get project's single bucket usage rollup.
Added new service method to call new query.
Added implementation for IsAuthenticated method which is used by new generated API.

Change-Id: I7cde5656d489953b6c7d109f236362eb465fa64a
2022-03-03 12:04:29 +00:00
Sean Harvey
d4a6524673 Makefile: use arm64 executable for arm64v8 storagenode images
This change fixes regression (one line of 7e63afb) that overwrote the
previous fix (10d7a63).

Change-Id: I892b0c0445ac0fbe6eb30c131edc96fd1d33734d
2022-03-02 14:30:29 +01:00
Fadila Khadar
e776c65172 satellite/checker: pieces in excluded countries are not healthy
Add a RepairExcludedCountryCodes config flag for overlay for providing a list of country codes to exclude nodes from target repair selection.

Mark segments with less than repairThreshold pieces in countries not in the RepairExcludedCountryCodes as not healthy.
With this change, the repair process is not affected. The segment will be removed from the repair queue by the repairer.

Another change will handle the logic at the repairer level.

Fixes https://github.com/storj/team-metainfo/issues/95

Change-Id: I9231b32de117a116488de055a3e94efcabb46e81
2022-03-02 09:59:09 +00:00
Erik van Velzen
a9bd983f04 sql: capitalize keywords
Capitalize some keywords which were overlooked

Change-Id: Ie2ad283669e2ca2650fcddfd8c7395a81bac09a8
2022-03-01 15:19:38 +00:00
Vitalii Shpital
ba6956db0f console/server, apigen: feature flag for new generated console api
Added a feture flag which will be used to indicate if new generated console api is used.
Fixed some comments from previous PR.

Change-Id: Ice31c998b0b347028a491c971a648fd1269bfd49
2022-02-28 23:00:12 +00:00
Michał Niewrzał
1da9697995 satellite/metainfo: use feature flag for begin/finish object copy metods
We want also to cover with feature flag code for begin/finish
object copy methods.

Change-Id: I97330e7d9b310de98408c47c2c1ed41551631bec
2022-02-28 15:09:40 +00:00
Erik van Velzen
7a01a2a134 satellite/metabase/metabasetest/create: return segments
Return segments when creating a test object so that it can be checked
that the correct segments are remaining after a delete action.

Change-Id: Ifc245948935ba278806e887672c03abc5f2c2654
2022-02-28 13:40:46 +00:00
Ivan Fraixedes
8caa4c4557 satellite/console: Don't lose ErrValiation error class
There was a defined type (`validationErrors`) for gathering several
validation errors and classify them with the `ErrValdiation errs.Class`.

`errs.Combine` doesn't maintain the classes of the errors to combine,
for example

```
var myClass errs.Class = "My error class"

err1 := myClass.Wrap(erros.New("error 1"))
err2 := myClass.Wrap(erros.New("error 2"))
err3 := errors.New("error 3")

combinedErr := errs.Combine(err1, err2, err3)
myClass.Has(combinedErr) // It returns false

// Even only passing errors with a class and with the same one for all
// of them
combinedErr := errs.Combine(err1, err2)
myClass.Has(combinedErr) // It returns false
```

Hence `validationErrors` didn't return what we expected to return when
calling its `Combine` method.

This commit delete the type and it replaces by `errs.Group` when there
are more than one error, and wrapping the `errs.Group.Err` returned
error with `ErrValiation` error class.

The bug caused the HTTP API server to return a 500 status code as you
can seee in the following log message extracted from the satellite
production logs:

```
code: 500
error: "console service: validation: full name can not be empty; validation: Your password needs at least 6 characters long; validation: mail: no address"
errorVerbose: "console service: validation: full name can not be empty; validation: Your password needs at least 6 characters long; validation: mail: no address
        storj.io/storj/satellite/console.(*Service).CreateUser:593
        storj.io/storj/satellite/console/consoleweb/consoleapi.(*Auth).Register:250
        net/http.HandlerFunc.ServeHTTP:2047
        storj.io/storj/private/web.(*RateLimiter).Limit.func1:90
        net/http.HandlerFunc.ServeHTTP:2047
        github.com/gorilla/mux.(*Router).ServeHTTP:210
        storj.io/storj/satellite/console/consoleweb.(*Server).withRequest.func1:464
        net/http.HandlerFunc.ServeHTTP:2047
        net/http.serverHandler.ServeHTTP:2879
        net/http.(*conn).serve:1930"
message: "There was an error processing your request"
```

The issues was that not being classified with `ErrValidation` class it
was not picked by the correct switch branch of the
`consoleapi.Auth.getStatusCode` method which is in the call chain to
`consoleapi.Auth.Register` method when it calls
`console.Service.CreateUser` and returns an error.

These changes should return the appropriated HTTP status code (Bad
Request) when `console.Service.CreateUser` returns a validation error.

Returning the appropriated HTTP statsus code also makes not to show this
as an error in the server logs because the Bad Request sttatus code gets
logged with debug level.

Change-Id: I869ea85788992ae0865c373860fbf93a40d2d387
2022-02-28 11:00:56 +00:00
Michał Niewrzał
c0297bae78 satellite/metabase: return object metadata with FinishCopyObject
Updates metadata and metainfo to return object metadata with
FinishCopyObject request.

https://github.com/storj/storj/issues/4474

Change-Id: I32cba5c20a943272e9b5964df1b3d6463ad212dc
2022-02-25 12:46:30 +00:00
Fadila Khadar
332e6afed8 satellite/metabasetest: order RawSegments by streamID and position
Change-Id: I669c0c2361c2c7989c8e946eb1fbf2f52074ce0e
2022-02-25 11:19:11 +00:00
Fadila Khadar
2d4760fd09 satellite/metainfo: BeginCopyObject and FinishCopyObject
Metainfo endpoints to use server-side copy.

Fixes https://github.com/storj/storj/issues/4475

Change-Id: Ided06aed9e6187d6d8f030e95dda019ba78fff95
2022-02-24 14:38:04 +01:00
Michał Niewrzał
fbe2680500 satellite/metainfo: add feature flag for server side copy code
We would like to disable in production those parts of code
which are now mixed with new server-side copy logic.

Change-Id: Iff50682bc9545207330f58dd19b5eee53d404d7f
2022-02-24 10:43:49 +00:00
Moby von Briesen
d12bb19972 satellite/console/consoleweb: Update CSP directives
Update the Content Security Policy to whitelist `blob:` for the img-src
and media-src directives. This is necessary to prevent CSP errors in the
object browser while loading previews and object maps.

Change-Id: Ic32bf0954f300c77ec4f0fe11fae63f0c7b622da
2022-02-23 17:30:35 -05:00
Jeff Wendling
ccb847a329 scripts/tests: write old uplink config yaml directly
sometimes these scripts want to have an access imported
after it has been potentially modified by having the
satellite address and node id added. it used to use the
uplink command to do this, but the cli api for that
has changed. rather than try to have the script detect
which uplink version is in use and call the right thing
it can always write out a valid yaml file and depend on
the new cli migrating it.

Change-Id: Ib82819699333f5f29e00117b99bfb10640033b94
2022-02-22 15:26:06 -05:00
Jeff Wendling
863b01bf09 testversions: skip some downloads for older uplinks
uplink command versions >= 1.48.0 always do multipart
uploads which cannot be downloaded by any of the
versions in the test < v1.27.6. so skip those tests.

Change-Id: I9644afbd14bfce9facfd87644d132f7d66367d62
2022-02-22 17:24:07 +00:00
Erik van Velzen
ac5f97a364 satellite/sattelitedb/test: improve test database name
Remove special characters from the test database name to improve
compatibility with external tools like the Cockroach web gui.

Change-Id: I3a6a368a5051e3064e7279b14eec4f2d4ff3c435
2022-02-22 14:45:01 +00:00
Sembeth
d015805d15
cmd/multinode: Add further documentation (#4556)
The text has been expanded a bit to clarify that it is necessary to create identity files with an example before using the Docker image.
Changed the <identity-dir> placeholder to <multinode-identity-dir> so no one confuses them with the storagenode identity files.
Changed the <storage-dir> placeholder to <multinode-config-dir> so no one confuses them with the storagenode 'config' folder.

fixed #4547
2022-02-22 14:42:54 +01:00
Fadila Khadar
b11d144bb6 satellite/metabase: GetSegmentByPosition for copied segments
Part of server-side copy.

For getting a copied segment GetSegmentByPosition needs to retrieve inline_data or remote_alias_pieces and other information from the original segment.

Fixes https://github.com/storj/storj/issues/4477

Change-Id: I283cbc546df6e1e2fa34a803c7ef280ecd0f65d7
2022-02-22 09:08:02 +00:00
Stefan Benten
ec2bd50bb4
scripts/deploy-storagenode.sh: adding multinode image to deployment
In order to tag the latest release for the multinode image, it makes
the most sense to do it at the same time as we release the storagenode
image.

Change-Id: I2d63c1f93858354ad1f9a4fce0ce45a8fda2716f
2022-02-21 14:37:04 +01:00
Stefan Benten
afa09b3c2e adding a short readme
Change-Id: I6fda2dd358895ae256a13dba6d033aa054795443
2022-02-21 13:08:48 +02:00
Stefan Benten
f89e030c3f final touches
Change-Id: I8fcc986f71f592507dc3f35ce4352647da46d461
2022-02-21 13:08:48 +02:00
Stefan Benten
76be9e6efd cmd/multinode,Makefile: build docker image for multinode dashboard
Closes #4547

We do not build an docker image for the multinode dashboard,
which makes monitoring for docker-focused environments harder.
This adds the basic image and ties it into CI/CD.

Change-Id: I14c01a7f1f0019f6f5c1b8fd75dc424fc362b18d
2022-02-21 13:08:48 +02:00
Stefan Benten
321bf26c85
cmd,private,satellite: add application_name to metabase connections
Currently the metainfo/metabase DB connections are missing the proper
application_name in order to differentiate and filter queries on the DB
side for analytics.
Without it, it is very time-consuming to correlate processes and their load.

This change adds the "check" on DB connection init and passes the fallbacks
in all places to catch connection strings, that do not set it.

Change-Id: Iea5cea8658bc63778ff89038e5c1c352bf482cfd
2022-02-20 22:29:41 +01:00
JT Olio
0178a13743 satellite/web: change cli install flow to use uplink again
depends on merging https://review.dev.storj.io/c/storj/storj/+/6639

Change-Id: Ie594c2a724ac85ce926caf40189f66ce29305d19
2022-02-18 16:46:30 +00:00
Cameron Ayer
1fa79d64f5 satellite/attribution: update value attribution report query
cmd/satellite/reports: show userAgent field and all partners

Change-Id: I9a763f6daa97358e9c22e1f75889ea14e0a3b7c2
2022-02-18 14:25:19 +00:00
Fadila Khadar
5c7c4fedde satellite/metabasetest: include copies in raw state.
It can be useful to compare object copies created during unit tests.
They appear in the segment_copies table as couple (stream_id, ancestor_stream_id).

Change-Id: Id335c3ff7084fe30346456d27e670aff329154ea
2022-02-18 12:48:42 +00:00
Ivan Fraixedes
9688c574b7
web/storagenode: Use an existing const instead value
Use an exiting constant for the ZK explorer URL rather than hard-coding
the URL value.

Change-Id: I029248c4fc2d1279f6b061ae94ca1036f0790b84
2022-02-18 11:49:18 +01:00
paul cannon
12b3fb5fb0 cmd/satellite: add fetch-pieces command
The "satellite fetch-pieces" command allows a satellite operator to
fetch as many pieces of a segment as possible, along with their
original order limits and hashes as provided by the storage nodes. The
fetched pieces and associated info will be stored on in a specified
folder as they are, rather than being RS-decoded or decrypted.

It is hoped that this will allow easier debugging of certain one-off
problems we've observed in the wild.

Change-Id: I42ae0e9ef0023538e42473a9be5a2460a3ac0f3a
2022-02-18 00:13:53 +00:00
Jeff Wendling
ce3a405644 makefile: remove uplink-image
Change-Id: I3552de12fa99892bcd7e83aa5b0fdddb268f7db5
2022-02-17 16:01:53 -05:00
Fadila Khadar
0358a08d6b satellite/metabase: BeginCopyObject and FinishCopyObject
Part of server-side copy implementation.

Creates the methods BeginCopyObject and FinishCopyObject in satellite/metabase/copy_object.go.

This commit makes it possible to copy objects and their segments in metabase.

https://github.com/storj/team-metainfo/issues/79

Change-Id: Icc64afce16e5e0e15b83c43cff36797bb9bd39bc
2022-02-17 13:37:39 +01:00
paul cannon
294d253923 satellite/payments: chore to migrate big.Float values out of db
All code on known satellites at this moment in time should know how to
populate and use the new numeric columns on the
stripecoinpayments_tx_conversion_rates and coinpayments_transactions
tables in the satellite db. However, there are still gob-encoded
big.Float values in the database from before these columns existed. To
get rid of those values, so that we can excise the gob-decoding code
from the relevant sections, however, we need something to read the gob
bytestrings and convert them to numeric values, a few at a time, until
they're all gone.

To accomplish that, this change adds two chores to be run in the
satellite core process- one for the coinpayments_transactions table, and
one for the stripecoinpayments_tx_conversion_rates table. They should
run relatively infrequently, so that we do not impose any undue load on
processing resources or the db.

Both of these chores work without using explicit sql transactions, but
should still be concurrent-safe, since they work by way of
compare-and-swap type operations.

If the satellite core process needs to be restarted, both of these
chores will start scanning for migrateable rows from the beginning of
the id space again. This is not ideal, but shouldn't be a problem (as
far as I can tell, there are only a few thousand rows at most in either
of these tables on any production satellite).

Change-Id: I733b7cd96760d506a1cf52735f598c6c3aa19735
2022-02-16 23:48:30 +00:00
Jeff Wendling
0a860a2e89 scripts: fix builds for new uplink command
the new uplink command expects to be able to migrate
and so we need to specify the --legacy-config-dir
flag sometimes as well. but unfortunately, the old
uplink will error if it gets a flag it doesn't
understand, so we have to set it as an env var.
but we can't use the env var for the --config-dir
flag all the time because the old uplink doesn't
look for that env var.

Change-Id: I019315192c0e6c348814527794342d823a5f9ec3
2022-02-16 21:40:06 +00:00
Jeff Wendling
e2e5882c86 cmd/uplink: fix migration for some old configs
some old configs had a value like

	access: <data>

in the yaml. this would end up causing migration to
create a json file where it had no access values and
a default name of the data. that's not what the command
expects to operate on, so now we fix that during
migration and add a little mini migration for any
users that may have hit it.

Change-Id: I4c98ca5d09d043fe9338738ef6b4f930f933892c
2022-02-16 21:13:52 +00:00
Mya
05a17ef42d deps: upgrade storj.io/common
In addition to upgrading the storj.io/common library, this change
moves off the TCPConnector in favor of the HybridConnector per
the deprecation warning.

Change-Id: I7e7e1e7568e8b95e4a99ad9caa158a799e68e1e3
2022-02-16 18:59:19 +00:00
Fadila Khadar
62eeb599ce docs/blueprints: server-side copy
https://github.com/storj/team-metainfo/issues/23

Change-Id: Ie4b7262cac6191aa1fe72b65cbaaa6f89d590a05
2022-02-16 17:26:23 +00:00
Moby von Briesen
538fa11636 web/satellite: Fix payment banner amount
Update payment banner to use 150GB as the free tier limits rather than
50GB. Fixes #4519

Change-Id: Idb35aeecf52fcf7ab0b38b8ecffe0fc99cec09d0
2022-02-16 16:21:33 +00:00
Michał Niewrzał
b24379aa91 satellite/metainfo: use single testplanet instance in tests
Tests refactoring to reuse testplanet instance between some
of tests. This should decrease resources needed to run those
tests.

Change-Id: I98f3041ec23085d3903b19acd339904973319ec1
2022-02-16 15:27:26 +00:00
AFranco
b181db6c2b
docs/testplan: Repo Security Executive Summary (#4376)
* This commit stubs out the test plan for fuzz testing.

* pushing small change before rebasing.

* This commit implements a rough draft of our security tests, most importantly fuzz tests.

* Moving testplan up a directory and renaming to indicate executive summary.

* Cleaning out fuzzing tests from cmd/uplink,

* updates to tools list
2022-02-16 09:18:29 -05:00
NickolaiYurchenko
70c8ccb435 web/satellite: inactivity timer to dashboard added
added InactivityTimerEnabled flag to enable/disable feature
added InactivityTimerDelay to configure delay time in seconds
default timer set up to 10 minutes
reset dom events: keypress, mouseover, mousedown, touchmove

Change-Id: Idb66067c2902b2cdbe1a972225319c8abff97927
2022-02-16 07:16:00 +00:00
NickolaiYurchenko
aff5cae56b web/satellite: filebrowser upload buttons UI/UX changes
https://www.figma.com/file/HlmasFJNHxs2lzGerq3WYH/Satellite-GUI-Public?node-id=11702%3A79450
2 buttons for file and folder upload converted into 1 with dropdown
icons changed
stylings changed

Change-Id: I13720ba940f93a6b3c1b1dae6ba92ee57f1e526c
2022-02-15 23:03:40 +00:00
NickolaiYurchenko
c6add65e75 web/satellite: download link hidden
download performed without adding link to address bar
added warning to not share link with external people

Change-Id: Ifa79582c51c77c000342c1e03e752dd157912975
2022-02-15 08:17:32 +00:00