Commit Graph

2464 Commits

Author SHA1 Message Date
Erik van Velzen
a9bd983f04 sql: capitalize keywords
Capitalize some keywords which were overlooked

Change-Id: Ie2ad283669e2ca2650fcddfd8c7395a81bac09a8
2022-03-01 15:19:38 +00:00
Vitalii Shpital
ba6956db0f console/server, apigen: feature flag for new generated console api
Added a feture flag which will be used to indicate if new generated console api is used.
Fixed some comments from previous PR.

Change-Id: Ice31c998b0b347028a491c971a648fd1269bfd49
2022-02-28 23:00:12 +00:00
Michał Niewrzał
1da9697995 satellite/metainfo: use feature flag for begin/finish object copy metods
We want also to cover with feature flag code for begin/finish
object copy methods.

Change-Id: I97330e7d9b310de98408c47c2c1ed41551631bec
2022-02-28 15:09:40 +00:00
Erik van Velzen
7a01a2a134 satellite/metabase/metabasetest/create: return segments
Return segments when creating a test object so that it can be checked
that the correct segments are remaining after a delete action.

Change-Id: Ifc245948935ba278806e887672c03abc5f2c2654
2022-02-28 13:40:46 +00:00
Ivan Fraixedes
8caa4c4557 satellite/console: Don't lose ErrValiation error class
There was a defined type (`validationErrors`) for gathering several
validation errors and classify them with the `ErrValdiation errs.Class`.

`errs.Combine` doesn't maintain the classes of the errors to combine,
for example

```
var myClass errs.Class = "My error class"

err1 := myClass.Wrap(erros.New("error 1"))
err2 := myClass.Wrap(erros.New("error 2"))
err3 := errors.New("error 3")

combinedErr := errs.Combine(err1, err2, err3)
myClass.Has(combinedErr) // It returns false

// Even only passing errors with a class and with the same one for all
// of them
combinedErr := errs.Combine(err1, err2)
myClass.Has(combinedErr) // It returns false
```

Hence `validationErrors` didn't return what we expected to return when
calling its `Combine` method.

This commit delete the type and it replaces by `errs.Group` when there
are more than one error, and wrapping the `errs.Group.Err` returned
error with `ErrValiation` error class.

The bug caused the HTTP API server to return a 500 status code as you
can seee in the following log message extracted from the satellite
production logs:

```
code: 500
error: "console service: validation: full name can not be empty; validation: Your password needs at least 6 characters long; validation: mail: no address"
errorVerbose: "console service: validation: full name can not be empty; validation: Your password needs at least 6 characters long; validation: mail: no address
        storj.io/storj/satellite/console.(*Service).CreateUser:593
        storj.io/storj/satellite/console/consoleweb/consoleapi.(*Auth).Register:250
        net/http.HandlerFunc.ServeHTTP:2047
        storj.io/storj/private/web.(*RateLimiter).Limit.func1:90
        net/http.HandlerFunc.ServeHTTP:2047
        github.com/gorilla/mux.(*Router).ServeHTTP:210
        storj.io/storj/satellite/console/consoleweb.(*Server).withRequest.func1:464
        net/http.HandlerFunc.ServeHTTP:2047
        net/http.serverHandler.ServeHTTP:2879
        net/http.(*conn).serve:1930"
message: "There was an error processing your request"
```

The issues was that not being classified with `ErrValidation` class it
was not picked by the correct switch branch of the
`consoleapi.Auth.getStatusCode` method which is in the call chain to
`consoleapi.Auth.Register` method when it calls
`console.Service.CreateUser` and returns an error.

These changes should return the appropriated HTTP status code (Bad
Request) when `console.Service.CreateUser` returns a validation error.

Returning the appropriated HTTP statsus code also makes not to show this
as an error in the server logs because the Bad Request sttatus code gets
logged with debug level.

Change-Id: I869ea85788992ae0865c373860fbf93a40d2d387
2022-02-28 11:00:56 +00:00
Michał Niewrzał
c0297bae78 satellite/metabase: return object metadata with FinishCopyObject
Updates metadata and metainfo to return object metadata with
FinishCopyObject request.

https://github.com/storj/storj/issues/4474

Change-Id: I32cba5c20a943272e9b5964df1b3d6463ad212dc
2022-02-25 12:46:30 +00:00
Fadila Khadar
332e6afed8 satellite/metabasetest: order RawSegments by streamID and position
Change-Id: I669c0c2361c2c7989c8e946eb1fbf2f52074ce0e
2022-02-25 11:19:11 +00:00
Fadila Khadar
2d4760fd09 satellite/metainfo: BeginCopyObject and FinishCopyObject
Metainfo endpoints to use server-side copy.

Fixes https://github.com/storj/storj/issues/4475

Change-Id: Ided06aed9e6187d6d8f030e95dda019ba78fff95
2022-02-24 14:38:04 +01:00
Michał Niewrzał
fbe2680500 satellite/metainfo: add feature flag for server side copy code
We would like to disable in production those parts of code
which are now mixed with new server-side copy logic.

Change-Id: Iff50682bc9545207330f58dd19b5eee53d404d7f
2022-02-24 10:43:49 +00:00
Moby von Briesen
d12bb19972 satellite/console/consoleweb: Update CSP directives
Update the Content Security Policy to whitelist `blob:` for the img-src
and media-src directives. This is necessary to prevent CSP errors in the
object browser while loading previews and object maps.

Change-Id: Ic32bf0954f300c77ec4f0fe11fae63f0c7b622da
2022-02-23 17:30:35 -05:00
Erik van Velzen
ac5f97a364 satellite/sattelitedb/test: improve test database name
Remove special characters from the test database name to improve
compatibility with external tools like the Cockroach web gui.

Change-Id: I3a6a368a5051e3064e7279b14eec4f2d4ff3c435
2022-02-22 14:45:01 +00:00
Fadila Khadar
b11d144bb6 satellite/metabase: GetSegmentByPosition for copied segments
Part of server-side copy.

For getting a copied segment GetSegmentByPosition needs to retrieve inline_data or remote_alias_pieces and other information from the original segment.

Fixes https://github.com/storj/storj/issues/4477

Change-Id: I283cbc546df6e1e2fa34a803c7ef280ecd0f65d7
2022-02-22 09:08:02 +00:00
Stefan Benten
321bf26c85
cmd,private,satellite: add application_name to metabase connections
Currently the metainfo/metabase DB connections are missing the proper
application_name in order to differentiate and filter queries on the DB
side for analytics.
Without it, it is very time-consuming to correlate processes and their load.

This change adds the "check" on DB connection init and passes the fallbacks
in all places to catch connection strings, that do not set it.

Change-Id: Iea5cea8658bc63778ff89038e5c1c352bf482cfd
2022-02-20 22:29:41 +01:00
Cameron Ayer
1fa79d64f5 satellite/attribution: update value attribution report query
cmd/satellite/reports: show userAgent field and all partners

Change-Id: I9a763f6daa97358e9c22e1f75889ea14e0a3b7c2
2022-02-18 14:25:19 +00:00
Fadila Khadar
5c7c4fedde satellite/metabasetest: include copies in raw state.
It can be useful to compare object copies created during unit tests.
They appear in the segment_copies table as couple (stream_id, ancestor_stream_id).

Change-Id: Id335c3ff7084fe30346456d27e670aff329154ea
2022-02-18 12:48:42 +00:00
paul cannon
12b3fb5fb0 cmd/satellite: add fetch-pieces command
The "satellite fetch-pieces" command allows a satellite operator to
fetch as many pieces of a segment as possible, along with their
original order limits and hashes as provided by the storage nodes. The
fetched pieces and associated info will be stored on in a specified
folder as they are, rather than being RS-decoded or decrypted.

It is hoped that this will allow easier debugging of certain one-off
problems we've observed in the wild.

Change-Id: I42ae0e9ef0023538e42473a9be5a2460a3ac0f3a
2022-02-18 00:13:53 +00:00
Fadila Khadar
0358a08d6b satellite/metabase: BeginCopyObject and FinishCopyObject
Part of server-side copy implementation.

Creates the methods BeginCopyObject and FinishCopyObject in satellite/metabase/copy_object.go.

This commit makes it possible to copy objects and their segments in metabase.

https://github.com/storj/team-metainfo/issues/79

Change-Id: Icc64afce16e5e0e15b83c43cff36797bb9bd39bc
2022-02-17 13:37:39 +01:00
paul cannon
294d253923 satellite/payments: chore to migrate big.Float values out of db
All code on known satellites at this moment in time should know how to
populate and use the new numeric columns on the
stripecoinpayments_tx_conversion_rates and coinpayments_transactions
tables in the satellite db. However, there are still gob-encoded
big.Float values in the database from before these columns existed. To
get rid of those values, so that we can excise the gob-decoding code
from the relevant sections, however, we need something to read the gob
bytestrings and convert them to numeric values, a few at a time, until
they're all gone.

To accomplish that, this change adds two chores to be run in the
satellite core process- one for the coinpayments_transactions table, and
one for the stripecoinpayments_tx_conversion_rates table. They should
run relatively infrequently, so that we do not impose any undue load on
processing resources or the db.

Both of these chores work without using explicit sql transactions, but
should still be concurrent-safe, since they work by way of
compare-and-swap type operations.

If the satellite core process needs to be restarted, both of these
chores will start scanning for migrateable rows from the beginning of
the id space again. This is not ideal, but shouldn't be a problem (as
far as I can tell, there are only a few thousand rows at most in either
of these tables on any production satellite).

Change-Id: I733b7cd96760d506a1cf52735f598c6c3aa19735
2022-02-16 23:48:30 +00:00
Mya
05a17ef42d deps: upgrade storj.io/common
In addition to upgrading the storj.io/common library, this change
moves off the TCPConnector in favor of the HybridConnector per
the deprecation warning.

Change-Id: I7e7e1e7568e8b95e4a99ad9caa158a799e68e1e3
2022-02-16 18:59:19 +00:00
Michał Niewrzał
b24379aa91 satellite/metainfo: use single testplanet instance in tests
Tests refactoring to reuse testplanet instance between some
of tests. This should decrease resources needed to run those
tests.

Change-Id: I98f3041ec23085d3903b19acd339904973319ec1
2022-02-16 15:27:26 +00:00
NickolaiYurchenko
70c8ccb435 web/satellite: inactivity timer to dashboard added
added InactivityTimerEnabled flag to enable/disable feature
added InactivityTimerDelay to configure delay time in seconds
default timer set up to 10 minutes
reset dom events: keypress, mouseover, mousedown, touchmove

Change-Id: Idb66067c2902b2cdbe1a972225319c8abff97927
2022-02-16 07:16:00 +00:00
Michał Niewrzał
ddb742daff satellite/metrics: speed up tests
For some reason very small interval for chore is causing
very long execution time for Pause() method and that was
the reason why a simple test was slow.

Change-Id: Iebfdea16e31f9865493de3e1bd7d69c6f9116fc0
2022-02-14 17:24:16 +00:00
Cameron
eefef2eb17 satellite/console: pass entire auth.User into users.Update in service.UpdateAccount
The Update method of the usersDB takes a console.User as an argument.
To update the columns in the DB, we have to migrate the fields from the
console.Users struct into a special dbx struct. If one of these fields
is left empty, then the zero value of that field's type will be used to
update the respective column.

In most cases where the users.Update method is called, the entire
console.User is apparently retrieved first, fields are updated, then it
is passed to users.Update. This is not the case for
service.UpdateAccount. Because these fields are not populated in the
user struct in UpdateAccount before it is passed into users.Update,
their respective columns in the database are overwritten with zero:

ProjectLimit, ProjectStorageLimit, ProjectBandwidthLimit,
ProjectSegmentLimit, PaidTier, MfaEnabled, MfaRecoveryCodes, MfaSecretKey

Solution: Do what is done in other places which call users.Update. Take
the console.User from the auth context, update the relevant fields on
that, then pass that in.

Change-Id: I3cbd560e8ea5397e5c27711fb40bb3907d987028
2022-02-11 15:33:04 +00:00
Fadila Khadar
b1d9a4426f satellite/metabase: adjust segment_copies table primary key.
segment_copies table has (stream_id, ancestor_stream_id) as primary key,
but it should have only stream_id as primary as a stream_id can only have one ancestor_stream_id.

https://github.com/storj/team-metainfo/issues/84

Change-Id: I0218295e86e449c9dfefa5a21319f3083bf80afd
2022-02-10 13:52:16 +00:00
Vitalii Shpital
07c71e34c2 satellite/console{gen}: GetUsersProjects endpoint
Initial implementation of auto-generated GetUsersProjects endpoint

Change-Id: If41bff2ea3ff9cfc87afeda9e5e5b3f586cbab33
2022-02-10 08:03:02 +00:00
Bill Thorp
cf03209c16 satellite/metainfo: only collect metric "other" once per user agent
A user-agent string can contain multiple "products", in the case of
Gateway-MT at least this includes the HTTP client's full user agent.
This means that "other" is often logged even when we know the Storj
product, and sometimes logged more than once per call to "collect".

This makes sure that "other" is only logged if a product isn't
identified, and only logged once.

Change-Id: I8536f7eb32877e36fec97dab7b8d477ccb10f92e
2022-02-09 18:18:55 +00:00
paul cannon
3540f9b3ad satellite/satellitedb: phase 2 getting rid of gob encodings in db
For a thorough explanation of the overall transition, see the message on
commit c053bdbd70.

This change will rename the columns containing gob-encoded big.Floats
and add new columns which will contain the equivalent data in a more
sql-friendly format.

The change should *not* break already-running satellite processes,
because all functionality touching these tables has already been taught
to work with these new columns if it sees any "undefined column" errors.

Change-Id: I229324376533e383c5d05064b8aedad149cf825b
2022-02-09 12:25:46 +00:00
Vitalii Shpital
150be885b7 satellitedb/projectaccounting, web/satellite: reworked bandwidth chart to show both allocated and settled bandwidth
Extended DB query.
Reworked charts UI functionality to show 2 lines if necessary.

Change-Id: I8ac4e4fa07676fc9fa7e9c078ecdeed62233b8e2
2022-02-08 22:07:59 +00:00
Stefan Benten
2d3c417c92 satellite/metainfo/version_collector: adding more known user agents:
transfer-sh will be set as of https://github.com/dutchcoders/transfer.sh/pull/467.
filezilla needs to be verified and duplicati is set per info from @TopperDEL
comet and orbiter are added as preparation.

Change-Id: I44d730a7b3ba1969068e48c2477b478831799cd1
2022-02-08 16:24:55 +00:00
Mya
0164682c37 satellite/oidc: move oidc into common package
Change-Id: I77702e0e46f15a09fee315b9076638e1412836f7
2022-02-08 09:46:54 -06:00
Michał Niewrzał
95921b8b39 satellite/metabase: add segment_copies table
Part of server-side copy implementation.

Adds new table to metabase for keeping references between
copied segments and its ancestors.

Fixes https://github.com/storj/team-metainfo/issues/84

Change-Id: I436d4b533a3d951bcd752e222e3b721cee7f0844
2022-02-08 11:15:19 +00:00
Stefan Benten
06944f062d satellite/{admin,payments,satellitedb}: add checks for deletion of free tier accounts
This change adds some more checks to the deletion process for projects and
users, since we ran into a race condition during invoicing, where projects
have been deleted before the invoicing was finished, leading to missing
references.
This PR changes the logic to block user deletion if we are in exactly that period,
while also allowing the deletion of projects/users on free tier during the month.

Change-Id: Ic0735205e6633762fb7e3c2fa13e744cdfa5ec32
2022-02-08 10:11:31 +00:00
Vitalii Shpital
b3e1be37ff satellite/projectaccounting: query to get daily project usage by date range
Finished implementing queries for both bandwidth and storage using pgx.Batch.
Fixed CSP styling issue.

Change-Id: I5f9e10abe8096be3115b4e1f6ed3b13f1e7232df
2022-02-04 22:32:37 +00:00
Vitalii Shpital
3c8e41e665 web/satellite: get object map and preview by signed request.
There is a sev-2 issue to add more browser caching.
In this PR I made object map and object preview to be fetched by signed request with non-public credentials using AWS SignatureV4 package.

Change-Id: Ib5013fa6d6af3faa97eed5168c11a13f9629cd87
2022-02-03 11:31:09 +00:00
Mya
4d0dd737b0 satellite/admin: add endpoints for oauth clients
Change-Id: I26aa81266f494be8aab0b5523217bad9405037a0
2022-02-02 09:47:52 -06:00
Michał Niewrzał
2e31ef3f29 satellite/metabase: better error message while move
Before this change we were returning full DB error message.
That can be very confusing for end user. This change is translating
error message into more user frindly version and fixes also DRPC
error status code.

Fixes https://github.com/storj/team-metainfo/issues/76

Change-Id: I29b06ab4ba50a0d14db7a822a2906d95d65ab524
2022-02-02 15:03:24 +00:00
Michał Niewrzał
bc161794fc satellite/metabase: drop DeleteObjectLatestVersion method
This method was never used, except tests.

Change-Id: Idc1e69b2e2971995b5c4e6cf78a2b5fc69f39ad2
2022-02-02 14:33:48 +00:00
Michał Niewrzał
5041ee0abc satellite/metabase: drop GetObjectLatestVersion method
We don't have different version of object than 1 so at the moment
this method is not needed. Also using GetObjectExactVersion
should be slightly more performant.

Change-Id: I78235d8ae22594cc1d6345dabcc915f41cd7797b
2022-02-02 09:40:53 +00:00
Michał Niewrzał
dffa7845f6 satellite/metainfo: split endpoint tests
We already split main code base, now we need to split test
to reflect new files structure (bucket/object/segment/other).

Fixes https://github.com/storj/team-metainfo/issues/12

Change-Id: Ica1054c4fc7df764483b03f204b4beba094df8e1
2022-02-02 07:46:14 +00:00
Michał Niewrzał
3a7fb4588f satellite/satellitedb: users/projects segment limit migration
We have here two migrations in fact. One is for existing users,
we need to check if its paying user (paid_tier) and set 1M for
them and 150K for others.
Second migration is to set limits for projects depends on owner.
If owner is a paying user (paid_tier) then project should have
1M limit, otherwise it should be 150K. In this case to make
migration faster initially projects table segment_limit was set
to 1M by default. With migration we are selecting all paying
users and we are setting 150K limit for all projects which owners
are not in paying users set.

Initially we had a concern if that query wil lbe quick enough to
be executed during deployment but after investigation CRDB
team confirms that this should take seconds for out DBs.

Fixes https://github.com/storj/team-metainfo/issues/70

Change-Id: I8be06e9f949b68b993e043cc15525e8483bf49ea
2022-02-01 14:35:12 +00:00
Egon Elbre
64c8de6ea5 mod: use vendored base58
Change-Id: I5aa29515928848c862500330218cc094618638d7
2022-01-31 15:54:33 +02:00
Michał Niewrzał
d393f6094c satellite/accounting: adjust tests to expose new uplink errors
Existing tests are checking errors by comparing error message.
We plan to expose storage/segment limit errors as a public uplink
API but before that will happen we need to update tests to handle
both cases as new uplink error will have a bit different error message
than current tested error.

https://github.com/storj/uplink/issues/77

Change-Id: Id2706323c60d050d96752e66e859d4ec051a69b9
2022-01-27 15:15:07 +00:00
Vitalii Shpital
b8c55fdd87 satellite/projectaccounting, satellite/console, web/satellite: implemented backend for bandwidth chart
Implemented endpoint and query to get bandwidth chart data for new project dashboard.
Connected backend with frontend.
Storage chart data is mocked right now.

Change-Id: Ib24d28614dc74bcc31b81ee3b8aa68b9898fa87b
2022-01-26 15:09:22 +00:00
Vitalii Shpital
5d6ee506b0 private/apigen: initial typings and methods
Initial typings and methods for the future API Management implementation.
Blueprint - https://review.dev.storj.io/c/storj/storj/+/6341

Change-Id: I9afe66a7610a3a80e764b317ddd211f125b1ea43
2022-01-26 14:39:22 +00:00
Michał Niewrzał
17f56ab636 satellite/metainfo: split endpoint into smaller files (object/segment)
This is part of change to split metainfo endpoint into smaller
files. It will be grouped by bucket/object/segment/other requests.

Tests will be split as a separate set of changes.

Updates https://github.com/storj/team-metainfo/issues/12

Change-Id: I6c691e4d0e192fe3ad7974d2d0ab5ced0d272f3c
2022-01-25 16:14:37 +00:00
Michał Niewrzał
9ce1a4e25d satellite/accounting: use AOST to get segment usage
For better performance we should use AOST for getting
data where being up to data it not very crucial. We don't
care about small differences for segment limit calculation.

Change-Id: I9b2d4f2bd15ebc9d1c46bc84dd51a2e9d9231506
2022-01-25 11:00:08 +00:00
Malcolm Bouzi
3451162583 web/console: Save timestamp when creating users to mark their first email verification notification.
This value will be updated whenever we send a reminder for a user to verify their email.

Change-Id: I7f08b16eccc6633b8eb3662eb7efed1f1f780f44
2022-01-24 16:53:51 +00:00
Michał Niewrzał
65c38c6cef satellite/metainfo: split metainfo endpoint into smaller files (buckets)
This is part of change to split metainfo endpoint into smaller
files. It will be grouped by bucket/object/segment/other requests.

Tests will be split as a separate set of changes.

Change-Id: I9b097dcc8fa889f985b7f4ef5f8f435a1ff0ef95
2022-01-24 13:47:51 +00:00
Michał Niewrzał
913718ef97 satellite/accounting: test bandwidth limit without cache
Additional test case to cover situation when cache value
expired and we need to get this value directly from DB.

Change-Id: I07bda67e19333e09a567104ce70f112fd47a7845
2022-01-24 13:24:50 +00:00
Egon Elbre
be02aa9b17 satellite/overlay: add test for UpdateCheckIn panic
The database table got invalid input and the resulting error
was not checked. This adds updates that contain invalid fields
to trigger different errors.

Change-Id: Iacea32cbef5599aab562c88e4113073596cc9996
2022-01-24 15:01:59 +02:00
Michał Niewrzał
4be84c3a6c satellite/accounting: fix how we put segments usage into cache
We had two problems here. First was how we were handling
errors message from GetProjectSegmentUsage. We were always
returning error, even for ErrKeyNotFound which should be used
to refresh value in cache and not propagated out of method.
Second, we were using wrong value to update cache. We used
current value from cache which obviously it's not what we intend.

Fixes https://github.com/storj/storj/issues/4389

Change-Id: I4d7ca7a1a0a119587db6a5041b44319102ef64f8
2022-01-24 11:57:50 +00:00
Michał Niewrzał
bfad42f29a satellite/metainfo: split metainfo endpoint into smaller files
This is part of change to split metainfo endpoint into smaller
files. It will be grouped by bucket/object/segment/other requests.

Tests will be split as a separate set of changes.

Change-Id: I5128c84e06c82777fe71460bf5f9a6e26e52a243
2022-01-24 08:24:24 +00:00
Stefan Benten
e7e39fe628
satellite/metainfo: reduce default user rate limit to 100
Currently the rate limit has kept per satellite api endpoint.
Since we run 9+ api endpoints in production, we do not need
a limit of 1000, since the intention was to allow 1000 total.
This change reduces the effective limit given 9 instances
down to 900, which should be close enough.

Change-Id: Ia579149ccc3a12e8febe0cfd5586b8a39de40f55
2022-01-21 23:37:59 +01:00
Michał Niewrzał
1c47163eee satellite/metainfo: add more logging for CreateBucket
We were returning pure non rpc errors in two cases.
This change added loging and correct rpc error as a
return.

Change-Id: I581ceb17dcdc00921dfa3c1057015c3b4d04308d
2022-01-21 15:51:37 +00:00
Malcolm Bouzi
c30c7def44 satellite/{console,payments},web/satellite: save signup promo code to newly registered users
Users signing up through a url containing a promo code will have that code applied to their stripe account instead of the free tier coupon.

Change-Id: I071041b0934648ef3f5bdb05b6ec97c400f89ae4
2022-01-21 15:02:22 +00:00
Cameron Ayer
251e432e5b satellite/satellitedb: drop contained from nodes and reputations
A few months ago we removed all references to the contained
column in nodes and reputations

bb21551a9c
and
56fe636123

But we never did the migration to drop the columns.
This commit will finally do that.

Change-Id: I82aa2f257b1fb14a2f1c4c4a1589f80895360ae4
2022-01-20 19:46:48 +00:00
Moby von Briesen
464b4b02fa satellite/console/consoleweb: Only replace some kinds of characters in name
The previous change (59648dc272) ends up removing a lot of characters
for valid non-English names. Instead, only replace URL characters such
as slashes, colons, and periods. Since someone may use these characters
to separate two parts of a name, e.g. Name1/Name2, replace these
characters with a hyphen.

Change-Id: I4cc3d1bdb05d525a83970cf1b42479414c9678e7
2022-01-20 11:13:26 -05:00
Clement Sam
2de6feefde satellite/contact: fix connection leak in PingMe endpoint
Fixes connection leak in pingMe endpoint.
includes other minor fixes.

Change-Id: I7c61f620565f46dd113d21a772de7c439be550e3
2022-01-20 14:20:24 +00:00
Moby von Briesen
59648dc272 satellite/console: Filter special characters out of name
When a user is created, but before verification or forgot password email
is sent, remove any special characters in the provided name. This
protects us against certain phishing attacks.

Change-Id: Ieddd3479da20eb80b9f1b56eb86c8f46bca2642c
2022-01-20 00:18:56 +00:00
Clement Sam
9f3c1f9cda satellite/contact: add PingMe endpoint
Change-Id: I832a72fafeacf76ad64a0129bcc6582cc4f9290d
2022-01-19 17:52:33 +00:00
Michał Niewrzał
95750b9d32 satellite/console: verify limits while test
Additional test case to be sure that update operation went well.

Change-Id: I7dbf1e3110f716712ba9747a1915b54112b89801
2022-01-19 16:39:36 +00:00
Qweder93
228f465d45 satellite/metainfo: сombine checks for storage and segment limit
We need to combine methods from accounting.Service (ExceedsStorageUsage and ExceedsSegmentUsage)
to run checks concurrently.

Resolves https://github.com/storj/team-metainfo/issues/73

Change-Id: I47831bca92457f16cfda789da89dbd460738ac97
2022-01-19 10:44:04 +00:00
Michał Niewrzał
270c6e24f2 satellite/console: fix default limit for free-tier segment usage limit
Free-tier segment usage limit was defined as 150k, not 140k. This change
is correcting that.

https://github.com/storj/team-metainfo/issues/8

Change-Id: I71ec0961930b19fd09b2b996e01acd406a8dcf8f
2022-01-19 09:21:13 +00:00
Mya
309ac41bfc satellite/satellitedb/dbx: add schema for oauth tokens
Change-Id: Ia980766a8f86b662545e7d8cae6ecf631e199809
2022-01-18 09:38:12 -06:00
Mya
83f7d3e045 satellite/satellitedb/dbx: add schema for oauth clients
Change-Id: Ib08c7074803805cca8ffaedbd89da54504842af7
2022-01-18 09:38:12 -06:00
Michał Niewrzał
36c07720bb satellite/metainfo: bound user agent metrics
We need to limit number of results for SeriesTag to avoid
overloading our metric system.

Change-Id: I3d9f4f4f6d489bf4ce87c6240f66d48b29b88abb
2022-01-18 12:42:09 +00:00
Mya
cf462fcf92 satellite/admin: update geofence endpoints to follow response conventions
Change-Id: Ieff91c16f91f9720e9c3d8ead0e13c47651d23f5
2022-01-13 12:33:10 -06:00
Michał Niewrzał
5e9643e1b8 satellite/metainfo: cleanup validation
Refactoring to do few things:
* move simple validation before validations with DB calls
* combine validation check/update for storage and segment
limits together

Change-Id: I6c2431ba236d4e388791d2e2d01ca7e0dd4439fc
2022-01-13 13:00:40 +01:00
Michał Niewrzał
18ab464b51 satellite/metainfo: UpdateProjectSegmentUsage enabled only with segment validation
updating project segment usage entry to redis is enabled only if
segment validation is enabled in config value, so no incompatible
changes will be added while deployment.

Change-Id: I1288cb9ff0a8a00f095dc94e20d2f14393e9a613
2022-01-13 12:12:25 +01:00
Ivan Fraixedes
3a987ac00f satellite/admin: Fix client API & docs
The following 2 commits added 2 new query parameters to set the `burst`
and `segments` limits for a project and also to new fields to the
response JSON object body to the "get project limits" endpoint:

* c911360eb5
* b7b010adc9

However, the API documentation and the Typescript client API (used by
the UI) weren't updated with them.

Later, the commit dc6128e9e2 updated the
Typescript client API with the `segments` limit but it didn't update the
documentation to reflect it.

This commit brings all things that were missed in those previous
commits.

Change-Id: Iff12cdd4a0d3c448cd73b57a98d171ba468d2c98
2022-01-13 09:31:01 +00:00
Michał Niewrzał
eb0d08d59b satellite/metainfo: collect versions of user tools
We want to know usage statistics for our main tools
like uplink-cli or rclone. Initially we will collect
only usage stats without relation to specific process
e.g. download or upload.

Change-Id: I203b1a6c07ae014e710368f77163f13fdf10763c
2022-01-12 13:49:15 +00:00
Yaroslav Vorobiov
e0b3c7152b satellite/nodestats: use overlay node for reputation status
Use overlay db node as primary source of truth for reputation status when node
query it's status via Endpoint.Get. If node was was disqualified in overlay reputation
status may be inconsistend with reputation table.

Change-Id: I1bf858a4b020324035847a9215ef29cf5432f6a4
2022-01-11 16:38:27 +00:00
Qweder93
b6625cadea satellite/metainfo: uncomment tests, renamed EncryptedPath into EncryptedObjectKey
moved tests back and renamed params that were renamed on uplink side

Change-Id: I4a9592bd955f2217c5809c235c394a4e78d337ea
2022-01-11 17:18:18 +02:00
Qweder93
332b673a05 satellite/metainfo: comment tests for rename
comment tests that contains fields needs
to rename on uplink side without breaking compatibility.
after rename tests will be moved back from comments.

Change-Id: I3bc4aff6ae7f6711ade956ac389f0d7e1a1ab91a
2022-01-10 17:46:32 +02:00
Qweder93
15c2b2cc10 satellite/metainfo: comment tests for rename
comment tests that contains fields needs
to rename on uplink side without breaking compatibility.
after rename tests will be moved back from comments.

Change-Id: I439783c62678c32805a85aa52bef1d2b767543a1
2022-01-10 14:44:43 +00:00
Qweder93
dc6128e9e2 satellite/admin: add update project segment limit on ui
Added possibility to update project segment limit on
satellite admin panel ui.

Resolves https://github.com/storj/team-metainfo/issues/5

Change-Id: I192d0c9adbab411556aa46673167b3ff7437d04b
2022-01-10 13:26:56 +00:00
Qweder93
8be655c487 satellite/metainfo: add segment limit validation, caching
We want to be able to limit the number of segments per project for users.
To limit this we need to check limit value associated with project
and value of used segments already in BeginMoveObject, BeginMoveSegment
and increment cache segments usage after each CommitSegment call.

Resolves https://github.com/storj/team-metainfo/issues/1

Change-Id: I6290e67c095a174b9d101c4521802d9bfe0453b8
2022-01-10 14:33:26 +02:00
Yingrong Zhao
1f8f7ebf06 satellite/{audit, reputation}: fix potential nodes reputation status
inconsistency

The original design had a flaw which can potentially cause discrepancy
for nodes reputation status between reputations table and nodes table.
In the event of a failure(network issue, db failure, satellite failure, etc.)
happens between update to reputations table and update to nodes table, data
can be out of sync.
This PR tries to fix above issue by passing through node's reputation from
the beginning of an audit/repair(this data is from nodes table) to the next
update in reputation service. If the updated reputation status from the service
is different from the existing node status, the service will try to update nodes
table. In the case of a failure, the service will be able to try update nodes
table again since it can see the discrepancy of the data. This will allow
both tables to be in-sync eventually.

Change-Id: Ic22130b4503a594b7177237b18f7e68305c2f122
2022-01-06 21:05:59 +00:00
littleskunk
b21cbc85f1
satellite/orders: log level warn for remaining "bucketName or projectID not set" (#4326)
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2022-01-06 16:31:26 +01:00
dlamarmorgan
ab37b65cfc satellite/{accounting,orders,satellitedb}: group bucket bandwidth rollups by time window
Batching of the order submissions can lead to combining the allocated
traffic totals for two completely different time windows, resulting
in incorrect customer accounting. This change will group the batched
order submissions by projectID as well as time window, leading to
distinct updates of a buckets bandwidth rollup based on the hour
window in which the order was created.

Change-Id: Ifb4d67923eec8a533b9758379914f17ff7abea32
2022-01-05 20:24:48 +00:00
Egon Elbre
a42b9d1a48 all: fix uses of email.com
email.com is not a domain that should be used for examples
nor tests.

Change-Id: I654d4287d02633d5ed9740e81a79150470eeaf25
2022-01-05 16:29:19 +02:00
Egon Elbre
04d04c0306 web/satellite: remove old object browser
Change-Id: Ief16a3887444df093c644831d86a9b3e865e5867
2022-01-05 16:29:19 +02:00
Malcolm Bouzi
12cb733b16 satellite/satellitedb: add last_verifcation_reminder column to users table
We want to issue a reminder to users when they don't verify their email within 24hrs of registering. This change only adds a column to the users table.

Change-Id: I92e2baeabf179338ffec01574d4752c0ccdba88b
2022-01-04 19:38:04 +00:00
Malcolm Bouzi
6a059b7f9d web/satellite: remove all references to the old onboarding flow
Change-Id: I1f42fabdb407a7eb9c34f0341d332e66c3ce288d
2022-01-04 18:40:08 +00:00
Moby von Briesen
5351c43757 satellite/analytics: track origin header and referer on signup
This allows us to distinguish between accounts created from the signup
page vs. from www.storj.io.
Also set a field `account_created=true` when we send so
that we can see when existing leads have created an account.

Change-Id: Ibef34825a08b6c68b8f2869625e576bb837520e5
2022-01-03 17:47:48 +00:00
Cameron Ayer
2a4ff6f469 satellite/satellitedb: select user_agent in apikeys method GetPagedByProjectID
The user_agent column was missing from the SQL query in
apikeys.GetPagedByProjectID. Add it.

Change-Id: I2201b0ecd1767194a6d2339dba25a0b58e1ee30a
2022-01-03 11:58:26 -05:00
Moby von Briesen
1bd74658df satellite/console: Disable segment limit checking during project update
This is not supported on the client side yet.

Change-Id: Id2df75278efe4b3a48be21d0992ee835e804fc3a
2021-12-23 14:35:45 -05:00
Michał Niewrzał
ff653f6f22 satellite/metainfo: stop using in tests unused method
We would like to remove it from uplink but to do that
we need to stop using it in storj tests.

Change-Id: I8f6af74d6b8b8e447ab8ed3a00517b40840dad58
2021-12-22 15:05:53 +01:00
Michał Niewrzał
db7985ca90 satellite/metainfo: set EncryptedMetadataEncryptedKey while listing objects
At some point we missed to add metadata key to list objects
response. Because of that uplink is taking key from pb.SteamMeta.
We need to clean it up.

Tests will be added on uplink side.

https://github.com/storj/uplink/issues/71

Change-Id: I3328e2f1b86bca15aeaf89f8e59cdca3c8e97742
2021-12-21 13:09:11 +00:00
Michał Niewrzał
87a9292127 satellite/metainfo/zombiedeletion: enable chore by default
We enabled this chore on different satellites
for testing. Works fine so we can enable this
by default for all configurations.

Change-Id: I987639685d8de5c7e5798adca30fe26bdac9e1d1
2021-12-21 13:26:18 +01:00
Stefan Benten
a6139f5a6b satellite/orders: log less decrypt order messages
Currently most of the satellite log is made up by this specific log
message and thus we should increase the log level. We already have
a monkit event tracking this case. In case we need to look into this
more we should just increase the satellite log level.

Change-Id: I27ebed3e6745701c66c83e8c52ddc836ad9d5f4e
2021-12-21 00:39:29 +01:00
Qweder93
b7b010adc9 satellite/admin: add project limit on segment count
Added possibility to set custom value of segments
per project for user/customer.

Change-Id: Ic6e1abe25269755e9e8819fbef065a87de359fce
2021-12-17 12:40:12 +00:00
Ivan Fraixedes
f87ce0cadf satellite/admin/ui: Fix component rendering
I introduced a change in the UI that was a better way to dynamically
render Svelte components
(https://review.dev.storj.io/c/storj/storj/+/5931/4..5) during the
review of the first version of it, however, while it works perfectly
on development mode it doesn't work when the assets are built for
production, failing silently, because the constructor name get renamed
due to the name mangling caused by the minifyer as stated in the
following issue: https://github.com/sveltejs/svelte/issues/6980

This commit use a different alternative not based on the constructor
name and it works fine with the production build.

Change-Id: I643c405f877a9206cf0e51b44d5138e5a9756a79
2021-12-17 11:00:31 +00:00
Ivan Fraixedes
5573ece848 satellite/admin/ui: Migrate to SvelteKit
Migrate the satellite admin UI web app from the Svelte template used to
generate a Svelte App scaffolding to SvelteKit.

There aren't any functional changes in the  application, however, the
commit has a lot because:

1. SvelteKit uses a different directory layout and constraints to it, so
   the files have been moved.
2. The files have changed its formatting due to the new default linter
   configurations that SvelteKit uses.
3. The linter detected some issues with using `object` and `any` types
   in Typescript, so they have been replaced by better general types
   (e.g. Record).

The  migration allows to use the new tooling rather than Rollup
directly, besides that will empower the future of it when it needs more
features (e.g. different routes, etc.).

Change-Id: Ifa6736c13585708337f6c5a59388077b784eaddd
2021-12-17 10:27:13 +00:00
Michał Niewrzał
60d03b9e42 satellite/metainfo: drop GetSegmentByLocation
Method is not used anymore. We need it during transition
from metainfo loop into segment loop.

Change-Id: Id32590ff9b02e06025efce1c32869e2cf9cd0bc5
2021-12-16 16:32:45 +01:00
Michał Niewrzał
3f0babffe8 satellite/{console,satellitedb}: add project segment limit to user
All limits we have for projects have also parent limits stored
with user data. New created project is first taking limits from
owner (user) limits.
This change is extending users table with project_segment_limit
column and adds functionality to get and set value for this
column.

Change-Id: Iff5e36c62b517652390b649fc05992475916ecff
2021-12-16 08:46:01 +00:00
Jeremy Wharton
9d13c649a2 satellite/{console,satellitedb}: Forbid creating users with used email
This change disallows creation of users possessing the same email.
If a user attempts to create an account with an email address
that's already used - whether it belongs to an active account or not -
he will be notified of unsuccessful account creation. If he attempts to
log in using an email address belonging to an inactive account,
he will be presented with a link allowing him to re-send the
verification email. Attempting to register with an email address
belonging to an existing account triggers a password reset email.

Change-Id: Iefd8c3bef00ecb1dd9e8504594607aa0dca7d82e
2021-12-15 17:24:35 +00:00
Michał Niewrzał
911540e76e satellite/orders: avoid logging "bucketName or projectID not set"
When we switched to segments loop we stopped
adding bucket name and project ID to order limit
metadata. This is causing lots of logging that we don't need.

Change-Id: Id3f878a170b7a6b801e8a838ee69165715985d60
2021-12-14 10:31:48 +00:00
dlamarmorgan
9a37399346 satellite/metainfo: remove Gateway-ST library name from UserAgent string
When trimming the UserAgent string, Library names such as Gateway-ST
should be removed.

Change-Id: I3d71fa8514364bf1d4d79b0dc00c85aed337ab45
2021-12-13 18:07:55 +00:00
Michał Niewrzał
c33d2f58cf satellite/metainfo: override StreamMeta only if needed
For backward compatibility we are overriding pb.StreamMeta
values returned as encrypted metadata. It turns out that we
should do it not when target values are missing but when
values to override exists.

This was causing problems after move operation. Details can be
found here https://github.com/storj/uplink/issues/70

Backward compatibility test will be added to storj/uplink testsuite.

Change-Id: I72e7a01226b1dd62902cb0d6ebb1ff91a4693005
2021-12-10 08:41:37 +00:00