Commit Graph

6364 Commits

Author SHA1 Message Date
NickolaiYurchenko
c6add65e75 web/satellite: download link hidden
download performed without adding link to address bar
added warning to not share link with external people

Change-Id: Ifa79582c51c77c000342c1e03e752dd157912975
2022-02-15 08:17:32 +00:00
Michał Niewrzał
ddb742daff satellite/metrics: speed up tests
For some reason very small interval for chore is causing
very long execution time for Pause() method and that was
the reason why a simple test was slow.

Change-Id: Iebfdea16e31f9865493de3e1bd7d69c6f9116fc0
2022-02-14 17:24:16 +00:00
Márton Elek
a2dd10ce14 web/storagenode: support polygon with transaction links
This patch fixes the link of "Transaction" on the storagenode web
console when polygon is used.

Tested with storj/storj-up based cluster and generating test payments.

Change-Id: Ia88954e055c6f41b92d77052afffebe633da549c
2022-02-14 13:44:26 +00:00
Cameron
eefef2eb17 satellite/console: pass entire auth.User into users.Update in service.UpdateAccount
The Update method of the usersDB takes a console.User as an argument.
To update the columns in the DB, we have to migrate the fields from the
console.Users struct into a special dbx struct. If one of these fields
is left empty, then the zero value of that field's type will be used to
update the respective column.

In most cases where the users.Update method is called, the entire
console.User is apparently retrieved first, fields are updated, then it
is passed to users.Update. This is not the case for
service.UpdateAccount. Because these fields are not populated in the
user struct in UpdateAccount before it is passed into users.Update,
their respective columns in the database are overwritten with zero:

ProjectLimit, ProjectStorageLimit, ProjectBandwidthLimit,
ProjectSegmentLimit, PaidTier, MfaEnabled, MfaRecoveryCodes, MfaSecretKey

Solution: Do what is done in other places which call users.Update. Take
the console.User from the auth context, update the relevant fields on
that, then pass that in.

Change-Id: I3cbd560e8ea5397e5c27711fb40bb3907d987028
2022-02-11 15:33:04 +00:00
Erik van Velzen
0796653b07 cmd/uplinkng: registeraccess via libuplink
Change the implementation of register and share so that it uses the
uplink method to contact the Auth Service. The network protocol switches
from HTTP to DRPC.

Closes https://github.com/storj/storj/issues/4324

Change-Id: Ib8fdb1665c6385bb39a546ba46a8df43a136df9c
2022-02-11 11:36:45 +00:00
Mya
56c668a2e3 private/testplanet: use info level logs in jenkins
Change-Id: Ie500a7bfcd23ca7470139341e961b2e9986d6a33
2022-02-10 11:47:06 -06:00
Fadila Khadar
b1d9a4426f satellite/metabase: adjust segment_copies table primary key.
segment_copies table has (stream_id, ancestor_stream_id) as primary key,
but it should have only stream_id as primary as a stream_id can only have one ancestor_stream_id.

https://github.com/storj/team-metainfo/issues/84

Change-Id: I0218295e86e449c9dfefa5a21319f3083bf80afd
2022-02-10 13:52:16 +00:00
Vitalii Shpital
07c71e34c2 satellite/console{gen}: GetUsersProjects endpoint
Initial implementation of auto-generated GetUsersProjects endpoint

Change-Id: If41bff2ea3ff9cfc87afeda9e5e5b3f586cbab33
2022-02-10 08:03:02 +00:00
Bill Thorp
cf03209c16 satellite/metainfo: only collect metric "other" once per user agent
A user-agent string can contain multiple "products", in the case of
Gateway-MT at least this includes the HTTP client's full user agent.
This means that "other" is often logged even when we know the Storj
product, and sometimes logged more than once per call to "collect".

This makes sure that "other" is only logged if a product isn't
identified, and only logged once.

Change-Id: I8536f7eb32877e36fec97dab7b8d477ccb10f92e
2022-02-09 18:18:55 +00:00
Jeff Wendling
9061dd309f cmd/uplinkng: become cmd/uplink
Change-Id: If426c32219d32044d715ab6dfa9718807f32cb9f
2022-02-09 17:02:21 +00:00
Vitalii Shpital
9d52112af8 testsuite/ui/satellite: test for new project dashboard
Implemented initial go-rod test for new project dashboard.

Change-Id: If3b0f936b5af791f8500d0911c6559fa367f8be5
2022-02-09 15:06:28 +00:00
paul cannon
3540f9b3ad satellite/satellitedb: phase 2 getting rid of gob encodings in db
For a thorough explanation of the overall transition, see the message on
commit c053bdbd70.

This change will rename the columns containing gob-encoded big.Floats
and add new columns which will contain the equivalent data in a more
sql-friendly format.

The change should *not* break already-running satellite processes,
because all functionality touching these tables has already been taught
to work with these new columns if it sees any "undefined column" errors.

Change-Id: I229324376533e383c5d05064b8aedad149cf825b
2022-02-09 12:25:46 +00:00
Vitalii Shpital
150be885b7 satellitedb/projectaccounting, web/satellite: reworked bandwidth chart to show both allocated and settled bandwidth
Extended DB query.
Reworked charts UI functionality to show 2 lines if necessary.

Change-Id: I8ac4e4fa07676fc9fa7e9c078ecdeed62233b8e2
2022-02-08 22:07:59 +00:00
Stefan Benten
2d3c417c92 satellite/metainfo/version_collector: adding more known user agents:
transfer-sh will be set as of https://github.com/dutchcoders/transfer.sh/pull/467.
filezilla needs to be verified and duplicati is set per info from @TopperDEL
comet and orbiter are added as preparation.

Change-Id: I44d730a7b3ba1969068e48c2477b478831799cd1
2022-02-08 16:24:55 +00:00
Mya
0164682c37 satellite/oidc: move oidc into common package
Change-Id: I77702e0e46f15a09fee315b9076638e1412836f7
2022-02-08 09:46:54 -06:00
Michał Niewrzał
95921b8b39 satellite/metabase: add segment_copies table
Part of server-side copy implementation.

Adds new table to metabase for keeping references between
copied segments and its ancestors.

Fixes https://github.com/storj/team-metainfo/issues/84

Change-Id: I436d4b533a3d951bcd752e222e3b721cee7f0844
2022-02-08 11:15:19 +00:00
Stefan Benten
06944f062d satellite/{admin,payments,satellitedb}: add checks for deletion of free tier accounts
This change adds some more checks to the deletion process for projects and
users, since we ran into a race condition during invoicing, where projects
have been deleted before the invoicing was finished, leading to missing
references.
This PR changes the logic to block user deletion if we are in exactly that period,
while also allowing the deletion of projects/users on free tier during the month.

Change-Id: Ic0735205e6633762fb7e3c2fa13e744cdfa5ec32
2022-02-08 10:11:31 +00:00
Jeff Wendling
4f4b67c6f6 cmd/uplinkng: add quic support
Change-Id: Ia7b4569ed39ef79ad085358e711f3f3fc0d8c398
2022-02-07 15:22:43 -05:00
Vitalii Shpital
b3e1be37ff satellite/projectaccounting: query to get daily project usage by date range
Finished implementing queries for both bandwidth and storage using pgx.Batch.
Fixed CSP styling issue.

Change-Id: I5f9e10abe8096be3115b4e1f6ed3b13f1e7232df
2022-02-04 22:32:37 +00:00
Erik van Velzen
4a26f0c4f1 cmd/storagenode: restore passing arguments
Through `docker run storjlabs/storagenode:latest --help` we have always
made available around 100 command-line arguments.

However if you now pass such an argument it will be passed to
storagenode-update and it may no longer be recognized. This will cause
the storagenode not to start.

This was introduced in
https://review.dev.storj.io/c/storj/storj/+/5426

This change restores previous functionality.

Change-Id: I06823283ff82ffda12aee48c4d83717bddfbfdac
2022-02-04 02:48:39 +00:00
Ivan Fraixedes
7f1dc74e37
cmd/storagenode: Change order load id in setup
Change the order of when the storage node setup node loads the identity
for avoiding to write anything in the disk in the case that there is an
error loading the identity.

This bug was reported by @onionjake Github username's and the specific
changes to make.

Closes #4387 #4396

Change-Id: I360fff3c23b160c9e055203d3526d749edfd9129
2022-02-03 16:00:29 +01:00
Egon Elbre
11a35520c3 mod: bump common, quic, golang/x packages
Change-Id: I3b1f8ec69bac59b60665f8e59e4183dd4a0a221b
2022-02-03 12:20:43 +00:00
Vitalii Shpital
3c8e41e665 web/satellite: get object map and preview by signed request.
There is a sev-2 issue to add more browser caching.
In this PR I made object map and object preview to be fetched by signed request with non-public credentials using AWS SignatureV4 package.

Change-Id: Ib5013fa6d6af3faa97eed5168c11a13f9629cd87
2022-02-03 11:31:09 +00:00
NickolaiYurchenko
778e7e100d web/satellite: logout after password reset added
Change-Id: I2522161e9cc7f657cd4df89d136f425239f10852
2022-02-02 17:38:06 +00:00
NickolaiYurchenko
3cd8e46e75 web/satellite: new project dashboard: buckets sections added
Change-Id: Ie3bb20e6d28b31b336413e7ba9028753de3853f7
2022-02-02 16:18:16 +00:00
Mya
4d0dd737b0 satellite/admin: add endpoints for oauth clients
Change-Id: I26aa81266f494be8aab0b5523217bad9405037a0
2022-02-02 09:47:52 -06:00
Michał Niewrzał
2e31ef3f29 satellite/metabase: better error message while move
Before this change we were returning full DB error message.
That can be very confusing for end user. This change is translating
error message into more user frindly version and fixes also DRPC
error status code.

Fixes https://github.com/storj/team-metainfo/issues/76

Change-Id: I29b06ab4ba50a0d14db7a822a2906d95d65ab524
2022-02-02 15:03:24 +00:00
Michał Niewrzał
bc161794fc satellite/metabase: drop DeleteObjectLatestVersion method
This method was never used, except tests.

Change-Id: Idc1e69b2e2971995b5c4e6cf78a2b5fc69f39ad2
2022-02-02 14:33:48 +00:00
Clement Sam
7e63afbef6 storagenode: docker image autoupdate binaries
Get storagenode and storagenode-updater binaries during
run of the container to not to release new docker image
on each new version of the storagenode binary.

Fixes https://github.com/storj/storj/issues/4176

Change-Id: I994c4942136a2cc7298eb0346238689eb406ae5b
2022-02-02 11:40:04 +00:00
Michał Niewrzał
5041ee0abc satellite/metabase: drop GetObjectLatestVersion method
We don't have different version of object than 1 so at the moment
this method is not needed. Also using GetObjectExactVersion
should be slightly more performant.

Change-Id: I78235d8ae22594cc1d6345dabcc915f41cd7797b
2022-02-02 09:40:53 +00:00
Michał Niewrzał
dffa7845f6 satellite/metainfo: split endpoint tests
We already split main code base, now we need to split test
to reflect new files structure (bucket/object/segment/other).

Fixes https://github.com/storj/team-metainfo/issues/12

Change-Id: Ica1054c4fc7df764483b03f204b4beba094df8e1
2022-02-02 07:46:14 +00:00
JT Olio
02bbcd7fbb satellite-ui: update wizard to instruct uplinkng usage
Change-Id: I30cc563d96b6b7a163918b64e9bd72b8e29ef65c
2022-02-01 18:22:04 +00:00
Michał Niewrzał
3a7fb4588f satellite/satellitedb: users/projects segment limit migration
We have here two migrations in fact. One is for existing users,
we need to check if its paying user (paid_tier) and set 1M for
them and 150K for others.
Second migration is to set limits for projects depends on owner.
If owner is a paying user (paid_tier) then project should have
1M limit, otherwise it should be 150K. In this case to make
migration faster initially projects table segment_limit was set
to 1M by default. With migration we are selecting all paying
users and we are setting 150K limit for all projects which owners
are not in paying users set.

Initially we had a concern if that query wil lbe quick enough to
be executed during deployment but after investigation CRDB
team confirms that this should take seconds for out DBs.

Fixes https://github.com/storj/team-metainfo/issues/70

Change-Id: I8be06e9f949b68b993e043cc15525e8483bf49ea
2022-02-01 14:35:12 +00:00
Márton Elek
2ae78db660 satellite/email: add delimiter to close the last part of Multipart emails
The existing implementation doesn't send proper rfc1341 compatible mails
according to https://www.w3.org/Protocols/rfc1341/7_2_Multipart.html as
the closing `--boundary_id--` is missing from our mails. (see 7.2.1 from
the standard).

Mailservers which are not very flexible, deny the mails.

Example mail log: (see the
missing boundary delimiter at the end).

```
Subject: Activate your email
From: "Storj DCS - EU1" <noreply@eu1.storj.io>
To: "..." <...>
...

--26d7220f6f1c9f6fb47b535319eb15dce513bb6e1d941a0efddf25e96712
Content-Type: text/html; charset=UTF-8

....

</body>
</html>
.
smtp: DATA error	{"msg_id":"6d82c9e3","reason":"unexpected EOF"}
smtp: 554 5.0.0 Internal server error (msg ID = 6d82c9e3)
```

This patch moves all the Multipart writing to a function to make sure
that the `wr.Close()` (which writes out the last part) is executed
BEFORE `body.Bytes()` (defer added it AFTER `body.Bytes()` was calculated)

Change-Id: I8f18fc81b1857b646470eab32e73d6cbdc50d2ad
2022-02-01 09:55:29 +00:00
Cameron
0d03473e00 cmd/partnerid-to-useragent-migration: LIMIT 1 in MigrateUsers id select query
The select statement is scanning the entire table.
Limit to one row.

Change-Id: Ie50d9777fb054b197484a83a468d67fa24096bb5
2022-01-31 23:57:30 +00:00
Cameron
1743a72f21 cmd/partnerid-to-useragent-migration: parallelize tests, use db test migration method
Use satellite.DB method TestingMigrateToLatest instead of
MigrateToLatest. TestingMigrateToLatest is much faster.

Also, run package tests in parallel.

Change-Id: I18bc0926dcfb80ace30d0b401e64ed919bfb966f
2022-01-31 18:37:13 +00:00
Jeff Wendling
b7ca2289e0 cmd/uplinkng: improvements to prepare for doc update
* improve setup wizard
* added access grant file support
* improved consistency across commands
* a couple bug fixes
* added access import and export

Change-Id: I30ad9d4771f15430904a503a4d465bc40be471b5
2022-01-31 17:37:54 +00:00
Cameron Ayer
9d3614fc1b cmd/partnerid-to-useragent-migration: add value_attributions migration and tests
Change-Id: I21b8a752720c1567840b58cc9380eff0b8abeece
2022-01-31 16:34:31 +00:00
Egon Elbre
64c8de6ea5 mod: use vendored base58
Change-Id: I5aa29515928848c862500330218cc094618638d7
2022-01-31 15:54:33 +02:00
Cameron Ayer
6b6e9901e2 cmd/partnerid-to-useragent-migration: add bucket_metainfos migration and tests
Change-Id: I8c207fb4f28f45a30b3ca6e02122dd701f044b03
2022-01-28 11:51:08 -05:00
Cameron Ayer
1f1a054de4 cmd/partnerid-to-useragent-migration: add api_keys migration and tests
Change-Id: Id8616fb30c09b8f502e506a65cae7367b6e0872e
2022-01-28 10:52:00 -05:00
Cameron Ayer
6d6d6776d8 cmd/partnerid-to-useragent-migration: add projects migration and tests
Change-Id: I23f931cb6ff8e047aa9b30a45b4e05c41d87c04e
2022-01-28 10:16:50 -05:00
Michał Niewrzał
d393f6094c satellite/accounting: adjust tests to expose new uplink errors
Existing tests are checking errors by comparing error message.
We plan to expose storage/segment limit errors as a public uplink
API but before that will happen we need to update tests to handle
both cases as new uplink error will have a bit different error message
than current tested error.

https://github.com/storj/uplink/issues/77

Change-Id: Id2706323c60d050d96752e66e859d4ec051a69b9
2022-01-27 15:15:07 +00:00
Egon Elbre
a1aa9f24ed private/testplanet: expose NewLogger for other tests
NewLogger contains bunch of logic that is useful for creating
a nice logger.

Change-Id: Idad57525d334f40b3faa70dbc4737b114992b0a4
2022-01-27 13:56:27 +02:00
Vitalii Shpital
b8c55fdd87 satellite/projectaccounting, satellite/console, web/satellite: implemented backend for bandwidth chart
Implemented endpoint and query to get bandwidth chart data for new project dashboard.
Connected backend with frontend.
Storage chart data is mocked right now.

Change-Id: Ib24d28614dc74bcc31b81ee3b8aa68b9898fa87b
2022-01-26 15:09:22 +00:00
Vitalii Shpital
5d6ee506b0 private/apigen: initial typings and methods
Initial typings and methods for the future API Management implementation.
Blueprint - https://review.dev.storj.io/c/storj/storj/+/6341

Change-Id: I9afe66a7610a3a80e764b317ddd211f125b1ea43
2022-01-26 14:39:22 +00:00
Jeff Wendling
67c58fdd56 cmd/uplinkng: add setup command
it's an alias to `access create` for backwards
compatibility.

Change-Id: I17b64da0b8c0ff56cd82b76af94ee578167a2269
2022-01-25 16:16:55 -05:00
Michał Niewrzał
17f56ab636 satellite/metainfo: split endpoint into smaller files (object/segment)
This is part of change to split metainfo endpoint into smaller
files. It will be grouped by bucket/object/segment/other requests.

Tests will be split as a separate set of changes.

Updates https://github.com/storj/team-metainfo/issues/12

Change-Id: I6c691e4d0e192fe3ad7974d2d0ab5ced0d272f3c
2022-01-25 16:14:37 +00:00
NickolaiYurchenko
33b9d24c47 web/satellite: NotificationItem: support link added
link to support added if support mentioned in message

Change-Id: I9da5b869b2bfed39671b9c622e5262e2a80d0f09
2022-01-25 14:31:42 +00:00
Michał Niewrzał
9ce1a4e25d satellite/accounting: use AOST to get segment usage
For better performance we should use AOST for getting
data where being up to data it not very crucial. We don't
care about small differences for segment limit calculation.

Change-Id: I9b2d4f2bd15ebc9d1c46bc84dd51a2e9d9231506
2022-01-25 11:00:08 +00:00