Commit Graph

6229 Commits

Author SHA1 Message Date
Cameron
1743a72f21 cmd/partnerid-to-useragent-migration: parallelize tests, use db test migration method
Use satellite.DB method TestingMigrateToLatest instead of
MigrateToLatest. TestingMigrateToLatest is much faster.

Also, run package tests in parallel.

Change-Id: I18bc0926dcfb80ace30d0b401e64ed919bfb966f
2022-01-31 18:37:13 +00:00
Jeff Wendling
b7ca2289e0 cmd/uplinkng: improvements to prepare for doc update
* improve setup wizard
* added access grant file support
* improved consistency across commands
* a couple bug fixes
* added access import and export

Change-Id: I30ad9d4771f15430904a503a4d465bc40be471b5
2022-01-31 17:37:54 +00:00
Cameron Ayer
9d3614fc1b cmd/partnerid-to-useragent-migration: add value_attributions migration and tests
Change-Id: I21b8a752720c1567840b58cc9380eff0b8abeece
2022-01-31 16:34:31 +00:00
Egon Elbre
64c8de6ea5 mod: use vendored base58
Change-Id: I5aa29515928848c862500330218cc094618638d7
2022-01-31 15:54:33 +02:00
Cameron Ayer
6b6e9901e2 cmd/partnerid-to-useragent-migration: add bucket_metainfos migration and tests
Change-Id: I8c207fb4f28f45a30b3ca6e02122dd701f044b03
2022-01-28 11:51:08 -05:00
Cameron Ayer
1f1a054de4 cmd/partnerid-to-useragent-migration: add api_keys migration and tests
Change-Id: Id8616fb30c09b8f502e506a65cae7367b6e0872e
2022-01-28 10:52:00 -05:00
Cameron Ayer
6d6d6776d8 cmd/partnerid-to-useragent-migration: add projects migration and tests
Change-Id: I23f931cb6ff8e047aa9b30a45b4e05c41d87c04e
2022-01-28 10:16:50 -05:00
Michał Niewrzał
d393f6094c satellite/accounting: adjust tests to expose new uplink errors
Existing tests are checking errors by comparing error message.
We plan to expose storage/segment limit errors as a public uplink
API but before that will happen we need to update tests to handle
both cases as new uplink error will have a bit different error message
than current tested error.

https://github.com/storj/uplink/issues/77

Change-Id: Id2706323c60d050d96752e66e859d4ec051a69b9
2022-01-27 15:15:07 +00:00
Egon Elbre
a1aa9f24ed private/testplanet: expose NewLogger for other tests
NewLogger contains bunch of logic that is useful for creating
a nice logger.

Change-Id: Idad57525d334f40b3faa70dbc4737b114992b0a4
2022-01-27 13:56:27 +02:00
Vitalii Shpital
b8c55fdd87 satellite/projectaccounting, satellite/console, web/satellite: implemented backend for bandwidth chart
Implemented endpoint and query to get bandwidth chart data for new project dashboard.
Connected backend with frontend.
Storage chart data is mocked right now.

Change-Id: Ib24d28614dc74bcc31b81ee3b8aa68b9898fa87b
2022-01-26 15:09:22 +00:00
Vitalii Shpital
5d6ee506b0 private/apigen: initial typings and methods
Initial typings and methods for the future API Management implementation.
Blueprint - https://review.dev.storj.io/c/storj/storj/+/6341

Change-Id: I9afe66a7610a3a80e764b317ddd211f125b1ea43
2022-01-26 14:39:22 +00:00
Jeff Wendling
67c58fdd56 cmd/uplinkng: add setup command
it's an alias to `access create` for backwards
compatibility.

Change-Id: I17b64da0b8c0ff56cd82b76af94ee578167a2269
2022-01-25 16:16:55 -05:00
Michał Niewrzał
17f56ab636 satellite/metainfo: split endpoint into smaller files (object/segment)
This is part of change to split metainfo endpoint into smaller
files. It will be grouped by bucket/object/segment/other requests.

Tests will be split as a separate set of changes.

Updates https://github.com/storj/team-metainfo/issues/12

Change-Id: I6c691e4d0e192fe3ad7974d2d0ab5ced0d272f3c
2022-01-25 16:14:37 +00:00
NickolaiYurchenko
33b9d24c47 web/satellite: NotificationItem: support link added
link to support added if support mentioned in message

Change-Id: I9da5b869b2bfed39671b9c622e5262e2a80d0f09
2022-01-25 14:31:42 +00:00
Michał Niewrzał
9ce1a4e25d satellite/accounting: use AOST to get segment usage
For better performance we should use AOST for getting
data where being up to data it not very crucial. We don't
care about small differences for segment limit calculation.

Change-Id: I9b2d4f2bd15ebc9d1c46bc84dd51a2e9d9231506
2022-01-25 11:00:08 +00:00
Malcolm Bouzi
3451162583 web/console: Save timestamp when creating users to mark their first email verification notification.
This value will be updated whenever we send a reminder for a user to verify their email.

Change-Id: I7f08b16eccc6633b8eb3662eb7efed1f1f780f44
2022-01-24 16:53:51 +00:00
Cameron Ayer
e6961e1fd7 cmd: add command for partner_id to user_agent satelliteDB migration
Value attribution codes were converted into UUIDs and stored
in the users, projects, api_keys, bucket_metainfos, and
value_attributions tables in the partner_id column. This
migration will lookup the appropriate partner name associated
with each of these UUIDs, and store the partner name directly
in the user_agent column within each table. If no corresponding
partner name exists for a partner_id, the partner_id value will
be stored instead.

Add migration for users table with tests.

Change-Id: I61254d9b81c474e76bcfc1c8cd863697c6ef44b6
2022-01-24 15:30:14 +00:00
Jeremy Wharton
c968bd0b73 docs/blueprints: Create API renovation blueprint
This change adds a blueprint outlining the renovation of the
current API system including guidelines for API code generation.

Change-Id: I5560cc8642eee44842829ecddf205886cd5b13f2
2022-01-24 14:24:31 +00:00
Michał Niewrzał
65c38c6cef satellite/metainfo: split metainfo endpoint into smaller files (buckets)
This is part of change to split metainfo endpoint into smaller
files. It will be grouped by bucket/object/segment/other requests.

Tests will be split as a separate set of changes.

Change-Id: I9b097dcc8fa889f985b7f4ef5f8f435a1ff0ef95
2022-01-24 13:47:51 +00:00
Michał Niewrzał
913718ef97 satellite/accounting: test bandwidth limit without cache
Additional test case to cover situation when cache value
expired and we need to get this value directly from DB.

Change-Id: I07bda67e19333e09a567104ce70f112fd47a7845
2022-01-24 13:24:50 +00:00
Egon Elbre
be02aa9b17 satellite/overlay: add test for UpdateCheckIn panic
The database table got invalid input and the resulting error
was not checked. This adds updates that contain invalid fields
to trigger different errors.

Change-Id: Iacea32cbef5599aab562c88e4113073596cc9996
2022-01-24 15:01:59 +02:00
Michał Niewrzał
4be84c3a6c satellite/accounting: fix how we put segments usage into cache
We had two problems here. First was how we were handling
errors message from GetProjectSegmentUsage. We were always
returning error, even for ErrKeyNotFound which should be used
to refresh value in cache and not propagated out of method.
Second, we were using wrong value to update cache. We used
current value from cache which obviously it's not what we intend.

Fixes https://github.com/storj/storj/issues/4389

Change-Id: I4d7ca7a1a0a119587db6a5041b44319102ef64f8
2022-01-24 11:57:50 +00:00
Michał Niewrzał
bfad42f29a satellite/metainfo: split metainfo endpoint into smaller files
This is part of change to split metainfo endpoint into smaller
files. It will be grouped by bucket/object/segment/other requests.

Tests will be split as a separate set of changes.

Change-Id: I5128c84e06c82777fe71460bf5f9a6e26e52a243
2022-01-24 08:24:24 +00:00
Stefan Benten
e7e39fe628
satellite/metainfo: reduce default user rate limit to 100
Currently the rate limit has kept per satellite api endpoint.
Since we run 9+ api endpoints in production, we do not need
a limit of 1000, since the intention was to allow 1000 total.
This change reduces the effective limit given 9 instances
down to 900, which should be close enough.

Change-Id: Ia579149ccc3a12e8febe0cfd5586b8a39de40f55
2022-01-21 23:37:59 +01:00
Michał Niewrzał
1c47163eee satellite/metainfo: add more logging for CreateBucket
We were returning pure non rpc errors in two cases.
This change added loging and correct rpc error as a
return.

Change-Id: I581ceb17dcdc00921dfa3c1057015c3b4d04308d
2022-01-21 15:51:37 +00:00
Malcolm Bouzi
c30c7def44 satellite/{console,payments},web/satellite: save signup promo code to newly registered users
Users signing up through a url containing a promo code will have that code applied to their stripe account instead of the free tier coupon.

Change-Id: I071041b0934648ef3f5bdb05b6ec97c400f89ae4
2022-01-21 15:02:22 +00:00
Cameron Ayer
251e432e5b satellite/satellitedb: drop contained from nodes and reputations
A few months ago we removed all references to the contained
column in nodes and reputations

bb21551a9c
and
56fe636123

But we never did the migration to drop the columns.
This commit will finally do that.

Change-Id: I82aa2f257b1fb14a2f1c4c4a1589f80895360ae4
2022-01-20 19:46:48 +00:00
Moby von Briesen
464b4b02fa satellite/console/consoleweb: Only replace some kinds of characters in name
The previous change (59648dc272) ends up removing a lot of characters
for valid non-English names. Instead, only replace URL characters such
as slashes, colons, and periods. Since someone may use these characters
to separate two parts of a name, e.g. Name1/Name2, replace these
characters with a hyphen.

Change-Id: I4cc3d1bdb05d525a83970cf1b42479414c9678e7
2022-01-20 11:13:26 -05:00
Clement Sam
2de6feefde satellite/contact: fix connection leak in PingMe endpoint
Fixes connection leak in pingMe endpoint.
includes other minor fixes.

Change-Id: I7c61f620565f46dd113d21a772de7c439be550e3
2022-01-20 14:20:24 +00:00
Clement Sam
8b40a071a0 storagenode: check if QUIC is properly configured
This change adds a check to confirm if UDP port if properly configured for QUIC

Resolves https://github.com/storj/storj/issues/4332
Partly resolves https://github.com/storj/storj/issues/4358

Change-Id: I9a66f26a115e48b4fcd168f50a7d0b4d81712f4e
2022-01-20 12:04:04 +00:00
Moby von Briesen
59648dc272 satellite/console: Filter special characters out of name
When a user is created, but before verification or forgot password email
is sent, remove any special characters in the provided name. This
protects us against certain phishing attacks.

Change-Id: Ieddd3479da20eb80b9f1b56eb86c8f46bca2642c
2022-01-20 00:18:56 +00:00
Clement Sam
9f3c1f9cda satellite/contact: add PingMe endpoint
Change-Id: I832a72fafeacf76ad64a0129bcc6582cc4f9290d
2022-01-19 17:52:33 +00:00
Michał Niewrzał
95750b9d32 satellite/console: verify limits while test
Additional test case to be sure that update operation went well.

Change-Id: I7dbf1e3110f716712ba9747a1915b54112b89801
2022-01-19 16:39:36 +00:00
NickolaiYurchenko
01287913ec web/satellite: graphql error logging added
most of project related errors fixed to show correct message
found out that update project mutation is broken

Change-Id: Ia3492d34e9a9ef0ce0f0ea4591bd48d5cdbb5f5a
2022-01-19 14:06:05 +00:00
Qweder93
228f465d45 satellite/metainfo: сombine checks for storage and segment limit
We need to combine methods from accounting.Service (ExceedsStorageUsage and ExceedsSegmentUsage)
to run checks concurrently.

Resolves https://github.com/storj/team-metainfo/issues/73

Change-Id: I47831bca92457f16cfda789da89dbd460738ac97
2022-01-19 10:44:04 +00:00
Michał Niewrzał
270c6e24f2 satellite/console: fix default limit for free-tier segment usage limit
Free-tier segment usage limit was defined as 150k, not 140k. This change
is correcting that.

https://github.com/storj/team-metainfo/issues/8

Change-Id: I71ec0961930b19fd09b2b996e01acd406a8dcf8f
2022-01-19 09:21:13 +00:00
NickolaiYurchenko
a81af2829e web/satellite: error while adding credit card handled
Change-Id: I995eb4f07a62c26052226ddbbf06ee332323a0dd
2022-01-18 17:53:07 +00:00
Mya
309ac41bfc satellite/satellitedb/dbx: add schema for oauth tokens
Change-Id: Ia980766a8f86b662545e7d8cae6ecf631e199809
2022-01-18 09:38:12 -06:00
Mya
83f7d3e045 satellite/satellitedb/dbx: add schema for oauth clients
Change-Id: Ib08c7074803805cca8ffaedbd89da54504842af7
2022-01-18 09:38:12 -06:00
Michał Niewrzał
36c07720bb satellite/metainfo: bound user agent metrics
We need to limit number of results for SeriesTag to avoid
overloading our metric system.

Change-Id: I3d9f4f4f6d489bf4ce87c6240f66d48b29b88abb
2022-01-18 12:42:09 +00:00
Jeremy Wharton
1007e6300e web/satellite: Distinguish between file and folder upload buttons
Previously, there was no difference between the "Upload File" and
"Upload Folder" buttons in the Object Browser aside from the text.
This change adds file and folder upload icons to their respective
buttons.

Change-Id: If9766c12420d9a9bc45348c0b9b0bdf6b42b8922
2022-01-16 02:15:00 +00:00
Mya
cf462fcf92 satellite/admin: update geofence endpoints to follow response conventions
Change-Id: Ieff91c16f91f9720e9c3d8ead0e13c47651d23f5
2022-01-13 12:33:10 -06:00
Michał Niewrzał
6bbe2fc692 cmd/uplinkng: fix flag/error messages
This change fixes some small issues with flag descriptions
and error messages.

Change-Id: I74a669d2566bf51ad871f16d64a92984ae75e712
2022-01-13 13:36:11 +00:00
Erik van Velzen
4f96a85642 cmd/uplink/share: register access via DRPC
Convert registering access with the edge services
from the HTTP protocol to DRPC protocol

Change-Id: Iba88dd0758c26f613cf501be9a20ead07d122d0b
2022-01-13 13:24:05 +00:00
Michał Niewrzał
5e9643e1b8 satellite/metainfo: cleanup validation
Refactoring to do few things:
* move simple validation before validations with DB calls
* combine validation check/update for storage and segment
limits together

Change-Id: I6c2431ba236d4e388791d2e2d01ca7e0dd4439fc
2022-01-13 13:00:40 +01:00
Michał Niewrzał
18ab464b51 satellite/metainfo: UpdateProjectSegmentUsage enabled only with segment validation
updating project segment usage entry to redis is enabled only if
segment validation is enabled in config value, so no incompatible
changes will be added while deployment.

Change-Id: I1288cb9ff0a8a00f095dc94e20d2f14393e9a613
2022-01-13 12:12:25 +01:00
Ivan Fraixedes
3a987ac00f satellite/admin: Fix client API & docs
The following 2 commits added 2 new query parameters to set the `burst`
and `segments` limits for a project and also to new fields to the
response JSON object body to the "get project limits" endpoint:

* c911360eb5
* b7b010adc9

However, the API documentation and the Typescript client API (used by
the UI) weren't updated with them.

Later, the commit dc6128e9e2 updated the
Typescript client API with the `segments` limit but it didn't update the
documentation to reflect it.

This commit brings all things that were missed in those previous
commits.

Change-Id: Iff12cdd4a0d3c448cd73b57a98d171ba468d2c98
2022-01-13 09:31:01 +00:00
Ivan Fraixedes
8e80a51b64 Makefile: Set NodeJS version to use in var
Add new variable in the Makefile to specify the used NodeJS version so
it can be referenced wherever else in the file rather than hard-coding
it.

Having in to a variable makes more visible and quicker to find out which
NodeJS version the project uses.

Change-Id: I87abc3b3e5489407638618f8199aec7c2f40ca9f
2022-01-13 08:52:26 +00:00
Stefan Benten
44a73b01d9 cmd/storagenode: use contact address instead of server address
Currently the address being used is most of the time just :28967, which is not the correct address to reach the node from the public on.
This change uses the designated contact external address value that contains the set and preferred way to reach the node.

Change-Id: I99e979c2541043755b81e65c36c4289bfa3f60f3
2022-01-12 15:15:09 +00:00
Michał Niewrzał
eb0d08d59b satellite/metainfo: collect versions of user tools
We want to know usage statistics for our main tools
like uplink-cli or rclone. Initially we will collect
only usage stats without relation to specific process
e.g. download or upload.

Change-Id: I203b1a6c07ae014e710368f77163f13fdf10763c
2022-01-12 13:49:15 +00:00