Commit Graph

6665 Commits

Author SHA1 Message Date
Jeremy Wharton
4e0acba9ba web/{multinode,satellite,storagenode}: Fix SVG styling
Our SVG loader was recently reworked so that snapshots of UI tests
are smaller, but this broke some SVG styles because SVG paths no longer
contain a scoped CSS attribute. This change adds deep selectors where
appropriate to repair the broken styles.

A scoped CSS attribute is added by Vue Loader to elements of components
with scoped styles to identify which component the element belongs to
and restrict CSS rules accordingly. Paths within imported SVGs no longer
contain this type of attribute, so the deep selector must be used to
style the SVG as if it were a child component.

Change-Id: If4beb903fa3b40aa208efa549b8fe23ccf2a547b
2022-06-08 21:54:59 +00:00
Jeremy Wharton
16d1296613 web/satellite: Fix new access grant flow dropdown
This change resolves an issue where clicking outside an access
grant deletion dropdown would produce errors instead of closing it.

Change-Id: I2e0d5bfc50e2abeffe2d181aa3319b3476138d07
2022-06-08 20:33:15 +00:00
Márton Elek
0831c8f390 makefile: use the ci-slim image for linting
Change-Id: I748d9fb6874a4b399985776e50202399467cfc2e
2022-06-08 14:03:10 +00:00
Egon Elbre
11dd0687c9 satellite/metabase: add missing monitoring
Change-Id: I8b1049aafc0d2c6029a68cac0e6fdc3d7b25b30d
2022-06-08 12:34:00 +00:00
Moby von Briesen
5b79970575 satellite/metainfo: Use project-level attribution at bucket level
If project.UserAgent is set, use this for bucket.UserAgent on bucket
creation. Otherwise, set bucket attribution as before (getting UserAgent
from request headers).

Tests were updated to create the bucket with a different user, added as
a project member. Otherwise, the tests do not catch the bug.

Change-Id: I7ecf79a8eac5957eed361cbea94823190f58b776
2022-06-07 17:08:50 +00:00
paul cannon
737d7c7dfc satellite/reputation: new ApplyUpdates() method
The ApplyUpdates() method on the reputation.DB interface acts like the
similar Update() method, but can allow for applying the changes from
multiple audit events, instead of only one.

This will be necessary for the reputation write cache, which will batch
up changes to each node's reputation in order to flush them
periodically.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I44cc47767ea2d9423166bb8fed080c8a11182041
2022-06-07 15:22:25 +00:00
Yaroslav Vorobiov
62774768d9 satellite/payments/storjscan: update config and wrap service errors
Change-Id: I4f75375e9499119cbcce78c5e77d1950e4bae38b
2022-06-07 14:21:42 +00:00
NickolaiYurchenko
e01daa538e web/satellite: bucket details page created
Change-Id: I0ee4dd4cabe7bad6f3d2ebdee455285529d81bdf
2022-06-07 16:49:02 +03:00
Vitalii
bb92b923d4 web/satellite: open bucket modal for new objects flow
Implemented new modal to open bucket.
This is a part of new objects flow.
It is hidden by feature flag.

Change-Id: I3839126933b8804c8adbd8949a29ac75771fab4a
2022-06-07 09:54:43 +00:00
Vitalii
f56504de2a apigen: project delete endpoint
Implemented project delete endpoint for REST API.
Added project usage status check service method to indicate if project can be deleted.
Updated project invoice status check method to indicate if project can be deleted.

Change-Id: I57dc96efb072517144252001ab5405446c9cdeb4
2022-06-07 12:23:24 +03:00
Paul Willoughby
911cc1e163 satellite/contact: reject privateIPs in PingMe and CheckIn endpoints
prevent network enumeration by rejecting privateIPs in PingMe and
Checkin endpoints

Closes storj/storj-private#32

Change-Id: I63f00483ff4128ebd5fa9b7b8da826a5706748c9
2022-06-07 08:09:14 +00:00
Vitalii
f0b28d6326 apigen: endpoint to get user
Implemented new GET user by request context endpoint.
Updated docs.

Change-Id: Iebb493e55f9456b89d7dbd234bb0b939b82b0ced
2022-06-06 16:31:19 +00:00
Vitalii
ba58530089 {satellite}/web,console,testsuite: remove old navigation structure
Removed old satellite UI navigation structure.
Removed old feature flag.

Change-Id: Ic998886cf2e30ebd44e67a20fc53888103fe4b8d
2022-06-06 16:43:05 +03:00
Michał Niewrzał
18ef3d3881 satellite/accounting: add metrics for total pending objects
We don't have metric to track how many pending objects we have in the
system. This change is using tally objects loop to collect pending
objects per bucket and at the end its combining all buckets values
into single metric for all pending objects in a system.

Change-Id: Iac7a6bfb48854f7e70127d275ea8fdd60c4eb8b7
2022-06-06 12:04:46 +00:00
hovex023
688b3907ab
web/satellite: Implement New Create Access Grant Wizard. (#4821)
Created and style modal-added open and close functionality
Added logic to access grants selection type
Changed CLI to API-Changed Types to radio buttons-Added date picker-Added buckets list(not yet populating with buckets)
Added permissions check box logic and new input field for buckets.
Added tooltips and hover logic for the tooltips
Added acknoledgement functionality to encrypt step
Added conditional logic to encrypt access button and the ascknoledgement box - removed blank comments from paymentMethods snapshot
Added logic to retrieve restricted api key, access credentials, and satellite address
Added copy functionality and download functionality for the credentials step

Change-Id: I8c8f02bc1ee38c3df42396cbd9bb3db2e7ff9cc4

Co-authored-by: cl-mitch <mitch.george@compozelabs.com>
2022-06-03 13:03:36 -05:00
Moby von Briesen
82e3932dbd web/satellite: Fix incorrect input placeholder
In the name step of the create bucket view for the new objects flow, fix
the placeholder for the input to mention "name" rather than "passphrase"

Change-Id: Ia7a64383b58d36f6aee1df952ed7cb300734e09d
2022-06-03 11:08:57 -04:00
dlamarmorgan
270204f352 satellite/{payments/storjscan,satellitedb}: Add wallet implementation
Add storjscan wallets implementation to the satellite. The wallets interface allows you to add and claim new wallets as called by the API. The storjscan specific implementation of this interface uses a wallets DB to associate the user to a wallet address, as well as a storjscan client to request and associate new wallets to the satellite.

Change-Id: I54081edb5545d4e3ee07cf1cce3d3e87cc00c4a1
2022-06-03 11:45:47 +00:00
cl-mitch
cbaca8b17e
web/satellite added new billing screen feature flag (#4836)
New feature flag implemented to enable the new billing flow.

Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
2022-06-02 09:30:27 -05:00
Ivan Fraixedes
261a6edf4f
satellite/admin/ui: Update NPM dependencies
Update all the NPM dependencies used by the Admin UI.

The dev dependencies correspond to the ones that are currently used by
an svelte app generated with the last svelte-kit version. They
deprecated some configuration options and changed some svelte
directives.

The only non-dev dependency is also updated to the last published.

Change-Id: I5f2192cab41e00efc3239237f8dc8f3d07816b63
2022-06-02 13:24:15 +02:00
Egon Elbre
763c04770f satellite/console/consoleweb: fix CSP values
script-src-elem is preferred over script-src in certain scenarios.
If it's absent, then the browser always uses script-src. By adding
script-src-elem it ended up blocking google recaptcha.

Change-Id: I9cf96e71e69054c4a034ca189db84fbe8903a59b
2022-06-02 10:44:48 +03:00
Jeremy Wharton
dba28f1d61 cmd/satellite: add project ID, bucket name to partner attribution CSV
This change adds project ID and bucket name columns to the generated
partner attribution report. Attribution values are now summed based on
their project ID and bucket name in addition to their user agent.
Additionally, the command to generate the attribution report has been
modified to optionally include only certain user agents.

Change-Id: I61a1d854379134f26b31467d9e83a787beb451dd
2022-06-01 20:27:12 +00:00
Vitalii
a61f0f6be3 satellite/projectaccounting: make daily usage query return correct allocated traffic
Updated daily project usage query to return correct allocated traffic.
If allocated egress has expired then we return settled egress.
If not then we return allocated egress - dead egress.

Fix for this issue
https://github.com/storj/storj/issues/4563

Change-Id: Ia15a50d3bb8d8cb1106936e17dbe0f1f5a40fa87
2022-06-01 15:24:52 +00:00
Vitalii
07e65cd338 satellite/projectaccounting: sum up bucket usages for daily usage query
Fixed daily usage query returning single bucket usage.
We sum up bucket usages now.
Also fixed https://github.com/storj/storj/issues/4559.

Change-Id: I2eb6299f1ef500d68150879195011b6fbb5f37ed
2022-06-01 12:50:10 +00:00
Márton Elek
84d02f7fbf metabase: use predefined snapshot for unit tests avoid migration
Change-Id: I6225ae32c312030e3559d8ed0faa137ffc3cc5b7
2022-06-01 12:07:00 +00:00
littleskunk
6cdd250019
satellite/console: enable new object flow / onboarding (#4851) 2022-06-01 14:04:03 +02:00
Ivan Fraixedes
aa913e63db cmd/storj-sim: Fix path to admin UI assets folder
The admin UI assets aren't inside of the `web` directory they are
directly in the `satellite` one.

The invalid path provoked that storj-sim generated a satellite
configuration with an invalid path to the Admin UI static assets
provoking that it didn't load the UI by default.

Change-Id: I49fb289377f51634057173690fbd8cf863ca9a9d
2022-05-31 18:54:54 +00:00
littleskunk
3d1b469f35
satellite/console: disable account activation reminder email (#4844)
* satellite/console: disable account activation reminder email
2022-05-31 17:36:40 +02:00
Márton Elek
9d8eaf3d7e metabasetest: use DROP instead of TRUNCATE
TRUNCATE requires table recreation which involves 'online schema change' with crdb.
(with psql it might be fater than DROP, that was the motivation of the original change)

`online schema change` is an async operation with crdb and it's eventually very slow therefore we try to avoid it.

This·reverts·commit·15bed0ed0e81d54fe4ffac9928bdf648f5e06ec6.¬

Change-Id: I93e1ab470962be77e3458d74c8787442c9d7bee0
2022-05-31 12:16:02 +00:00
Márton Elek
7c854fcde4 Makefile: make test run more safe
This patch suggests small improvements for the previously committed (very nice <3) unit test helper

 * Use non-default ports for postgres + cockroach. This would help to run same services for ad-hoc unit test (for IDE development for example)
 * Use the same flags for cockroach what we use for unit tests (mem store supposed to make tests faster)
 * Use `docker-compose down -v` for removing all networks and annynmous volumes
 * Remove connection limit (I have seen related problems in my jenkins runs)
 * Omit unused databases to avoid getting failures (eg. omit psql when crdb is used)
 * Do not drop cockroach database (full database will be dropped, anyway...): it also makes testing faster

Change-Id: Iadc04f8617a2825ea4f10072bd023a9c86883680
2022-05-31 10:58:26 +00:00
Michał Niewrzał
ffbb43ddbc cmd/uplink: fix how we are collecting errors while copy in parallel
Main issue was that when one part copy failed while being inside
goroutine (limiter) and another part was still collecting src/dst parts
it was possible to drop errors from failed part copy. It was possible
bacause on fail context was canceled and if we were still getting
part src/dst then it was returning error immediately and error
group with errors from goroutine was ignored.

Change-Id: I75c6799eba358741629795f2971c7a964cb2c9ce
2022-05-31 10:18:51 +00:00
Vitalii
4f668f54c0 web/satellite: fix main content width
Fixed main content width to be appropriate.
In case of very long access grant name (100+ symbols) main content gets overlapped with window border. That's an unhandled edge case.

Change-Id: I42ef709edc8454f054baa52231c605afadc471e2
2022-05-31 12:27:12 +03:00
Ivan Fraixedes
2438c92d56
satellite/admin/ui: Add disable MFA endpoint
Add the admin API endpoint for disabling an user's multifacdtor
authentication to the satellite admin UI.

Remove a couple of commented code lines too.

Change-Id: Iaee7efe7a3d4d38bdd6541311447a9726806f0f1
2022-05-30 15:23:35 +02:00
Stefan Benten
25b45c47ee
satellite/admin: allow disabling of MFA
We have a couple of support tickets so far that require us to
disable the mfa on accounts. Since we currently had no other
way than doing a SQL War Crime, it makes sense to add it to the
admin API.

Change-Id: Ib16735c1961380b04345a3495d4eebee5fa0bc41
2022-05-30 13:56:11 +02:00
Stefan Benten
62a64bae3b
satellite/admin: allow deletion of paid tier user projects when invoiced
Currently we have a bug in which we would require that a project of
a paid tier user needs to be two months unused before we can delete it.
This change fixes it and reduces it back to the normal next billing cycle.

Change-Id: I28610b6c45c68943fd4f2621233bccc06cab28a0
2022-05-30 13:54:51 +02:00
Stefan Benten
8673583582 satellite/admin: fix console config handling
An older change plummed the full console config as subconfig of
the admin api configuration in. This bloated the generated satellite
configuration unnecessarily while also allow for confusion/mistakes.

Change-Id: Icf49cc1f147711e37e85f6eac1143fab8ddf1659
2022-05-27 22:26:06 +00:00
Fadila Khadar
792c49e5ad satellite/metabase: simplify delete query
when deleting an object that has been copied multiple times, we look for an ancestor_stream_id by taking the min of all copies stream_id.

This change simplifies this process by picking any stream_id as a new ancestor by using 'distinct on'.

Fixes https://github.com/storj/storj/issues/4745

Change-Id: Iffb519b82d2ae2ed73af48fa0e86f87384e0158f
2022-05-27 20:09:50 +00:00
Cameron
b379fe89d2 satellite/core.go: hook up flag to enable email reminders
Change-Id: I973bb3a709e806e7744679b3543217c1cab85f9a
2022-05-27 13:16:14 -04:00
Vitalii
69ad49f473 satellite/server: fixed CSP for hcaptcha
Fixed CSP errors for hcaptcha

Change-Id: Ie928e206c652c97d36bcbdaf8436ae4a33afed8d
2022-05-27 14:28:31 +00:00
Michał Niewrzał
4f2fae4f28 cmd/uplink: better error handling for parallel transfer
Few improvements were made to how we are handling errors
while doing parallel upload/download for single object:
* unhide error under 'context canceled' which was shown in most of
cases
* add part number to error message
* don't try to commit if any error occurs while operation
* combine errors into more readable form, example:

---
failed to download part 3: uplink: eestream: failed to download stripe 0:
error retrieving piece 00: ecclient: piecestore: rpc: tcp connector failed: rpc: dial tcp 97.119.158.36:28967: i/o timeout
...
error retrieving piece 89: ecclient: piecestore: rpc: tcp connector failed: rpc: dial tcp 161.129.152.194:28967: i/o timeout
failed to download part 1: uplink: eestream: failed to download stripe 0:
error retrieving piece 01: io: read/write on closed pipe
...
error retrieving piece 97: io: read/write on closed pipe
failed to download part 2: uplink: eestream: failed to download stripe 0:
error retrieving piece 00: io: read/write on closed pipe
...
error retrieving piece 01: ecclient: piecestore: rpc: tcp connector failed: rpc: dial tcp 180.183.132.234:28967: operation was canceled
error retrieving piece 96: io: read/write on closed pipe
	main.(*cmdCp).parallelCopy:418
	main.(*cmdCp).copyFile:262
	main.(*cmdCp).Execute:156
	main.(*external).Wrap:123
	github.com/zeebo/clingy.(*Environment).dispatchDesc:126
	github.com/zeebo/clingy.(*Environment).dispatch:53
	github.com/zeebo/clingy.Environment.Run:34
	main.main:26
	runtime.main:250
---

Change-Id: I9bb70b3f754567761fa8d17bef8ef59b0709e33b
2022-05-27 14:00:35 +00:00
Vitalii
3f13cd7de5 web/satellite: reworked add PM and Create Project modals
Reworked "add payment method" and "create project prompt" modals to use new common VModal component.

Change-Id: I73039e5d30077e35f05501a48f39142b799da875
2022-05-27 10:18:56 +00:00
Ivan Fraixedes
54aa3adbf7 scripts: Fix lint to show out on error
`os/exec.Cmd.CombineOutput` runs the command, hence, it cannot be used
after calling the `Run` method.

Because `CombineOutput` already runs the command, we can use it directly
instead of `Run`.

Without this change if the command returns an non-zero code we get an
error because of the command already started and we don't get the
output.

Example removing a copyright notice from one file and running the linter
(only showing the affected printed line)

Without this fix

  2022/05/26 15:48:40 [/storj check-copyright] error  exec: already started

With this fix

  2022/05/26 16:22:40 [/storj check-copyright] error missing copyright certificate/doc.go: %!w(<nil>)

NOTE the `%!w(<nil)` is a bug in the check-copyright linter.

Change-Id: I40b64842028399b92a8982bfb143e1f87f92467b
2022-05-27 09:32:48 +00:00
dlamarmorgan
007d4190c2 satellite/{payments/billing,satellitedb}: Add payment billing DB
Add billing DB to the satellite. This DB will hold all transactions on the users account and can be used to compute the users current usable account balance.

Change-Id: I056416efc169e5e5e30c9f30cd8bc766b7bc8073
2022-05-27 08:56:31 +00:00
Paul Willoughby
1d97b2c855 cmd/uplink: add use - for stdout,stdin to cp help
Change-Id: Ife3a0972d1be119a73eaefc0e23407b74fe03f54
2022-05-26 10:27:20 -06:00
Vitalii
d916b26e0e apigen: create api key endpoint
Implemented new service method for generating API keys.
Implemented new endpoint.
Improved multiple endpoint groups handling.

Change-Id: Iba26fbf9123707b5b4c2d5e8c5a35d507404f24a
2022-05-26 16:00:23 +00:00
Cameron
55821605e8 satellite/console: add monkit metrics around user registraion/login
github issue: https://github.com/storj/storj/issues/4807

Change-Id: Id56ec73ec91b07b639b8011f0f916b4adbb01be6
2022-05-26 10:44:47 -04:00
Michał Niewrzał
c934f45bfc satellite/metabase: drop IterateObjectsAllVersions
We are not using this method and most probably we
won't need to list objects with all statuses at once.
Removing for now.

Change-Id: I7aa0468c5f635ee2fb1fe51db382595c6343dd9c
2022-05-26 14:12:20 +00:00
Fadila Khadar
4725a3878c satellite/metainfo: parallel deletion tests
- parallel deletion of 50 objects and their 50 copies (one copy per object)
This test is skipped because it's creating deadlocks that are not automatically retried on postgres
- parallel deletion of 1 object and its 50 copies.

Fixes https://github.com/storj/storj/issues/4745

Change-Id: Id7a28251c06bb12b5edcc88721f60bf7a4bc0492
2022-05-26 11:00:42 +00:00
Mya
b74340b1d6 scripts/lint: fix timing of lint tasks
Change-Id: I068cad315fd05ee235413d8230d454507b0a60e4
2022-05-25 13:24:03 -05:00
Mya
9153f37055 Makefile: run lint locally in docker
Our linting process depends heavily on custom tools and linting
configuration. To help improve this process for running on local
developer machines, we can run the various tasks in our existing CI
container.

Change-Id: I60407686ce9233cc4f16e3724c5e8d44367aa200
2022-05-25 12:30:15 -05:00
Mya
4b61cc9e9c Makefile: update test target to support a local execution workflow
This change attempts to run tests as close to CI as possible. It
introduces a docker-compose file that deploys some supporting
services. While this change makes it possible to run tests locally,
it does not address the core time it takes to run tests end to end.

make test/postgres  1417.87s user 263.38s system 291% cpu 9:37.73 total

Resolves https://github.com/storj/dev-enablement/issues/10

Change-Id: I4d9adc125992c1b1c133e8fe108384fa3c66ff7a
2022-05-25 16:47:06 +00:00