Commit Graph

6626 Commits

Author SHA1 Message Date
JT Olio
098ee71e03 test-sim-versions: write what bash step is executing
Change-Id: If1231f69601c1774bc6ce116f45105cc1b2f4e22
2022-06-10 19:33:03 +00:00
Yaroslav Vorobiov
da97fafb0c satellite/payments/storjscan: add payments DB
Change-Id: Ia6f0f62912c9a60d2c513459959634179eb9506c
2022-06-10 13:44:27 +01:00
prerna-parashar
cc0518f473
satellite/analytics: Added segment.io page calls to track all the pages (#4880)
satellite/analytics: send analytics 'page visit' api requests when the user navigates around the UI
2022-06-09 11:54:23 -07:00
Cameron
b4c77d89a6 web/satellite: add free/pro tier badge to account area
Add a badge to the account area indicating whether the user is paid tier
or free tier.

github issue: https://github.com/storj/storj/issues/4747

Change-Id: I3d435113fada48d4bda6faeef27680eb2850427b
2022-06-09 18:02:57 +00:00
Cameron
5cfa7ca460 satellite/console: add cutoff to email reminders
There are multiple entries in the users table with the same email
address. This is because in the past users were able to register
multiple times if the email was not verified. This is no longer
the case. If a user tries to register with an unverified email
already in the DB, we send a verification email instead of
creating another entry. However, since these old entries in the
table with duplicate emails were never cleaned up, the email
reminder chore will send out email verification reminders to them.
A single person will get one separate email per entry in the DB
with their email and where status = 0.

Since the multiple entries with the same email problem was solved
a while ago, just add a constraint to GetUnverifiedNeedingReminder
to only select users created after a cutoff. Once the DB is
migrated to remove the duplicate emails, we can remove the cutoff.

github issue: https://github.com/storj/storj/issues/4853

Change-Id: I07d77d43109bcacc8909df61d4fb49165a99527c
2022-06-09 16:45:35 +00:00
JT Olio
c761acc0b5 web/ratelimiter: support disabling
Change-Id: I6f2d77016c4917389514b75b5eab64223cc69b2d
2022-06-09 15:33:35 +00:00
Michał Niewrzał
7e387af010 cmd/uplink: add buffering while writing to stdout
Current pipelining to stdout is synchronous so we don't have any
advantage from using --parallelism flag. This change adds buffer
while writing to stdout. Each part is first read into the buffer
and flushed only when all data was read from this part.

https://github.com/storj/uplink/issues/105

Change-Id: I07bec0f4864dc4fccb42224e450d85d4d196f2ee
2022-06-09 15:10:04 +00:00
Márton Elek
7608ea7af0 Makefile: make test should work with any directory name
docker-compose exec is more reliable as it doesn't depend on the name of the directory (the first part of the container name).

I tried to run the `make test` comman on Jenkins, but it failed as the checkout directory was storj_main (or storj_debug).

Change-Id: I85b060ebd0705a2a9fcaefa17033c61a0509c4a2
2022-06-09 13:37:56 +00:00
Márton Elek
35c022ebf9 satellite/accounting: don't use testplanet in rollup_test if not necessary
TestRollupNoDeletes is very flaky (passes locally but fails in the main branch build).

The exact reason is not clear, but stopping the loop seems to be async, the following lines may not stop the loops immediatelly which is a potential problem:

```
satellitePeer.Accounting.Rollup.Loop.Pause()
satellitePeer.Accounting.Tally.Loop.Pause()
```

Fortunatelly these test check only the database interfaces. Instead of testplanet.Run we can run only satellitedbtest.Run which is faster and more predictable (no background loops).

Other potential problem: comment claims that the default of DeleteTallies is false:

```
// In testplanet the setting config.Rollup.DeleteTallies defaults to false.
```

But it seems to be true (rollup.go):

```
DeleteTallies bool          `help:"option for deleting tallies after they are rolled up" default:"true"`
```

This is also fixed in the patch (as we need set it explicit), but TBH it can be fixed with testplanet, too.

Change-Id: Id7ec80d5c069bed2c556f4d001c71aa23fc5af23
2022-06-09 13:11:45 +00:00
Vitalii
cce0a18bf8 web/satellite: merge HeaderedInput and HeaderlessInput components
Merged two components into single one to have single source of truth.
Also this may fix some go-rod test problems.

Change-Id: Iffa86d1b3c24a0e2a551335ecda721a93ff616e3
2022-06-09 10:27:59 +00:00
Vitalii
289be4f1e9 web/satellite: removed confusing line smoothing for chart lines
Fix for this issue
https://github.com/storj/storj/issues/4564

Removed line smoothing for chart lines.

Change-Id: Idb1b9c6786307c48234487517a56fe1a8b76ae98
2022-06-09 08:40:45 +00:00
Jeremy Wharton
4e0acba9ba web/{multinode,satellite,storagenode}: Fix SVG styling
Our SVG loader was recently reworked so that snapshots of UI tests
are smaller, but this broke some SVG styles because SVG paths no longer
contain a scoped CSS attribute. This change adds deep selectors where
appropriate to repair the broken styles.

A scoped CSS attribute is added by Vue Loader to elements of components
with scoped styles to identify which component the element belongs to
and restrict CSS rules accordingly. Paths within imported SVGs no longer
contain this type of attribute, so the deep selector must be used to
style the SVG as if it were a child component.

Change-Id: If4beb903fa3b40aa208efa549b8fe23ccf2a547b
2022-06-08 21:54:59 +00:00
Jeremy Wharton
16d1296613 web/satellite: Fix new access grant flow dropdown
This change resolves an issue where clicking outside an access
grant deletion dropdown would produce errors instead of closing it.

Change-Id: I2e0d5bfc50e2abeffe2d181aa3319b3476138d07
2022-06-08 20:33:15 +00:00
Márton Elek
0831c8f390 makefile: use the ci-slim image for linting
Change-Id: I748d9fb6874a4b399985776e50202399467cfc2e
2022-06-08 14:03:10 +00:00
Egon Elbre
11dd0687c9 satellite/metabase: add missing monitoring
Change-Id: I8b1049aafc0d2c6029a68cac0e6fdc3d7b25b30d
2022-06-08 12:34:00 +00:00
Moby von Briesen
5b79970575 satellite/metainfo: Use project-level attribution at bucket level
If project.UserAgent is set, use this for bucket.UserAgent on bucket
creation. Otherwise, set bucket attribution as before (getting UserAgent
from request headers).

Tests were updated to create the bucket with a different user, added as
a project member. Otherwise, the tests do not catch the bug.

Change-Id: I7ecf79a8eac5957eed361cbea94823190f58b776
2022-06-07 17:08:50 +00:00
paul cannon
737d7c7dfc satellite/reputation: new ApplyUpdates() method
The ApplyUpdates() method on the reputation.DB interface acts like the
similar Update() method, but can allow for applying the changes from
multiple audit events, instead of only one.

This will be necessary for the reputation write cache, which will batch
up changes to each node's reputation in order to flush them
periodically.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I44cc47767ea2d9423166bb8fed080c8a11182041
2022-06-07 15:22:25 +00:00
Yaroslav Vorobiov
62774768d9 satellite/payments/storjscan: update config and wrap service errors
Change-Id: I4f75375e9499119cbcce78c5e77d1950e4bae38b
2022-06-07 14:21:42 +00:00
NickolaiYurchenko
e01daa538e web/satellite: bucket details page created
Change-Id: I0ee4dd4cabe7bad6f3d2ebdee455285529d81bdf
2022-06-07 16:49:02 +03:00
Vitalii
bb92b923d4 web/satellite: open bucket modal for new objects flow
Implemented new modal to open bucket.
This is a part of new objects flow.
It is hidden by feature flag.

Change-Id: I3839126933b8804c8adbd8949a29ac75771fab4a
2022-06-07 09:54:43 +00:00
Vitalii
f56504de2a apigen: project delete endpoint
Implemented project delete endpoint for REST API.
Added project usage status check service method to indicate if project can be deleted.
Updated project invoice status check method to indicate if project can be deleted.

Change-Id: I57dc96efb072517144252001ab5405446c9cdeb4
2022-06-07 12:23:24 +03:00
Paul Willoughby
911cc1e163 satellite/contact: reject privateIPs in PingMe and CheckIn endpoints
prevent network enumeration by rejecting privateIPs in PingMe and
Checkin endpoints

Closes storj/storj-private#32

Change-Id: I63f00483ff4128ebd5fa9b7b8da826a5706748c9
2022-06-07 08:09:14 +00:00
Vitalii
f0b28d6326 apigen: endpoint to get user
Implemented new GET user by request context endpoint.
Updated docs.

Change-Id: Iebb493e55f9456b89d7dbd234bb0b939b82b0ced
2022-06-06 16:31:19 +00:00
Vitalii
ba58530089 {satellite}/web,console,testsuite: remove old navigation structure
Removed old satellite UI navigation structure.
Removed old feature flag.

Change-Id: Ic998886cf2e30ebd44e67a20fc53888103fe4b8d
2022-06-06 16:43:05 +03:00
Michał Niewrzał
18ef3d3881 satellite/accounting: add metrics for total pending objects
We don't have metric to track how many pending objects we have in the
system. This change is using tally objects loop to collect pending
objects per bucket and at the end its combining all buckets values
into single metric for all pending objects in a system.

Change-Id: Iac7a6bfb48854f7e70127d275ea8fdd60c4eb8b7
2022-06-06 12:04:46 +00:00
hovex023
688b3907ab
web/satellite: Implement New Create Access Grant Wizard. (#4821)
Created and style modal-added open and close functionality
Added logic to access grants selection type
Changed CLI to API-Changed Types to radio buttons-Added date picker-Added buckets list(not yet populating with buckets)
Added permissions check box logic and new input field for buckets.
Added tooltips and hover logic for the tooltips
Added acknoledgement functionality to encrypt step
Added conditional logic to encrypt access button and the ascknoledgement box - removed blank comments from paymentMethods snapshot
Added logic to retrieve restricted api key, access credentials, and satellite address
Added copy functionality and download functionality for the credentials step

Change-Id: I8c8f02bc1ee38c3df42396cbd9bb3db2e7ff9cc4

Co-authored-by: cl-mitch <mitch.george@compozelabs.com>
2022-06-03 13:03:36 -05:00
Moby von Briesen
82e3932dbd web/satellite: Fix incorrect input placeholder
In the name step of the create bucket view for the new objects flow, fix
the placeholder for the input to mention "name" rather than "passphrase"

Change-Id: Ia7a64383b58d36f6aee1df952ed7cb300734e09d
2022-06-03 11:08:57 -04:00
dlamarmorgan
270204f352 satellite/{payments/storjscan,satellitedb}: Add wallet implementation
Add storjscan wallets implementation to the satellite. The wallets interface allows you to add and claim new wallets as called by the API. The storjscan specific implementation of this interface uses a wallets DB to associate the user to a wallet address, as well as a storjscan client to request and associate new wallets to the satellite.

Change-Id: I54081edb5545d4e3ee07cf1cce3d3e87cc00c4a1
2022-06-03 11:45:47 +00:00
cl-mitch
cbaca8b17e
web/satellite added new billing screen feature flag (#4836)
New feature flag implemented to enable the new billing flow.

Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
2022-06-02 09:30:27 -05:00
Ivan Fraixedes
261a6edf4f
satellite/admin/ui: Update NPM dependencies
Update all the NPM dependencies used by the Admin UI.

The dev dependencies correspond to the ones that are currently used by
an svelte app generated with the last svelte-kit version. They
deprecated some configuration options and changed some svelte
directives.

The only non-dev dependency is also updated to the last published.

Change-Id: I5f2192cab41e00efc3239237f8dc8f3d07816b63
2022-06-02 13:24:15 +02:00
Egon Elbre
763c04770f satellite/console/consoleweb: fix CSP values
script-src-elem is preferred over script-src in certain scenarios.
If it's absent, then the browser always uses script-src. By adding
script-src-elem it ended up blocking google recaptcha.

Change-Id: I9cf96e71e69054c4a034ca189db84fbe8903a59b
2022-06-02 10:44:48 +03:00
Jeremy Wharton
dba28f1d61 cmd/satellite: add project ID, bucket name to partner attribution CSV
This change adds project ID and bucket name columns to the generated
partner attribution report. Attribution values are now summed based on
their project ID and bucket name in addition to their user agent.
Additionally, the command to generate the attribution report has been
modified to optionally include only certain user agents.

Change-Id: I61a1d854379134f26b31467d9e83a787beb451dd
2022-06-01 20:27:12 +00:00
Vitalii
a61f0f6be3 satellite/projectaccounting: make daily usage query return correct allocated traffic
Updated daily project usage query to return correct allocated traffic.
If allocated egress has expired then we return settled egress.
If not then we return allocated egress - dead egress.

Fix for this issue
https://github.com/storj/storj/issues/4563

Change-Id: Ia15a50d3bb8d8cb1106936e17dbe0f1f5a40fa87
2022-06-01 15:24:52 +00:00
Vitalii
07e65cd338 satellite/projectaccounting: sum up bucket usages for daily usage query
Fixed daily usage query returning single bucket usage.
We sum up bucket usages now.
Also fixed https://github.com/storj/storj/issues/4559.

Change-Id: I2eb6299f1ef500d68150879195011b6fbb5f37ed
2022-06-01 12:50:10 +00:00
Márton Elek
84d02f7fbf metabase: use predefined snapshot for unit tests avoid migration
Change-Id: I6225ae32c312030e3559d8ed0faa137ffc3cc5b7
2022-06-01 12:07:00 +00:00
littleskunk
6cdd250019
satellite/console: enable new object flow / onboarding (#4851) 2022-06-01 14:04:03 +02:00
Ivan Fraixedes
aa913e63db cmd/storj-sim: Fix path to admin UI assets folder
The admin UI assets aren't inside of the `web` directory they are
directly in the `satellite` one.

The invalid path provoked that storj-sim generated a satellite
configuration with an invalid path to the Admin UI static assets
provoking that it didn't load the UI by default.

Change-Id: I49fb289377f51634057173690fbd8cf863ca9a9d
2022-05-31 18:54:54 +00:00
littleskunk
3d1b469f35
satellite/console: disable account activation reminder email (#4844)
* satellite/console: disable account activation reminder email
2022-05-31 17:36:40 +02:00
Márton Elek
9d8eaf3d7e metabasetest: use DROP instead of TRUNCATE
TRUNCATE requires table recreation which involves 'online schema change' with crdb.
(with psql it might be fater than DROP, that was the motivation of the original change)

`online schema change` is an async operation with crdb and it's eventually very slow therefore we try to avoid it.

This·reverts·commit·15bed0ed0e81d54fe4ffac9928bdf648f5e06ec6.¬

Change-Id: I93e1ab470962be77e3458d74c8787442c9d7bee0
2022-05-31 12:16:02 +00:00
Márton Elek
7c854fcde4 Makefile: make test run more safe
This patch suggests small improvements for the previously committed (very nice <3) unit test helper

 * Use non-default ports for postgres + cockroach. This would help to run same services for ad-hoc unit test (for IDE development for example)
 * Use the same flags for cockroach what we use for unit tests (mem store supposed to make tests faster)
 * Use `docker-compose down -v` for removing all networks and annynmous volumes
 * Remove connection limit (I have seen related problems in my jenkins runs)
 * Omit unused databases to avoid getting failures (eg. omit psql when crdb is used)
 * Do not drop cockroach database (full database will be dropped, anyway...): it also makes testing faster

Change-Id: Iadc04f8617a2825ea4f10072bd023a9c86883680
2022-05-31 10:58:26 +00:00
Michał Niewrzał
ffbb43ddbc cmd/uplink: fix how we are collecting errors while copy in parallel
Main issue was that when one part copy failed while being inside
goroutine (limiter) and another part was still collecting src/dst parts
it was possible to drop errors from failed part copy. It was possible
bacause on fail context was canceled and if we were still getting
part src/dst then it was returning error immediately and error
group with errors from goroutine was ignored.

Change-Id: I75c6799eba358741629795f2971c7a964cb2c9ce
2022-05-31 10:18:51 +00:00
Vitalii
4f668f54c0 web/satellite: fix main content width
Fixed main content width to be appropriate.
In case of very long access grant name (100+ symbols) main content gets overlapped with window border. That's an unhandled edge case.

Change-Id: I42ef709edc8454f054baa52231c605afadc471e2
2022-05-31 12:27:12 +03:00
Ivan Fraixedes
2438c92d56
satellite/admin/ui: Add disable MFA endpoint
Add the admin API endpoint for disabling an user's multifacdtor
authentication to the satellite admin UI.

Remove a couple of commented code lines too.

Change-Id: Iaee7efe7a3d4d38bdd6541311447a9726806f0f1
2022-05-30 15:23:35 +02:00
Stefan Benten
25b45c47ee
satellite/admin: allow disabling of MFA
We have a couple of support tickets so far that require us to
disable the mfa on accounts. Since we currently had no other
way than doing a SQL War Crime, it makes sense to add it to the
admin API.

Change-Id: Ib16735c1961380b04345a3495d4eebee5fa0bc41
2022-05-30 13:56:11 +02:00
Stefan Benten
62a64bae3b
satellite/admin: allow deletion of paid tier user projects when invoiced
Currently we have a bug in which we would require that a project of
a paid tier user needs to be two months unused before we can delete it.
This change fixes it and reduces it back to the normal next billing cycle.

Change-Id: I28610b6c45c68943fd4f2621233bccc06cab28a0
2022-05-30 13:54:51 +02:00
Stefan Benten
8673583582 satellite/admin: fix console config handling
An older change plummed the full console config as subconfig of
the admin api configuration in. This bloated the generated satellite
configuration unnecessarily while also allow for confusion/mistakes.

Change-Id: Icf49cc1f147711e37e85f6eac1143fab8ddf1659
2022-05-27 22:26:06 +00:00
Fadila Khadar
792c49e5ad satellite/metabase: simplify delete query
when deleting an object that has been copied multiple times, we look for an ancestor_stream_id by taking the min of all copies stream_id.

This change simplifies this process by picking any stream_id as a new ancestor by using 'distinct on'.

Fixes https://github.com/storj/storj/issues/4745

Change-Id: Iffb519b82d2ae2ed73af48fa0e86f87384e0158f
2022-05-27 20:09:50 +00:00
Cameron
b379fe89d2 satellite/core.go: hook up flag to enable email reminders
Change-Id: I973bb3a709e806e7744679b3543217c1cab85f9a
2022-05-27 13:16:14 -04:00
Vitalii
69ad49f473 satellite/server: fixed CSP for hcaptcha
Fixed CSP errors for hcaptcha

Change-Id: Ie928e206c652c97d36bcbdaf8436ae4a33afed8d
2022-05-27 14:28:31 +00:00
Michał Niewrzał
4f2fae4f28 cmd/uplink: better error handling for parallel transfer
Few improvements were made to how we are handling errors
while doing parallel upload/download for single object:
* unhide error under 'context canceled' which was shown in most of
cases
* add part number to error message
* don't try to commit if any error occurs while operation
* combine errors into more readable form, example:

---
failed to download part 3: uplink: eestream: failed to download stripe 0:
error retrieving piece 00: ecclient: piecestore: rpc: tcp connector failed: rpc: dial tcp 97.119.158.36:28967: i/o timeout
...
error retrieving piece 89: ecclient: piecestore: rpc: tcp connector failed: rpc: dial tcp 161.129.152.194:28967: i/o timeout
failed to download part 1: uplink: eestream: failed to download stripe 0:
error retrieving piece 01: io: read/write on closed pipe
...
error retrieving piece 97: io: read/write on closed pipe
failed to download part 2: uplink: eestream: failed to download stripe 0:
error retrieving piece 00: io: read/write on closed pipe
...
error retrieving piece 01: ecclient: piecestore: rpc: tcp connector failed: rpc: dial tcp 180.183.132.234:28967: operation was canceled
error retrieving piece 96: io: read/write on closed pipe
	main.(*cmdCp).parallelCopy:418
	main.(*cmdCp).copyFile:262
	main.(*cmdCp).Execute:156
	main.(*external).Wrap:123
	github.com/zeebo/clingy.(*Environment).dispatchDesc:126
	github.com/zeebo/clingy.(*Environment).dispatch:53
	github.com/zeebo/clingy.Environment.Run:34
	main.main:26
	runtime.main:250
---

Change-Id: I9bb70b3f754567761fa8d17bef8ef59b0709e33b
2022-05-27 14:00:35 +00:00