Commit Graph

6693 Commits

Author SHA1 Message Date
prerna-parashar
14ebba0d77
satellite/analytics: Track events generated from navigation and dashboard (#4795)
satellite/analytics: Track events generated from navigation and dashboard
2022-05-18 12:49:10 -07:00
Cameron
0e0437cc9f web/satellite: change button to 3 dots instead of 2
Edit svg in payment-methods-container__card-container__dots-container to draw 3 dots instead of two.

Change-Id: Ie447abf9a2e15b619cf937fe1b1c263e2eeca3e8
2022-05-18 17:50:38 +00:00
paul cannon
ac7752ac3c satellite/reputation: return full Info from reputation update
This will let us update our reputation cache when writing through to the
db.

Since the information is already being fetched from the db and returned
to the application, the extra cpu load here should be minimal.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I2b8619f2c0d541893c7d3e7d33b1863b96775ebd
2022-05-18 17:22:06 +00:00
cl-mitch
f08732f30d
web/satellite: make preliminary changes for new access grants flow (#4801)
Made some small styling changes to access grants and common components
Added svg and png files to be used in the new access grant modal

Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
Co-authored-by: hovex023 <97616907+hovex023@users.noreply.github.com>
2022-05-18 12:10:26 -05:00
Michał Niewrzał
d90ce467fc cmd/uplink: bring back --metadata for cp command
At some point uplink cli lost ability to set metadata. This change
brings back this functionality for 'cp' operation.

https://github.com/storj/storj/issues/3848

Change-Id: Ia5f60eb577fcab8a38d94730d8cdc6e0338d3b46
2022-05-18 15:58:53 +00:00
Erik van Velzen
bdcd285685 satellite/metabase: add codeowners
Add codeowners file for part of the code mostly owned by metainfo team.
This is to try out the feature and see how the team members like it.

Change-Id: I914d5f245271b800232b20c0c5c6a704739cfb6b
2022-05-18 16:13:06 +02:00
Michał Niewrzał
f9a3f19443 scripts: add 'uplink cp' tests for stdin/stdout
Uplink can upload from stdin and download to stdout. We had
such tests for old binary but now we were missing it.

Change-Id: I5110a9f531f5cc21277fa53611995fb5b556ff16
2022-05-18 08:42:04 +00:00
Malcolm Bouzi
087e57d037 web/satellite: create chore that will resend verification emails to unverified users
We want to remind unverified users to verify their emails:
once after 24 hours has passed and again after 5 days has passed.

Add mailservice.Service to satellite core because it is needed by the
chore for sending emails. To add the mailservice.Service to the core,
we create a helper function in satellite/peer.go to avoid duplicating
the code in both api.go and core.go. In addition to the chore, this
change adds methods to users.DB to get unverified users in need of
reminder.

Change-Id: I4e515bdf43f922788b4f965b2efb34fa32288bd1
2022-05-18 08:08:33 +00:00
Jeff Wendling
836fa188fa cmd/uplink: allow listing invalid access grants
if you somehow get an invalid access grant in your
config file, it'd be nice to be able to list it
and delete it and stuff.

Change-Id: I7e335bf32353f294d5abb6a7c5f8f3aa18f2f6a7
2022-05-17 20:00:36 +00:00
Vitalii
48b76d7147 web/satellite: fix chart's scaling
Made chart's y-axis always start at 0

Fix for this issue https://github.com/storj/storj/issues/4558

Change-Id: Id4dae7a46b5e7ba5f11912765cd46607503959a8
2022-05-17 19:32:21 +00:00
Qweder93
50b826e582 satellite/metabase: drop GetProjectSegmentCount and related code
Resolves: https://github.com/storj/storj/issues/4744

Change-Id: I66b12e17de461242745e785dedd397fbe3b386fa
2022-05-17 19:01:51 +00:00
Qweder93
a447b27ffc satellite/accounting: modify tally.go to use returned segments values
modify tally to calculate how we need to update segments in the live accounting
cache with UpdateProjectSegmentUsage method. adjust accounting.ExceedsUploadLimits
to use only cache for segment validation, if cache returns 0 or key not found then
we shouldn't reject such project as its possible that we won't have this value before first object iteration
https://github.com/storj/storj/issues/4744

Change-Id: I32c22d7fb71236e354653ba8719e029fc71f04c7
2022-05-17 17:20:02 +00:00
paul cannon
aa728bd6ea satellite/satellitedb: add reputations.disqualification_reason
We added nodes.disqualification_reason recently, but we didn't add a
corresponding column in the reputations table (despite having a
corresponding `disqualified` column there).

Without this change, the (very useful and informative) assignments to
updateFields.DisqualificationReason in reputations.go have no effect.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I77404902ca64b56aed72f1de76b303fe82b76aab
2022-05-17 10:09:36 -05:00
Márton Elek
82b1450a46 private/blockchain: generic types for wallet/transactions
Change-Id: I32a77c58ee519059bf6cd42e0db829d5104843a4
2022-05-17 12:05:11 +00:00
Michał Niewrzał
230424eace satellite/metainfo: add IPFS datastore user agent to metrics
Change-Id: I2ac14c97c64542669acef58ca6ded79984d0bd11
2022-05-17 13:09:22 +02:00
Michał Niewrzał
3ebb05ecca satellite/metainfo: add user-agent bandwidth metric
The existing versionCollector metrics can tell us how many times various
metainfo endpoints are called, but they don't tell us how many bytes a
client is transferring.  We currently can't collect precise information
on this, but we can collect information on how much planned traffic is
requested via order limits.

The implementation as provided is intended to measure objects sizes
before erasure encoding is taken into account.

Change-Id: I2f1d2a7831630e8439ecf5342e933df259151792
2022-05-17 09:57:39 +00:00
Michał Niewrzał
7b253075bb satellite/accounting: test if server-side copy affects bandwidth
Doing server-side copy operation should not affect user monthly
bandwidth. This test covers that case.

Fixes https://github.com/storj/storj/issues/4717

Change-Id: I84ffab96b84851f395ea3a34d88f7dba424ec440
2022-05-17 06:24:15 +00:00
Clement Sam
87ea2a4794 cmd/storagenode: make supervisor HTTP Server configurable
The current supervisord condifguration sets up the HTTP server
to listen on a tcp socket which is private i.e. available only
on localhost. This poses a regression where multiple containers
cannot be run if the host network interface is used when docker
container is run with `--network host` option.

This change adds a new env variable `SUPERVISOR_SERVER`, with
potential values `unix | private_port | public_port`, where
`unix` is set as the default value.

By default, the HTTP server is now set to listen on a UNIX
domain socket.
The file path is set to `/etc/supervisor/supervisor.sock`
instead of the /tmp directory since some systems
periodically delete older files in /tmp. If the socket file is
deleted, supervisorctl will be unable to connect to supervisord.

When SUPERVISOR_SERVER is set to `public_port` or `private_port`,
the HTTP server is set to listen on a TCP socket.

Resolves https://github.com/storj/storj/issues/4661

Change-Id: I224836dcae0293bcfe49874f2748be7723944687
2022-05-16 20:06:24 +00:00
JT Olio
5fb9ee3cfa uplink, satellite: use bgp hostnames
Change-Id: I58f5011d3019f8267fa8cbd3096b2cfe42eb5f8b
2022-05-16 18:14:37 +00:00
Cameron
bfad64a6f0 satellite/satellitedb: add verification_reminders column to users table
When a new user registers, we send a verification request to their
email. Currently, if they do not verify their email, we take no further
action. We want to send these users reminders: one after about one day
and one after about 5 days. To do this we will use this new
verification_reminders column.

It will look something like this:

```
SELECT email FROM users
WHERE status = 0
   AND (
          (verification_reminders = 0 AND created_at < now() - 'INTERVAL 1d')
          OR (verification_reminders = 1 AND created_at < now() - 'INTERVAL 5d')
   )
```

Change-Id: If0620e08c97e9e337c9563481d665c5bd462693b
2022-05-16 16:37:16 +00:00
Cameron
772397172f satellite/console: implement WebappSessions db
Create WebappSessions interface in consoleauth package.
Interface implements the DB for webapp_sessions table.

https://github.com/storj/storj/blob/main/docs/blueprints/webapp-session-management.md

Change-Id: Ib56f238c20b58f2877046fc384232add253ee82b
2022-05-16 15:06:04 +00:00
Cameron
18eedaa62f satellite/satellitedb/dbx: add method to delete webapp sessions by user ID
Change-Id: I4552622fdf40c6ecd53046e597250bbe713f510c
2022-05-16 14:37:34 +00:00
Michał Niewrzał
277f8d2ae0 satellite/accounting: don't expire segments usage key
Initialy we wanted to put segment usage into cache values retrieved
directly from metabase but it cause performance issues. Now we
will be collecting segment usage from tally object loop and those
values will be put into a cache but becuse we cannot get those values
on demand we shouldn't expire cache value at all because objects
loop requires sustencial amount of time to be executed.

Part of solution for https://github.com/storj/storj/issues/4744

Change-Id: I3b37e23badeecebed0c95064156e85b38038bfe2
2022-05-16 13:00:16 +00:00
Vitalii
747f2a09f7 testsuite/satellite: updated test for new onboarding cli flow
Updated onboarding cli flow go-rod test

Change-Id: I72a5e3e8b650434fdd2375d19d79099b9b90dd20
2022-05-16 08:28:31 +00:00
Vitalii
166fc5a2b4 web/satellite: progress bar for onboarding cli flow
Added progress bar for updated onboarding cli flow

Change-Id: I1fc53f990e78717f89cb0c2808d2439b5b0356c4
2022-05-16 08:28:24 +00:00
Vitalii
b3df7f4a1a web/satellite: onboarding cli flow updates
Updated UI and added back access grant creation steps.

Change-Id: I32b84601bb975fe0551562c9b1ebf6050de6dca7
2022-05-16 08:28:18 +00:00
Vitalii
b544dc917e web/satellite: update onboarding overview screen
Made updates to onboarding overview screen to correspond to newest designs.
https://www.figma.com/file/HlmasFJNHxs2lzGerq3WYH/Satellite-GUI-Public?node-id=11691%3A87378

Change-Id: I6f21e73b69222b9719934fe3034ed714317929e9
2022-05-16 08:28:11 +00:00
Stefan Benten
7afdb15fc8
cmd/uplink: adding Length Method to MultiReadHandle
This changes allows fetching the file size more easily (for supported
files) in order to afterwards calculate the multipart part size
accordingly.

Change-Id: Idabba4c2ee794ee471973889f5843174a7acad35
2022-05-13 21:31:45 +02:00
Stefan Benten
5e4ec0b3be
cmd/uplink: adjust multipart part size based on file size
This change allows the uplink to bump the part size based on the
content length that is being copied. This ensures we are staying
below the 10k part limit currently enforced on the satellites.

If the user specifies the flag, it will error out if the value
chosen by the user is too low. Otherwise it will use it.

Change-Id: I00d30f603d941c2f7703ba19d5923e668629a7b9
2022-05-13 21:31:23 +02:00
Cameron
0633aca607 satellite/console: create new consoleauth service
We want to send email verification reminders to users from the satellite
core, but some of the functionality required to do so exists in the
satellite console service. We could simply import the console service
into the core to achieve this, but the service requires a lot of
dependencies that would go unused just to be able to send these emails.

Instead, we break out the needed functionality into a new service which
can be imported separately by the console service and the future email
chore.

The consoleauth service creates, signs, and checks the expiration of auth
tokens.

Change-Id: I2ad794b7fd256f8af24c1a8d73a203d508069078
2022-05-13 16:27:07 +00:00
Moby von Briesen
763bfc0913 satellite/console,web/satellite: Implement hCaptcha
Adds a new configuration for hcaptcha enabled, secretkey, and sitekey.
If both reCAPTCHA and hCaptcha are configured as "enabled", reCAPTCHA
will be used.

Change-Id: I73cc6e133d8da3555e0ed8b2b377cf9eb263e6dc
2022-05-13 14:57:45 +00:00
Vitalii
aba7de6c32 web/satellite: added create project prompt modal for free tier users
Added new modal when free tier user tries to create new project.
This modal prompts to upgrade account.

Change-Id: I0e787e35b099387a08186c7df7fc080092d2366c
2022-05-13 14:02:44 +00:00
Vitalii
95ae739d9c satellite/projectaccounting: fetch bucket usage since bucket creation instead of project creation
Fix for this customer issue
https://github.com/storj/customer-issues/issues/34

By this change we fetch bucket usage since its creation instead of using project's createdAt timestamp.

Change-Id: Ic0ea5d169056a5bd64ed143d13954d794da6e1d2
2022-05-13 13:27:42 +00:00
Moby von Briesen
7d35f3e977 Revert "web/satellite: use aws-sdk v3"
This reverts commit 2e3e5547e2.

The S3 client is returning an error related to region with the new
changes.

Change-Id: I989b8a46c0e97c1278517e1de07fe42d5950cf54
2022-05-13 09:31:12 +00:00
Erik van Velzen
db1cc8ca95 satellite/repair/checker: buffer repair queue
Integrate previous changes. Speed up the segment loop by batch inserting
into repair queue.

Change-Id: Ib9f4962d91960d21bad298f7771345b0dd270276
2022-05-12 16:28:05 +00:00
Michał Niewrzał
9b49ffe48a mod: bump storj/private dependency
Change-Id: I5067c0cb3f1d33d9704ecdbcbb0f36cda9dc0bb0
2022-05-12 14:39:51 +00:00
Egon Elbre
51c409844a web/satellite: delete unused Dockerfile
Change-Id: I85a8f822b0bca958dc27183c437624813851bb24
2022-05-12 11:27:07 +03:00
Egon Elbre
4eb3dd572c web/satellite: display authorize area icon only when set
When the icon wasn't defined, it showed an empty image icon instead.

Change-Id: I8fc731f99dbd1a5b29fa84c6c14be1bfe4c51f67
2022-05-12 09:26:45 +03:00
Egon Elbre
2e3e5547e2 web/satellite: use aws-sdk v3
While there isn't a significant win neither the prod javascript size nor
the number of dependencies, it does reduce the number of deprecated
packages.

Change-Id: Ib8edeb6916844500d74ae067464be9f28f542605
2022-05-12 09:17:26 +03:00
Egon Elbre
4791ba5d50 satellite/{mailservice,oidc}: improvements to debugging
Things that make debugging easier.
* Added logging to automatic link clicking to make it obvious, when it
  fails.
* Added monitoring to oidc.
* Made dbx create calls noreturn for oauth_*

Change-Id: I37397b4e84ce5bfd82954aed9c38fdfd52595f24
2022-05-11 19:59:42 +00:00
Jeff Wendling
89ccfe2dd7 cmd/uplink: fix recursive copy and improve tests
recursive copy had a bug with relative local paths.

this fixes that bug and changes the test framework
to use more of the code that actually runs in uplink
and only mocks out the direct interaction with the
operating system.

Change-Id: I9da2a80bfda8f86a8d05879b87171f299f759c7e
2022-05-11 15:17:16 -04:00
paul cannon
41c5879f7c satellite: more detailed goroutine labels
This will apply an appropriate "subsystem" label to goroutines which are
part of the core, api, repairer, admin, or gc subsystems.

It will also label goroutines whose job it is to watch for slow shutdown
of lifecycle groups (there are a lot of these).

Finally, this will also label goroutines whose job it is to wait on the
toplevel errgroup of a subsystem.

Change-Id: I560b5fff4a0101300d6c9a67609c2d80d7424486
2022-05-11 17:50:55 +00:00
Egon Elbre
89cc544b34 web/satellite: avoid querystring
querystring dependency is deprecated in favor of URLSearchParams.

Change-Id: If7006cca1c590fff3b50440240307c00c2a23399
2022-05-11 16:39:39 +00:00
Vitalii
dedccbd2e4 satellite/console, web/satellite: limit failed login attempts
Added account locking on 3 or more login attempts.
Includes both password and MFA failed attempts on login.
Unlock account on successful password reset.

Change-Id: If4899b40ab4a77d531c1f18bfe22cee2cffa72e0
2022-05-11 14:49:11 +00:00
Vitalii
ca6a0fc844 web/satellite: use chunkhash instead of hash in webpack config
Replaced 'hash' with 'chunkhash' in webpack config because regular 'hash' was deprecated.

Change-Id: I9d2823707698ea3cfa2afeda38bba1830c1b2918
2022-05-11 17:18:37 +03:00
Erik van Velzen
928375a67c satellite/repair/queue: buffer batch insert
Implement a buffer for inserting repair items into the queue in a batch.

Part of https://github.com/storj/storj/issues/4727

Change-Id: I718472b2f2b1f4993c3d6f15c44923776407155a
2022-05-11 09:02:20 +00:00
Michał Niewrzał
6175731166 private/testplanet: use simpler access for tests
This change created access grant manually to save some cpu
by avoiding root key derivation and avoid dialing satellite
for project id. It affects only testplanet tests.

Change-Id: I6742bcf699cca51e658f147e6df72c6b3db78d10
2022-05-10 20:10:10 +00:00
Kaloyan Raev
15bed0ed0e satellite/metabase: use TRUNCATE in TestingDeleteAll
TRUNCATE is faster than DELETE when deleting all rows.

As almost every metabase test case calls TestingDeleteAll, this change
should give some slight test speed-up.

Change-Id: Ib477962b6deb93edd60d6db2f1be6ede1b4b2381
2022-05-10 19:13:04 +00:00
Moby von Briesen
a8b1e1a81d web/satellite/jest.config: Always use UTC
Update jest config to use UTC so that undesired timestamp changes are
not made to snapshots when they are updated in American timezones.

Right now, if you are in an American timezone, there are diffs like this
in the snapshot files after updating them:

-  <p class="user-container__date">1/1/1970</p>
+  <p class="user-container__date">12/31/1969</p>

The hacky solution is to manually set timezone to Europe/Amsterdam in
the terminal before updating snapshots, but this commit should fix that
automatically.

Change-Id: I7cbbda0493b229fcd22b863dd688201436edc18a
2022-05-10 17:46:37 +00:00
prerna-parashar
490254213e
satellite/analytics: Track events generated from Upgrade Account Banner (#4770)
satellite/analytics: Track events generated from Upgrade Account banner
2022-05-10 10:01:27 -07:00