Commit Graph

6655 Commits

Author SHA1 Message Date
Márton Elek
55de50eea7 console: stub endpoints for native token payments
Change-Id: I3f99ab44332eb8c9e2a3834a932eee72f44bd490
2022-05-23 13:28:00 +00:00
hovex023
460af183a0
web/satellite: Add new access grants flow (#4778)
Created and styled Access Grant Flow modal. Added open and close functionality
Added proper text for access grant create buttons
Added logic to access grants selection type
Added permissions check box logic and new input field for buckets.
Added tooltips and hover logic for the tooltips
Added acknowledgement functionality to encrypt step
Added conditional logic to encrypt access button and the acknowledgement box 
Added a prop that shows the scrollbar for the bucket selector
Added name validation - needs proper error message
Implemented download utility and simplified true/false method

Co-authored-by: cl-mitch <mitch.george@compozelabs.com>
2022-05-23 08:04:56 -05:00
Márton Elek
31560c94f8 satellite/metabase/test: extend default time diff for comparison
Tests are intermittently fail with similar error:

```
    --- FAIL: TestDeletePendingObject/Cockroach (15.85s)

    test.go:380:
        	Error Trace:	test.go:380
        	           				delete_test.go:221
        	Error:      	Should be zero, but was   metabase.DeleteObjectResult{
        	           	 	Objects: []metabase.Object{
        	           	 		{
        	           	 			ObjectStream: {ProjectID: {0x0f, 0x40, 0x70, 0x41, ...}, BucketName: "txxywyg4", ObjectKey: "\xbb+$\x17\x80\xc6\xcaC\xa3\xdb\xc3z*\xa8\xbe\xaf", Version: 1, ...},
        	           	- 			CreatedAt:    s"2022-05-20 14:40:15.995376773 +0200 CEST",
        	           	+ 			CreatedAt:    s"2022-05-20 14:40:21.04949 +0200 CEST",
        	           	 			ExpiresAt:    nil,
        	           	 			Status:       1,
        	           	 			... // 9 identical fields
        	           	 		},
        	           	 	},
        	           	 	Segments: {{RootPieceID: {0x01, 0x00, 0x00, 0x00, ...}, Pieces: {{...}}}, {RootPieceID: {0x01, 0x00, 0x00, 0x00, ...}, Pieces: {{...}}}},
        	           	 }
        	Test:       	TestDeletePendingObject/Cockroach/with_segments
        --- FAIL: TestDeletePendingObject/Cockroach/with_segments (0.68s)
```

Looks like we shouldn't have an assumption that all tests can be finished in 5 seconds, especially not in highly parallel environment.

These tests use `time.Now` at the beginning and compare the time saved in the database (usually filled by the database).

The difference shouldn't be higher than 20 seconds (before this commit: 5 seconds) which assumes that the records are saved in this timeframe...

Change-Id: Ia6f52897d13f88c6857c05d728bf8e72ab863c9b
2022-05-23 10:24:58 +00:00
NickolaiYurchenko
99237d5c78 web/satellite: new bucket creation flow
old bucket creation flow removed
new flow added
name and passphrase splitted into separate views
demo bucket will not be created automatically
bucket creation progress bar added

Change-Id: I2a1d7d77c3038caaafb3c06bdb0ac5dd1ad17599
2022-05-22 11:55:55 +00:00
paul cannon
951a842fd4 satellite/{reputation,satellitedb}: move addAudit
This functionality will be needed in both packages, so here we move it
into the more general reputation-code package and export it for use in
satellitedb.

This also removes the related UpdateAuditHistory() signature from the
reputation DB interface, since it doesn't have anything to do with the
db. It doesn't need to be a method, either.

Finally, this changes the test for addAudit to be a plain test function
instead of using testplanet.Run(). It didn't need a whole testplanet
setup or any databases.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I90f6a909e5404f03ad776b95cfa2f248308c57c1
2022-05-20 15:37:31 +00:00
Yaroslav Vorobiov
0bf12523e1 satellite/payments/storjscan: add client and list all payments API call
Change-Id: I1f5065b3d15cc93f4b42868941e82e04af364565
2022-05-20 15:10:17 +00:00
cl-mitch
2b0016af62
web/satellite: My Access Table style changes and search functionality (#4767)
Styled Access Grant Table according to new flow UX
Added search functionality to My Access Table. 
Separated search styles into 2 components


Co-authored-by: Moby von Briesen <mobyvb@gmail.com>
Co-authored-by: hovex023 <97616907+hovex023@users.noreply.github.com>
2022-05-20 09:54:26 -05:00
Michał Niewrzał
7a2d2a36ca satellite: use more optimal monkit call for loop observers methods
Recently we applied this optimization to metrics observer and time
used by its method dropped from 12m to 3m for us1 (220m segments).
It looks that it make sense to apply the same code to all observers.

Change-Id: I05898aaacbd9bcdf21babc7be9955da1db57bdf2
2022-05-20 11:03:41 +00:00
Michał Niewrzał
c5edf77237 satellite/metabase: fix flaky TestCommitInlineSegment
If TestCommitInlineSegment tests are taking longer time
then zombieDeadline created at the beginning of test
can be too far in the past. Creating zombieDeadline for each case
should avoid flakines.

Change-Id: Ieb011e8e470f6f1c32cf9365c8ae819317de6738
2022-05-20 10:20:56 +00:00
Clement Sam
d3ada57872 web/storagenode: remove QUIC status tooltip when QUIC is configured
Resolves https://github.com/storj/storj/issues/4448

Change-Id: Id69340eb843f3ffdd9e1660d239bda65dbf28be4
2022-05-19 21:28:07 +00:00
Egon Elbre
5d6d5331f5 private/testplanet: don't cleanup cockroach
Márton found out that DROP DATABASE is rather slow on CRDB, and it makes
a significant impact when running the whole testsuite. In sum of test
times it's ~2.5h compared to ~2h. And the end-to-end ~20m to ~16m.

This adds a new flag STORJ_TEST_COCKROACH_NODROP for enabling this
behavior in the CI environment.

Fixes https://github.com/storj/dev-enablement/issues/6

Change-Id: I5a6616c32dc6596a96ba3d203f409368307d7438
2022-05-19 15:27:37 +03:00
Michał Niewrzał
456aea727e satellite: use PieceIDDeriver for derivation
We can use PieceIDDeriver in all places where we are deriving id from
the same id multiple times. We have serveral such places: gc, segment
deletion, segment validation, order limit creation. Using it should
save some resources.

Change-Id: I24668d516c0f7cea4aec6470614067734149501d
2022-05-19 06:31:42 +00:00
prerna-parashar
14ebba0d77
satellite/analytics: Track events generated from navigation and dashboard (#4795)
satellite/analytics: Track events generated from navigation and dashboard
2022-05-18 12:49:10 -07:00
Cameron
0e0437cc9f web/satellite: change button to 3 dots instead of 2
Edit svg in payment-methods-container__card-container__dots-container to draw 3 dots instead of two.

Change-Id: Ie447abf9a2e15b619cf937fe1b1c263e2eeca3e8
2022-05-18 17:50:38 +00:00
paul cannon
ac7752ac3c satellite/reputation: return full Info from reputation update
This will let us update our reputation cache when writing through to the
db.

Since the information is already being fetched from the db and returned
to the application, the extra cpu load here should be minimal.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I2b8619f2c0d541893c7d3e7d33b1863b96775ebd
2022-05-18 17:22:06 +00:00
cl-mitch
f08732f30d
web/satellite: make preliminary changes for new access grants flow (#4801)
Made some small styling changes to access grants and common components
Added svg and png files to be used in the new access grant modal

Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
Co-authored-by: hovex023 <97616907+hovex023@users.noreply.github.com>
2022-05-18 12:10:26 -05:00
Michał Niewrzał
d90ce467fc cmd/uplink: bring back --metadata for cp command
At some point uplink cli lost ability to set metadata. This change
brings back this functionality for 'cp' operation.

https://github.com/storj/storj/issues/3848

Change-Id: Ia5f60eb577fcab8a38d94730d8cdc6e0338d3b46
2022-05-18 15:58:53 +00:00
Erik van Velzen
bdcd285685 satellite/metabase: add codeowners
Add codeowners file for part of the code mostly owned by metainfo team.
This is to try out the feature and see how the team members like it.

Change-Id: I914d5f245271b800232b20c0c5c6a704739cfb6b
2022-05-18 16:13:06 +02:00
Michał Niewrzał
f9a3f19443 scripts: add 'uplink cp' tests for stdin/stdout
Uplink can upload from stdin and download to stdout. We had
such tests for old binary but now we were missing it.

Change-Id: I5110a9f531f5cc21277fa53611995fb5b556ff16
2022-05-18 08:42:04 +00:00
Malcolm Bouzi
087e57d037 web/satellite: create chore that will resend verification emails to unverified users
We want to remind unverified users to verify their emails:
once after 24 hours has passed and again after 5 days has passed.

Add mailservice.Service to satellite core because it is needed by the
chore for sending emails. To add the mailservice.Service to the core,
we create a helper function in satellite/peer.go to avoid duplicating
the code in both api.go and core.go. In addition to the chore, this
change adds methods to users.DB to get unverified users in need of
reminder.

Change-Id: I4e515bdf43f922788b4f965b2efb34fa32288bd1
2022-05-18 08:08:33 +00:00
Jeff Wendling
836fa188fa cmd/uplink: allow listing invalid access grants
if you somehow get an invalid access grant in your
config file, it'd be nice to be able to list it
and delete it and stuff.

Change-Id: I7e335bf32353f294d5abb6a7c5f8f3aa18f2f6a7
2022-05-17 20:00:36 +00:00
Vitalii
48b76d7147 web/satellite: fix chart's scaling
Made chart's y-axis always start at 0

Fix for this issue https://github.com/storj/storj/issues/4558

Change-Id: Id4dae7a46b5e7ba5f11912765cd46607503959a8
2022-05-17 19:32:21 +00:00
Qweder93
50b826e582 satellite/metabase: drop GetProjectSegmentCount and related code
Resolves: https://github.com/storj/storj/issues/4744

Change-Id: I66b12e17de461242745e785dedd397fbe3b386fa
2022-05-17 19:01:51 +00:00
Qweder93
a447b27ffc satellite/accounting: modify tally.go to use returned segments values
modify tally to calculate how we need to update segments in the live accounting
cache with UpdateProjectSegmentUsage method. adjust accounting.ExceedsUploadLimits
to use only cache for segment validation, if cache returns 0 or key not found then
we shouldn't reject such project as its possible that we won't have this value before first object iteration
https://github.com/storj/storj/issues/4744

Change-Id: I32c22d7fb71236e354653ba8719e029fc71f04c7
2022-05-17 17:20:02 +00:00
paul cannon
aa728bd6ea satellite/satellitedb: add reputations.disqualification_reason
We added nodes.disqualification_reason recently, but we didn't add a
corresponding column in the reputations table (despite having a
corresponding `disqualified` column there).

Without this change, the (very useful and informative) assignments to
updateFields.DisqualificationReason in reputations.go have no effect.

Refs: https://github.com/storj/storj/issues/4601

Change-Id: I77404902ca64b56aed72f1de76b303fe82b76aab
2022-05-17 10:09:36 -05:00
Márton Elek
82b1450a46 private/blockchain: generic types for wallet/transactions
Change-Id: I32a77c58ee519059bf6cd42e0db829d5104843a4
2022-05-17 12:05:11 +00:00
Michał Niewrzał
230424eace satellite/metainfo: add IPFS datastore user agent to metrics
Change-Id: I2ac14c97c64542669acef58ca6ded79984d0bd11
2022-05-17 13:09:22 +02:00
Michał Niewrzał
3ebb05ecca satellite/metainfo: add user-agent bandwidth metric
The existing versionCollector metrics can tell us how many times various
metainfo endpoints are called, but they don't tell us how many bytes a
client is transferring.  We currently can't collect precise information
on this, but we can collect information on how much planned traffic is
requested via order limits.

The implementation as provided is intended to measure objects sizes
before erasure encoding is taken into account.

Change-Id: I2f1d2a7831630e8439ecf5342e933df259151792
2022-05-17 09:57:39 +00:00
Michał Niewrzał
7b253075bb satellite/accounting: test if server-side copy affects bandwidth
Doing server-side copy operation should not affect user monthly
bandwidth. This test covers that case.

Fixes https://github.com/storj/storj/issues/4717

Change-Id: I84ffab96b84851f395ea3a34d88f7dba424ec440
2022-05-17 06:24:15 +00:00
Clement Sam
87ea2a4794 cmd/storagenode: make supervisor HTTP Server configurable
The current supervisord condifguration sets up the HTTP server
to listen on a tcp socket which is private i.e. available only
on localhost. This poses a regression where multiple containers
cannot be run if the host network interface is used when docker
container is run with `--network host` option.

This change adds a new env variable `SUPERVISOR_SERVER`, with
potential values `unix | private_port | public_port`, where
`unix` is set as the default value.

By default, the HTTP server is now set to listen on a UNIX
domain socket.
The file path is set to `/etc/supervisor/supervisor.sock`
instead of the /tmp directory since some systems
periodically delete older files in /tmp. If the socket file is
deleted, supervisorctl will be unable to connect to supervisord.

When SUPERVISOR_SERVER is set to `public_port` or `private_port`,
the HTTP server is set to listen on a TCP socket.

Resolves https://github.com/storj/storj/issues/4661

Change-Id: I224836dcae0293bcfe49874f2748be7723944687
2022-05-16 20:06:24 +00:00
JT Olio
5fb9ee3cfa uplink, satellite: use bgp hostnames
Change-Id: I58f5011d3019f8267fa8cbd3096b2cfe42eb5f8b
2022-05-16 18:14:37 +00:00
Cameron
bfad64a6f0 satellite/satellitedb: add verification_reminders column to users table
When a new user registers, we send a verification request to their
email. Currently, if they do not verify their email, we take no further
action. We want to send these users reminders: one after about one day
and one after about 5 days. To do this we will use this new
verification_reminders column.

It will look something like this:

```
SELECT email FROM users
WHERE status = 0
   AND (
          (verification_reminders = 0 AND created_at < now() - 'INTERVAL 1d')
          OR (verification_reminders = 1 AND created_at < now() - 'INTERVAL 5d')
   )
```

Change-Id: If0620e08c97e9e337c9563481d665c5bd462693b
2022-05-16 16:37:16 +00:00
Cameron
772397172f satellite/console: implement WebappSessions db
Create WebappSessions interface in consoleauth package.
Interface implements the DB for webapp_sessions table.

https://github.com/storj/storj/blob/main/docs/blueprints/webapp-session-management.md

Change-Id: Ib56f238c20b58f2877046fc384232add253ee82b
2022-05-16 15:06:04 +00:00
Cameron
18eedaa62f satellite/satellitedb/dbx: add method to delete webapp sessions by user ID
Change-Id: I4552622fdf40c6ecd53046e597250bbe713f510c
2022-05-16 14:37:34 +00:00
Michał Niewrzał
277f8d2ae0 satellite/accounting: don't expire segments usage key
Initialy we wanted to put segment usage into cache values retrieved
directly from metabase but it cause performance issues. Now we
will be collecting segment usage from tally object loop and those
values will be put into a cache but becuse we cannot get those values
on demand we shouldn't expire cache value at all because objects
loop requires sustencial amount of time to be executed.

Part of solution for https://github.com/storj/storj/issues/4744

Change-Id: I3b37e23badeecebed0c95064156e85b38038bfe2
2022-05-16 13:00:16 +00:00
Vitalii
747f2a09f7 testsuite/satellite: updated test for new onboarding cli flow
Updated onboarding cli flow go-rod test

Change-Id: I72a5e3e8b650434fdd2375d19d79099b9b90dd20
2022-05-16 08:28:31 +00:00
Vitalii
166fc5a2b4 web/satellite: progress bar for onboarding cli flow
Added progress bar for updated onboarding cli flow

Change-Id: I1fc53f990e78717f89cb0c2808d2439b5b0356c4
2022-05-16 08:28:24 +00:00
Vitalii
b3df7f4a1a web/satellite: onboarding cli flow updates
Updated UI and added back access grant creation steps.

Change-Id: I32b84601bb975fe0551562c9b1ebf6050de6dca7
2022-05-16 08:28:18 +00:00
Vitalii
b544dc917e web/satellite: update onboarding overview screen
Made updates to onboarding overview screen to correspond to newest designs.
https://www.figma.com/file/HlmasFJNHxs2lzGerq3WYH/Satellite-GUI-Public?node-id=11691%3A87378

Change-Id: I6f21e73b69222b9719934fe3034ed714317929e9
2022-05-16 08:28:11 +00:00
Stefan Benten
7afdb15fc8
cmd/uplink: adding Length Method to MultiReadHandle
This changes allows fetching the file size more easily (for supported
files) in order to afterwards calculate the multipart part size
accordingly.

Change-Id: Idabba4c2ee794ee471973889f5843174a7acad35
2022-05-13 21:31:45 +02:00
Stefan Benten
5e4ec0b3be
cmd/uplink: adjust multipart part size based on file size
This change allows the uplink to bump the part size based on the
content length that is being copied. This ensures we are staying
below the 10k part limit currently enforced on the satellites.

If the user specifies the flag, it will error out if the value
chosen by the user is too low. Otherwise it will use it.

Change-Id: I00d30f603d941c2f7703ba19d5923e668629a7b9
2022-05-13 21:31:23 +02:00
Cameron
0633aca607 satellite/console: create new consoleauth service
We want to send email verification reminders to users from the satellite
core, but some of the functionality required to do so exists in the
satellite console service. We could simply import the console service
into the core to achieve this, but the service requires a lot of
dependencies that would go unused just to be able to send these emails.

Instead, we break out the needed functionality into a new service which
can be imported separately by the console service and the future email
chore.

The consoleauth service creates, signs, and checks the expiration of auth
tokens.

Change-Id: I2ad794b7fd256f8af24c1a8d73a203d508069078
2022-05-13 16:27:07 +00:00
Moby von Briesen
763bfc0913 satellite/console,web/satellite: Implement hCaptcha
Adds a new configuration for hcaptcha enabled, secretkey, and sitekey.
If both reCAPTCHA and hCaptcha are configured as "enabled", reCAPTCHA
will be used.

Change-Id: I73cc6e133d8da3555e0ed8b2b377cf9eb263e6dc
2022-05-13 14:57:45 +00:00
Vitalii
aba7de6c32 web/satellite: added create project prompt modal for free tier users
Added new modal when free tier user tries to create new project.
This modal prompts to upgrade account.

Change-Id: I0e787e35b099387a08186c7df7fc080092d2366c
2022-05-13 14:02:44 +00:00
Vitalii
95ae739d9c satellite/projectaccounting: fetch bucket usage since bucket creation instead of project creation
Fix for this customer issue
https://github.com/storj/customer-issues/issues/34

By this change we fetch bucket usage since its creation instead of using project's createdAt timestamp.

Change-Id: Ic0ea5d169056a5bd64ed143d13954d794da6e1d2
2022-05-13 13:27:42 +00:00
Moby von Briesen
7d35f3e977 Revert "web/satellite: use aws-sdk v3"
This reverts commit 2e3e5547e2.

The S3 client is returning an error related to region with the new
changes.

Change-Id: I989b8a46c0e97c1278517e1de07fe42d5950cf54
2022-05-13 09:31:12 +00:00
Erik van Velzen
db1cc8ca95 satellite/repair/checker: buffer repair queue
Integrate previous changes. Speed up the segment loop by batch inserting
into repair queue.

Change-Id: Ib9f4962d91960d21bad298f7771345b0dd270276
2022-05-12 16:28:05 +00:00
Michał Niewrzał
9b49ffe48a mod: bump storj/private dependency
Change-Id: I5067c0cb3f1d33d9704ecdbcbb0f36cda9dc0bb0
2022-05-12 14:39:51 +00:00
Egon Elbre
51c409844a web/satellite: delete unused Dockerfile
Change-Id: I85a8f822b0bca958dc27183c437624813851bb24
2022-05-12 11:27:07 +03:00
Egon Elbre
4eb3dd572c web/satellite: display authorize area icon only when set
When the icon wasn't defined, it showed an empty image icon instead.

Change-Id: I8fc731f99dbd1a5b29fa84c6c14be1bfe4c51f67
2022-05-12 09:26:45 +03:00