Commit Graph

5316 Commits

Author SHA1 Message Date
Yingrong Zhao
babe01f229 scripts/tests/rollingupgrade: fix rollingupgrade test backwards
compatibility

Change-Id: I1551f993bc8ca5e6fac07da7710dd46dbb031e47
2021-02-08 21:37:24 +00:00
Yaroslav Vorobiov
1e328f3c30 satellite/compensation: add wallet features to invoice csv
Change-Id: I90a72c0a4c8d07604096913b6680263b6defc0a2
2021-02-08 22:13:48 +02:00
Yaroslav Vorobiov
966535e9de {storagenode,satellite}/nodeoperator: add wallet features
Change-Id: Iac7eb40a52b8fddcc573aebaad2e3a30a10cded9
2021-02-08 22:09:45 +02:00
VitaliiShpital
570dcfad3e web/satellite: add segment event for Generate Gateway Credentials button
WHAT:
we want to track how many people click Generate Gateway Credentials button

WHY:
user tracking

Change-Id: Iad6a38a552f5773bbea7c07bac3562365a5f57d9
2021-02-08 18:31:28 +00:00
VitaliiShpital
fcd18ef48a web/satellite: rework signup verify redirect flow
WHAT:
people who sign up on US2 are not redirected to verifying page. From now on we have to set verify URL to make redirect happen

WHY:
user experience

Change-Id: I96c51a2c4f9cb6376cbfea639675b32918b58bee
2021-02-08 18:00:57 +00:00
VitaliiShpital
fd095e604f web/satellite: fix for access grants creation datepicker
WHAT:
when user chooses the same date in datepicker it sets grant's NotBefore and NotAfter restrictions as the same date which makes access grant unusable

WHY:
bug fix

Change-Id: I416606610ebddfd1fa881cce89d2beeded48457f
2021-02-08 17:37:20 +00:00
Yingrong Zhao
89529237a2 web/satellite, web/marketing: remove referral program related UI code
Change-Id: Ia815a72c0d17760bb63d9a37bb18b88952209368
2021-02-08 11:07:35 -05:00
Yingrong Zhao
3b49d3cddf satellite: remove referral program related code
This PR removes all back-end related referral program code including the
marketing portal.

We will have a separate PR for front-end code and database migration to
drop `offers` and `usercredits` table

Change-Id: If59f952cddfe0558a7dc03a0eac7cc1081517f88
2021-02-08 13:52:50 +00:00
Qweder93
dc6a8cbcf3 storagenode/suspension: fixed timings of downtime suspension notifications
Change-Id: Ie5b4f369efc0e66910b2f939aa8c5b11dc92946c
2021-02-08 15:24:29 +02:00
NickolaiYurchenko
e86041a5cf web: stylint config moved from package.json
Change-Id: Ic16ef4190947e5af447072f24055ca9294526956
2021-02-08 10:48:06 +00:00
NickolaiYurchenko
cd57606209 web/storagenode: size package refactored
Change-Id: I139478c760b3eb480990a1ff3aec6f8b90c218ac
2021-02-08 12:19:58 +02:00
Egon Elbre
0c223f288a satellite/metainfo/metabase: don't use random node ID-s
We will add a cache to nodes, so using completely random nodes wouldn't
show the actual performance.

Change-Id: I94f18283712812f05f7795efd3c7cf57499fa52c
2021-02-08 11:36:05 +02:00
Egon Elbre
d0a7ab60ec satellite/metainfo/metabase: add NodeAliasCache
This implements a node alias cache that automatically refreshes
and adds new nodes when needed.

Change-Id: I4af121f7307b7b7ac4f87ad465f02d4eeb17d42f
2021-02-08 11:36:03 +02:00
Natalie Villasana
3cc1a0b393 dbutil/cockroachutil: make crdb retry when tcp error
Adds checks in the existing NeedsRetry cockroachutil method
to see if the error is from the net package and a tcp error,
in which case it should retry. Also adds checks for if the
error contains EOF-- we don't retry in this case because it's
unclear if the query succeeded or failed.
These checks are needed to prevent our app from crashing when
it gets "connection reset by peer" and other tcp errors that
we've seen frequently coming from the metainfo loop.

Change-Id: I194d4b120082393cb6dbda2dd86b44f4696a66a4
2021-02-08 08:49:22 +00:00
Egon Elbre
0260966de4 satellite/metainfo/metabase: add NodeAliasMap
We need to keep an inmemory cache to avoid lookups into aliases table.
This adds the inmemory state of the cache.

Change-Id: Ief2b9bb19e10b46839b9208472dfc3035eb49af3
2021-02-08 10:28:22 +02:00
Malcolm Bouzi
8b49b210a1
Projects list view (#4035)
web/satellite: add new projects list management page
- add manage projects link to Projects navigation dropdown
- add create project cta to page
- add pagination of projects page
- render dates, number of users and project name to list table
- update snapshots
2021-02-05 15:57:03 -05:00
Jeff Wendling
0ef696a15e cla: remove .clabot
moved to a shared repo at https://github.com/storj/clabot-config
so that we only need to have one list for multiple repositories.

Change-Id: Icc5e2393d8a963d8e0ff5f5d4a1e5c410dc71aa4
2021-02-05 18:05:37 +00:00
NickolaiYurchenko
12fde343c1 web/multinode: delete node option added to context menu
Change-Id: I56d362fc319b4785714bd45877e6f37755530033
2021-02-05 13:56:42 +00:00
Egon Elbre
322c3a167b satellite/metainfo/metabase: add node alias table
This is first step in supporting node aliases. It adds a table
that automatically assigns aliases to nodes inserted into the table.

Change-Id: Ibdf40097c3c1e5b371500203f8db203505a48adc
2021-02-05 14:34:15 +02:00
Jeff Wendling
e114cfe86d satellite/satellitedb: fix broken query
Change-Id: I6d412a673d75264bf9751c6f15b1fb0ab94e1394
2021-02-04 16:00:51 -05:00
Caleb Case
f011a5e8cc
Use WithNonce when creating caveats
This ensures the caveats are unique even when they contain the same
permissions and will result in unique macaroons. This is important to
ensure revocation doesn't impact more macaroons than intended.

Change-Id: I6354edd0119f2d85eaf580f2d1926a3de9151b88
2021-02-04 13:46:32 -05:00
Qweder93
cdf58a1ffd multinode/payouts: EarnedPerSatellite added
private/multinodepb: multinode.proto updated

Change-Id: Iaeffd31ae124ac5b2716ca1b2d05e764372e2d94
2021-02-04 18:10:17 +02:00
Malcolm Bouzi
db3a3088f9
satellite/satellitedb: add professional user fields to db interface (#4034) 2021-02-04 10:00:15 -05:00
Egon Elbre
615586a471 {satellite,storagnode}/internalpb: bump gogo to v1.3.2
Change-Id: I585a3197630b9b675a4b540cf25612e1ed39b599
2021-02-04 11:09:13 +00:00
Qweder93
b92a86e524 storagenode/payouts: earned amount at satellite and list of paying satellites IDs added to DB
Change-Id: I66e34d6d80e37481d6eaae4c7bd017a7e096a751
2021-02-04 10:19:08 +00:00
Michał Niewrzał
9a60011774 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ia90f29be432e207c4125f7f955c912978eabe59a
2021-02-04 09:38:08 +01:00
Yingrong Zhao
21dfd99883 bump storj.io/private to latest
Change-Id: If4f08cd8cd8cd7cb09cd7d7051d966b08fe96c2e
2021-02-04 00:54:28 +00:00
Ivan Fraixedes
9c9f481469 satellite/orders: Remove deprecated endpoint
Remove the orders Settlement endpoint because it isn't used and it was
already always returning an error.

Change-Id: I81486fbe7044a1444182173bc0693698ee7cfe7e
2021-02-03 23:47:07 +00:00
Yingrong Zhao
0de3709f71 pkg/server: implement hybrid connector for tcp fallback
This PR introduces a new connector that will try to establish a tcp and
a quic connection concurrently and returns the connection that's
ready first. If both connections are ready around the same time, it will
prioritize/return quic connection.

Change-Id: Ib87175321516125526aa01c96386bdc8d3b863b0
2021-02-03 14:22:50 -05:00
Yingrong Zhao
53d189d712 private/testplanet: add reconfiguration for disabling QUIC listener and
TCP listener

Change-Id: I067656e755c4a85ada161ac80d52778ba91f1045
2021-02-03 18:43:00 +00:00
JT Olio
c8e42df2f8 satellitedb: reorder migrations 140-144
These changes are independently tracked on
https://github.com/storj/storj/tree/jt/migration-reorder

The point of this is to make the distributed column
migration, needed for SNO invoice generation, the very
next one, so we can release it as a point release.

Change-Id: I26e1c03629c4f079b9ad12485e2b71a715d82b3b
2021-02-03 18:28:42 +00:00
Yingrong Zhao
7e80badaf9 pkg/server,pkg/quic: accept an existing conn to create quic listener and
allow disabling tcp/quic

In order to have more control of a server so that we can
simulate connection failures in `testplanet`, this PR changes
quic.Listener to accept an existing UDPConn instead of relying on the
quic-go library to create the UDPConn.
This PR also adds two flags on the `server.Config` struct to allow
enabling/disabling tcp/tls listener and quic listener. By default, they
are both set to true.
    - `DisableTCPTLS`: internal flag, disables tcp/tls listener.
    - `DisableQUIC`: hidden flag, disables quic listener
By making the `DisableQUIC` a hidden flag, it allows storagenode operators to
have the ability to disable quic traffic in case their set up can't work
with udp traffic.

Change-Id: I853b12435d988b9c41ad9b873fd57480d792e378
2021-02-03 12:04:29 -05:00
Jeff Wendling
a754c3984b satellite/console/consolewasm: assert less about error
this changes from a satellite error to a local encryption
error with the upcoming permissions changes where we only
include keys for the paths that are allowed.

Change-Id: I7aa37cfbaee31a1e54afe0423b283b9f41d9345f
2021-02-03 10:36:58 -05:00
Ethan
9506b67ca2 satellite/projectaccounting: Improve performance of ProjectAccounting.getBuckets
Limit bucket name lookup to date range of the calling methods since we only need distinct bucket names for that time period.
Adds new index and removes an index specific to project ID since it is no longer needed.

Change-Id: Ic07bbfb1c32280e0c0e39f8da020b284e1e5d974
2021-02-03 14:05:12 +00:00
Fadila Khadar
f34519c969 metainfo/metabase: iterate through pending objects by key
Change-Id: I451453d52278a2ff15ade1a6fccede136ab314be
2021-02-03 11:44:05 +01:00
Kaloyan Raev
8d25c47897 satellite/repair: fix comment in TestRepairExpiredSegment
Change-Id: Ib91e81f6ba0a7f65daed157b78f7a1a108984930
2021-02-03 10:09:49 +02:00
Cameron Ayer
a17934cb51 satellite/satellitedb: remove reference to uptime counts
Change-Id: I26ac540b720a8ba5d6ca44526900228352dcaf4e
2021-02-02 14:51:27 -05:00
Jeff Wendling
759bdd6794 satellite/compensation: add total-paid and total-distributed to invoices
Change-Id: Id4414867917cbf8aad77795f764d6381e88d9a34
2021-02-02 18:14:31 +00:00
Kaloyan Raev
038bd0a4da satellite/repair/repairer: fix repair for pending objects
https://storjlabs.atlassian.net/browse/PG-160

Change-Id: Ice7a0dcfc591bcde85a355cf95fff1eb3411f508
2021-02-02 19:50:10 +02:00
Kaloyan Raev
6f3d0c4ad5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/repair/repair_test.go
	satellite/repair/repairer/segments.go

Change-Id: Ie51a56878bee84ad9f2d31135f984881a882e906
2021-02-02 19:19:04 +02:00
Kaloyan Raev
339d1212cd satellite/repair: don't remove expired segments from repair queue
It's impossible to time correctly this check. The segment may expire
just at the time we upload the repaired pieces to new storage nodes.
They will reject this as expired and the repair will fail.

Also, we penalize storage nodes with audit failure only if they fail
piece hash verification, i.e. return incorrect data, but only if they
have already deleted the piece.

So, it would be best if the repair service does not care about object
expiration at all. This is a responsibility of another service.

Removing this check will also simplify how we migrate this code
correctly to the metabase.

Change-Id: I09f7b372ae2602daee919a8a73cd0475fb263cd2
2021-02-02 16:13:01 +00:00
Caleb Case
22bc69ad6d
cmd/uplink: Add more details to access inspect
Change-Id: Ia7e600c214d6097b48b9c868532b8fbe21b0035c
2021-02-02 10:25:26 -05:00
Egon Elbre
0bbe2f1d61 satellite/metainfo: add unimplemented ListPendingObjectStreams
Change-Id: I6c0fd240ce5be82c1f464470a6f147289b1cdf9d
2021-02-02 16:51:36 +02:00
Qweder93
e383722812 storagenode/payouts: Distributed added to paystubs
Change-Id: I02ba1f681be84765a0a5dae7af17d2a6674e7cca
2021-02-02 13:51:43 +00:00
Ivan Fraixedes
cc0d88f9c3
satellite/satellitedb: Fix GE flaky test
Fix an issue due to copy-paste problem that made that the Graceful Exit
test to be flaky.

The test uses a time created at the beginning of the test for avoiding
to get undeterministic time differences due to the fact of the response
time variation by the DB queries, however some part of the test were
using a current time rather than this base time, so they have been
addressed.

Change-Id: I4786f06209e041269875c07798a44c2850478438
2021-02-02 13:24:42 +01:00
Kaloyan Raev
b519bb377d satellite/metainfo/metabase: GetSegmentByLocation
We need this method to fix repairing pending objects. In another PR, it
will replace the GetObjectLatestVersion + GetSegmentByPosition calls
that are currently executed.

Change-Id: I4c5c2ab604edf898452b6fd21b86d4d3f970ce79
2021-02-01 21:52:12 +02:00
Ivan Fraixedes
d93944c57b satellite/orders: Delete unused methods & DB tables
Delete satellite order methods and DB tables which aren't used anymore
after we have done a refactoring on the orders to stuck bucket
information in the orders' encrypted metadata.

There are also configuration parameters and a satellite chore that
aren't needed anymore after the orders refactoring.

Change-Id: Ida3682b95921df70792284b42c96d2508bf8ca9c
2021-02-01 18:01:29 +00:00
Ivan Fraixedes
076804eac9 cmd/satellite: Add command for GE data cleanup
Add a command to the satellite for cleaning up the Graceful Exit (a.k.a
GE) transfer queue items of nodes that have exited.

The commit adds to the GE satellite DB a couple of new methods, and its
corresponding test, for performing the operations of the new command.

Change-Id: I29a572a59689d63b24990ac13c52e76d65aaa917
2021-02-01 17:30:58 +00:00
Jeff Wendling
1cf3d89a56 satellite/satellitedb: add distributed column and migration
using redash i manually checked that the only times the sum of
the payments does not match the paid column is for 2020-12 and
if it does not match then there are no payments.

Change-Id: I71ce0571de7e38e21548d7d6757b25abc3bfa781
2021-02-01 16:33:14 +00:00
JT Olio
6e2450ecf7 uplink cli: make uplink share --register output access keys again
this used to work prior to https://review.dev.storj.io/c/storj/storj/+/3466

Change-Id: I9f02410f0e3768f0df0c5309c3107609fcbf7742
2021-02-01 15:50:02 +00:00