Commit Graph

5014 Commits

Author SHA1 Message Date
Kaloyan Raev
b519bb377d satellite/metainfo/metabase: GetSegmentByLocation
We need this method to fix repairing pending objects. In another PR, it
will replace the GetObjectLatestVersion + GetSegmentByPosition calls
that are currently executed.

Change-Id: I4c5c2ab604edf898452b6fd21b86d4d3f970ce79
2021-02-01 21:52:12 +02:00
Kaloyan Raev
6f441960ec satellite/metainfo/metabase: fix ParseSegmentKey to parse part numbers
ParseSegmentKey did not parse part numbers correctly from an encoded
SegmentKey.

Change-Id: I4ec7e15e0a094ed2a659c8c09d2f024eaceda678
2021-02-01 15:18:18 +02:00
Kaloyan Raev
d0612199f0 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/config.go
	satellite/metainfo/metainfo_test.go

Change-Id: I95cf3c1d020a7918795b5eec63f36112fdb86749
2021-02-01 14:32:12 +02:00
Egon Elbre
5d895fb404 storagenode/{payouts,console}: use same time for all calculations
When using calling time.Now() multiple times, they can cross
month boundary causing errors in calculations.

Change-Id: I66b5be7598f3bf475b4b5fe0dcce82eee55b3134
2021-02-01 13:59:50 +02:00
Egon Elbre
ee1f67bb18 go.mod: bump quic to fix Go 1.16rc1
Change-Id: Ief21af5302c36a91d19755e58a2e36fae61cd15b
2021-02-01 13:32:56 +02:00
Egon Elbre
8a3db08f68 storagenode/payouts/estimatedpayouts: fix calculations
Change-Id: Iaa01ebd06a32c19d3ddc46b52524020e51212a7b
2021-02-01 13:32:56 +02:00
Cameron Ayer
c5ecca1e1d web/storagenode: remove uptime columns and references
Full scope:
storagenode/{console,nodestats,notifications,reputation,storagenodedb},
web/storagenode

These columns are deprecated. They used to be for the uptime reputation
system which has been replaced by downtime tracking with audits.

Change-Id: I151d6569577d89733ac97af21a1d885323522b21
2021-01-29 15:36:59 -05:00
Egon Elbre
b7a0739219 satellite/overlay: use DownloadSelectionCache for getting node IPs
Change-Id: Ib8f4eedb2bf465767050693a1e961b37a294ca06
2021-01-29 16:47:10 +02:00
Egon Elbre
54e01d37f9 satellite/overlay: add DownloadSelectionCache
Change-Id: Ic0779280172325f8d03f55a2e9673722f72bdd44
2021-01-29 16:47:06 +02:00
Cameron Ayer
89e682b4d7 satellite/repair/checker: add 29/80/130-52 to default repair overrides
Change-Id: I2e5a7538fdf33f3869fcb65fc88f7abb10faad79
2021-01-28 16:55:16 -05:00
Yingrong Zhao
52d6852e58 pkg/server: add retry logic for random port assignment
Change-Id: I70464e344a79dce8eadb9513d2a990faf3b2cca8
2021-01-28 14:44:22 -06:00
Brandon Iglesias
fff10b041c
quick update to readme (#4033)
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2021-01-28 10:16:57 -05:00
Egon Elbre
19e3dc4ec0 satellite/overlay: rename NodeSelectionCache to UploadSelectionCache
It wasn't obvious that NodeSelectionCache was only for uploads.

Change-Id: Ifeeaa6fdb50a4b7916245b48d8634d70ac54459c
2021-01-28 14:56:53 +02:00
Egon Elbre
a97b5c8552 go.mod: bump quic to master, such it compiles with Go 1.16
Change-Id: I4af35cd561955094a34fe3c96b0c532f325ac505
2021-01-28 11:13:02 +02:00
Kaloyan Raev
4d32bdaefb satellite/satellitedb: drop bucket_metainfos_name_project_id_key index
This index is obsolete and duplicates a similiar (project_id, name)
index on the same table.

Moreover, it might confuse CockroachDB which of the two index to use,
which may might affect DB performance.

Change-Id: If8d1df8347714942cea9dca82864ba5f4973bed3
2021-01-28 09:06:22 +02:00
Yingrong Zhao
824fd6f131 pkg/quic: add backward compatibility for qtls
Change-Id: I2560074ece5b61d9ddb236269172325e1b5de83e
2021-01-27 15:24:50 -05:00
Jeff Wendling
ca86820b8b satellite/snopayouts: use dbx + some refactorings
Change-Id: I8f3973d2377f071bcea2f61e0fc21d913ffa7ea8
2021-01-27 17:59:16 +00:00
Jeff Wendling
66e15fb7f1 satellite/compensation: remove ytd paid amounts
they aren't right and we aren't using them.

Change-Id: I5ca024e38d055696696886278863e941b5bc51bf
2021-01-27 17:31:01 +00:00
Yingrong Zhao
02845e7b8f pkg/server,private/testplanet: start to listen on quic
This PR introduces a new listener that can listen for quic traffic on
both storagenodes and satellites.

Change-Id: I5eb5bc82c37dde20d3be2ec8fa5f69c18fae0af0
2021-01-27 11:03:42 -05:00
Kaloyan Raev
0f1961b7d1 satellite/metainfo/metabase: improve subquery comparision performance
Comparing the result from a subquery with the "IN" operator instead of
"=" makes a huge difference in the execution time of the SQL query on
CockroachDB.

Change-Id: I76e8f75a7bc95951667345d1ed9bd60f9aef3edb
2021-01-27 17:45:27 +02:00
Stefan Benten
f18cb24522
.clabot: add harrymaurya05 and gregoirevda (#4032)
Adding both as they signed the CLA.
2021-01-27 15:34:23 +01:00
Egon Elbre
da68f0cda7 cmd/metainfo-migration: add fastpb for faster unmarshaling
Change-Id: I7f9856fd48477cf54ade62b74935c26c411998aa
2021-01-27 16:26:12 +02:00
Caleb Case
d2148edcdb
cmd/uplink/cmd/setup.go: Access Grant vs API Key error messaging
Provide a clearer error message to users who confuse the API Key with
the Access Grant and suggest the right command to them.

Change-Id: If73ae8cde140b68a19f4cfc3f59bb88a3b74c9c1
2021-01-26 14:09:59 -05:00
crawter
93e8413617 web/mnd: my nodes page ui
Change-Id: I96403a6ae6ce232eee08208895e03c6a7aed74cb
2021-01-26 17:45:07 +00:00
crawter
9820145e14 web/mnd: nodes domain, api and store
Change-Id: I022c5153dfc85a25eebce6e8ba91b97e906736cb
2021-01-26 17:25:57 +00:00
crawter
135d846aff mnd/console/server: endpoint with index.html added
Change-Id: Ic5154feaa995bf5c26c851024079f7e82612f306
2021-01-26 17:16:11 +00:00
Malcolm Bouzi
24d60384c5
satellite/satellitedb: add columns for professional users (#4028)
Co-authored-by: Egon Elbre <egonelbre@gmail.com>
2021-01-26 11:38:53 -05:00
Isaac Hess
c92bda7e75 tally monkit: change location to monitor piecesize
When we observed the value for total piecesizes stored in the network,
we were doing it after converting them to byte-hours, rather than using
the actual piece sizes. This fixes that issue.

Change-Id: I1564d21b519f70eb59f298d97dbd777baf127723
2021-01-26 15:37:02 +00:00
littleskunk
0b2568d712
satellite/overlay/straynodes: increase development duration without contact
Stopping storj-sim for over 5 minutes caused nodes to be disqualified. Set development max duration without contact to 300 days.
2021-01-26 12:24:39 +02:00
Michał Niewrzał
3fc0d2a83e satellite/metainfo: add testing method from multipart-upload branch
We wanto have single uplink branch for standard and multipart-upload satellite but some tests are using helper methods from multipart. This change adds methods used by uplink test.

Change-Id: I82352ed56674ff7e8743b58061ba594018e78e3b
2021-01-26 09:13:12 +00:00
Michał Niewrzał
50bea3ab1a satellite/metainfo: adjust tests to changes in uplink
One of uplink method changed its signature and we need to fix test on satellite side.

Change-Id: Ib89ea6aa25c57bac11bc3e0e9c2c89a4b9debd7c
2021-01-26 08:54:35 +00:00
Moby von Briesen
8263f18321 satellite/console: Add graphql query for owned projects
Change-Id: If47183d46cb7552ecdddbb3e536c36d958fad6d0
2021-01-25 17:43:04 +00:00
Qweder93
c139cbd76b storagenode/payouts: fix CurrentMonthExpectations timezone handling. Estimations based on node's join date.
On servers with non-UTC it would have calculated a different month boundary.
If node joined in current month calculations will be related on amount of days node've been working.

Change-Id: Ie572b197f50c6cdff5a044a53dfb5b9138f82f24
2021-01-25 19:03:30 +02:00
Ivan Fraixedes
49c8e94480
scripts: Add test Satellite working w/o Redis
Create a storj-sim test that checks that uplinks operations works when
satellite runs and can connect to Redis and when it cannot connect to
simulate a Redis downtime. Also verifies that the satellite can start
despite of Redis being downtime.

This test currently doesn't pass and it will be the one used to verify
the work that has to be done to make sure that the satellite allow the
clients to perform their operations despite of Redis being unavailable.
We require these changes before we deploy any customer face satellite on
a multi-region architecture.

NOTE that this test will be added later on to Jenkins to run this test
every time that we apply changes and at that time we'll see if it has to
be adjusted for being able to run on Jenkins because as it's now it may
not work because the scripts start and stop a Redis docker container.

Change-Id: I22acb22f0ca594583e36b45c88f8c03bac73b329
2021-01-25 16:02:59 +01:00
Egon Elbre
a700a1bdab satellite/metainfo/metabase: add benchmark
Change-Id: I4289aac252f9c951548444aa5a82e007e9f47806
2021-01-25 14:59:24 +00:00
Kaloyan Raev
ba2907be1b satellite/metainfo: do not expire satStreamID
We are checking if satStreamID is created in the last 48 hours. If it is
older we treat is as expired an fail to unmarshal it.

Since the satStreamID is also the Upload ID for multipart uploads, this
means that all calls fail for multipart uploads older than 48 hours.
Even aborting old multipart uploads is not possible.

To resolve this issue, we should stop checking satStreamID for
expiration.

Change-Id: Ieaf53ed3cd800cdd08843676c2d9490b007d962e
2021-01-25 13:35:20 +00:00
Egon Elbre
b8fd8c775a satellite/metainfo/metabase: fix FixedSegmentSize for gapped parts
Parts that have segment index gaps should be treated similarly how
multipart objects are, because direct calculation of the segment does
not work.

Change-Id: I2717eac36f085b5100f3d600fcf0ce056202a9eb
2021-01-25 13:30:56 +02:00
Egon Elbre
2ce829a8b1 satellite/metainfo/metabase: fix segment index in tests
Change-Id: Iaf9591229454d54d6e6927893738be678926b31c
2021-01-25 13:27:24 +02:00
Egon Elbre
c44368c489 cmd/storj-sim: fix port assignment for multiple satellites
Currently first satellite GC would've conflicted with second satellites
public RPC port. Instead assign "satellite workers" a new peer class.

Change-Id: Id6bdaa17243556482e88da708c5147149788f6be
2021-01-22 14:08:39 +02:00
Yingrong Zhao
ae03a0540c pkg/quic: add quic implementation
Due to the issues with the licensing and go version requirement by the
quic-go library, we can not introduce the library into libuplink at the
moment. Therefore, we decided to put our quic integration code into
storj/storj. It will unblock us from rolling out quic to storagenodes.

Change-Id: If9d765da10c45947a4e3a3a11e4679bab69bcf08
2021-01-21 21:28:30 +00:00
Jessica Grebenschikov
b7d8dee5e9 satellite/console/wasm: add js tests
Change-Id: I8b1b0e81500836e0408e0517edb6c696698ab5f7
2021-01-21 20:18:03 +00:00
Michał Niewrzał
3c13aae61e satellite/metainfo: remove unused method
CreateGetOrderLimits is not used anymore because we have CreateGetOrderLimits2. We need to remove old method and fix name of second.

Change-Id: I59148b8d28fc9dbab7d452c884319125a02745d1
2021-01-21 17:00:13 +01:00
Jessica Grebenschikov
1f1d9fce58 satellite/console/wasm: add test to confirm wasm size isnt growing
Change-Id: I975a9f8ac3f6b98cc213140fdd7a99557efe14c8
2021-01-21 15:48:49 +00:00
VitaliiShpital
6c3b7a233e web/satellite: add beta messaging for EU and US2 sats
WHAT:
add a banner with a message saying: gateway MT is in beta for EU and US2 sats

WHY:
user experience?

Change-Id: I0964c4499cdaf12dbd9dba082910fff4ff6f9a12
2021-01-21 15:05:18 +00:00
Malcolm Bouzi
fa2dae0aca
Pin Project Owner in Team tab to top of members list (#4021)
* web/satellite: Pin Project Owner in Team tab to top of members list

* web/satellite:Pin Project Owner to top of members list
- input missing semicolon

* web/satellite: Pin Project Owner to top of members list
- refactor project owner variable instantion
- check for existance of owner before returning members list

* web/satellite: Pin project owner to top of members list
- changed returned members list variable name
- add comment explaining unshift logic
2021-01-21 10:03:02 -05:00
Michał Niewrzał
ce675b7707 satellite/metainfo/metabase: remove sql transaction while committing segments
Change-Id: Ia2cb1df63e1cbb8b9d36b0161a75b9c9cb3a56f9
2021-01-21 13:20:42 +00:00
Stefan Benten
ff102d6ffe
Makefile: update to go v1.15.7 (#4026) 2021-01-20 19:08:35 +01:00
Michał Niewrzał
292caa9043 satellite/metainfo/metabase: allow to set Encryption while committing object
In some cases we need to set encryption parameters later, with CommitObject method. This change makes Encryption optional with BeginObject* methods and mandatory with CommitObject if not set earlier.

Change-Id: I812c9b0e8fc213ca32d4758e0e68227e0e9bdd32
2021-01-20 18:28:25 +01:00
Michał Niewrzał
2d087c54b1 satellite/metainfo/metabase: use PlainSize for fixed segment calculation
In the past we were storing fixed segment size with StreamInfo, encrypted in metadata. The value was unencrypted size of segment, not encrypted one.

Change-Id: Id6b18440c674223eabbb152b1636c83e1ab6462c
2021-01-20 16:40:27 +00:00
Moby von Briesen
0a48071854 satellite/console: Add pagination fields for ListProjectsByOwnerID
Add ProjectsCursor type for pagination
Add PageCount, CurrentPage, and TotalCount ProjectsPage
This allows us to mimic the logic of GetBucketTotals and the
implementation of BucketUsages in graphql for the new ProjectsByOwnerID
functionality.

Change-Id: I4e1613859085db65971b44fcacd9813d9ddad8eb
2021-01-20 16:15:29 +00:00