Commit Graph

314 Commits

Author SHA1 Message Date
Michał Niewrzał
12402eb729 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I38adf8218c1415c7ea1910f8bd6bed13544b0f03
2021-02-17 08:50:38 +01:00
Michał Niewrzał
8685c6c9cd private/testplanet: use better method in test
In ec client in uplink we have two methods Put and PutSingleResult. Logic is the same but result for PutSingleResult is combined into single var. We would like to remove unused Put method but to do this it needs to be replaced in test.

Change-Id: Ia65eff3ecc9e68d5b3180b557ea82fa31d3c969c
2021-02-16 13:24:01 +00:00
Jeff Wendling
fe5b6e1725 sat/console/consolewasm: restrict enc access to api key
Change-Id: Ie263ffc7343e59ee9a90996b259bb04258a78f29
2021-02-11 17:52:32 -05:00
Michał Niewrzał
5a23d9c167 metainfo-migrator: improve performance
Change-Id: I258605737c37dfdd8f0070dbadf0e3a9c6358cc3
2021-02-11 17:33:22 +01:00
Michał Niewrzał
908a96ae30 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I075aaff42ca3f5dc538356cedfccd5939c75e791
2021-02-11 11:48:23 +01:00
Michał Niewrzał
dc2bec9f89 satellite/metainfo: collect uplink versions with monkit
This change adds metrics for uplink version and method which was used by this version.

Change-Id: I05fc425a024805cfcd6d8add810a2a0749405f4f
2021-02-10 17:47:13 +01:00
Qweder93
49884b5aad multinode/console: payouts added
Change-Id: I003b6a6286e39f43b3eb5e50de54b0a3d62f5abc
2021-02-10 09:45:16 +00:00
Rafael Gomes
d9d6edde71 go.mod: bump monkit-jaeger to replace dashes by underscore
Change-Id: I51fd920901ff559e9def795e9183bbd6d06e4045
2021-02-09 15:24:00 -03:00
Caleb Case
f011a5e8cc
Use WithNonce when creating caveats
This ensures the caveats are unique even when they contain the same
permissions and will result in unique macaroons. This is important to
ensure revocation doesn't impact more macaroons than intended.

Change-Id: I6354edd0119f2d85eaf580f2d1926a3de9151b88
2021-02-04 13:46:32 -05:00
Egon Elbre
615586a471 {satellite,storagnode}/internalpb: bump gogo to v1.3.2
Change-Id: I585a3197630b9b675a4b540cf25612e1ed39b599
2021-02-04 11:09:13 +00:00
Michał Niewrzał
9a60011774 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ia90f29be432e207c4125f7f955c912978eabe59a
2021-02-04 09:38:08 +01:00
Yingrong Zhao
21dfd99883 bump storj.io/private to latest
Change-Id: If4f08cd8cd8cd7cb09cd7d7051d966b08fe96c2e
2021-02-04 00:54:28 +00:00
Ivan Fraixedes
9c9f481469 satellite/orders: Remove deprecated endpoint
Remove the orders Settlement endpoint because it isn't used and it was
already always returning an error.

Change-Id: I81486fbe7044a1444182173bc0693698ee7cfe7e
2021-02-03 23:47:07 +00:00
Kaloyan Raev
6f3d0c4ad5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/repair/repair_test.go
	satellite/repair/repairer/segments.go

Change-Id: Ie51a56878bee84ad9f2d31135f984881a882e906
2021-02-02 19:19:04 +02:00
Egon Elbre
0bbe2f1d61 satellite/metainfo: add unimplemented ListPendingObjectStreams
Change-Id: I6c0fd240ce5be82c1f464470a6f147289b1cdf9d
2021-02-02 16:51:36 +02:00
Natalie Villasana
91bd4191dd satellite/accounting: add rollup archiver chore
The rollup archiver chore moves bucket bandwidth rollups and
storagenode rollups that are older than a given duration
to two new archive tables.

Change-Id: I1626a3742ad4271bc744fbcefa6355a29d49c6a5
2021-02-01 09:29:54 -05:00
Kaloyan Raev
d0612199f0 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/config.go
	satellite/metainfo/metainfo_test.go

Change-Id: I95cf3c1d020a7918795b5eec63f36112fdb86749
2021-02-01 14:32:12 +02:00
Egon Elbre
ee1f67bb18 go.mod: bump quic to fix Go 1.16rc1
Change-Id: Ief21af5302c36a91d19755e58a2e36fae61cd15b
2021-02-01 13:32:56 +02:00
Egon Elbre
a97b5c8552 go.mod: bump quic to master, such it compiles with Go 1.16
Change-Id: I4af35cd561955094a34fe3c96b0c532f325ac505
2021-01-28 11:13:02 +02:00
Yingrong Zhao
02845e7b8f pkg/server,private/testplanet: start to listen on quic
This PR introduces a new listener that can listen for quic traffic on
both storagenodes and satellites.

Change-Id: I5eb5bc82c37dde20d3be2ec8fa5f69c18fae0af0
2021-01-27 11:03:42 -05:00
Michał Niewrzał
50bea3ab1a satellite/metainfo: adjust tests to changes in uplink
One of uplink method changed its signature and we need to fix test on satellite side.

Change-Id: Ib89ea6aa25c57bac11bc3e0e9c2c89a4b9debd7c
2021-01-26 08:54:35 +00:00
Qweder93
c139cbd76b storagenode/payouts: fix CurrentMonthExpectations timezone handling. Estimations based on node's join date.
On servers with non-UTC it would have calculated a different month boundary.
If node joined in current month calculations will be related on amount of days node've been working.

Change-Id: Ie572b197f50c6cdff5a044a53dfb5b9138f82f24
2021-01-25 19:03:30 +02:00
Egon Elbre
a700a1bdab satellite/metainfo/metabase: add benchmark
Change-Id: I4289aac252f9c951548444aa5a82e007e9f47806
2021-01-25 14:59:24 +00:00
Yingrong Zhao
ae03a0540c pkg/quic: add quic implementation
Due to the issues with the licensing and go version requirement by the
quic-go library, we can not introduce the library into libuplink at the
moment. Therefore, we decided to put our quic integration code into
storj/storj. It will unblock us from rolling out quic to storagenodes.

Change-Id: If9d765da10c45947a4e3a3a11e4679bab69bcf08
2021-01-21 21:28:30 +00:00
Kaloyan Raev
c24ada7114 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum

Change-Id: Icf7c029e9d800e5f6a9fdd208c36f28e05468690
2021-01-20 17:35:57 +02:00
Kaloyan Raev
ea48322dd3 satellite/metainfo: use deterministic signing for satStreamID
So we can have stable UploadID for multipart uploads.

Change-Id: Iac6780394c8cc0f96c0b9c4b850b92ed3627a9b0
2021-01-19 15:56:29 +00:00
Qweder93
6ba8f6c8a9 storanode, satellite: payout renamed to payouts, expected estimation payouts added, console api for audits reworked
Change-Id: I4aa5e99bffaa87d0a800a429a4c83aa498ad4b7b
2021-01-18 10:56:03 +00:00
Kaloyan Raev
424d2787eb go.mod: bump deps to uplink v1.4.5
Change-Id: Ic5a4efc78bb9c140a4104fd96c514477e225cba4
2021-01-14 17:51:29 +02:00
Kaloyan Raev
6dff40f5c5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/metainfo.go

Change-Id: Ib5c49f3c911c58319855a171f9ce73657da976d9
2021-01-14 14:33:59 +02:00
Egon Elbre
d11c2b709e go.mod: bump storj.io/common
* Add missing endpoints.
* Fix deprecated packages and funcs.

Change-Id: I756090c46a4d15eabf6d413a593cdc64c5809bc7
2021-01-13 14:51:08 +00:00
Michał Niewrzał
ec88d21a3c Merge 'main' branch.
Change-Id: I6e8162d1a6caf75e89c9f9c9f9522730aebf83ae
2021-01-11 10:26:58 +01:00
Kaloyan Raev
1e44be2900 go.mod: bump deps to uplink v1.4.4
Change-Id: I203259f0f324284ac2991e2cabf4bd471cd8ad5d
2021-01-06 13:07:17 +00:00
Michał Niewrzał
ad3e3a38c5 Merge 'main' branch
Change-Id: Ia0db1b1f9ef3e0671d3f2208881b0abc3064e200
2021-01-04 12:13:45 +01:00
Moby von Briesen
edbee53888 satellite,storagenode: Pass audit history over GetStats endpoint
Full prefix: satellite/{overlay,nodestats},storagenode/{reputation,nodestats}

Allow the storagenode to receive its audit history data from the
satellite via the satellite's GetStats endpoint.

The storagenode does not save this data for use in the API yet.

Change-Id: I9488f4d7a4ccb4ccf8336b8e4aeb3e5beee54979
2020-12-30 19:13:26 +00:00
JT Olio
f787aa4647 go.mod: bump uplink to v1.4.3
Change-Id: Ia6ff4f3a84897aaebe7aef7680f17b94b59cb9de
2020-12-23 11:18:26 +00:00
Michal Niewrzal
57f374af24 Merge 'master' branch
Change-Id: Idf6b10ea7ca94e4d232e6a3b6a38ef2e646ba197
2020-12-15 08:26:53 +01:00
Qweder93
c630037f34 multinodepb: diskspace, reputation and status added
Change-Id: I470fa8b59ce7f00f2fbedbd0c0878fb5fff0590c
2020-12-14 17:01:16 +00:00
Michal Niewrzal
b3acc1101a Merge 'master' branch
Change-Id: Iee99400c7095770e61cde94b3b2c8eb0ddec463d
2020-12-10 15:42:52 +01:00
Michal Niewrzal
c2a97aeb14 satellite/satellitedb: add ListAllBuckets method
We need to be able to list all buckets in DB without knowing project ID.
This method will be used to list buckets for metainfo loop
implementation based on metabase.

Change-Id: Iac75af0eee4f31e80a15577575a8249cbca787b2
2020-12-10 14:19:27 +00:00
Egon Elbre
588dff6d78 Upgrade to uplink v1.4.2
Change-Id: Ib1f5e70c76710d9c49a39347e4ed95bd4671efab
2020-12-10 15:47:11 +02:00
Kaloyan Raev
3269401515 deps: update to latest uplink from multipart-upload branch
This resolves an issue in Uplink CLI with listing size 0 for files
uploaded with multipart upload.

Change-Id: I80e0b11a96f87ed6a87eb5301034c08dbc09e8aa
2020-12-07 14:54:51 +00:00
Michal Niewrzal
218bbeaffa Merge 'master' branch
Change-Id: Ica5c25607a951076dd9f77e35e308062f71ce3f0
2020-12-07 15:05:52 +01:00
Caleb Case
c1bdb88d3c
Upgrade to uplink v1.4.1
Change-Id: Ie823f9b72d41f3da8e929d96a870fffc2e7ec12b
2020-12-04 11:38:19 -05:00
Caleb Case
6ba8dd58e6 Update to uplink v1.4.0
Change-Id: Ia01fec012031ceb2867b80a37e6a6c2a7d179cca
2020-12-02 13:44:32 +00:00
Jessica Grebenschikov
b261110352 satellite/orders: get bucketID from encrypted metadata in order instead of serial_numbers table
We want to stop using the serial_numbers table in satelliteDB. One of the last places using the serial_numbers table is when storagenodes settle orders, we look up the bucket name and project ID from the serial number from the serial_numbers table.

Now that we have support to add encrypted metadata into the OrderLimit, this PR makes use of that and now attempts to read the project ID and bucket name from the encrypted orderLimit metadata instead of from the serial_numbers table. For backwards compatibility and to ensure no errors, we will still fallback to the old way of getting that info from the serial_numbers table, but this will be removed in the next release as long as there are no errors.

All processes that create orderLimits must have an orders.encryption-keys set. The services that create orderLimits (and thus need to encrypt the order metadata) are the satellite apiProcess, the repair process, audit service (core process), and graceful exit (core process). Only the satellite api process decrypts the order metadata when storagenodes settle orders. This means that the same encryption key needs to be provided in the config for the satellite api process, repair process, and the core process like so:
orders.include-encrypted-metadata=true
orders.encryption-keys="<"encryptionKeyID>=<encryptionKey>"

Change-Id: Ie2c037971713d6fbf69d697bfad7f8b672eedd66
2020-12-01 15:29:32 +00:00
Michal Niewrzal
e4c4ab06ba satellite/metainfo: add testing methods for getting objects and segments
Change-Id: I4229d80090609df27d998a90bdb3137469dba042
2020-12-01 12:44:08 +00:00
Michal Niewrzal
5a7bc9657d Merge 'master' branch
Change-Id: If583132a821274dc4b78cf5f72b853ba8460c619
2020-11-30 12:57:22 +01:00
Michal Niewrzal
21602e0494 satellite/metainfo: enable commented test
Test was commented to make uplink refactoring possible. Now we can bring
back this test.

Change-Id: I0511b76073efaafed8aac97f8e845dcec93dd059
2020-11-30 10:49:23 +01:00
Michal Niewrzal
cfb45a785c satellite/metainfo/metabase: add TotalPlainSize to objects table
We need to know how large is object in general and we shouldn't query
segments for that.

Change-Id: Ie316e1ec38bfc39065e6046b79b55d58f3ce9d53
2020-11-25 14:35:28 +01:00
Michal Niewrzal
46102c1942 satellite/metainfo: change ListSegments required permission to
Read/Download

We should treat listing segments as a part of download process, not
listing.

Change-Id: Iae72c97c77976c427f668a96486af613d58da3b9
2020-11-24 17:50:33 +01:00