Commit Graph

514 Commits

Author SHA1 Message Date
Michał Niewrzał
12402eb729 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I38adf8218c1415c7ea1910f8bd6bed13544b0f03
2021-02-17 08:50:38 +01:00
Michał Niewrzał
8685c6c9cd private/testplanet: use better method in test
In ec client in uplink we have two methods Put and PutSingleResult. Logic is the same but result for PutSingleResult is combined into single var. We would like to remove unused Put method but to do this it needs to be replaced in test.

Change-Id: Ia65eff3ecc9e68d5b3180b557ea82fa31d3c969c
2021-02-16 13:24:01 +00:00
Jeff Wendling
fe5b6e1725 sat/console/consolewasm: restrict enc access to api key
Change-Id: Ie263ffc7343e59ee9a90996b259bb04258a78f29
2021-02-11 17:52:32 -05:00
Michał Niewrzał
5a23d9c167 metainfo-migrator: improve performance
Change-Id: I258605737c37dfdd8f0070dbadf0e3a9c6358cc3
2021-02-11 17:33:22 +01:00
Michał Niewrzał
908a96ae30 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: I075aaff42ca3f5dc538356cedfccd5939c75e791
2021-02-11 11:48:23 +01:00
Michał Niewrzał
dc2bec9f89 satellite/metainfo: collect uplink versions with monkit
This change adds metrics for uplink version and method which was used by this version.

Change-Id: I05fc425a024805cfcd6d8add810a2a0749405f4f
2021-02-10 17:47:13 +01:00
Qweder93
49884b5aad multinode/console: payouts added
Change-Id: I003b6a6286e39f43b3eb5e50de54b0a3d62f5abc
2021-02-10 09:45:16 +00:00
Rafael Gomes
d9d6edde71 go.mod: bump monkit-jaeger to replace dashes by underscore
Change-Id: I51fd920901ff559e9def795e9183bbd6d06e4045
2021-02-09 15:24:00 -03:00
Caleb Case
f011a5e8cc
Use WithNonce when creating caveats
This ensures the caveats are unique even when they contain the same
permissions and will result in unique macaroons. This is important to
ensure revocation doesn't impact more macaroons than intended.

Change-Id: I6354edd0119f2d85eaf580f2d1926a3de9151b88
2021-02-04 13:46:32 -05:00
Egon Elbre
615586a471 {satellite,storagnode}/internalpb: bump gogo to v1.3.2
Change-Id: I585a3197630b9b675a4b540cf25612e1ed39b599
2021-02-04 11:09:13 +00:00
Michał Niewrzał
9a60011774 Merge remote-tracking branch 'origin/main' into multipart-upload
Change-Id: Ia90f29be432e207c4125f7f955c912978eabe59a
2021-02-04 09:38:08 +01:00
Yingrong Zhao
21dfd99883 bump storj.io/private to latest
Change-Id: If4f08cd8cd8cd7cb09cd7d7051d966b08fe96c2e
2021-02-04 00:54:28 +00:00
Ivan Fraixedes
9c9f481469 satellite/orders: Remove deprecated endpoint
Remove the orders Settlement endpoint because it isn't used and it was
already always returning an error.

Change-Id: I81486fbe7044a1444182173bc0693698ee7cfe7e
2021-02-03 23:47:07 +00:00
Kaloyan Raev
6f3d0c4ad5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/repair/repair_test.go
	satellite/repair/repairer/segments.go

Change-Id: Ie51a56878bee84ad9f2d31135f984881a882e906
2021-02-02 19:19:04 +02:00
Egon Elbre
0bbe2f1d61 satellite/metainfo: add unimplemented ListPendingObjectStreams
Change-Id: I6c0fd240ce5be82c1f464470a6f147289b1cdf9d
2021-02-02 16:51:36 +02:00
Natalie Villasana
91bd4191dd satellite/accounting: add rollup archiver chore
The rollup archiver chore moves bucket bandwidth rollups and
storagenode rollups that are older than a given duration
to two new archive tables.

Change-Id: I1626a3742ad4271bc744fbcefa6355a29d49c6a5
2021-02-01 09:29:54 -05:00
Kaloyan Raev
d0612199f0 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/config.go
	satellite/metainfo/metainfo_test.go

Change-Id: I95cf3c1d020a7918795b5eec63f36112fdb86749
2021-02-01 14:32:12 +02:00
Egon Elbre
ee1f67bb18 go.mod: bump quic to fix Go 1.16rc1
Change-Id: Ief21af5302c36a91d19755e58a2e36fae61cd15b
2021-02-01 13:32:56 +02:00
Egon Elbre
a97b5c8552 go.mod: bump quic to master, such it compiles with Go 1.16
Change-Id: I4af35cd561955094a34fe3c96b0c532f325ac505
2021-01-28 11:13:02 +02:00
Yingrong Zhao
02845e7b8f pkg/server,private/testplanet: start to listen on quic
This PR introduces a new listener that can listen for quic traffic on
both storagenodes and satellites.

Change-Id: I5eb5bc82c37dde20d3be2ec8fa5f69c18fae0af0
2021-01-27 11:03:42 -05:00
Michał Niewrzał
50bea3ab1a satellite/metainfo: adjust tests to changes in uplink
One of uplink method changed its signature and we need to fix test on satellite side.

Change-Id: Ib89ea6aa25c57bac11bc3e0e9c2c89a4b9debd7c
2021-01-26 08:54:35 +00:00
Qweder93
c139cbd76b storagenode/payouts: fix CurrentMonthExpectations timezone handling. Estimations based on node's join date.
On servers with non-UTC it would have calculated a different month boundary.
If node joined in current month calculations will be related on amount of days node've been working.

Change-Id: Ie572b197f50c6cdff5a044a53dfb5b9138f82f24
2021-01-25 19:03:30 +02:00
Egon Elbre
a700a1bdab satellite/metainfo/metabase: add benchmark
Change-Id: I4289aac252f9c951548444aa5a82e007e9f47806
2021-01-25 14:59:24 +00:00
Yingrong Zhao
ae03a0540c pkg/quic: add quic implementation
Due to the issues with the licensing and go version requirement by the
quic-go library, we can not introduce the library into libuplink at the
moment. Therefore, we decided to put our quic integration code into
storj/storj. It will unblock us from rolling out quic to storagenodes.

Change-Id: If9d765da10c45947a4e3a3a11e4679bab69bcf08
2021-01-21 21:28:30 +00:00
Kaloyan Raev
c24ada7114 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum

Change-Id: Icf7c029e9d800e5f6a9fdd208c36f28e05468690
2021-01-20 17:35:57 +02:00
Kaloyan Raev
ea48322dd3 satellite/metainfo: use deterministic signing for satStreamID
So we can have stable UploadID for multipart uploads.

Change-Id: Iac6780394c8cc0f96c0b9c4b850b92ed3627a9b0
2021-01-19 15:56:29 +00:00
Qweder93
6ba8f6c8a9 storanode, satellite: payout renamed to payouts, expected estimation payouts added, console api for audits reworked
Change-Id: I4aa5e99bffaa87d0a800a429a4c83aa498ad4b7b
2021-01-18 10:56:03 +00:00
Kaloyan Raev
424d2787eb go.mod: bump deps to uplink v1.4.5
Change-Id: Ic5a4efc78bb9c140a4104fd96c514477e225cba4
2021-01-14 17:51:29 +02:00
Kaloyan Raev
6dff40f5c5 Merge remote-tracking branch 'origin/main' into multipart-upload
Conflicts:
	go.mod
	go.sum
	satellite/metainfo/metainfo.go

Change-Id: Ib5c49f3c911c58319855a171f9ce73657da976d9
2021-01-14 14:33:59 +02:00
Egon Elbre
d11c2b709e go.mod: bump storj.io/common
* Add missing endpoints.
* Fix deprecated packages and funcs.

Change-Id: I756090c46a4d15eabf6d413a593cdc64c5809bc7
2021-01-13 14:51:08 +00:00
Michał Niewrzał
ec88d21a3c Merge 'main' branch.
Change-Id: I6e8162d1a6caf75e89c9f9c9f9522730aebf83ae
2021-01-11 10:26:58 +01:00
Kaloyan Raev
1e44be2900 go.mod: bump deps to uplink v1.4.4
Change-Id: I203259f0f324284ac2991e2cabf4bd471cd8ad5d
2021-01-06 13:07:17 +00:00
Michał Niewrzał
ad3e3a38c5 Merge 'main' branch
Change-Id: Ia0db1b1f9ef3e0671d3f2208881b0abc3064e200
2021-01-04 12:13:45 +01:00
Moby von Briesen
edbee53888 satellite,storagenode: Pass audit history over GetStats endpoint
Full prefix: satellite/{overlay,nodestats},storagenode/{reputation,nodestats}

Allow the storagenode to receive its audit history data from the
satellite via the satellite's GetStats endpoint.

The storagenode does not save this data for use in the API yet.

Change-Id: I9488f4d7a4ccb4ccf8336b8e4aeb3e5beee54979
2020-12-30 19:13:26 +00:00
JT Olio
f787aa4647 go.mod: bump uplink to v1.4.3
Change-Id: Ia6ff4f3a84897aaebe7aef7680f17b94b59cb9de
2020-12-23 11:18:26 +00:00
Michal Niewrzal
57f374af24 Merge 'master' branch
Change-Id: Idf6b10ea7ca94e4d232e6a3b6a38ef2e646ba197
2020-12-15 08:26:53 +01:00
Qweder93
c630037f34 multinodepb: diskspace, reputation and status added
Change-Id: I470fa8b59ce7f00f2fbedbd0c0878fb5fff0590c
2020-12-14 17:01:16 +00:00
Michal Niewrzal
b3acc1101a Merge 'master' branch
Change-Id: Iee99400c7095770e61cde94b3b2c8eb0ddec463d
2020-12-10 15:42:52 +01:00
Michal Niewrzal
c2a97aeb14 satellite/satellitedb: add ListAllBuckets method
We need to be able to list all buckets in DB without knowing project ID.
This method will be used to list buckets for metainfo loop
implementation based on metabase.

Change-Id: Iac75af0eee4f31e80a15577575a8249cbca787b2
2020-12-10 14:19:27 +00:00
Egon Elbre
588dff6d78 Upgrade to uplink v1.4.2
Change-Id: Ib1f5e70c76710d9c49a39347e4ed95bd4671efab
2020-12-10 15:47:11 +02:00
Kaloyan Raev
3269401515 deps: update to latest uplink from multipart-upload branch
This resolves an issue in Uplink CLI with listing size 0 for files
uploaded with multipart upload.

Change-Id: I80e0b11a96f87ed6a87eb5301034c08dbc09e8aa
2020-12-07 14:54:51 +00:00
Michal Niewrzal
218bbeaffa Merge 'master' branch
Change-Id: Ica5c25607a951076dd9f77e35e308062f71ce3f0
2020-12-07 15:05:52 +01:00
Caleb Case
c1bdb88d3c
Upgrade to uplink v1.4.1
Change-Id: Ie823f9b72d41f3da8e929d96a870fffc2e7ec12b
2020-12-04 11:38:19 -05:00
Caleb Case
6ba8dd58e6 Update to uplink v1.4.0
Change-Id: Ia01fec012031ceb2867b80a37e6a6c2a7d179cca
2020-12-02 13:44:32 +00:00
Jessica Grebenschikov
b261110352 satellite/orders: get bucketID from encrypted metadata in order instead of serial_numbers table
We want to stop using the serial_numbers table in satelliteDB. One of the last places using the serial_numbers table is when storagenodes settle orders, we look up the bucket name and project ID from the serial number from the serial_numbers table.

Now that we have support to add encrypted metadata into the OrderLimit, this PR makes use of that and now attempts to read the project ID and bucket name from the encrypted orderLimit metadata instead of from the serial_numbers table. For backwards compatibility and to ensure no errors, we will still fallback to the old way of getting that info from the serial_numbers table, but this will be removed in the next release as long as there are no errors.

All processes that create orderLimits must have an orders.encryption-keys set. The services that create orderLimits (and thus need to encrypt the order metadata) are the satellite apiProcess, the repair process, audit service (core process), and graceful exit (core process). Only the satellite api process decrypts the order metadata when storagenodes settle orders. This means that the same encryption key needs to be provided in the config for the satellite api process, repair process, and the core process like so:
orders.include-encrypted-metadata=true
orders.encryption-keys="<"encryptionKeyID>=<encryptionKey>"

Change-Id: Ie2c037971713d6fbf69d697bfad7f8b672eedd66
2020-12-01 15:29:32 +00:00
Michal Niewrzal
e4c4ab06ba satellite/metainfo: add testing methods for getting objects and segments
Change-Id: I4229d80090609df27d998a90bdb3137469dba042
2020-12-01 12:44:08 +00:00
Michal Niewrzal
5a7bc9657d Merge 'master' branch
Change-Id: If583132a821274dc4b78cf5f72b853ba8460c619
2020-11-30 12:57:22 +01:00
Michal Niewrzal
21602e0494 satellite/metainfo: enable commented test
Test was commented to make uplink refactoring possible. Now we can bring
back this test.

Change-Id: I0511b76073efaafed8aac97f8e845dcec93dd059
2020-11-30 10:49:23 +01:00
Michal Niewrzal
cfb45a785c satellite/metainfo/metabase: add TotalPlainSize to objects table
We need to know how large is object in general and we shouldn't query
segments for that.

Change-Id: Ie316e1ec38bfc39065e6046b79b55d58f3ce9d53
2020-11-25 14:35:28 +01:00
Michal Niewrzal
46102c1942 satellite/metainfo: change ListSegments required permission to
Read/Download

We should treat listing segments as a part of download process, not
listing.

Change-Id: Iae72c97c77976c427f668a96486af613d58da3b9
2020-11-24 17:50:33 +01:00
Michal Niewrzal
5be4d413b0 satellite/metainfo: set PlainSize for inline segment
Plain size is required by metabase implementation.

Change-Id: I6739744bff4ce485e0066f2a49acdf38560d31fb
2020-11-20 13:54:14 +01:00
Kaloyan Raev
05fe497a6f satellite/metainfo: add stream id when listing pending objects
Change-Id: Idcdffd70b82c0fe4f79635fc193604b66101ef76
2020-11-19 14:22:09 +02:00
Michal Niewrzal
9bdc8ebf75 satellite/metainfo: adjust GetObject to handle
EncryptedMetadataEncryptedKey

Change-Id: If159fdcc9f37e9449a7d8459a277366d6d6b5802
2020-11-17 16:38:10 +00:00
Michal Niewrzal
cd7d911b26 satellite/metainfo: implement ListSegments with metabase
Change-Id: I78fb2400fed0bb01c5e05bd5bbcddef5e4a3bdf0
2020-11-16 08:33:35 +00:00
Michal Niewrzal
3fe16f4003 satellite/metainfo: upload/download with metabase
This change is adjusting metainfo endpoint to use metabase for uploading
and downloading remote objects. Inline segments will be added later.

Change-Id: I109d45bf644cd48096c47361043ebd8dfeaea0f3
2020-11-11 12:13:52 +00:00
Michal Niewrzal
7dde184cb5 Merge 'master' branch
Change-Id: I6070089128a150a4dd501bbc62a1f8b394aa643e
2020-11-10 11:58:59 +00:00
Egon Elbre
b892a00143 mod: bump dependencies and reenable test
We shouldn't have any EOF issues with recent drpc fix, let's reenable
and see whether it's still flaky.

Change-Id: I0de312bcb087c7f70ec9d3281d73d86f971845d5
2020-11-10 10:32:21 +00:00
jessicagreben
c4c29e370a wasm: add webassembly code for creating access grant in console web UI
Change-Id: I3c6d9afc660f3d959d6138db84341e9460b877a1
2020-11-04 12:08:30 -08:00
Egon Elbre
716068a1e0 Merge branch 'master'.
Change-Id: Ic14325edc291573582dce0cea3e04991a820b48b
2020-11-02 13:02:01 +02:00
Egon Elbre
9674ce40e6 mod: bump dependencies
Change-Id: I61beaf9793609f7614478cfa604b2721d9010b52
2020-10-30 16:17:55 +02:00
Michal Niewrzal
809eb14ac2 satellite/metainfo/metabase: move metainfo PoC into storj repo
Change-Id: I39356d8bc7305b4a8ea0c1fb5603010ad72a68b9
2020-10-29 09:44:55 +01:00
Caleb Case
be84616e69
Update to uplink v1.3.1
Change-Id: Iae1e6a63c00d07fee9047638ddc9a414780ce0e9
2020-10-14 09:39:29 -04:00
Egon Elbre
2f4bb114d4 go.mod: bump common to remove sha256-simd
Change-Id: I77aeb3b4bea87e8d5bb83b05e86b61a80a695f2a
2020-10-14 12:47:37 +03:00
Caleb Case
cf1748158a
Bump Dependencies
Change-Id: I4c8a4438e74379a490a19f1f88ea9dac7715dbbd
2020-10-09 09:33:49 -04:00
Yingrong Zhao
f13084983d bump common to latest
Change-Id: I55ca40a7b2c58d70225da07b7a62d5ad60451881
2020-10-01 15:39:09 -04:00
Yingrong Zhao
c085a17a52 bump common and uplink to latest
Change-Id: I717f0214dd9973acd51b7732c5d64587f610c805
2020-10-01 15:38:58 +00:00
Egon Elbre
139ff3427b go.mod: bump dependencies
Change-Id: I2b1bf014a60ef6ddc4b1b546f58393c34ffe22fc
2020-09-30 11:28:37 +03:00
Egon Elbre
c23a8e3b81 go.mod: update pgx to v4.9.0
Fix query to use TextArray instead of VarcharArray.
Fix queries to use the correct type.

Change-Id: Ibb7e55adba277d05778118d81ca697470e72c374
2020-09-29 19:03:08 +00:00
Kaloyan Raev
b409b53f7f cmd/satellite: command for verifying piece hashes
Jira: https://storjlabs.atlassian.net/browse/PG-69

There are a number of segments with piece_hashes_verified = false in
their metadata) on US-Central-1, Europe-West-1, and Asia-East-1
satellites. Most probably, this happened due to a bug we had in the
past. We want to verify them before executing the main migration to
metabase. This would simplify the main migration to metabase with one
less issue to think about.

Change-Id: I8831af1a254c560d45bb87d7104e49abd8242236
2020-09-29 10:58:24 +00:00
Egon Elbre
888bfaae4b cmd/satellite: only add google profiler to satellite
Previously uplink, storagenode etc. included google cloud profiler,
however they don't need it.

Change-Id: Ibc95cb03d667a3844672eecd49fa455a6acc3866
2020-09-25 18:56:59 +03:00
JT Olio
1a094fd6da storj: bump storj.io/private
Change-Id: Ia2927010332bb2e1d140603299341a8549093233
2020-09-10 16:21:12 -06:00
Egon Elbre
b45cad5eed bump common to latest
This makes tests pass on Go 1.15.

Change-Id: I8407f16dcfc0b306aa685da30dcfedea6d3b515d
2020-09-05 10:51:58 +03:00
Yingrong Zhao
cccd62fff9 bump uplink version to v1.3.0
Change-Id: I9e7833713ca00f19bfb662eeaf61ef57bdc3bbbd
2020-09-04 09:57:35 -04:00
Yingrong Zhao
af773ec8a6 cmd/uplink: use DeleteBucketWithObjects for bucket force deletion
This PR updates `uplink rb --force` command to use the new libuplink API
`DeleteBucketWithObjects`.
It also updates `DeleteBucket` endpoint to return a specific error
message when a given bucket has concurrent writes while being deleted.

Change-Id: Ic9593d55b0c27b26cd8966dd1bc8cd1e02a6666e
2020-09-02 16:39:20 +00:00
Qweder93
36d752e92d storagenode/reputation: offline_under_review_at added
Change-Id: Ia7ec79b2d6f20fe29de0c36223f9485380d2845c
2020-09-02 18:48:28 +03:00
Qweder93
7d9897b7af storagenode/nodestats: online_score added
Change-Id: I84b50a6cace306e5f10d53a2073fe8810d4d2960
2020-09-02 17:45:01 +03:00
Isaac Hess
ba1a113e2e satellite/metainfo: Add test for GetObjectIPs
This PR adds the integration test for GetObjectIPs, testing all repos.

Change-Id: Id4d4f19c427a2e320b2a82efa68150fa7f5f86fb
2020-08-31 10:27:56 -06:00
Cameron Ayer
3e343b683b cmd/segment-reaper: add metrics for zombie segments count
Change-Id: I106c6795946283165ba3de8465e5898346da1a3f
2020-08-26 18:42:59 +00:00
Yingrong Zhao
bd5213f68b satellite/metainfo: implement batch delete for DeleteBucket
This PR changes DeleteBucket to be able to delete all objects within a
bucket if `DeleteAll` is set in `BucketDeleteRequest`.
It also changes `DeleteBucket` API to treat `ErrBucketNotFound` as a
successful delete operation instead of returning an error back to the
client.

Change-Id: I3a22c16224c7894f2d0c2a40ba1ae8717fa1005f
2020-08-24 13:28:09 +00:00
Isaac Hess
a14887e20b satellite/metainfo: Add GetObjectIPs method
This adds the unimplemented GetObjectIPs method to metainfo endpoint so
we can import new common protobuf definitions.

Change-Id: I154f26baccb6bb3c66de3eb25611930545c9754b
2020-08-11 11:39:15 -06:00
Caleb Case
00f9882ad5 bump to uplink v1.2.0
Also fixup config and related tests for storj.io/common
bee93690ab909580993f402b594ad5a14760c39f

Change-Id: I6897d925cd28b139c1fc2b43e63b08c0cc3fb688
2020-08-10 17:11:09 +00:00
Qweder93
4ee1b2d45a storagenode/console: added list of all audits per satellite to sno dashboard/satellites
Change-Id: I52e58748d6467f372d9a308347fc77e400d137e2
2020-08-10 12:55:07 +00:00
Caleb Case
7735b797c0
Bump dependencies for various fixes.
Change-Id: I79e4a985aef38f48ef49683eb19ce9dc0b3a2f85
2020-07-29 11:27:35 -04:00
Jeff Wendling
bdb3a10d39 mod: bump drpc
Change-Id: I584a2f55a391ce26e9fb1675e9f8a678febe649f
2020-07-29 14:24:07 +00:00
JT Olio
c89746a383 storagenode: use hardcoded metric application name
Change-Id: If11f226d03bc4bf70e2465569f8d578750718ae7
2020-07-28 16:24:15 +00:00
Egon Elbre
1f9ee4a536 mod: bump common
Update testrand.Metadata to prevent flaky test.

Change-Id: I8085f694403e738e3c6cbd8370945cf2c7dc17d3
2020-07-23 12:51:16 +00:00
Michal Niewrzal
18b3495248 satellite/metainfo: restore tests disabled to remove StreamID from
BeginDeleteObject method on uplink side

Change-Id: Id10cdeb367759f9949dc557b291726fd331d38ba
2020-07-22 13:54:39 +00:00
Jessica Grebenschikov
8abb907010 satellite/orders: add settle orders with window
Why: We need a way to cut down on database traffic due to bandwidth
measurement and tracking.

What: This changeset is the Satellite side of settling orders in 1 hr windows.
See design doc for more details: https://review.dev.storj.io/c/storj/storj/+/1732

Change-Id: I2e1c151e2e65516ebe1b7f47b7c5f83a3a220b31
2020-07-13 15:41:29 -07:00
paul cannon
bbdb351e5e all: use jackc/pgx in place of lib/pq
What:

Use the github.com/jackc/pgx postgresql driver in place of
github.com/lib/pq.

Why:

github.com/lib/pq has some problems with error handling and context
cancellations (i.e. it might even issue queries or DML statements more
than once! see https://github.com/lib/pq/issues/939). The
github.com/jackx/pgx library appears not to have these problems, and
also appears to be better engineered and implemented (in particular, it
doesn't use "exceptions by panic"). It should also give us some
performance improvements in some cases, and even more so if we can use
it directly instead of going through the database/sql layer.

Change-Id: Ia696d220f340a097dee9550a312d37de14ed2044
2020-07-13 15:54:41 +00:00
Egon Elbre
29ccda5d47 cmd/uplink/cmd: remove spf13/cast dependency
Change-Id: I8013014b0b37966128e673155d851705ee1d43db
2020-07-13 11:10:48 +00:00
Qweder93
b639ec08d4 satellite/heldamount: payments added, endpoind for payments added
Change-Id: Ia2b9580bc353ef614680230c6f82c5bf6ded49c4
2020-07-01 18:15:01 +03:00
Isaac Hess
4ce1c099b1 satellite/metainfo: Add RevokeAPIKey to batch
Change-Id: I53b233a5f9f1c8ce7d0bd30e88998f2cdee37762
2020-06-22 13:50:07 -06:00
Isaac Hess
34b019d16a satellite/metainfo: Handle revocation request
Logic to handle a request to revoke a macaroon.

Change-Id: I5d5c93fcc2e026b0aaf82bfdfacc822185b10f9f
2020-06-22 13:50:07 -06:00
Isaac Hess
2d727bb14e satellite: Check macaroon revocation
When a request comes in on the satellite api and we validate the
macaroon, we now also check if any of the macaroon's tails have been
revoked.

Change-Id: I80ce4312602baf431cfa1b1285f79bed88bb4497
2020-06-22 13:50:07 -06:00
Egon Elbre
6cc7fd5f31 satellite/metainfo: remove old endpoints
Change-Id: Ic85c27caeea08879714e8303b285b6136f9a7fd1
2020-06-16 16:12:14 +03:00
Isaac Hess
05a981bc13 satellite/metainfo: Add RevokeAPIKey stub endpoint
Change-Id: I07fb997dda48ea204aab11f656a1a37762684339
2020-06-16 07:03:02 -06:00
Michal Niewrzal
d414f98250 storj: update to uplink v1.1.1
Change-Id: Ia62f3a84cbca89320714e819fc445606365f0647
2020-06-16 10:55:07 +00:00
Qweder93
0826c8d87f satellite/heldamount: fix dimension of usage_at_rest
Change-Id: If1518ad41736912d15fb2c882c9e236c16f85a51
2020-06-11 13:07:51 +00:00
stefanbenten
dfeb4dc9c2 satellite/payments: add finalize invoice command
Change-Id: Ib74dc8fd352e9576a302f5923439f5e50f69c9f0
2020-06-10 14:48:05 +00:00
Qweder93
e52809d53e cmd/storagenode: add check if satellites available to gracefulexit
Change-Id: I8747507593d810bbdec0d140de0600ee147011c3
2020-06-10 13:38:36 +00:00
JT Olio
d9b1c4520e storagenode-updater: fix logic to use minimum version correctly
See:
  https://review.dev.storj.io/c/storj/storj/+/2052
  https://review.dev.storj.io/c/storj/private/+/2053

Change-Id: I30952c6e3893194ca876c12ecb7e53944e66aa2e
2020-06-08 17:43:10 +00:00
Egon Elbre
0de1b5fa9f ci: fix golangci-lint failures
Change-Id: Ia8c4dd03439ce4e3f85890d45255b8ea1c94b50a
2020-06-05 12:55:35 +00:00
Caleb Case
34ee3ef970
bump uplink to v1.0.7
Change-Id: Iab4ebded301b00bcf53c6dea012c35ce68eee8ba
2020-06-04 09:35:55 -04:00
Yingrong Zhao
9d7713cdd0 script/testdata: update tracing agent default address
Change-Id: I730994f16f135c4b8643a52f4cf499487e4af326
2020-06-03 23:46:18 +00:00
Egon Elbre
d6c90b7ab5 pkg/macaroon: remove aliased package
Change-Id: I8b06e9a794d00752948f190623006e1afffdb031
2020-05-29 13:34:20 +00:00
Caleb Case
2cf4f4d1a1 bump uplink to v1.0.6
Change-Id: Idcabd742c2bb6394b27466a5244915a6675dbf15
2020-05-29 13:11:36 +00:00
Michal Niewrzal
84892631c8 private/testplanet: remove old libuplink from testplanet
Change-Id: Ib1553f84d0b3ae12a5b00382f0f53357b6a273e2
2020-05-28 13:50:23 +00:00
Egon Elbre
bef84a5f9d storagenode: remove dependency to overlay.NodeDossier
This is the last dependency from storage node to satellite.

Change-Id: I12f7abb91e84f823ba5af126c6e2979519838612
2020-05-21 08:37:13 +03:00
Qweder93
49ad90dcd8 storagenode/reputation: unknown_score (unknown_alpha / unknown_alpha+unknown_beta) added to reputation stats, https://storjlabs.atlassian.net/browse/SG-326
Change-Id: I0b29ad736f7a11c7e57a846b6891f4b40755aa48
2020-05-20 11:25:14 +00:00
Egon Elbre
ed627144ed all: use DialNodeURL throughout the codebase
Change-Id: Iaf9ae3aeef7305c937f2660c929744db2d88776c
2020-05-20 10:36:30 +00:00
Kaloyan Raev
aac1e3c45f satellite/payments: move inspector commands to satellite cli
This allows to seeing logs in the output of the invoice commands.

Existing ensure-stripe-customer commands is moved from the 'reports' to
the new 'billing' root command.

Change-Id: I752c7ab6ca59bfac8e0f174a45d2ab45fc18e467
2020-05-18 16:12:45 +00:00
Kaloyan Raev
49571f1a23 satellite/payments: all invoice commands require period
To avoid including multiple months in a single invoice, we need all
inspector's invoice commands to run in for specific period.

See https://storjlabs.atlassian.net/browse/USR-725

Change-Id: I3637dc189234f02350daca8d897c21765762ea55
2020-05-14 11:50:19 +00:00
Ethan
159df8b2e4 Add logging listener for retrieving and setting log levels
See https://storjlabs.atlassian.net/browse/SM-752

These changes allow us to change the log level at runtime through a handler off of the debug endpoint.

Examples of changing the log level on storj-sim

To get the current level for the satellite api process:
curl -XGET 'http://127.0.0.1:10009/logging' --header 'Content-Type: text/plain'

To change the log level:
curl -XPUT 'http://127.0.0.1:10009/logging' --header 'Content-Type: text/plain' --data-raw '{"level":"error"}'

Change-Id: I05d164b290929fa06b6d78c01075ee41f8238044
2020-05-12 16:38:06 -04:00
Egon Elbre
93f1fe49e3 mod: update to latest common
Change-Id: Ie938b40c82ce22d017a0566f03fd32dbef7d6829
2020-05-11 17:26:12 +00:00
Egon Elbre
e6d5ce6b77 all: remove grpc
It seems everyone has migrated to drpc.

Change-Id: Ica6b2d0bdef68c6603083f2963458843eca71e9e
2020-05-10 06:36:09 +00:00
Egon Elbre
a549d523dc bump drpc and common
Change-Id: I231d72793cc07cefc59d013fce02c03d5638eabc
2020-05-08 10:50:57 +03:00
Qweder93
b754806b23 storagenode/reputation: unknown_audit_reputation_alpha and beta added to db, and reputation endpoint
Change-Id: I5d92268851bb9a202cc5d6b4d403467e6f692726
2020-05-05 15:48:04 +00:00
Egon Elbre
b7495d414d mod: update sqlite to v3.31.1
Change-Id: Ia2885f5385ffa227b8576c1d9e1c8bbba0c5d13d
2020-05-05 10:52:36 +00:00
Michal Niewrzal
16081f53d2 storj: update to uplink 1.0.5
Change-Id: Ie4f01356edc43c3ee2d124a69f4da1eb75a17618
2020-05-04 08:09:18 +00:00
Michal Niewrzal
b2acd93a78 storagenode/inspector: adjust Uptime type change in protobuf
Change-Id: I4f5110b534ac4f419b74f1a3dd72f8600e0a53a8
2020-04-29 09:28:37 +00:00
Yingrong Zhao
c5309a3f91 cmd/uplink: set sample rate for tracing to be 1 when tracing is enabled
when tracing is enabled, we should also set sampling rate to
a non-zero value. For now, we will set it to 1.
Uplink CLI users should be able to override it with the sample
flag.

Change-Id: I8bcf514fb14c2a1c4349b7957dd24ec23e4a85e5
2020-04-28 20:15:28 +00:00
Yingrong Zhao
9b4a3f8fcc cmd/uplink: use tracing.enabled flag
Previously we are using tracing.sampled to be the switch for turning on/off tracing.
However we would like to separate sampling rate from being the switch,
so we can set sampling rate to be 0 but still intialize tracing for
satellite and storagenodes

Change-Id: I27e6ba25ea6f6b612b4e1a57cf1301889ded41ec
2020-04-27 17:54:57 +00:00
Isaac Hess
db0371703f storagenode/pieces: Return UnhandledCount to satellite
When we receive a piece deletion request, include the number of piece
IDs we couldn't add to the queue in the reponse

Change-Id: Ibebbe92ac50105bb5c74b18211ed38d468eb33f3
2020-04-27 08:56:56 -06:00
Michal Niewrzal
c52fc964d5 Upgrade storj.io/uplink to v1.0.4
Change-Id: I5248fcfdb3ccd3c9b60d38f245b1eb6c0e37bf38
2020-04-27 07:35:53 +00:00
Yingrong Zhao
0bdcf123cf bump monkit, monkit-jaeger, and private to latest
Also bump storj.io/common and sync repo

Change-Id: If8e60db6bdf0af8077b7befcb1da304c3c4dcae4
2020-04-22 12:30:37 -04:00
Qweder93
805e328c47 storagenode/heldamount payments removed
Change-Id: I87cc04f43d182a4190a571ef417be85d02db9d34
2020-04-21 17:15:31 +00:00
Yingrong Zhao
8375a09c89 cmd: remove InitTracing from satellite and storagenode main.go file
Change-Id: I4addbe7d0645f66abfb3e98d74d17035e9624e69
2020-04-20 14:06:26 -04:00
Michal Niewrzal
884638a294 scripts: reduce segment size for integration tests
This will speedup integration tests.

Change-Id: Ifb1fadd01692390457971bfe7f9ba73a0796abb2
2020-04-20 15:13:05 +02:00
Qweder93
6e3585e394 satellite/heldamount/endpoint : GetAllPaystubs added
Change-Id: Ic8cdd9db8b2a68796f9579c7fed2d49d9054bd64
2020-04-19 19:21:54 +03:00
Egon Elbre
7e0e74c65c satellite/metainfo: attribution based on useragent
Automatically attach attribution information to bucket during
BeginObject or CreateBucket when the UserAgent is set.

Change-Id: I405cb26c5a2f7394b30e3f2cf5d2214c8781eb8b
2020-04-17 10:47:14 +03:00
Qweder93
c498e52cf2 satellite/nodestats/endoint: reputation stats updated
Change-Id: I03a44c4c3465d4f8f23fcbbc742148f5e49544e2
2020-04-16 22:28:58 +03:00
Kaloyan Raev
e35cbfb6ee bump storj.io/common
Change-Id: I280e7d69973f1508ae09fbe9cf2a8ceea9001dc3
2020-04-13 17:05:13 +00:00
Moby von Briesen
14b3704f56 storagenode: add suspended status to storagenode dashboard/api
* Add migration to storagenode reputation table to add suspended
timestamp
* Send suspended info to storagenode from satellite nodestats endpoint
* Add suspended status to storagenode api
* Add an indicator on the storagenode dashboard informing operator of
the satellites the node is suspended on

Change-Id: Ie3669f6069cc0258ba76ec99d17006e1b5fd9c8a
2020-04-09 13:36:23 +00:00
Egon Elbre
11a44cdd88 all: don't depend on gogo/proto directly
Change-Id: I8822dea0d1b7b99e0b828e0373a0308a42dde2be
2020-04-08 17:32:15 +00:00
Qweder93
3a9422cc9a satellite/nodestats: add pricing model to endpoint
Change-Id: Iddace8e437216a343458f440b543cee61164f233
2020-04-08 14:29:51 +00:00
Egon Elbre
cf26951a5b satellite/satellitedb/pbold: remove dead code
Change-Id: I7464773c20b8f99a601ca9cc4bee804f1ac14cf9
2020-04-08 15:22:31 +03:00
Yingrong Zhao
96e58d21b4 cmd;pkg/server: init tracing collector in all processes
Add tracing handler in drpc server.
Initializing tracing collector in admin, satellite api, garbage
collection, satellite core, repaier, storagenode.
Change-Id: Ie98420e35dfc6913836ebd82b517d9d12877aefc

Change-Id: I91057b6265a4ac8bde033dfde692b8a28acca99f
2020-04-07 17:20:59 -04:00
Michal Niewrzal
2d1a6968b0 satellite/metainfo: revert returning segment size and inline segment
size with BeginObject

Such solution will add one round trip to satellite during upload so for
now we are reverting this until we will have solution for this.

Change-Id: Ic2d826448ab7b0318cd6922df05deee9167cf2f0
2020-04-06 13:36:34 +02:00
Egon Elbre
6492b13d81 all: remove old uuid
Change-Id: I3a137f73456f010c37d3933dbe12cbbb840b809f
2020-04-02 19:30:36 +03:00
Michal Niewrzal
c178a08cb8 satellite/metainfo: add max segment size and max inline size to
BeginObject response

We want to control inline segment size and segment size on satellite
side. We need to return such information to uplink like with redundancy
scheme.

Change-Id: If04b0a45a2757a01c0cc046432c115f475e9323c
2020-04-02 12:41:28 +00:00
Egon Elbre
8f73fb7a32 all: simplify uuid usage
uuid.UUID implements driver.Value so it can be directly used as a
scannable result.

Replace uses of dbutil.BytesToUUID with uuid.FromBytes.

Change-Id: I51a670185ceb3cc2199d5aa2b76bc3fc191ca8fe
2020-04-02 05:48:58 +00:00
Kaloyan Raev
473cd4695f bump storj.io/uplink to v1.0.3
Change-Id: I9466214260d139eed155c2d8fd725572cd5f0425
2020-04-01 11:12:59 +00:00
Egon Elbre
0a69da4ff1 all: switch to storj.io/common/uuid
Change-Id: I178a0a8dac691e57bce317b91411292fb3c40c9f
2020-03-31 19:16:41 +03:00
JT Olio
f28100b73f bump storj.io/private
Change-Id: I4ddd5c34521602967b89bd18e2a71a6f1e29f436
2020-03-27 21:57:35 +00:00
Michal Niewrzal
75961fe8ce cmd/gateway: remove gateway command from repository and adjust `make
install-sim`

Change-Id: Ie55989ec0d702f6037843a8b6664f4f4f6347446
2020-03-27 07:42:42 +00:00
Yingrong Zhao
fecd36f6b6 bump uplink version to latest
Change-Id: I57c7dae60e4b521bf6f5a07b107f5c2ef18fc168
2020-03-26 09:03:53 +00:00
Egon Elbre
c715c75fea pkg/server: add counters for grpc calls
This will help to determine how many grpc calls are made to the
satellite.

Also remove the grpc funcs that have been added to upstream.

Change-Id: I91878f4fd10f9bfe601c94222c102eaaf4d35963
2020-03-25 21:38:13 +02:00
Yingrong Zhao
b7b19289d1 bump storj.io/common to latest
Change-Id: I16e337660ce8e1ef332cc842dbf4cfa067b9b98b
2020-03-25 09:08:40 -04:00
Kaloyan Raev
0a50d6bdce bump storj.io/uplink to v1.0.1
Change-Id: I55d44aee1a02fe662408a3c48a3e97d6721994af
2020-03-25 10:41:34 +02:00
Yingrong Zhao
a731472496 bump storj.io/common to latest and storj.io/drpc to v0.0.11
Change-Id: I7a6e823b441eeff4621dfdf2d6577be76c9761c8
2020-03-24 15:17:10 -04:00
Egon Elbre
326c0cebde storage/boltdb: update to etcd/bbolt v1.3.4
bbolt v1.3.4 has pointer usage fixes.

Change-Id: I5e0fc4782711d01c09ced579f25a4f8fbc8de85c
2020-03-24 12:33:34 +02:00
Michal Niewrzal
f0aeda3091 storj: remove from storj/pkg packages moved to storj/private repo
* debug
* traces
* cfgstruct
* process

Package `storj/private/version` will be removed as a separate change.

Change-Id: Iadc40faa782e6225513b28218952f02d9c240a9f
2020-03-24 09:56:29 +01:00
Isaac Hess
ddf87286c5 cmd/statreceiver: Remove statreceiver code
Change-Id: Ic572a1623a2008c49d584b769d13e07e4470bce0
2020-03-23 13:00:13 -06:00
Egon Elbre
6a7571f73e cmd/s3-benchmark: move to storj.io/benchmark
Change-Id: Idca2b836bdf876ca28eb5cabc9bfae1d576e4a3e
2020-03-23 19:09:42 +02:00
Egon Elbre
1b6ab173a8 private/context2: moved to storj.io/common/context2
Change-Id: Ic1dd1ed645ff3e1057c9b2b143e2c3ddf29d678e
2020-03-20 14:39:46 +00:00
JT Olio
b2590cf283
bump uplink to 1.0.0 (#3816)
What: bumps uplink to 1.0.0
Why: we just released it!
2020-03-19 15:38:27 -06:00
Kaloyan Raev
10b032e484 libuplink: return deleted bucket/object (step 4)
Switch back to the original DeleteBucket and DeleteObject methods.

Next step: remove the DeleteBucketReturnDeleted and
DeleteObjectReturnDeleted from storj.io/uplink.

Change-Id: I273a305326d411e51ce354ce72fcc6ecadf4dd5f
2020-03-19 13:32:07 +02:00
Kaloyan Raev
78b253c774 libuplink: return deleted bucket/object (step 2)
step 1 in https://review.dev.storj.io/c/storj/uplink/+/1236

Now the old libuplink uses the temporary DeleteBucketReturnDeleted and
DeleteObjectReturnDeleted methods. This way, in the next step, we will
be able to change the DeleteBucket and DeleteObject methods to return
the deleted bucket/object.

Change-Id: I2e638be1960bca6ce1456c92849fcdd6d93e5252
2020-03-18 17:26:23 +00:00
Michal Niewrzal
19685ad81a go.mod: bump common and uplink versions
Change-Id: I9894a1bdbc11b9785d5c0744675d720545f398c0
2020-03-18 15:25:54 +00:00
Stefan Benten
52590197c2
satellite/payments: More Cleanup and Satellite command to ensure we have stripe customers (#3805) 2020-03-16 20:34:15 +01:00
Kaloyan Raev
27f811a9e1 metainfo: delete methods return the deleted item
This only happens if Read or List permission is granted together with
the Delete permission

Change-Id: I68b5f04a476bddabe499809ac98097aac75732a8
2020-03-16 16:26:16 +02:00
Isaac Hess
85b6316ce8 statreceiver: Update to filter on packet headers
The admission/v3 protocol now supports arbitrary key/value headers to be
included in each packet of metrics. This commit creates support for
this, so the lua config file can declare a filter taking into account
the key/value headers.

Change-Id: I41de8c018d33304ccf46ec221ae689d55c5fb1ee
2020-03-12 13:57:07 -06:00
Isaac Hess
8e274a5ce4 cmd/uplink: Enable telemetry on ctx
Change-Id: I26ffbbb72b9f2ca71b629ba12637c74c2aa072c3
2020-03-12 14:52:40 +00:00
Michal Niewrzal
c20cf25f35 cmd: migrate uplink CLI to new API
Change-Id: I8f8fcc8dd9a68aac18fd79c4071696fb54853a60
2020-03-09 13:26:29 +00:00
Jessica Grebenschikov
bcb0453db2 upgrade dependencies for trace db debug endpoint
Change-Id: I4de658b361bb39ce28dc31b982895bb4f45b580a
2020-03-04 07:35:34 +00:00
Cameron Ayer
7244a6a84e storagenode/{contact, piecestore}: implement low disk notification with cooldown
When a storagenode begins to run low on capacity, we want to notify
the satellite before completely running out of space. To achieve this,
at the end of an upload request, the SN checks if its available space has
fallen below a certain threshold. If so, trigger a notification to the
satellites.

The new NotifyLowDisk method on the monitor chore is implemented using the
common/syn2.Cooldown type, which allows us to execute contact only once
within a given timeframe; avoiding hammering the satellites with requests.
This PR contains changes to the storagenode/contact package, namely moving
methods involving the actual satellite communication out of Chore and into
Service. This allows us to ping satellites from the monitor chore

Change-Id: I668455748cdc6741291b61130d8ef9feece86458
2020-03-03 10:45:37 -05:00
Egon Elbre
decb2ec69a private/processgroup: moved to storj.io/common/processgroup
Change-Id: I1ec0bb440dda757d8f9a6f564a0084dde2f9cc84
2020-03-03 10:50:33 +00:00
Egon Elbre
64330c55b3 all: use pbgrpc
common/pb moved grpc to a separate package common/pb/pbgrpc.
This updates this repository to use it.

Change-Id: I2de2a190688871cf9cb61f7ea511f8a01e264e4e
2020-02-26 21:27:47 +02:00
Egon Elbre
8822e98c1f cmd/gateway: simplify module handling
Change-Id: If6ed158a6c9568fa33f69ca2d52e231ee4fcb0cb
2020-02-26 17:59:45 +00:00
Jessica Grebenschikov
e19e3c1101 pkg/process:
Now that we are trying to identify the root cause of the satellite load limitations (i.e. currently the satellite has a max ability of 400 rps for uploads and we need this to be higher), we are using the golang diagnostic tools to collect insight into what the bottlenecks are.  We currently have a debug endpoint to gather some cpu and mem data, but it could be useful to have continuous profiling. GCP stackdriver has support for continuous profiling so lets set that up and see if it is helpful to gather more data.

This PR adds support for [GCP continuous profiler](https://cloud.google.com/profiler) which allows enabling continuous cpu/mem profiling and the stats are sent to stackdriver in google cloud console.

To enable the continuous profiling for a storj component, do the following:
- prereq: the workload must be running in GKE and have Stackdriver Profiling IAM role permissions
- provide the config flag `debug.profilename` in the config.yaml file for the workload (i.e. satellite api process, etc). The profilename should be the workload name, for example "satellite-api".
- once the above config flag is provided, the profiler will be initialized and profiling stats will automatically be sent to GCP project where the workload is running and viewable in the Stackdriver Profile page in the console

The current implementation assumes the workload is running in GKE, however if we find if useful we can add support to enable this from anywhere. But for simplicity, its configured this way assuming the main goal is to enable in production systems.

Change-Id: Ibf8ebe2df7bf06fdd4951ee6a1e48854dd36ad47
2020-02-25 09:04:23 -08:00
Egon Elbre
29452d82a5 go.mod: unlock graphql dependency and bump to latest
Change-Id: I40026f6c8de155e024f5fbb51105546065393034
2020-02-25 13:17:49 +02:00
Egon Elbre
9752d01884 private/prompt: remove dependency to go-prompt
Change-Id: Ida8ef731ce806cec076343dc77d72a3b0d7736b4
2020-02-25 13:09:41 +02:00
JT Olio
50a21de9dc traces: fix memory leak for long running traces that aren't being collected
for real this time. i'm so ashamed

Change-Id: Ib05bb50d8e947dec2d872fd53e71eec561c2d0e8
2020-02-24 15:02:26 -07:00
Egon Elbre
e30f7b35b6 cmd/gateway: use a separate repository
Change-Id: Idbb0b2b6cf0e60c6d5d91218c24524d72285cf26
2020-02-24 10:03:03 +02:00
Yingrong Zhao
5011e78311 storagenode/piecestore: remove unused DeletePiece endpoint
With commit: 3331b443e7, satellite will
start calling `DeletePieces`. Therefore, we can remove the old endpoint
once the above commit is deployed with all satellites

Change-Id: I0124bc00a7cb808d119eb59f8fcd7fadf68158bb
2020-02-21 21:03:49 +00:00
Michal Niewrzal
54e38b8986 pkg/miniogw: gateway implementation with new libuplink
Change-Id: I170c3a68cfeea33b528eeb27e6aecb126ecb0365
2020-02-21 16:20:38 +01:00
Egon Elbre
5342dd9fe6 go.mod: update uplink
Change-Id: I867a6a1eef8aa5d60bb676e5112b98c4192ce811
2020-02-21 16:08:12 +02:00
Ivan Fraixedes
0a8f268a7e
go.mod: Update golang.org/x/crypto to fix vulnerability
Update the golang.org/x/crypto package to fix the vulnerability
CVE-2020-9283.

See https://groups.google.com/forum/#!topic/golang-nuts/XDqhhjZViNk

Change-Id: I7c841c0bae0f55dad0c7de19ac70c730d11733f0
2020-02-21 11:30:05 +01:00
Yingrong Zhao
e6da8d0249 satellite/metainfo: use global limiter for DeletePieces Service
we want to return back to the user as quick as possible but also keep
deleting remaining pieces on the storagenodes

Change-Id: I04e9e7a80b17a8c474c841cceae02bb21d2e796f
2020-02-19 12:17:36 +00:00
Egon Elbre
892b190db6 satellite/admin: add project limit modification and authorization token
Change-Id: If9a7214a940b8544f8023c2cd82da21f19d3f521
2020-02-17 07:56:16 +00:00
JT Olio
900cc47772 traces: fix memory leak for long running traces that aren't being collected
Change-Id: I7576e5f420e83c52c5bdb65a30f68aa8ee3d3cc8
2020-02-13 03:16:31 -07:00
Jeff Wendling
05a240050e storagenode: monitor available space and bandwidth
Change-Id: I5763597327c5b32982faab8910c136c6c8dc18c5
2020-02-13 07:07:29 +00:00
Qweder93
eeaaa8aa98 satellite/payments/stripecoinpayments: added ApplyInvoiceCredits
Change-Id: I7ed9d8397c0aa59d4ce0d40d1e50d13929e0fe5f
2020-02-12 20:06:08 +02:00
Michal Niewrzal
cea4c25f53 mod: bump common and uplink version
Change-Id: Ia063d33c087dd91a46c008e154b078f11fa21527
2020-02-12 14:33:54 +00:00
Egon Elbre
34f38bf6ce mod: upgrade miniredis to latest
miniredis 2.5.0 had a bug with matching keys with newlines.

Change-Id: I9bcf998459be6d7d4e03bca3589e989e5ed2304d
2020-02-06 13:31:17 +00:00
Jeff Wendling
7999d24f81 all: use monkit v3
this commit updates our monkit dependency to the v3 version where
it outputs in an influx style. this makes discovery much easier
as many tools are built to look at it this way.

graphite and rothko will suffer some due to no longer being a tree
based on dots. hopefully time will exist to update rothko to
index based on the new metric format.

it adds an influx output for the statreceiver so that we can
write to influxdb v1 or v2 directly.

Change-Id: Iae9f9494a6d29cfbd1f932a5e71a891b490415ff
2020-02-05 23:53:17 +00:00
Jessica Grebenschikov
dd9d18f152 upgrade drpc so that we have the monkit metric capability
Change-Id: Icdd08478aeff4fbd7148975eca8a21fac41289d7
2020-02-03 17:05:54 +00:00
Jeff Wendling
d20db90cff private/dbutil/txutil: create new transactions for retries
it was noticed that if you had a long lived transaction A that
was blocking some other transaction B and A was being aborted
due to retriable errors, then transaction B was never given
priority. this was due to using savepoints to do lightweight
retries.

this behavior was problematic becaue we had some queries blocked
for over 16 hours, so this commit addresses the issue with two
prongs:

    1. bound the amount of time we will retry a transaction
    2. create new transactions when a retry is needed

the first ensures that we never wait for 16 hours, and the value
chosen is 10 minutes. that should be long enough for an ample
amount of retries for small queries, and huge queries probably
shouldn't be retried, even if possible: it's more preferrable to
find a way to make them smaller.

the second ensures that even in the case of retries, queries that
are blocked on the aborted transaction gain priority to run.

between those two changes, the maximum stall time due to retries
should be bounded to around 10 minutes.

Change-Id: Icf898501ef505a89738820a3fae2580988f9f5f4
2020-02-01 18:34:28 +00:00
Michal Niewrzal
a181e0b627 libuplink: adjust tests to changes in encryption store
We move PathCipher to encryption.Store and we need to adjust
storj/uplink for those changes. Uplink repo is also using libuplink to
run tests so we need first adjust storj/storj libuplink and later
storj/uplink.

Change-Id: I84f23e6bad18ac139f72c19939dc526f9f46d88b
2020-01-30 22:00:24 +00:00
Yaroslav Vorobiov
083b396c16 satellite/payments: allow floating point numbers for pricing
Change-Id: I78b60134cf043746efef5371b761939a10f75aaf
2020-01-28 22:52:13 -05:00
Jessica Grebenschikov
54dbaaece2 satellite/orders: create as many orderLimits as needed to download a file
Change-Id: I2a39483d35037d9940913c035a78a93ea692ce9f
2020-01-28 20:04:11 +00:00
Egon Elbre
f4317d257a mod: bump uplink and common
Change-Id: I83874539c705e4b22940bc15ea990fe879dde721
2020-01-27 08:20:10 -05:00
Jeff Wendling
16bb374deb storagenode/piecestore: add large timeouts to read/write operations
this is to help protect against intentional or unintentional
slowloris style problems where a client keeps a tcp connection
alive but never sends any data. because grpc is great, we have
to spawn a separate goroutine for every read/write to the stream
so that we can return from the server handler to cancel it if
necessary. yep. really.

additionally, we update the rpcstatus package to do some stack
trace capture and add a Wrap method for the times where we want
to just use the existing error.

also fixes a number of TODOs where we attach status codes to the
returned errors in the endpoints.

Change-Id: Id8bb8ff84aa34e0f711b0cf9bce3908b36a1d3c1
2020-01-23 19:20:49 +00:00
Yingrong Zhao
5de4f66553 scripts/tests: change multisegment file to be 128kb
To cover a special case: an object that has 2 remote segments
and 1 inline segment.

Change-Id: Ia8d82bb67fc6cf76af9c7f44cd738cab6df591e9
2020-01-22 17:12:11 +00:00
Michal Niewrzal
86f194769f uplink: adjust to changes in storj/uplink
This change is adjusting code base to changes in storj/uplink.

https://review.dev.storj.io/c/storj/uplink/+/643

Change-Id: Ieca87f9f5983e391bf4b4fec8b9d5491fd32bfa1
2020-01-20 22:06:19 +00:00
Egon Elbre
64fb2d3d2f Revert "dbutil: statically require all databases accesses to use contexts"
This reverts commit 8e242cd012.

Revert because lib/pq has known issues with context cancellation.
These issues need to be resolved before these changes can be merged.

Change-Id: I160af51dbc2d67c5449aafa406a403e5367bb555
2020-01-15 07:28:00 +00:00
JT Olio
c01cbe0130 satellitedb: save out all db-touching traces
Change-Id: Ib1e192221f9da813fd9cbb55f620a047b82c9523
2020-01-14 18:47:45 -05:00
JT Olio
8e242cd012 dbutil: statically require all databases accesses to use contexts
this will allow for some nice runtime analysis down the road.
also, this allows for wrapping database handles in a way that
can interact with these contexts

requires https://review.dev.storj.io/c/storj/dbx/+/514

Change-Id: Ib087b7cd73296dd2c1e0331314da34d861f61d2b
2020-01-14 18:20:47 -05:00
crawter
41d5e86306 satellite/payments: coupon addition removed
Change-Id: I92781d9133603fdefd58b19a6f0ac6b1c6df3ac6
2020-01-14 16:24:48 +00:00
crawter
a57ce18f58 satellite/payments: coupons, coupons usage, invoice generation with pricing model applied
Change-Id: Ic5d5a2fc116388647efe46896cfccc2038c77537
2020-01-14 12:45:00 +00:00
Yingrong Zhao
ee87846f0b satellite/contact: add placeholder for GetTime endpoint
Change-Id: I42f8479708f0558350c2280a398d84d145e8118f
2020-01-14 06:38:47 +00:00
Egon Elbre
b252fcdc01 mod: bump uplink dependency
Change-Id: I67bee25c7cae28445e5091ed519359cfb5ea1af5
2020-01-09 15:38:02 +00:00
VitaliiShpital
a4e5c18877 satellite/payments: mock methods added to endpoint to match pb PaymentsServer
Change-Id: Ic8ff44cbe0b2368021a5d83cf86ce0dd2b670fd7
2020-01-09 14:30:21 +02:00
Egon Elbre
8d8d57c3b5 mod: update sqlite module to v2.0.2
This updates SQLite amalgamation from 3.29.0 to 3.30.1.

The module contains fixes for races.

Change-Id: Ic6a06a43ba404de0091d8a2f7444a8f4b1d5d54c
2020-01-08 21:21:15 +02:00
Egon Elbre
082ec81714
uplink: move to storj.io/uplink (#3746) 2020-01-08 15:40:19 +02:00
Egon Elbre
00c0c51b1c cmd/uplink: fix TestSetGetMeta flakiness
testrand.Path was also returning folders which has different behavior
for cp.

Change-Id: Ia53a2709bf3e768b3b7063a6137ec474c2622cb2
2020-01-08 12:25:25 +00:00
Egon Elbre
05e2b0ca86 mod: bump common and some other packages
Change-Id: Icf73220f5182f5a2fd1431471825948c300c0bea
2020-01-07 18:09:59 +02:00
Jeff Wendling
2549c601e9 all: bump storj.io/common dependency
Change-Id: I2e9ba0a76380d99d8650ae6921ed0a3ffe436536
2020-01-06 22:41:06 +00:00
Natalie Ventura Villasana
aa3e183c2e
satellite/gracefulexit: add ge eligibility check
Adds check to see if storage nodes are eligible to initiate
graceful exit, by checking their CreatedAt date and seeing if
their "age" is greater than the new config value:
NodeMinAgeInMonths
The default for this value is 6 months for now.

https://storjlabs.atlassian.net/browse/V3-3357

Change-Id: Ib807ab8987ddb5a38a27a83886490f73fe8c5816
2019-12-31 09:31:58 -05:00
Egon Elbre
6615ecc9b6 common: separate repository
Change-Id: Ibb89c42060450e3839481a7e495bbe3ad940610a
2019-12-27 14:11:15 +02:00
Egon Elbre
ef8bc88328 ci: use external repository
Change-Id: If26a005df45f6067240511d603fb4dd613f92b79
2019-12-19 12:05:49 +00:00
Jennifer Johnson
7c5f777a4f satellitedb/repairqueue: runs a different implementation of the query within Select() for postgres vs cockroach
Change-Id: Ie34dbdb9d870d7d9f8f269702b6b3bad0c55b98e
2019-12-04 21:32:02 +00:00
JT Olio
031ba86de5
argon2: choose a steady parallelism value (#3630)
* argon2: choose a steady parallelism value

Change-Id: I6006da7d7980cda88f5f08ee759612df23a8132d

* whoops, not cruft

Change-Id: Ied9039f9a9be1d0f6ff3c7d5c4839a83fc7b4b1f

* fix broken test file

Change-Id: I07288cd6cef32ba387f2f003febff5c297e50997

* fix linting error

Change-Id: Icdbda8b709cc100a86f3859303c40edb8dff1e0f
2019-11-22 14:00:04 -07:00
Jeff Wendling
f3b20215b0 pkg/{rpc,server,tlsopts}: pick larger defaults for buffer sizes
these may not be optimal but they're probably better based on
our previous testing. we can tune better in the future now that
the groundwork is there.

Change-Id: Iafaee86d3181287c33eadf6b7eceb307dda566a6
2019-11-18 21:22:49 +00:00
Jeff Wendling
ecd2ef4a21 all: build release fully dprc and test in mixed mode
Change-Id: I3bded3edf25a0b113601c8b12ecf1337f596649b
2019-11-15 10:03:18 -07:00
Jeff Wendling
aaead2a29e
makefile: bump drpc and build storj-sim binaries like release (#3506)
Change-Id: I8cfd99226aefc2cfb5b18929799f56b3905cae53
2019-11-06 12:55:20 -07:00
Jess G
8d92c288e2
satellitedb: separate migration into subcommand (#3436)
* separate sadb migration, add version check

* update checkversion to do same validation as migration

* changes per CR

* add sa migration to storj-sim

* add different debug port in storj-sim for migration

* add wait for exit for storj-sim migration

* update sa docker entrypoint to support migration

* storj-sim satellite parts all wait for migration

* upgrade golang-migrate/migrate to v4 because bug

* fix go mod tidy
2019-11-02 13:09:07 -07:00
Jeff Wendling
d820b5abf0 drpc: bump version
Change-Id: I4c368a0200ea07dc4e1b1d14a750e64377e513dc
2019-11-01 23:27:02 +00:00
Jeff Wendling
506d9715da
drpc: version bump (#3386)
Change-Id: Ied42b1bbbe8bff0b3e5c10d29f895b70617373e6
2019-10-29 12:13:57 -06:00
Jeff Wendling
51d5d8656a pkg/rpc: drpc connection pooling
keep a pool of connections open when dialing for drpc. this
makes it so that long lived clients (like lib/uplink's Project)
don't continue to use a bad connection forever. it also allows
for concurrent rpcs.

Change-Id: If649b286050e4f09c413fadc3e1ce88f5fc6e600
2019-10-22 18:15:24 -06:00
Yingrong Zhao
743a0fc38b storagenode/cmd: create start graceful exit CLI (#3202) 2019-10-11 09:58:12 -04:00
Isaac Hess
89c68890b3
Bump drpc version (#3235) 2019-10-10 13:48:47 -06:00
Yehor Butko
0cc23add5b
satellite/payments - payment account setup (#3187) 2019-10-10 20:12:23 +03:00
Jeff Wendling
c1fbfea7fa drpc: bump to latest version
Change-Id: I8426c2dd7f6263050c746c2724524ff687c7298a
2019-10-04 15:09:10 -06:00
Jeff Wendling
098cbc9c67 all: use pkg/rpc instead of pkg/transport
all of the packages and tests work with both grpc and
drpc. we'll probably need to do some jenkins pipelines
to run the tests with drpc as well.

most of the changes are really due to a bit of cleanup
of the pkg/transport.Client api into an rpc.Dialer in
the spirit of a net.Dialer. now that we don't need
observers, we can pass around stateless configuration
to everything rather than stateful things that issue
observations. it also adds a DialAddressID for the
case where we don't have a pb.Node, but we do have an
address and want to assert some ID. this happened
pretty frequently, and now there's no more weird
contortions creating custom tls options, etc.

a lot of the other changes are being consistent/using
the abstractions in the rpc package to do rpc style
things like finding peer information, or checking
status codes.

Change-Id: Ief62875e21d80a21b3c56a5a37f45887679f9412
2019-09-25 15:37:06 -06:00
Michal Niewrzal
68d281db44 autoupdater: use blang/semver (#3063) 2019-09-20 17:22:40 +03:00
Egon Elbre
149a2f8d3f
go.mod: update to grpc 1.23.1 and update golang.org/x (#3072) 2019-09-19 15:41:19 +03:00
Jeff Wendling
0dcbd3dc08 bootstrap/satellite/certificate/storagenode: register drpc services
Change-Id: Id29f14b76a8c9cb2be31001b9a7a4356a4bda183
2019-09-12 15:09:46 -06:00
Jeff Wendling
007662d49e pkg/server: serve drpc listeners along with grpc
The fundamental problem is that both drpc and grpc servers
want to close the listener and they both want to ignore the
error from Accept after the listener is closed. There's no
way to do this in a race free way. Fortunately, the mux
hands out listeners that can be independently closed. That
means they can both do their own shutdown logic where they
ignore the error, and then after they're closed, the code
orchestrating the servers can close the listeners.

The final weird bit is that the server's Close method is
required to wait until the Run method has exited (or at
least enough for the listeners to definitely be closed)
because tests depend on that behavior, so we have to add
some channels/mutexes/onces to ensure that Run has exited
and that a new call can't start after Close is called.

Change-Id: I7c4ef293f7963f83138815f51824fd5b8d09ce15
2019-09-12 19:18:30 +00:00
Egon Elbre
0d4fd6ceeb
cmd/uplink: fix progressbar data race (#2987) 2019-09-10 11:51:49 +03:00
Jeff Wendling
60eba990eb use-drpc: use protoc-gen-drpc to generate protobufs
Change-Id: I5c23256068e30864022dba5137c499796ab9d6ad
2019-09-06 13:28:27 -06:00
paul cannon
bf5a16b823 jenkins: work around git.apache.org failures (#2944) 2019-09-04 11:30:59 +03:00
Egon Elbre
00b2e1a7d7 all: enable staticcheck (#2849)
* by having megacheck in disable it also disabled staticcheck

* fix closing body

* keep interfacer disabled

* hide bodies

* don't use deprecated func

* fix dead code

* fix potential overrun

* keep stylecheck disabled

* don't pass nil as context

* fix infinite recursion

* remove extraneous return

* fix data race

* use correct func

* ignore unused var

* remove unused consts
2019-08-22 13:40:15 +02:00
Egon Elbre
b3a3d8df06
go.mod: update grpc-go (#2778) 2019-08-14 21:43:21 +03:00
Egon Elbre
7fcbec48f7 go.mod: update grpc-go and go (#2777) 2019-08-14 06:18:10 -04:00
Yaroslav Vorobiov
4cf2b96731
storagenode/nodestats: fix test duration (#2748) 2019-08-09 14:12:32 +03:00
Egon Elbre
9c4d2eb5bf go.mod: use pseudo version (#2741) 2019-08-08 11:09:36 -05:00
Egon Elbre
ad5a1d4a02
go.mod: use custom grpc that doesn't have tracing support (#2739) 2019-08-08 17:24:28 +03:00
Jeff Wendling
21a3bf89ee cmd/uplink: use scopes to open (#2501)
What: Change cmd/uplink to use scopes

It moves the fields that will be subsumed by scopes into an explicit legacy section and hides their configuration flags.

Why: So that it can read scopes in from files and stuff
2019-08-05 11:01:20 -06:00
Egon Elbre
ebbf0e1462
uplink/storage: don't import mock in production code (#2687) 2019-08-02 11:01:36 +03:00
Egon Elbre
9ba8b53ed5 pkg/auth: use grpc.WithPerRPCCredentials (#2670) 2019-07-31 13:57:13 +02:00
Egon Elbre
d52f764e54
protocol: implement new piece signing and verification (#2525) 2019-07-11 16:51:40 -04:00
Yaroslav Vorobiov
7aca0eb284 storagenode/dashboard: show console address (#2456) 2019-07-06 15:40:58 +02:00
Egon Elbre
f1e670c9be go.mod: update grpc to 1.22.0 (#2444) 2019-07-03 14:25:10 +02:00
Faris Huskovic
ebd976ec28
satellite/marketing: Create New Offer (#2186)
* update UI to reflect final mockups

* implement create handler and render offers table data to UI

* fix line-height unit and remove important from selectors

* update file names and ids for clarity

* shorten 'label' in ids

* localize global vars, fix endpoint names, remove unnecessary receiver, fix comments

* fix unnecessary initialization of pointer

* correct file-naming conventions

* register timeConverter in an init func for safety and remove unnecessary important from css

* consolidate create endpoints and add comments

* register timeConverter in init func

* add copyright to files

* introduce require pkg

* add proper http server unit test

* update linting and create offers concurrently in unit test

* fix getOffers comment

* add copy-right to unit-test

* fix data-races

* fix linting

* remove converter in NewServer

* requested changes in progress

* add require for checking status code

* renamed template file

* fix 400 handler

* fix missing copyright and remove extra line

* fix build

* run goroutine for testing parallel

* evaluate reqType with switch stmt and promp for credit amount in cents

* fix lint issue

* add default case

* remove unnecessary var

* fix range scope error

* remove empty lines and use long form for struct field

* fix merge conflicts

* fix template reference

* fix modal id

* not delete package

* add currency formatting and requested changes

* markup formatting

* lean out currency logic and move wait outside loop

* pass ToDollars func to home template

* fix lint
2019-06-28 10:34:10 -04:00
Jeff Wendling
1a65e42d33 rename EncryptionCtx back to EncryptionAccess (#2368)
* rename EncryptionCtx back to EncryptionAccess

Change-Id: I5e58915a59979ad6f2e83d36e191b2bbf3ba2ba2

* missed some stuff

Change-Id: Ib65fb186d7f854c0406e5fd7d11498e8941da59e

* oops protolock

Change-Id: I85a5ab4bafb9dd6a804d3dcd17a70811702f07e4

* retrigger

Change-Id: I2d0e3f02b4dbae0299b090e9617662b4437980b0
2019-06-27 23:18:24 -06:00
Jeff Wendling
efcdaa43a3
lib/uplink: encryption context (#2349)
* lib/uplink: encryption context

Change-Id: I5c23dca3286a46b713b30c4997e9ae6e630b2280

* lib/uplink: bucket operation examples

Change-Id: Ia0f6e69f365dcff0cf11c731f51b30842bce053b

* lib/uplink: encryption key sharing test cases

Change-Id: I3a172d565f33f4e591402cdcb9460664a7cc7fbe

* fix encrypted path prefix restriction issue

Change-Id: I8f3921f9d52aaf4b84039de608b8cbbc88769554

* implement panics in libuplink encryption code

todo on cipher suite selection as well as an api concern

Change-Id: Ifa39eb3cc4b3443f7d96f9304df9b2ac4ec4085d

* implement GetProjectInfo api call to get salt

Change-Id: Ic5f6b3be9ea35df48c1aa214ab5d355fb328e2cf

* some fixes and accessors for encryption store

Change-Id: I3bb61f6712a037900e2a96e72ad4029ec1d3f718

* general fixes to builds/tests/etc

Change-Id: I9930fa96acb3b221d9a001f8e274af5729cc8a47

* java bindings changes

Change-Id: Ia2bd4c9c69739c8d3154d79616cff1f36fb403b6

* get libuplink examples passing

Change-Id: I828f09a144160e0a5dd932324f78491ae2ec8a07

* fix proto.lock file

Change-Id: I2fbbf4d0976a7d0473c2645e6dcb21aaa3be7651

* fix proto.lock again

Change-Id: I92702cf49e1a340eef6379c2be4f7c4a268112a9

* fix golint issues

Change-Id: I631ff9f43307a58e3b25a58cbb4a4cc2495f5eb6

* more linting fixes

Change-Id: I51f8f30b367b5bca14c94b15417b9a4c9e7aa0ce

* bug fixed by structs bump

Change-Id: Ibb03c691fce7606c35c08721b3ef0781ab48a38a

* retrigger

Change-Id: Ieee0470b6a2d07168a1578552e8e7f271ae93a13

* retrigger

Change-Id: I753d63853171e6a436c104ce176048892eb974c5

* semantic merge conflict

Change-Id: I9419448496de90340569047a6a16a1b858a7978a

* update total to match prod defaults

Change-Id: I693d55c1ebb28b5803ee1d26e9e198decf82308b

* retrigger

Change-Id: I28b74d5d6202f61aa3866fe407d423f6a0a14b9e

* retrigger

Change-Id: I6fd054885c715f602e2cef623fd464c42e88742c

* retrigger

Change-Id: I6a01bae88c72406d4ed5a8f13bf8a2b3c650bd2d
2019-06-27 17:36:51 +00:00
Kaloyan Raev
ae36979804 Take advantage of Is and IsFunc from zeebo/errs (#2310) 2019-06-26 09:38:07 +02:00
Yehor Butko
e5fd0287e4
V3-1819 Storage node operator server and service started (#2112)
* V3-1819 Storage node operator server and service started
2019-06-20 14:52:32 +03:00
Faris Huskovic
7795ba8169
update UI to reflect final mockups (#2175)
* update UI to reflect final mockups

* fix line-height unit and remove important from selectors

* update file names and ids for clarity

* shorten 'label' in ids

* correct file-naming conventions

* add copyright to files

* check if break in imports is causing lint error

* resolve lint

* tidy go mod

* fix shorthands
2019-06-17 21:57:04 -04:00