Commit Graph

190 Commits

Author SHA1 Message Date
Yingrong Zhao
0bdcf123cf bump monkit, monkit-jaeger, and private to latest
Also bump storj.io/common and sync repo

Change-Id: If8e60db6bdf0af8077b7befcb1da304c3c4dcae4
2020-04-22 12:30:37 -04:00
Qweder93
805e328c47 storagenode/heldamount payments removed
Change-Id: I87cc04f43d182a4190a571ef417be85d02db9d34
2020-04-21 17:15:31 +00:00
Yingrong Zhao
8375a09c89 cmd: remove InitTracing from satellite and storagenode main.go file
Change-Id: I4addbe7d0645f66abfb3e98d74d17035e9624e69
2020-04-20 14:06:26 -04:00
Michal Niewrzal
884638a294 scripts: reduce segment size for integration tests
This will speedup integration tests.

Change-Id: Ifb1fadd01692390457971bfe7f9ba73a0796abb2
2020-04-20 15:13:05 +02:00
Qweder93
6e3585e394 satellite/heldamount/endpoint : GetAllPaystubs added
Change-Id: Ic8cdd9db8b2a68796f9579c7fed2d49d9054bd64
2020-04-19 19:21:54 +03:00
Egon Elbre
7e0e74c65c satellite/metainfo: attribution based on useragent
Automatically attach attribution information to bucket during
BeginObject or CreateBucket when the UserAgent is set.

Change-Id: I405cb26c5a2f7394b30e3f2cf5d2214c8781eb8b
2020-04-17 10:47:14 +03:00
Qweder93
c498e52cf2 satellite/nodestats/endoint: reputation stats updated
Change-Id: I03a44c4c3465d4f8f23fcbbc742148f5e49544e2
2020-04-16 22:28:58 +03:00
Kaloyan Raev
e35cbfb6ee bump storj.io/common
Change-Id: I280e7d69973f1508ae09fbe9cf2a8ceea9001dc3
2020-04-13 17:05:13 +00:00
Moby von Briesen
14b3704f56 storagenode: add suspended status to storagenode dashboard/api
* Add migration to storagenode reputation table to add suspended
timestamp
* Send suspended info to storagenode from satellite nodestats endpoint
* Add suspended status to storagenode api
* Add an indicator on the storagenode dashboard informing operator of
the satellites the node is suspended on

Change-Id: Ie3669f6069cc0258ba76ec99d17006e1b5fd9c8a
2020-04-09 13:36:23 +00:00
Egon Elbre
11a44cdd88 all: don't depend on gogo/proto directly
Change-Id: I8822dea0d1b7b99e0b828e0373a0308a42dde2be
2020-04-08 17:32:15 +00:00
Qweder93
3a9422cc9a satellite/nodestats: add pricing model to endpoint
Change-Id: Iddace8e437216a343458f440b543cee61164f233
2020-04-08 14:29:51 +00:00
Egon Elbre
cf26951a5b satellite/satellitedb/pbold: remove dead code
Change-Id: I7464773c20b8f99a601ca9cc4bee804f1ac14cf9
2020-04-08 15:22:31 +03:00
Yingrong Zhao
96e58d21b4 cmd;pkg/server: init tracing collector in all processes
Add tracing handler in drpc server.
Initializing tracing collector in admin, satellite api, garbage
collection, satellite core, repaier, storagenode.
Change-Id: Ie98420e35dfc6913836ebd82b517d9d12877aefc

Change-Id: I91057b6265a4ac8bde033dfde692b8a28acca99f
2020-04-07 17:20:59 -04:00
Michal Niewrzal
2d1a6968b0 satellite/metainfo: revert returning segment size and inline segment
size with BeginObject

Such solution will add one round trip to satellite during upload so for
now we are reverting this until we will have solution for this.

Change-Id: Ic2d826448ab7b0318cd6922df05deee9167cf2f0
2020-04-06 13:36:34 +02:00
Egon Elbre
6492b13d81 all: remove old uuid
Change-Id: I3a137f73456f010c37d3933dbe12cbbb840b809f
2020-04-02 19:30:36 +03:00
Michal Niewrzal
c178a08cb8 satellite/metainfo: add max segment size and max inline size to
BeginObject response

We want to control inline segment size and segment size on satellite
side. We need to return such information to uplink like with redundancy
scheme.

Change-Id: If04b0a45a2757a01c0cc046432c115f475e9323c
2020-04-02 12:41:28 +00:00
Egon Elbre
8f73fb7a32 all: simplify uuid usage
uuid.UUID implements driver.Value so it can be directly used as a
scannable result.

Replace uses of dbutil.BytesToUUID with uuid.FromBytes.

Change-Id: I51a670185ceb3cc2199d5aa2b76bc3fc191ca8fe
2020-04-02 05:48:58 +00:00
Kaloyan Raev
473cd4695f bump storj.io/uplink to v1.0.3
Change-Id: I9466214260d139eed155c2d8fd725572cd5f0425
2020-04-01 11:12:59 +00:00
Egon Elbre
0a69da4ff1 all: switch to storj.io/common/uuid
Change-Id: I178a0a8dac691e57bce317b91411292fb3c40c9f
2020-03-31 19:16:41 +03:00
JT Olio
f28100b73f bump storj.io/private
Change-Id: I4ddd5c34521602967b89bd18e2a71a6f1e29f436
2020-03-27 21:57:35 +00:00
Michal Niewrzal
75961fe8ce cmd/gateway: remove gateway command from repository and adjust `make
install-sim`

Change-Id: Ie55989ec0d702f6037843a8b6664f4f4f6347446
2020-03-27 07:42:42 +00:00
Yingrong Zhao
fecd36f6b6 bump uplink version to latest
Change-Id: I57c7dae60e4b521bf6f5a07b107f5c2ef18fc168
2020-03-26 09:03:53 +00:00
Egon Elbre
c715c75fea pkg/server: add counters for grpc calls
This will help to determine how many grpc calls are made to the
satellite.

Also remove the grpc funcs that have been added to upstream.

Change-Id: I91878f4fd10f9bfe601c94222c102eaaf4d35963
2020-03-25 21:38:13 +02:00
Yingrong Zhao
b7b19289d1 bump storj.io/common to latest
Change-Id: I16e337660ce8e1ef332cc842dbf4cfa067b9b98b
2020-03-25 09:08:40 -04:00
Kaloyan Raev
0a50d6bdce bump storj.io/uplink to v1.0.1
Change-Id: I55d44aee1a02fe662408a3c48a3e97d6721994af
2020-03-25 10:41:34 +02:00
Yingrong Zhao
a731472496 bump storj.io/common to latest and storj.io/drpc to v0.0.11
Change-Id: I7a6e823b441eeff4621dfdf2d6577be76c9761c8
2020-03-24 15:17:10 -04:00
Egon Elbre
326c0cebde storage/boltdb: update to etcd/bbolt v1.3.4
bbolt v1.3.4 has pointer usage fixes.

Change-Id: I5e0fc4782711d01c09ced579f25a4f8fbc8de85c
2020-03-24 12:33:34 +02:00
Michal Niewrzal
f0aeda3091 storj: remove from storj/pkg packages moved to storj/private repo
* debug
* traces
* cfgstruct
* process

Package `storj/private/version` will be removed as a separate change.

Change-Id: Iadc40faa782e6225513b28218952f02d9c240a9f
2020-03-24 09:56:29 +01:00
Isaac Hess
ddf87286c5 cmd/statreceiver: Remove statreceiver code
Change-Id: Ic572a1623a2008c49d584b769d13e07e4470bce0
2020-03-23 13:00:13 -06:00
Egon Elbre
6a7571f73e cmd/s3-benchmark: move to storj.io/benchmark
Change-Id: Idca2b836bdf876ca28eb5cabc9bfae1d576e4a3e
2020-03-23 19:09:42 +02:00
Egon Elbre
1b6ab173a8 private/context2: moved to storj.io/common/context2
Change-Id: Ic1dd1ed645ff3e1057c9b2b143e2c3ddf29d678e
2020-03-20 14:39:46 +00:00
JT Olio
b2590cf283
bump uplink to 1.0.0 (#3816)
What: bumps uplink to 1.0.0
Why: we just released it!
2020-03-19 15:38:27 -06:00
Kaloyan Raev
10b032e484 libuplink: return deleted bucket/object (step 4)
Switch back to the original DeleteBucket and DeleteObject methods.

Next step: remove the DeleteBucketReturnDeleted and
DeleteObjectReturnDeleted from storj.io/uplink.

Change-Id: I273a305326d411e51ce354ce72fcc6ecadf4dd5f
2020-03-19 13:32:07 +02:00
Kaloyan Raev
78b253c774 libuplink: return deleted bucket/object (step 2)
step 1 in https://review.dev.storj.io/c/storj/uplink/+/1236

Now the old libuplink uses the temporary DeleteBucketReturnDeleted and
DeleteObjectReturnDeleted methods. This way, in the next step, we will
be able to change the DeleteBucket and DeleteObject methods to return
the deleted bucket/object.

Change-Id: I2e638be1960bca6ce1456c92849fcdd6d93e5252
2020-03-18 17:26:23 +00:00
Michal Niewrzal
19685ad81a go.mod: bump common and uplink versions
Change-Id: I9894a1bdbc11b9785d5c0744675d720545f398c0
2020-03-18 15:25:54 +00:00
Stefan Benten
52590197c2
satellite/payments: More Cleanup and Satellite command to ensure we have stripe customers (#3805) 2020-03-16 20:34:15 +01:00
Kaloyan Raev
27f811a9e1 metainfo: delete methods return the deleted item
This only happens if Read or List permission is granted together with
the Delete permission

Change-Id: I68b5f04a476bddabe499809ac98097aac75732a8
2020-03-16 16:26:16 +02:00
Isaac Hess
85b6316ce8 statreceiver: Update to filter on packet headers
The admission/v3 protocol now supports arbitrary key/value headers to be
included in each packet of metrics. This commit creates support for
this, so the lua config file can declare a filter taking into account
the key/value headers.

Change-Id: I41de8c018d33304ccf46ec221ae689d55c5fb1ee
2020-03-12 13:57:07 -06:00
Isaac Hess
8e274a5ce4 cmd/uplink: Enable telemetry on ctx
Change-Id: I26ffbbb72b9f2ca71b629ba12637c74c2aa072c3
2020-03-12 14:52:40 +00:00
Michal Niewrzal
c20cf25f35 cmd: migrate uplink CLI to new API
Change-Id: I8f8fcc8dd9a68aac18fd79c4071696fb54853a60
2020-03-09 13:26:29 +00:00
Jessica Grebenschikov
bcb0453db2 upgrade dependencies for trace db debug endpoint
Change-Id: I4de658b361bb39ce28dc31b982895bb4f45b580a
2020-03-04 07:35:34 +00:00
Cameron Ayer
7244a6a84e storagenode/{contact, piecestore}: implement low disk notification with cooldown
When a storagenode begins to run low on capacity, we want to notify
the satellite before completely running out of space. To achieve this,
at the end of an upload request, the SN checks if its available space has
fallen below a certain threshold. If so, trigger a notification to the
satellites.

The new NotifyLowDisk method on the monitor chore is implemented using the
common/syn2.Cooldown type, which allows us to execute contact only once
within a given timeframe; avoiding hammering the satellites with requests.
This PR contains changes to the storagenode/contact package, namely moving
methods involving the actual satellite communication out of Chore and into
Service. This allows us to ping satellites from the monitor chore

Change-Id: I668455748cdc6741291b61130d8ef9feece86458
2020-03-03 10:45:37 -05:00
Egon Elbre
decb2ec69a private/processgroup: moved to storj.io/common/processgroup
Change-Id: I1ec0bb440dda757d8f9a6f564a0084dde2f9cc84
2020-03-03 10:50:33 +00:00
Egon Elbre
64330c55b3 all: use pbgrpc
common/pb moved grpc to a separate package common/pb/pbgrpc.
This updates this repository to use it.

Change-Id: I2de2a190688871cf9cb61f7ea511f8a01e264e4e
2020-02-26 21:27:47 +02:00
Egon Elbre
8822e98c1f cmd/gateway: simplify module handling
Change-Id: If6ed158a6c9568fa33f69ca2d52e231ee4fcb0cb
2020-02-26 17:59:45 +00:00
Jessica Grebenschikov
e19e3c1101 pkg/process:
Now that we are trying to identify the root cause of the satellite load limitations (i.e. currently the satellite has a max ability of 400 rps for uploads and we need this to be higher), we are using the golang diagnostic tools to collect insight into what the bottlenecks are.  We currently have a debug endpoint to gather some cpu and mem data, but it could be useful to have continuous profiling. GCP stackdriver has support for continuous profiling so lets set that up and see if it is helpful to gather more data.

This PR adds support for [GCP continuous profiler](https://cloud.google.com/profiler) which allows enabling continuous cpu/mem profiling and the stats are sent to stackdriver in google cloud console.

To enable the continuous profiling for a storj component, do the following:
- prereq: the workload must be running in GKE and have Stackdriver Profiling IAM role permissions
- provide the config flag `debug.profilename` in the config.yaml file for the workload (i.e. satellite api process, etc). The profilename should be the workload name, for example "satellite-api".
- once the above config flag is provided, the profiler will be initialized and profiling stats will automatically be sent to GCP project where the workload is running and viewable in the Stackdriver Profile page in the console

The current implementation assumes the workload is running in GKE, however if we find if useful we can add support to enable this from anywhere. But for simplicity, its configured this way assuming the main goal is to enable in production systems.

Change-Id: Ibf8ebe2df7bf06fdd4951ee6a1e48854dd36ad47
2020-02-25 09:04:23 -08:00
Egon Elbre
29452d82a5 go.mod: unlock graphql dependency and bump to latest
Change-Id: I40026f6c8de155e024f5fbb51105546065393034
2020-02-25 13:17:49 +02:00
Egon Elbre
9752d01884 private/prompt: remove dependency to go-prompt
Change-Id: Ida8ef731ce806cec076343dc77d72a3b0d7736b4
2020-02-25 13:09:41 +02:00
JT Olio
50a21de9dc traces: fix memory leak for long running traces that aren't being collected
for real this time. i'm so ashamed

Change-Id: Ib05bb50d8e947dec2d872fd53e71eec561c2d0e8
2020-02-24 15:02:26 -07:00
Egon Elbre
e30f7b35b6 cmd/gateway: use a separate repository
Change-Id: Idbb0b2b6cf0e60c6d5d91218c24524d72285cf26
2020-02-24 10:03:03 +02:00