Commit Graph

739 Commits

Author SHA1 Message Date
Jeff Wendling
f93d764998 cmd/uplink: properly path escape shares
Fixes #3829

Change-Id: I0f900a392239fcc064cea968fc7a5bc373ee26a1
2020-05-04 17:00:25 +00:00
Yingrong Zhao
0b80d60491 cmd/uplink: disable tlemetry if user opt out
Change-Id: I793e6428d04e4f422d1f8292e99b233c70d9dd6e
2020-05-04 14:46:49 +00:00
Egon Elbre
c630cf2490 storagenode/pieces: implement buffering for writing
Currently uploads can cause a lot of IOPS, reduce this by introducing a
in-memory buffer on-top of the file.

Change-Id: I5f4e3e01c0a36258271d180b922107de447bcb59
2020-05-04 06:01:32 +00:00
Yaroslav Vorobiov
516b8cf2be storagenode-updater: add recovery for windows service restart
Reimplement windows service restart part using svc, add recovery
for failed service startup. Added restart-service cmd, to execute
self restart in a separate process.

Addressed issues:
https://storjlabs.atlassian.net/browse/SG-49
https://storjlabs.atlassian.net/browse/SG-136
https://storjlabs.atlassian.net/browse/SG-137

Change-Id: Ic51d9a99e8c1c10800c6c60ff4e218321c674fea
2020-05-01 09:07:05 +00:00
Egon Elbre
8928399d02 all: rename CreateTables to MigrateToLatest
CreateTables hasn't been quite true for a while now, rename to
MigrateToLatest to be clearer in it's behavior.

Change-Id: Ida48e95122a5d9b7a814e922d3698e00024a2ba7
2020-04-30 07:21:17 +00:00
Michal Niewrzal
b2acd93a78 storagenode/inspector: adjust Uptime type change in protobuf
Change-Id: I4f5110b534ac4f419b74f1a3dd72f8600e0a53a8
2020-04-29 09:28:37 +00:00
Yingrong Zhao
c5309a3f91 cmd/uplink: set sample rate for tracing to be 1 when tracing is enabled
when tracing is enabled, we should also set sampling rate to
a non-zero value. For now, we will set it to 1.
Uplink CLI users should be able to override it with the sample
flag.

Change-Id: I8bcf514fb14c2a1c4349b7957dd24ec23e4a85e5
2020-04-28 20:15:28 +00:00
Jeff Wendling
42f63c6538 satellite/compensation: add offline status tracking
Change-Id: I52e615d3db186416ee95029dc72df626f0e69ad7
2020-04-28 19:35:59 +00:00
Natalie Villasana
6f84be133a satellite/metainfo: add MigrateToLatest to PointerDB
In cases like the segment reaper script connecting to the metainfodb,
we don't want a db migration to happen automatically when we call
metainfo.NewStore. This adds MigrateToLatest method for postgreskv
and cockroackv, and calls MigrateToLatest in places where NewStore used
to create tables.

Change-Id: I682d0f26d609af0601dfdb32a24866cdf5d32a7e
2020-04-28 17:26:35 +00:00
Yingrong Zhao
9b4a3f8fcc cmd/uplink: use tracing.enabled flag
Previously we are using tracing.sampled to be the switch for turning on/off tracing.
However we would like to separate sampling rate from being the switch,
so we can set sampling rate to be 0 but still intialize tracing for
satellite and storagenodes

Change-Id: I27e6ba25ea6f6b612b4e1a57cf1301889ded41ec
2020-04-27 17:54:57 +00:00
Yingrong Zhao
0b8699bcb5 cmd: add prompt for enabling tracing during uplink cli setup
We want to make tracing to be opt-in.
For now, we will use `tracing.sample` as the toggle config to enable or
disable tracing and default to sample every traces from uplink cli.
If user wants to change the default sampling rate, they can do so by
using the `--tracing.sample` flag to override the default value

Change-Id: I6f25dac0f43024c50a8aaf6c549e6a514211f834
2020-04-21 20:57:10 +00:00
Matt Robinson
1e295a48e7
add container image for segment-reaper (#3855) 2020-04-21 17:48:40 +03:00
Yingrong Zhao
8375a09c89 cmd: remove InitTracing from satellite and storagenode main.go file
Change-Id: I4addbe7d0645f66abfb3e98d74d17035e9624e69
2020-04-20 14:06:26 -04:00
JT Olio
5f38f8f1fe satellite/gc: use hostname for metric instance ids instead of node id
currently production uses a different application suffix for gc
services, so chronograf can distinguish between gc processes and core
processes, but it'd be nice to be a bit more consistent with repairers
and api servers

Change-Id: Icb96fed006c59d7afd730317d35636a6e4573b58
2020-04-20 14:52:44 +00:00
Cameron Ayer
a6f0be2047 cmd/inspector: decode path
Change-Id: I5c1eb7842ba5fc76c19c339e5be8f108d3f21fc9
2020-04-16 20:12:08 +00:00
Egon Elbre
d3ce845f82 satellite: revert log lines used to figure out node id
Currently storj-sim relies on the log lines to be exactly the same,
when they change it cannot find the necessary information from log.

Change-Id: Ia039915ef3375a7cf60f107b2c05c958de15b6d5
2020-04-15 17:07:56 +03:00
Michal Niewrzal
90bac51544 cmd/uplink/cmd: improve error message when config.yaml or --access flag
is missing

https://storjlabs.atlassian.net/browse/USR-78

Change-Id: I8221822d164279b85f24573df0ac05fcd6b3881e
2020-04-15 13:22:02 +00:00
Kaloyan Raev
a2ce836761 remove sugar logging
Change-Id: I6b6ca9704837cb3f5f5449ba7f55661487814d9f
2020-04-15 12:37:47 +00:00
JT Olio
e2d5b403e6 cmd/uplink: support --force (like awscli) for rb
Change-Id: If835c6dd08ee95e7c66ba7e4c7451cb3f0f95442
2020-04-14 18:10:54 +00:00
Moby von Briesen
68946c86b4 cmd/storagenode: update colored storagenode dashboard to have correct indendation
Change-Id: I8f8fc2322c42e7e1098437abf4aad28bd754e79c
2020-04-14 13:21:17 +00:00
Egon Elbre
11a44cdd88 all: don't depend on gogo/proto directly
Change-Id: I8822dea0d1b7b99e0b828e0373a0308a42dde2be
2020-04-08 17:32:15 +00:00
Yingrong Zhao
96e58d21b4 cmd;pkg/server: init tracing collector in all processes
Add tracing handler in drpc server.
Initializing tracing collector in admin, satellite api, garbage
collection, satellite core, repaier, storagenode.
Change-Id: Ie98420e35dfc6913836ebd82b517d9d12877aefc

Change-Id: I91057b6265a4ac8bde033dfde692b8a28acca99f
2020-04-07 17:20:59 -04:00
Jeff Wendling
2ded64ba2c satellite/compensation: more fixes to get prod running smoothly
Change-Id: I13a76d9d49222fb10796415a015f224d4084fde3
2020-04-07 10:10:27 +00:00
sixcorners
06cddb053e
cmd/identity: improve CA key message (#3819) 2020-04-03 13:00:45 +02:00
sixcorners
4f8cf53c9e
Remove VOLUME /root/.local/share/storj/storagenode (#3820) 2020-04-03 00:05:20 +02:00
Egon Elbre
6492b13d81 all: remove old uuid
Change-Id: I3a137f73456f010c37d3933dbe12cbbb840b809f
2020-04-02 19:30:36 +03:00
Jess G
e725535873
cmd/satellite: add gc to entrypoint (#3842)
Change-Id: I49816628bc4301006e4a54b5582b3d759806b5e7
2020-04-02 08:39:13 -07:00
Egon Elbre
1024bf9ce1 all: simplify uuid usage
Instead of uuid.Parse, use uuid.FromString.
This removes a bunch of pointer management logic.

Change-Id: Id25bd174eb43c71d00b450158a198abafd8958f2
2020-04-02 13:45:19 +00:00
Egon Elbre
8f73fb7a32 all: simplify uuid usage
uuid.UUID implements driver.Value so it can be directly used as a
scannable result.

Replace uses of dbutil.BytesToUUID with uuid.FromBytes.

Change-Id: I51a670185ceb3cc2199d5aa2b76bc3fc191ca8fe
2020-04-02 05:48:58 +00:00
Jeff Wendling
ffe7a3c211 satellite/compensation: make surge percent an int64 for strictcsv
Change-Id: I1783bf73ee68ca9beb8a03f5928873fab0bbe95d
2020-04-01 14:06:33 +00:00
Egon Elbre
644df8dcdc private/version: minimal fix for tag-release.sh
Previous split to a storj.io/private repository broke tag-release.sh
script. This is the minimal temporary fix to make things work.

This links the build information to specified variables and sets them
inline. This approach, of course, is very fragile.

Change-Id: I73db2305e6c304146e5a14b13f1d917881a7455c
2020-04-01 13:46:45 +00:00
Egon Elbre
0a69da4ff1 all: switch to storj.io/common/uuid
Change-Id: I178a0a8dac691e57bce317b91411292fb3c40c9f
2020-03-31 19:16:41 +03:00
Jeff Wendling
e2ff2ce672 satellite: compensation package and commands
Change-Id: I7fd6399837e45ff48e5f3d47a95192a01d58e125
2020-03-30 14:08:14 -06:00
Michal Niewrzal
75961fe8ce cmd/gateway: remove gateway command from repository and adjust `make
install-sim`

Change-Id: Ie55989ec0d702f6037843a8b6664f4f4f6347446
2020-03-27 07:42:42 +00:00
Yingrong Zhao
b7b19289d1 bump storj.io/common to latest
Change-Id: I16e337660ce8e1ef332cc842dbf4cfa067b9b98b
2020-03-25 09:08:40 -04:00
Michal Niewrzal
fdf40a7526 storj: remove storj/private/version package which was moved to
`storj/private` repo

Change-Id: I81c3f5b9d5e4fe7bca760999eb045ee9734e5e2e
2020-03-24 14:31:33 +00:00
Michal Niewrzal
f0aeda3091 storj: remove from storj/pkg packages moved to storj/private repo
* debug
* traces
* cfgstruct
* process

Package `storj/private/version` will be removed as a separate change.

Change-Id: Iadc40faa782e6225513b28218952f02d9c240a9f
2020-03-24 09:56:29 +01:00
Isaac Hess
ddf87286c5 cmd/statreceiver: Remove statreceiver code
Change-Id: Ic572a1623a2008c49d584b769d13e07e4470bce0
2020-03-23 13:00:13 -06:00
Egon Elbre
6a7571f73e cmd/s3-benchmark: move to storj.io/benchmark
Change-Id: Idca2b836bdf876ca28eb5cabc9bfae1d576e4a3e
2020-03-23 19:09:42 +02:00
Michal Niewrzal
e258de518c cmd/storj-sim: remove identity dir from gateway configuration
We are not generating identity for gateway during setup anymore, its
generated on-fly by libuplink. With this change we can remove
`--identity-dir` from gateway.

Change-Id: I63298115d4399564b2f29541b8dc16e3b3acdcf2
2020-03-23 14:11:51 +00:00
Kaloyan Raev
98ac16fb5f cmd/wizard: improve prompt for satellite
Change-Id: Iff83ccf30c7b4059cc970c14d29e7d11e497c4bb
2020-03-23 13:38:24 +02:00
Isaac Hess
05da31a97b statreceiver: Add name of buffer for better logging/debugging
Change-Id: If845908543c951111ac13ac6262b4e7c50dcd22c
2020-03-20 15:00:20 +00:00
Egon Elbre
1b6ab173a8 private/context2: moved to storj.io/common/context2
Change-Id: Ic1dd1ed645ff3e1057c9b2b143e2c3ddf29d678e
2020-03-20 14:39:46 +00:00
Michal Niewrzal
d994584e88 cmd/storj-sim: remove removed flags from storj-sim gateway setup
Change-Id: I27d585740e8441b41bce5010480cc8e4f5deb9ce
2020-03-20 12:09:55 +01:00
Egon Elbre
0781a91ff4 linksharing: move to storj.io/linksharing
Change-Id: If3bca64f80de9c244414e2aff28bb8b3bf9e87a2
2020-03-19 16:49:00 +00:00
Jessica Grebenschikov
5142874144 satellite/gc: move garbage collection to its own process
Change-Id: I7235aa83f7c641e31c62ba9d42192b2232dca4a5
2020-03-18 16:44:01 +00:00
Stefan Benten
49a30ce4a7
satellite/payments: Set proper defaults for the release (#3806)
* Slight adjustments to the migration

Change-Id: I68ae81c010c3414fde2845df16ab124f8d17834b

* Change Coupon Value

Change-Id: I0f241d09e5f716f1d1b3f0688643ba7f614d83c4

* Change AlphaUsage to 5GB

Change-Id: I5d25c6b5750684510cda8b14a27f38d5b2b07408

* change config lock

Change-Id: Ib7c7a54555ba2387c9aa8dd60a0501b0ee6491dd

* Use Scan properly

Change-Id: Ie39cf4644e3ddd703a254e2f5e616763dd805235

* Fix Config Lock

Change-Id: I558ecc1c1becfaaefc7aea5ad2fe83fd6bf6b561
2020-03-16 22:53:12 +01:00
Stefan Benten
52590197c2
satellite/payments: More Cleanup and Satellite command to ensure we have stripe customers (#3805) 2020-03-16 20:34:15 +01:00
Matt Robinson
44ade00e8a
Add inspector to satellite image (#3801)
Change-Id: I77f0a5397bc4073d3e46d759e2e99eb908b175c6

Co-authored-by: Stefan Benten <mail@stefan-benten.de>
Co-authored-by: littleskunk <jens.heimbuerge@googlemail.com>
2020-03-16 09:05:59 -04:00
Isaac Hess
85b6316ce8 statreceiver: Update to filter on packet headers
The admission/v3 protocol now supports arbitrary key/value headers to be
included in each packet of metrics. This commit creates support for
this, so the lua config file can declare a filter taking into account
the key/value headers.

Change-Id: I41de8c018d33304ccf46ec221ae689d55c5fb1ee
2020-03-12 13:57:07 -06:00