Commit Graph

3033 Commits

Author SHA1 Message Date
Egon Elbre
edf8318c0e
jenkins: update to golangci-lint 1.21.0 (#3289)
* jenkins: update to golint-1.21.0
* don't use underscores in var
* make handles private to satisfy linter

lib\uplinkc\universe.go:32:39: exported method Add returns unexported type storj.io/storj/lib/uplinkc._Ctype_long, which can be annoying to use (golint)
   func (m *Universe) Add(x interface{}) Handle {

* disable wsl explicitly
2019-10-17 10:42:22 +03:00
JT Olio
863e4cacbe
examples/scope: small scope exploder command line tool (#3266)
this provides a small tool that breaks a scope into
its constituent parts

Change-Id: If011dd457a175eeb6e013a17d1aee4f3edfb8b0c
2019-10-16 22:40:41 -06:00
Jess G
34764e5c97
cmd/satellite: create API subcommand (#3280)
* create sa api subcommand

* add api to storj-sim

* require postgres

* pass live accounting cache into satellite.NewAPI
2019-10-16 13:34:25 -07:00
Natalie Villasana
855fca003d satellite/metrics: create a metrics chore (#3263)
* add metrics counter and chore

* updates metrics observer interval release default and dev default to 15min

* add more specific check for remote pointers

* add Counter field to metrics chore, add counter tests

* rm redundant ObjectCount suffix

* make pointer check easier to read

* change metrics.Config.Interval to ChoreInterval

* rm unneeded var

* fix comment

* update satellite config lock
2019-10-16 14:08:33 -04:00
Yingrong Zhao
e5099f31f3
add context.Clean and correct rpc error code (#3295) 2019-10-16 13:50:01 -04:00
Cameron
76ad83f12c
satellite/accounting: add redis support to live accounting (#3213)
* set up redis support in live accounting

* move live.Service interface into accounting package and rename to Cache, pass into satellite

* refactor Cache to store one int64 total, add IncrBy method to redis client implementation

* add monkit tracing to live accounting
2019-10-16 12:50:29 -04:00
Ivan Fraixedes
21c6737b10
uplink/ecclient: clarify defer logic in putPiece (#3247)
Refactoring the 'defer' function logic to just only have what's important to not forget before returning but simplifying its logic for making easy to understand the overall function logic.
2019-10-16 18:05:22 +02:00
littleskunk
6e7607239c
satellite/repair: improve logging (#3287)
* satellite/repair: improve logging

* use Stringer wherever possible
2019-10-16 17:28:56 +02:00
Nikolai Siedov
c1033fc1ed
satellite/console: move token to separate endpoint (#3292) 2019-10-16 18:01:15 +03:00
JT Olio
bbe89ec9d1
add siedov mykola to clabot (#3293)
Change-Id: I10a498233d181706a8b3149b18c015d20a101b8d
2019-10-16 08:12:07 -06:00
littleskunk
2301a8287f Satellite/PieceHashValidation: Increase time window from 2h to 24h to avoid timezone issues (#3291) 2019-10-16 06:47:08 -06:00
littleskunk
eeb38245ff
satellite/audit: improve logging (#3285) 2019-10-16 13:48:05 +02:00
Bryan White
951c2891b9
{versioncontrol,internal/version}: add rollout to versioncontrol server (#3176) 2019-10-16 10:16:59 +02:00
Isaac Hess
ed6b88a12d piecestore: update usage before completing upload (#3286)
The upload code currently updates the usage in a deferred call to saveOrder().
The consequence is that in the success case, the RPC is completed before
the usage has been updated.

This change repurposes the deferred call to update usage in the
failure case, while explicitly updating the usage before completing the
RPC.

This fixes some test flakiness when using dRPC. gRPC waits until the final status is written before a Recv call completes, and the final status is written by the server after the handler function has exited. In practice this means that the client is blocked until the defer call is also finished. So this change will not change performance at all.

It has two advantages:

(1) It fixes test flakiness

and, more importantly:

(2) reduces the chances that someone will accidentally write a flaky test in the future
2019-10-15 20:17:17 -06:00
Yingrong Zhao
87e3764390
storagenode/cmd: add exit-status command for graceful exit (#3264)
* add exit-status command

* remove todo and fix format

* fix status display

* change startExit to exit progress

* fix linting error

* add successful column in exit progress

* fix test

* remove extra new line

* fix TYPOS

* format the percentage better
2019-10-15 18:07:32 -04:00
Ethan Adams
37ab84355f
satellite/gracefulexit: protobuf field name updates (#3284)
rename piece_id to original_piece_id
2019-10-15 15:59:12 -04:00
Yehor Butko
a5f4bbee22
satellite/payments: dbx scheme renamed, userID placed on Account level (#3281) 2019-10-15 21:05:45 +03:00
Andrew Harding
4962c6843e
piecestore: fix test flakiness around upload/download usage tracking (#3282) 2019-10-15 11:22:15 -06:00
Natalie Villasana
cf430d2d73
scripts: add check-monitoring script to detect changes to monkit calls (#3114) 2019-10-15 13:00:14 -04:00
Simon Guindon
abb5b6c499
storagenode/piecestore: Fix to ignore both gRPC and dRPC EOF errors. (#3274)
* Fix to ignore both gRPC and dRPC EOF errors.

* Fix to ignore both gRPC and dRPC EOF errors.
2019-10-15 12:13:53 -04:00
Ethan Adams
1ad2ba7e3e
storagenode/gracefulexit: Add graceful exit chore and worker. (#3262)
Adds graceful exit chore and worker for V3-2614
2019-10-15 11:29:47 -04:00
Yehor Butko
4d43b67ff9
add credit card to payment account (#3279) 2019-10-15 17:50:28 +03:00
Ivan Fraixedes
9caa3181d3
uplink/piecestore: Check SN piece hash timestamp (#3246)
Uplink must verify that every piece upload to a storage node return a
hash whose timestamp isn't older than the maximum elapsed time allowed
by the Satellite.

We cannot leave this check only to the Satellite site, because if there
is no error reported by this matter, the uplink cuts down the long tail.
When uplink submits the result uploads including these invalid ones, the
Satellite filters out the invalid ones and that can provoke that it gets
less than the optimal threshold amount of valid upload results, so it
rejects the request.

Detecting the error at this stage will allow the uplink to detect these
uploads as invalid and avoid to cut down the long tail prematurely.
2019-10-15 16:07:18 +02:00
Nikolay Yurchenko
875c7dfe72
web/satellite: project members frontend selection caching added (#3238) 2019-10-15 16:22:41 +03:00
Vitalii Shpital
f1867a954b
web/satellite: project members sorting fixed (#3231) 2019-10-15 15:24:53 +03:00
Nikolay Yurchenko
a34730a60b
web/satellite: api keys selection caching added (#3239) 2019-10-15 15:07:18 +03:00
Yehor Butko
501816770b
satellite/payments: receive credit cards (#3249) 2019-10-15 14:23:54 +03:00
Michal Niewrzal
57ff0af6ba
mobile: add EncryptionRestrictions (#3260) 2019-10-15 01:26:09 -07:00
littleskunk
5b20c716e6 satellite/repair: dont error on deleted segments (#3252) 2019-10-15 05:39:28 +02:00
Marc Schubert
93d5eeda31 Update dial.go (#3261)
What:
Bring back partial nodeID to debug.trace-out

Why:
The information is useful for interpreting the trace file and was there up drpc. I just bring it back.
https://github.com/storj/storj/blob/v0.21.3/pkg/transport/transport.go#L76

Please describe the tests:

Test 1:
Test 2:
Please describe the performance impact:
No impact.
2019-10-14 15:44:15 -06:00
JT Olio
694177e217 pkg/pb: regen gracefulexit.pb.go (#3270) 2019-10-14 17:06:04 -04:00
Jess G
87a426f228 internal/testplanet: add satellite.API to testplanet (#3237) 2019-10-14 16:01:53 -04:00
JT Olio
1b66517664 contact: small typo
Change-Id: If9126ae518b5672bfd9163a8c9fc518727d5138b
2019-10-14 13:21:05 -06:00
Jennifer Li Johnson
b185dbbee2
satellite/discovery: remove discovery related code (#3175) 2019-10-14 10:57:01 -04:00
littleskunk
96aeedcdee
OrderLimit/GracePeriod: Increase time window from 1h to 24h (#3255)
* OrderLimit/GracePeriod: Increase time window from 1h to 24h

* update satellite config lock
2019-10-13 17:40:24 +02:00
JT Olio
6ede140df1
pkg/rpc: defeat MITM attacks in most cases (#3215)
This change adds a trusted registry (via the source code) of node address to node id mappings (currently only for well known Satellites) to defeat MITM attacks to Satellites. It also extends the uplink UI such that when entering a satellite address by hand, a node id prefix can also be added to defeat MITM attacks with unknown satellites.

When running uplink setup, satellite addresses can now be of the form 12EayRS2V1k@us-central-1.tardigrade.io (not even using a full node id) to ensure that the peer contacted is the peer that was expected. When using a known satellite address, the known node ids are used if no override is provided.
2019-10-12 14:34:41 -06:00
Stefan Benten
dd21953fd6 Change Error Handling to return more clear messages to the user/client (#3254) 2019-10-12 20:51:20 +02:00
Ethan Adams
78ccf14837 fix interface and EOF check (#3251) 2019-10-12 07:06:20 -06:00
Isaac Hess
e567f27634
storagenode/piecestore: Change test to use ioutil.ReadAll to attempt to reduce test flake (#3250) 2019-10-11 15:57:59 -06:00
Ethan Adams
a1275746b4
satellite/gracefulexit: Implement the 'process' endpoint on the satellite (#3223) 2019-10-11 17:18:05 -04:00
Cameron
d17be58237 remove random sleep in storagenode contact (#3243) 2019-10-11 16:44:18 -04:00
JT Olio
a5d1776539 audits: missing continue
Change-Id: Ifcac8e61ebd8c59407e01c791adc60d9f88ff1b7
2019-10-11 13:55:27 -06:00
Brandon Iglesias
84156d6912 docs/blueprints: updating term to held amount (#3248) 2019-10-11 21:05:50 +03:00
Vitalii Shpital
0560b2bac4
web/storagenode: chart y and x axis reworked (#3179) 2019-10-11 20:07:30 +03:00
Vitalii Shpital
78a71ad3b6
web/storagenode: node status updated (#3220) 2019-10-11 19:28:47 +03:00
Yaroslav Vorobiov
535742d37e
satellite/payments: add coinpayments HTTP client (#3181) 2019-10-11 18:51:14 +03:00
Yehor Butko
451909b3ec
satellite/payments: account balance (#3242) 2019-10-11 18:00:35 +03:00
Isaac Hess
9256399872
CI: test drpc and grpc (#3163)
* wip: test drpc

* Add parallel intregration test

* Add jenkinsfile.drpc

* Remove unnecessary jenkinsfile items

* testing: GOFLAGS=-drpc (#3236)

* Use GOFLAGS

* add debug

* revert tags

* revert changes

* move goflags to the correct place

* add sanity check
2019-10-11 08:30:06 -06:00
Yingrong Zhao
743a0fc38b storagenode/cmd: create start graceful exit CLI (#3202) 2019-10-11 09:58:12 -04:00
Vitalii Shpital
91fcabd83c
web/satellite: notification message text breaking fixed (#3221) 2019-10-11 13:53:22 +03:00