Commit Graph

1002 Commits

Author SHA1 Message Date
Egon Elbre
1a1a084477
testcontext: sanitize folder name (#2195) 2019-06-13 15:46:08 +03:00
aligeti
8e8a5eb89c
initial value attribution check development (#2091)
* initial value attribution check development

* code reveiew updates
2019-06-12 21:35:37 -04:00
Natalie Villasana
5314c950cc
pkg/audit: add more reverify tests (#2144) 2019-06-11 16:14:40 -04:00
Jess G
d84cd719ff
add const stripesPerBlock const to calc blocksize (#2163)
* add const stripesPerBlock, update comments in Pad, add speed to progressbar

* change size of erasure share size, update comments

* missing copyright

* update tests with stripesPerBlock
2019-06-11 10:14:05 -07:00
Kaloyan Raev
252c8ac189
Add email to self node info (#2171) 2019-06-11 16:30:28 +03:00
Ivan Fraixedes
f624b213a3
reputation: Add configuration parameters (#2150)
* reputation: Add configuration parameters
 Add the configuration parameters which will be used by the algorithm
 which will calculate the storage node reputation.
 Because the reputation calculation is based on audit and uptime check
 results some configuration parameters are in pkg/audit, others in
 pkg/discovery and other in the satellite which will combine the both
 reputation results to obtain the storage node reputation for repair and
 uplink.
* satellite-config: Refresh lock file with new params
  Refresh the Satellite configuration yaml lock file with the new
  parameters added in this branch.
2019-06-11 12:14:01 +02:00
Kaloyan Raev
f0880b9b3c
Audit service should download erasure shares in parallel (#2164) 2019-06-11 11:00:59 +03:00
Egon Elbre
749846b42b
Update golangci-lint (#2159) 2019-06-10 11:52:09 +03:00
ethanadams
8f2dca8437 Re-enabling and fixing repairer tests (#2099)
* Disabled discovery service by changiing from Stop() to Pause()

Paused to solve race condition.  If discovery is running, it may mark a node "up" after they've been manually marked "down" in this test.

* Extend to the repair timeout

Fixes intermittent test failures when repairs were taking more than 2 seconds.

* Re-enabled test. Disabled discovery service by changiing from Stop() to Pause()

* Changed back to Stop.

* Revert "Changed back to Stop."

This reverts commit 46d410e72dfae63e0c44915be42784cc9a7b5abf.

* re-enabling TestIdentifyInjuredSegments

* Changed Pause to Stop.  Commented on timeout change

* testing...

* temporarily skipping audit tests

* changing back to discover Stop for testing via jenkins

* Revert "changing back to discover Stop for testing via jenkins"

This reverts commit 6aa8558b11a0053c30e0c8b2dbf0d6c0cb34ee6c.

* Changing back to Stop().  Depends on PR 2137

* Revert "temporarily skipping audit tests"

This reverts commit 1940ed9b315d663a0eb6c95521780cbcb48cb121.

* Removed reference to Graveyard since its been removed
2019-06-10 09:06:21 +02:00
Simon Guindon
3a7ebea76e
Moved monkit tracing after the limit nil check & fix node id formatting. (#2156)
* Moved monkit tracing after the limit nil check & fix node id formatting.

* Fixed to keep function traced even if limits is nil.
2019-06-07 18:34:16 -04:00
Natalie Villasana
16acb58546 rename verifier test files (#2154)
Despite the confusing git trail, this is two pure file renames, no other changes.
2019-06-07 16:02:36 -05:00
Ivan Fraixedes
44359e8bb3
pkg/kademlia: Fix bug using ref of var range loop (#2149)
Fix a bug in a range loop which used the reference of the variable assigned in the same loop to be appended in a slice, turning out that the slice will contain the same reference rather the a reference for each value of the ranged items.
2019-06-07 15:08:09 +02:00
Kaloyan Raev
8ddb1dee0d
Improve error analysis in audit service (#2125)
Co-authored-by: Natalie Ventura Villasana <nat@storj.io>
2019-06-07 15:38:41 +03:00
JT Olio
43d4f3daf5 discovery: remove graveyard (#2145) 2019-06-07 08:40:51 +03:00
Simon Guindon
a55df84bf7
Adding RS values to the info log and NodeID to tracing. (#2130) 2019-06-06 15:51:00 -04:00
Jess G
bf3d168cf6
makes sure all uplink cli configs get passed to libuplink, add stripeSize (#2103)
* makes sure all uplink cli configs get passed to libuplink, add stripSize

* update comment

* update defaults for uplink config blocksize

* changes per CR, update uplink config defaults

* pass shareSize from uplink config

* move block size validation to kvmeta pkg

* fix tests

* shareSize default 1k, rm config option blocksize

* rm printing err to stdout
2019-06-06 11:55:10 -07:00
Egon Elbre
91a0ece959
Remove bwagreement leftovers (#2140) 2019-06-06 18:16:14 +03:00
Egon Elbre
28a1201590 pkg/bwagreements: remove service (#2138) 2019-06-06 15:57:58 +02:00
Ivan Fraixedes
503fe0e5ec pkg/audit: Fix code style named result params (#2136) 2019-06-06 15:55:06 +03:00
JT Olio
2d19475d61 monitor macaroon parse/serialization perf (#2132)
Change-Id: I3c9556194e23af011db77b6b743abf3e5799f3f0
2019-06-06 11:35:03 +02:00
Natalie Villasana
25d7dda135 add disqualified check to node selection queries (#2102)
add disqualified check to queries, skip TestStatDB
2019-06-05 20:21:32 -04:00
Kaloyan Raev
099cf921d3 Fix flaky audit test (#2093)
* Show full unexpected error when assertion fails

* restart jenkins

* restart jenkins

* restart jenkins

* restart jenkins

* Even better display of the unexpected error

* Fix flakiness using kaloyan-raev/errs fork

* Add more logs

* restart jenkins

* restart jenkins

* Better print for unexpected errors

* "error" instead of "err"

* restart jenkins

* Switch to zeebo/errs v1.2.0

* restart jenkins

* restart jenkins

* restart jenkins

* restart jenkins

* restart jenkins

* restart jenkins

* Remove some extra logs

* restart jenkins
2019-06-05 12:33:35 -06:00
JT Olio
f1641af802 storage: add monkit task to missing places (#2122)
* storage: add monkit task to missing places

Change-Id: I9e17a6b14f7c25bbf698eeecf32785e9add3f26e

* fix tests

Change-Id: Id078276fa3de61a28eb3d01d4e751732ecbb173f

* import order

Change-Id: I814e33755b9f10b5219af37cd828cd75eb3da1a4

* remove part of other commit

Change-Id: Idaa4c95cd65e97567fb466de49718db8203cfbe1
2019-06-05 16:23:10 +02:00
JT Olio
ccb158c99b
pkg/auth: add monkit task to missing places (#2123)
What: add monkit.Task to a bunch of functions that are missing it

Why: this will significantly help our instrumentation, data collection, and tracing about what's going on in the network
2019-06-05 07:47:01 -06:00
paul cannon
d15eaed588 add capability of logging all GRPC calls/payloads (#2067) 2019-06-04 14:55:24 +02:00
JT Olio
3fe8343b6c repairer: fix config comments (#2105) 2019-06-04 14:13:31 +02:00
JT Olio
9c5708da32 pkg/*: add monkit task to missing places (#2109) 2019-06-04 13:36:27 +02:00
Cameron
e077b0d380
rename VetNode to IsVetted (#2097)
* rename VetNode to IsVetted
2019-06-03 10:53:30 -04:00
Egon Elbre
8c0c518621 pkg/storj: use proper unquoting of json data in NodeID and PieceID (#2094) 2019-06-03 15:29:07 +02:00
Kaloyan Raev
2ab95b533e
Check errors for possible outcomes from audit's DownloadShares (#2072) 2019-06-03 12:17:09 +03:00
JT Olio
e60ff9dcbb
process/metrics: have metrics suffix default to dev/release status (#2073)
What: this will make it so release binaries default to whatever-release instead of whatever-dev in metrics collection

Why: So we can monitor release binaries with default configuration without getting drowned out by dev binaries
2019-05-31 16:47:48 -06:00
Jeff Wendling
140251882e
fix bug for setting flag only values in process setup (#2089)
* fix bug for setting flag only values in process setup

when the code was changed to directly load values into the config
structs, it was missed that some configuration is only defined
through flags, but can be loaded from config files still.

so, we need to propogate the settings to the flag only values.

* add test for setting propagation

* fix linting error
2019-05-31 21:15:50 +00:00
Maximillian von Briesen
294fddcece
Add audit monkit stats (#2087) 2019-05-31 16:46:25 -04:00
Fadila
5b730e3073
Make maxReverifyCount configurable (#2071)
* make max reverify count configurable
2019-05-31 17:23:00 +02:00
aligeti
4ad5120923
Checker service refactor (v3-1871) (#2082)
*  refactor the checker service

* monkit update
2019-05-31 10:12:49 -04:00
Natalie Villasana
6db9388082 add disqualified column to nodes table (#2086)
* add disqualified column to nodes table, update migrate script and testdata

* fix crazy formatting of postgres.v25.sql
2019-05-30 17:38:23 -04:00
Cameron
590b1a5a1d
Satellite voucher service (#2043)
* set up voucher service skeleton, basic test

* add VetNode db method

* basic test for VetNode

* encode and sign voucher functions

* fill out and sign vouchers

* test pass/fail voucher request

* match EncodeVoucher to other Encode functions
2019-05-30 15:52:33 -04:00
Jennifer Li Johnson
24c8132975
rewrites encode methods for readability (#2084) 2019-05-30 15:01:55 -04:00
Stefan Benten
4b75752d6b
Relocate IP Lookup to ensure it is always set (#2061) 2019-05-30 19:35:04 +02:00
aligeti
934ebf9cbf
Added the irreparable repair functionality (#1955)
* Added the irreparable repair functionality
2019-05-30 11:18:20 -04:00
Jeff Wendling
e74cac52ab
Command line flags features and cleanup (#2068)
* change BindSetup to be an option to Bind
* add process.Bind to allow composite structures
* hack fix for noprefix flags
* used tagged version of structs

Before this PR, some flags were created by calling `cfgstruct.Bind` and having their fields create a flag. Once the flags were parsed, `viper` was used to acquire all the values from them and config files, and the fields in the struct were set through the flag interface.

This doesn't work for slices of things on config structs very well, since it can only set strings, and for a string slice, it turns out that the implementation in `pflag` appends an entry rather than setting it.

This changes three things:

1. Only have a `Bind` call instead of `Bind` and `BindSetup`, and make `BindSetup` an option instead.
2. Add a `process.Bind` call that takes in a `*cobra.Cmd`, binds the struct to the command's flags, and keeps track of that struct in a global map keyed by the command.
3. Use `viper` to get the values and load them into the bound configuration structs instead of using the flags to propagate the changes.

In this way, we can support whatever rich configuration we want in the config yaml files, while still getting command like flags when important.
2019-05-29 17:56:22 +00:00
ethanadams
16e3b77cf5
Enable Scopelint Linter (#2049)
* added scopelint and correcte issues found

* corrected scopelint issue

* made updates based on Ivan's suggestions

Most were around naming conventions
Some were false positives, but I kept them since the test.Run could eventually be changed to run in parallel, which could cause a bug
Others were false positives.  Added // nolint: scopelint
2019-05-29 09:30:16 -04:00
ethanadams
268dc6b7e4
Enable gocritic linter (#2051)
* first round cleanup based on go-critic

* more issues resolved for ifelsechain and unlambda checks

* updated from master and gocritic found a new ifElseChain issue

* disable appendAssign. i reports false positives

* re-enabled go-critic appendAssign and disabled lint check at code line level

* fixed go-critic lint error

* fixed // nolint add gocritic specifically
2019-05-29 09:14:25 -04:00
Michal Niewrzal
f731267e8c Per-project usage limiting (#2036)
What: Changes to support custom usage limit for the project. With this implementation by default project usage limit is taken from configuration flag. If project DB field usage_limit will be set to value larger than 0 it will become custom usage limit and we will be used to verify is limit was exceeded.

Whats changed:

usage_limit (bigint) field added to projects table (with migration)
things related to project usage moved from metainfo endpoint to project usage type
accounting.ProjectAccounting extended with GetProjectUsageLimits() method
Why: We need to have different usage limits per project. https://storjlabs.atlassian.net/browse/V3-1814
2019-05-28 09:36:52 -06:00
Maximillian von Briesen
c07162beef address potential divide by 0` (#2065) 2019-05-28 08:54:30 -06:00
Maximillian von Briesen
5a4ff2c855 add repair monkit stats (#2045)
* add repair monkit stats

* rename values, use meter instead of counter, use success threshold instead of repair threshold

* Counter -> Meter

* add repair segment size

* update names and use ratios for healthy before/after repair

* restart jenkins
2019-05-28 16:10:26 +02:00
Natalie Villasana
aa6ff17b70 add Reverify to auditing (#2041)
Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
Co-authored-by: Kaloyan Raev <kaloyan@storj.io>
2019-05-27 14:13:47 +03:00
Michal Niewrzal
4e8b5cb972 Fix flaky TestProjectUsageBandwidth (#2047) 2019-05-25 21:24:12 +02:00
Ivan Fraixedes
5c50aac661 pkg/audit: Readdress ignored Verifier tests (#2048) 2019-05-24 19:57:07 +03:00
Jeff Wendling
1bd52b9f90 server side macaroons (#1945)
What: Adds macaroon support to the server side

Why: So that api keys are now macaroons
2019-05-24 10:51:27 -06:00