Commit Graph

1293 Commits

Author SHA1 Message Date
Jess G
f11bf46a11
Jg/1967 mv bucket metadata uplink (#2505)
* add bucketstore, add init uplink bucket

* update uplink to use bucket rpc

* fix tests

* wrap metainfo client errors

* add allowedBucket struct, fix tests

* update comment

* add paging

* updates per CR

* add test for pagination

* fix lint

* fix uplink test so its easier tyo understand

* fix gateway pagination bug

* changes per cr

* fix bug w/allowedBuckets, add test to catch
2019-07-12 08:57:02 -04:00
Ivan Fraixedes
f420b29d35
[V3-1927] Repairer uploads to max threshold instead of success… (#2423)
* pkg/datarepair: Add test to check num upload pieces
  Add a new test for ensuring the number of pieces that the repair process
  upload when a segment is injured.
* satellite/orders: Don't create "put order limits" over total
  Repair must not create "put order limits" more than the total count.
* pkg/datarepair: Update upload repair pieces test
  Update the test which checks the number of pieces which are uploaded
  during a repair for using the same excess over the success threshold
  value than the implementation.
* satellites/orders: Limit repair put order for not being total
  Limit the number of put orders to be used by repair for only uploading
  pieces to a % excess over the successful threshold.
* pkg/datarepair: Change DataRepair test to pass again
  Make some changes in the DataRepair test to make pass again after the
  repair upload repaired pieces only until a % excess over success
  threshold.
  Also update the steps description of the DataRepair test after it has been
  changed, to match on what's now, besides to leave it more generic for
  avoiding having to update it on minimal future refactorings.
* satellite: Make repair excess optimal threshold configurable
  Add a new configuration parameter to the satellite for being able to
  configure the percentage excess over the optimal threshold, used for
  determining how many pieces should be repaired/uploaded, rather than
  having the value hard coded.
* repairer: Add configurable param to segments/repairer
  Add a new parameters to the segment/repairer to calculate the maximum
  number of excess nodes, based on the optimal threshold, that repaired
  pieces can be uploaded.
  This new parameter has been added for not returning more nodes than the
  number of upload orders for data repair satellite service calculate for
  repairing pieces.
* pkg/storage/ec: Update log message in clien.Repair
* satellite: Update configuration lock file
2019-07-12 00:44:47 +02:00
Michal Niewrzal
8a9c63809b tests for PiecePublicKey/PiecePrivateKey (#2526) 2019-07-11 17:11:04 -04:00
Egon Elbre
d52f764e54
protocol: implement new piece signing and verification (#2525) 2019-07-11 16:51:40 -04:00
Maximillian von Briesen
8b507f3d73 Address concerns with storagenode Retain endpoint (#2527) 2019-07-11 16:04:21 -04:00
Michal Niewrzal
268c629ba8
Replace base64 encoding for path segments (#2345) 2019-07-11 13:26:07 -04:00
Kaloyan Raev
0f36b160fc Higher dial timeout for TestDownloadSharesDownloadTimeout (#2500) 2019-07-10 17:45:09 -04:00
Maximillian von Briesen
de85d17069
Add checker metrics (#2487)
checker_segment_total_count - Number of total segments in pointer during checker iteration
checker_segment_healthy_count - Number of healthy segments in pointer during checker iterationn
time_since_checker_queue - Seconds elapsed between checker queue and beginning repair
time_for_repair - Seconds elapsed between beginning repair and ending repair/dequeueing
2019-07-10 17:27:46 -04:00
Jeff Wendling
10547cc1ea
segments: send in the object path to the initial CreateSegment… (#2518)
otherwise, api key restictions will fail because we look like we're
asking to put to the bucket metadata path.
2019-07-10 11:33:55 -04:00
Cameron
c29f033e7d
Move kademlia dialer into separate package (#2466)
* move kademlia.Dialer into kademliaclient package
2019-07-10 10:36:37 -04:00
JT Olio
a79c7d77f3 overlay cache: slight modification of node-is-online rules (#2490) 2019-07-09 22:36:09 -04:00
Alexander Leitner
1c5db71faf
Change protobuf expirations to use time.Time (#2509)
* Change protobuf expirations to use time.Time instead of timestamp.Timestamp
2019-07-09 17:54:00 -04:00
Jeff Wendling
671648012a Update password based key derivation design doc (#2367) 2019-07-09 00:07:03 -04:00
Ivan Fraixedes
50d601ab07 pkg/identity: Use identity error class (#2488) 2019-07-08 22:52:52 -04:00
JT Olio
887b784f54 discovery: use fetch info to ping (#2491) 2019-07-08 22:10:17 -04:00
Jess G
f9696d6c5e
satellite/metainfo: add buckets RPC and database (#2460)
* add db interface and methods, add sa metainfo endpoints and svc

* add bucket metainfo svc funcs

* add sadb bucekts

* bucket list gets all buckets

* filter buckets list on macaroon restrictions

* update pb cipher suite to be enum

* add conversion funcs

* updates per comments

* bucket settings should say default

* add direction to list buckets, add tests

* fix test bucket names

* lint err

* only support forward direction

* add comments

* minor refactoring

* make sure list up to limit

* update test

* update protolock file

* fix lint

* change per PR
2019-07-08 15:32:18 -07:00
Alexander Leitner
3587e1a579 Change pointerdb pointer to use time.Time for Creation date (#2483) 2019-07-09 00:16:50 +02:00
Egon Elbre
674742d1a7
satellite/datarepair: use reliability cache (#1976) 2019-07-09 01:04:35 +03:00
Ivan Fraixedes
a786e4249c pkg: Align errs Class messages (#2485)
Align 2 errs Class messages with the rest of them.
2019-07-08 17:37:12 -04:00
Alexander Leitner
19ab9852f2
Update node.proto to use time.Time instead of timestamp (#2482) 2019-07-08 14:24:42 -04:00
Alexander Leitner
dcf8e2936b
Update vouchers to use time.Time instead of timestamp (#2478)
* Update vouchers to use time.Time instead of timestamp
2019-07-08 13:07:30 -04:00
Simon Guindon
8d15d774b6
Removing trace of encodedPiece.Read() because it's fast. (#2477)
* Removing trace of encodedPiece.Read() because it's fast.

* Added comment about why tracing was removed.
2019-07-08 11:16:13 -04:00
Stefan Benten
ccef5eee46
Add proper Version Handling to Identity, Gateway and Uplink Binary (#2471) 2019-07-08 10:45:20 -04:00
Yaroslav Vorobiov
524eb24c83 storagenode/nodestats: combine stats into single RPC call (#2455)
* change satellite nodestats endpoint
2019-07-08 17:33:43 +03:00
Alexander Leitner
88732188cb
Update inspector timestamp to time.time (#2464)
* Update inspector timestamp
2019-07-08 10:06:12 -04:00
Michal Niewrzal
094e1b8b90
Add 'Old' suffix to some metainfo methods/messages (#2462) 2019-07-08 15:33:15 +02:00
Yaroslav Vorobiov
7aca0eb284 storagenode/dashboard: show console address (#2456) 2019-07-06 15:40:58 +02:00
Ivan Fraixedes
ea9a8b37bc pkg/datarepair/checker: Fix typo in doc comment (#2461) 2019-07-05 22:20:58 +02:00
Kaloyan Raev
f9ed0dc1a8
Improve stability of TestDownloadSharesDownloadTimeout (#2210) 2019-07-05 19:04:15 +03:00
Ivan Fraixedes
e0a8937c6c
pkg/storage/ec: Make minor improvements in client.Repair (#2454)
* Fix some log message to actually report the number of pieces needed to
  repaired for reaching the successful/optimal threshold.
* Remove some unneeded `nil` check conditional.
2019-07-05 12:39:10 +02:00
Jeff Wendling
8b07df37f5 refactor some store encryption stuff (#2434) 2019-07-05 11:36:35 +03:00
Yaroslav Vorobiov
42c125d697
Satellite nodestats endpoint add audit check (#2426) 2019-07-04 13:41:40 +03:00
Michal Niewrzal
5176e4f096 Rename BucketItem into Bucket (#2442)
* Rename BucketItem into Bucket
2019-07-03 15:44:02 -04:00
aligeti
ae8b9698f9
Rename/remove EncryptionScheme -> EncryptionParame… (#2363)
* rename/remove EncryptionScheme -> EncryptionParameters
2019-07-03 14:07:44 -04:00
Cameron
d499d162f4
implement storj.NodeURL in trusted satellites (#2388)
* implement storj.NodeURL in trusted satellites
2019-07-03 13:29:18 -04:00
Michal Niewrzal
2ee5bada2c
Add pieceNum to PieceID derivation function (#2193) 2019-07-03 18:53:15 +02:00
Michal Niewrzal
61dfa61e3a
Add timestamp and piece size to piece hash (#2198) 2019-07-03 18:14:37 +02:00
Maximillian von Briesen
2f6e193c32
Remove long tail timer in ecclient (#2433)
* remove long tail timer in ecclient
2019-07-03 11:00:24 -04:00
Michal Niewrzal
46b5c30f35
Fix test-sim debug-addr error (#2420) 2019-07-03 16:10:51 +02:00
Jeff Wendling
42e124cd08
monitor optimal wait fraction (#2435)
* monitor optimal wait fraction

Change-Id: I1c76da5e8031237cf78ce5a0774732dd5e558ea1

* monitor other times about the upload

Change-Id: Iae81c80fb1446fbf4b3dd04fc6b238f2ede96545
2019-07-02 20:49:35 +00:00
Egon Elbre
9e26149a2d pkg/cfgstruct: add pflag support to bind (#2425) 2019-07-02 13:53:01 -04:00
Alexander Leitner
6d55bbdb57
OrderLimit creation date time limit (#2412)
* Limit by order creation
2019-07-02 12:06:12 -04:00
Andrew Harding
0de4cf31cf
libuplink scope type (#2382) 2019-07-02 09:45:23 -06:00
Kaloyan Raev
d32c907440
overlay.UpdateStats removes node from containment mode (#2419) 2019-07-02 18:16:25 +03:00
Yaroslav Vorobiov
f4dfb8fb95
Add daily node space usage to satellite nodestats endpoint (#2361) 2019-07-02 13:42:09 +03:00
Maximillian von Briesen
52e5a4eee3 pass logger into repairer and ecclient (#2365) 2019-07-02 13:08:02 +03:00
Michal Niewrzal
3337087a94
Add storj.StreamID (#2407) 2019-07-02 11:16:06 +02:00
littleskunk
a2362f92dc
Rollback uptime disqualification (#2417) 2019-07-02 10:39:36 +02:00
Jennifer Li Johnson
a7aa854df8
Routing table antechamber should expire old nodes that failed connection (#2387) 2019-07-01 23:20:12 -04:00
littleskunk
9e62423f47
reduce vetting requirement (#2416) 2019-07-02 01:02:23 +02:00
Jess G
ea55ae63e3
add bucket metainfo rpc (#2383)
* add bucket metainfo rpc

* updates per design doc

* rm bucketmeta rpc

* add changes per comments

* update Bucket name since same name already exists

* add placeholder methods to fulfill pb.MetainfoServer interface

* update proto lock

* add bucketListItem message

* why failing lint?

* add back ciphersuite but with correct types

* fix proto errs

* update time type

* add proto lock

* rm unneeded import

* update protolock package
2019-07-01 15:17:30 -07:00
Jennifer Li Johnson
699ccea19f
Creates Routing Table Antechamber (#2318) 2019-07-01 17:20:19 -04:00
littleskunk
fb66867856 fix repair queue (#2405) 2019-07-01 19:00:53 +02:00
Natalie Villasana
4ba212107b
add back testrand (#2408) 2019-07-01 12:39:04 -04:00
Ivan Fraixedes
b6ae6a4e2c
pkg/datarepair/checker: Fix doc comments (#2401) 2019-07-01 18:13:18 +02:00
Egon Elbre
385c046723
pkg/pb: rename Order2 to Order, OrderLimit2 to OrderLimit (#2406) 2019-07-01 18:54:11 +03:00
Natalie Villasana
3ffb42483f
account for disqualified nodes in data repair tests (#2315) 2019-07-01 11:34:42 -04:00
Natalie Villasana
3f643551e7 remove flakiness in TestDataRepair and TestSegmentStoreRepair (#2335)
* stop audit loop in repair tests to prevent possible timeout
2019-07-01 11:15:45 -04:00
Kaloyan Raev
4f0e437965
Audit tests cleanup (#2402) 2019-07-01 17:02:00 +03:00
Ivan Fraixedes
fd2e708587 pkg/storage/segments: Abort repaired test unmet requirements (#2403)
Abort the repairer test when asserting results which are required later
on by the test for being able to continue.
2019-07-01 08:49:40 -04:00
Maximillian von Briesen
b750bc2d0f
Keep some unhealthy pieces in the event of a partial repair (#2380)
* if repair is partial, keep saving "unhealthy" pieces that are not duplicates
2019-06-28 15:48:51 -04:00
Egon Elbre
7e0c5d51d0
pkg/storj: add NodeURL type for verified url-s (#2379) 2019-06-28 21:43:39 +03:00
Egon Elbre
827fb92b47
satellite/console: nicer error handling in tests (#2378) 2019-06-28 16:04:50 +03:00
Jeff Wendling
1a65e42d33 rename EncryptionCtx back to EncryptionAccess (#2368)
* rename EncryptionCtx back to EncryptionAccess

Change-Id: I5e58915a59979ad6f2e83d36e191b2bbf3ba2ba2

* missed some stuff

Change-Id: Ib65fb186d7f854c0406e5fd7d11498e8941da59e

* oops protolock

Change-Id: I85a5ab4bafb9dd6a804d3dcd17a70811702f07e4

* retrigger

Change-Id: I2d0e3f02b4dbae0299b090e9617662b4437980b0
2019-06-27 23:18:24 -06:00
aligeti
811168e2c4
Uplink bucket attribution extension (#2353)
Attribution info added to the bucket info
2019-06-27 19:25:36 -04:00
Egon Elbre
e83ebd7cde
jenkins: avoid using goimports and distribute load better (#2359) 2019-06-27 21:52:50 +03:00
Jeff Wendling
efcdaa43a3
lib/uplink: encryption context (#2349)
* lib/uplink: encryption context

Change-Id: I5c23dca3286a46b713b30c4997e9ae6e630b2280

* lib/uplink: bucket operation examples

Change-Id: Ia0f6e69f365dcff0cf11c731f51b30842bce053b

* lib/uplink: encryption key sharing test cases

Change-Id: I3a172d565f33f4e591402cdcb9460664a7cc7fbe

* fix encrypted path prefix restriction issue

Change-Id: I8f3921f9d52aaf4b84039de608b8cbbc88769554

* implement panics in libuplink encryption code

todo on cipher suite selection as well as an api concern

Change-Id: Ifa39eb3cc4b3443f7d96f9304df9b2ac4ec4085d

* implement GetProjectInfo api call to get salt

Change-Id: Ic5f6b3be9ea35df48c1aa214ab5d355fb328e2cf

* some fixes and accessors for encryption store

Change-Id: I3bb61f6712a037900e2a96e72ad4029ec1d3f718

* general fixes to builds/tests/etc

Change-Id: I9930fa96acb3b221d9a001f8e274af5729cc8a47

* java bindings changes

Change-Id: Ia2bd4c9c69739c8d3154d79616cff1f36fb403b6

* get libuplink examples passing

Change-Id: I828f09a144160e0a5dd932324f78491ae2ec8a07

* fix proto.lock file

Change-Id: I2fbbf4d0976a7d0473c2645e6dcb21aaa3be7651

* fix proto.lock again

Change-Id: I92702cf49e1a340eef6379c2be4f7c4a268112a9

* fix golint issues

Change-Id: I631ff9f43307a58e3b25a58cbb4a4cc2495f5eb6

* more linting fixes

Change-Id: I51f8f30b367b5bca14c94b15417b9a4c9e7aa0ce

* bug fixed by structs bump

Change-Id: Ibb03c691fce7606c35c08721b3ef0781ab48a38a

* retrigger

Change-Id: Ieee0470b6a2d07168a1578552e8e7f271ae93a13

* retrigger

Change-Id: I753d63853171e6a436c104ce176048892eb974c5

* semantic merge conflict

Change-Id: I9419448496de90340569047a6a16a1b858a7978a

* update total to match prod defaults

Change-Id: I693d55c1ebb28b5803ee1d26e9e198decf82308b

* retrigger

Change-Id: I28b74d5d6202f61aa3866fe407d423f6a0a14b9e

* retrigger

Change-Id: I6fd054885c715f602e2cef623fd464c42e88742c

* retrigger

Change-Id: I6a01bae88c72406d4ed5a8f13bf8a2b3c650bd2d
2019-06-27 17:36:51 +00:00
Egon Elbre
27c92ffc10 jenkins: Stop docker container after 18m (#2358)
* remove unnecessary argument

* remove kademlia lookup short circuiting

* fix ident

* add timeout to docker container

* add comments to dockerfile
2019-06-27 10:09:37 -06:00
Egon Elbre
2128b460b4
cmd/uplink/cmd: don't create benchmark data on init (#2351) 2019-06-26 22:08:14 +03:00
Egon Elbre
615bfca135 Fix TestGetSignee flakiness (#2350)
* add IsCanceled

* fixes to error handling

* fix imports

* retrigger jenkins
2019-06-26 09:30:37 -06:00
JT Olio
fbe9696e92 pkg/kademlia: clean up peer discovery (#2252) 2019-06-26 16:16:46 +03:00
JT Olio
3925e84580 pkg/eestream: plumb contexts through (#2187) 2019-06-26 16:05:58 +03:00
Egon Elbre
7b66e0cd7c Use dial to clarify that it's internally closing the connection. (#2347) 2019-06-26 15:14:48 +03:00
Egon Elbre
b6ad3e9c9f
internal/testrand: new package for random data (#2282) 2019-06-26 13:38:51 +03:00
Philip Hutchins
11016b5067 Moving verbose info logging to debug (#2346) 2019-06-26 12:14:25 +03:00
Kaloyan Raev
ae36979804 Take advantage of Is and IsFunc from zeebo/errs (#2310) 2019-06-26 09:38:07 +02:00
Egon Elbre
caa2fcf62b satellite/orders: don't panic (#2331) 2019-06-26 09:26:33 +02:00
Egon Elbre
c7679b9b30
Fix some leaks and add notes about close handling (#2334) 2019-06-25 23:00:51 +03:00
Stefan Benten
57ef352b3c
Update Wizard to allow more easily addition of satellites and move package to cmd folder (#2340) 2019-06-25 20:25:31 +02:00
Stefan Benten
2cc01c7899 Rename Satellites in the Wizard (#2339)
* Rename Satellites

* Fixing Typo and changing names for gateway

* Unify Setup wizard across gateway and uplink
2019-06-25 11:29:52 -06:00
Fadila
20de18feec [v3-1952 test 6] Disqualification is a one way ticket. Storage Node can't get around it. (#2328) 2019-06-25 13:10:22 -04:00
Egon Elbre
c28f800098
Skip TestDataRepair and TestUplinksParallel, because they are flaky (#2337) 2019-06-25 19:30:39 +03:00
Jess G
e5c48fab74
fix ordersDB methods to take correct args (#2314)
* fix orderdDB methods to take correct args

* update tally to save projectID in correct format

* update var names in splitBucket test

* changes per CR comments
2019-06-25 08:58:42 -07:00
Egon Elbre
414648d660
Fix some metainfo.Client leaks (#2327) 2019-06-25 18:36:23 +03:00
Maximillian von Briesen
35f2ab5ded
Enable node selection tests (#2316) 2019-06-25 11:26:47 -04:00
Yaroslav Vorobiov
eb5dc4ba2b
Add node stats GRPC endpoint to the satellite (#2281) 2019-06-25 13:40:32 +03:00
Egon Elbre
6502143e79
fix import ordering (#2322) 2019-06-25 12:46:29 +03:00
Ivan Fraixedes
fa802dc429
pkg/audit: Improve Reporter.ReporterAudits doc (#2266)
Add what the method returns because from the method name and the
signature name, it wasn't clear why and when a report is returned.
2019-06-25 11:23:41 +02:00
Jeff Wendling
c35c8e4c24 allow reading bucket metadata with restricted keys (#2321)
Change-Id: I47d3a2f5f02744ae6c51d54963cdf2dff24134e2
2019-06-24 17:36:09 -06:00
JT Olio
8c57434ded
pkg/process/metrics: add an instance prefix (#2190)
* pkg/process/metrics: add an instance prefix

the distinction between which satellite is sending which
data should go in the instance field, not the suffix or application
fields. (un)fortunately, the instance id is deliberately not
configurable because we don't want it to be easy to accidentally
have multiple applications collide with the same instance id.

so we're currently stuffing the human readable instance in the
suffix. :(

perhaps a reasonable tradeoff would be an optional instance
prefix that allows operators to put their domain name in
the instance

Change-Id: I6fcc8498be908c5740439cc00f77474ad151febd

* linting

Change-Id: I9f9a44fa9a2634ef5e4f89548d42d57ce9e4450e
2019-06-24 16:45:37 -06:00
Fadila
70f28ae41b
[v3-1952 tests 4 & 5] DQ nodes should not be used for download or upload (#2272) 2019-06-25 00:04:06 +02:00
Jeff Wendling
30f790a040
Create and use an encryption.Store (#2293)
* add path implementation

This commit adds a pkg/paths package which contains two types,
Encrypted and Unencrypted, to statically enforce what is contained
in a path. It's part of a refactoring of the code base to be more
clear about what is contained in a storj.Path at all the layers.

Change-Id: Ifc4d4932da26a97ea99749b8356b4543496a8864

* add encryption store

This change adds an encryption.Store type to keep a collection
of root keys for arbitrary locations in some buckets. It allows
one to look up all of the necessary information to encrypt paths,
decrypt paths and decrypt list operations.

It adds some exported functions to perform encryption on paths
using a Store.

Change-Id: I1a3d230c521d65f0ede727f93e1cb389f8be9497

* add shim around streams store

This commit changes no functionality, but just reorganizes the code
so that changes can be made directly to the streams store
implementation without affecting callers.

It also adds a Path type that will be used at the interface boundary
for the streams store so that it can be sure that it's getting well
formed paths that it expects.

Change-Id: I50bd682995b185beb653b00562fab62ef11f1ab5

* refactor streams to use encryption store

This commit changes the streams store to use the path type as
well as the encryption store to handle all of it's encryption
and decryption.

Some changes were made to how the default key is returned in
the encryption store to have it include the case when the bucket
exists but no paths matched. The path iterator could also be
simplified to not report if a consume was valid: that information
is no longer necessary.

The kvmetainfo tests were changed to appropriately pass the
subtests *testing.T rather than having the closure it executes
use the parent one. The test framework now correctly reports
which test did the failing.

There are still some latent issues with listing in that listing
for "a/" and listing for "a" are not the same operation, but we
treat them as such. I suspect that there are also issues with
paths like "/" or "//foo", but that's for another time.

Change-Id: I81cad4ba2850c3d14ba7e632777c4cac93db9472

* use an encryption store at the upper layers

Change-Id: Id9b4dd5f27b3ecac863de586e9ae076f4f927f6f

* fix linting failures

Change-Id: Ifb8378879ad308d4d047a0483850156371a41280

* fix linting in encryption test

Change-Id: Ia35647dfe18b0f20fe13763b28e53294f75c38fa

* get rid of kvmetainfo rootKey

Change-Id: Id795ca03d9417e3fe9634365a121430eb678d6d5

* Fix linting failure for return with else

Change-Id: I0b9ffd92be42ffcd8fef7ea735c5fc114a55d3b5

* fix some bugs adding enc store to kvmetainfo

Change-Id: I8e765970ba817289c65ec62971ae3bfa2c53a1ba

* respond to review feedback

Change-Id: I43e2ce29ce2fb6677b1cd6b9469838d80ec92c86
2019-06-24 19:23:07 +00:00
JT Olio
e17908a266
metainfo: add project info rpc for getting a project salt using an api key (#2311)
The API exposed in #2302 needs to be able to get a project-level salt (https://github.com/storj/storj/pull/2302/files#diff-72b734f4a2e408a544fef071d2565fc8R200), see https://github.com/storj/storj/blob/master/docs/design/password-key-derivation.md also

I realized this morning that if we need it later this week, we need to get it into today's release.

Please describe the tests: tests will come as part of #2302

Please describe the performance impact: none
2019-06-24 11:15:45 -06:00
Maximillian von Briesen
fd6a4d96f2
change uptime dq threshold to 0.4 (#2313)
* change uptime dq threshold to 0.4

* update config lock
2019-06-24 12:18:32 -04:00
Stefan Benten
01beaa289a
Mask IP Addresses to subnets (#2305) 2019-06-24 17:33:18 +02:00
Fadila
8226024ca8
Do not use disqualified nodes when asking for get order limits (#2303)
Add checks about disqualification when creating order limits
2019-06-24 16:46:10 +02:00
Kaloyan Raev
75d71f70c6 Stop pinging disqualified nodes (#2306) 2019-06-24 14:39:47 +02:00
Michal Niewrzal
fdeb834801
Bucket name validation (#2244) 2019-06-24 11:52:25 +02:00
Jennifer Li Johnson
e285fe1997
Don't require encryption keys for project or bucket management (#2291) 2019-06-23 22:06:14 -04:00
Cameron
1283036e37
add storage node voucher request service (#2158)
* add voucher service on storage node

* config field tag syntax, go routines for requests

* hook up voucher service in storagenode/peer.go

* add voucher config to testplanet

* add voucher config to testplanet

* add voucher response status INVALID, ACCEPTED, REJECTED

* add a test for vouchers service

* handle no row from GetValid, test it

* add trust pool to voucher service

* use trusted list to get satellites

* verify vouchers upon receipt

* test VerifyVoucher
2019-06-21 18:48:52 -04:00
Jeff Wendling
81f1bc19dc add functions for password/root key derivation (#2294)
This commit adds two functions that implement the algorithms
described in the password key derivation design document. They
will be used during setup to derive bucket level root keys or
default passwords to use when buckets do not have their own
independent key.

Change-Id: Ie7fb2d8d549ba7465d0722716a2c1ac0ad907286
2019-06-21 18:21:16 -04:00
Bill Thorp
8f47fca5d3
Remove audit / uptime ratio fields (#2247)
* removed ratios
2019-06-21 13:14:53 -04:00
Ivan Fraixedes
3d6b25a043
[v3-1952 test 1 & 3] pkg/audit: Add DQ test for too many failed audits (#2265)
* pkg/audit: Add DQ test for too many failed audits
  
  Add an integration test which checks that a node which fails several
  audits gets disqualified but not before it reaches the audit reputation
  disqualification cut-off.

* internal/testplanet: Set DQ cut-off config values
  Set the values of the Overlay cache DQ cut-off configuration parameters
  used by testplanet.
2019-06-21 18:27:19 +02:00
ethanadams
4f2e893e68
Fix the way project_id is stored in bucket_storage_tallies and bucket_bandwidth_rollups (#2283)
* fixing issues where projectID is stored as the byte representation of a UUID string, instead of bytes of the UUID

* added test for spitBucketID
2019-06-21 11:38:37 -04:00
Maximillian von Briesen
bfcfe39313
Enable and fix statdb tests (#2270) 2019-06-21 11:23:15 -04:00
Natalie Villasana
169fc9594c
pkg/audit: fix maxRetries bug (#2271) 2019-06-21 11:10:03 -04:00
Maximillian von Briesen
d103dd2c4e
Include information about disqualified nodes in the storage node payment report (#2239) 2019-06-21 10:21:15 -04:00
JT Olio
aa25c4458f kvmetainfo: merge with storage/buckets (#2277) 2019-06-21 13:29:31 +02:00
JT Olio
76b54458e9 satellite: send external address in order limits (#2278) 2019-06-21 12:19:52 +03:00
JT Olio
06006effb3 lib/uplink: minor fixes (#2257) 2019-06-20 23:50:13 +02:00
Jess G
daf166d423
fix tally test flakiness (#2250)
* fix test flakiness

* update tests to rm uplink.upload

* rm metainfo change

* add test case of no objects in bucket
2019-06-20 12:15:13 -07:00
Natalie Villasana
edb3d1cbf8
pkg/overlay: update node selection config values for reputation (#2264) 2019-06-20 15:01:50 -04:00
Ivan Fraixedes
5b31086757
pkg/audit: Move test helper funcs to separated file (#2259)
Move 2 helper function used for test which relay on testplanet from the
test file where they were created to separated file to contain them
because they are not only used in the test file were initially they were
created.
2019-06-20 17:38:08 +02:00
Natalie Villasana
9386187fe6
add disqualification and new reputation system into overlay cache (#2227) 2019-06-20 09:56:04 -04:00
Fadila
1726a6c073
pkg/bloomfilter: implementation and benchmark results (#2113) 2019-06-20 12:27:20 +02:00
Kaloyan Raev
964c87c476 Fix checks around repair threshold (#2246) 2019-06-19 22:13:11 +02:00
Jess G
ddcf4fc2a3
add support to hide config settings (#2241)
* add hide support for config settings

* updates per CR to unit test

* check err for lint
2019-06-19 07:27:44 -07:00
Fadila
0eb8ac58c1
Create protocol buffer for garbage collection (#2222) 2019-06-19 15:56:46 +02:00
Egon Elbre
8f4a6afc8a Fixes for value attribution (#2238)
* Move value attribution to satellite/

* remove unnecessary conversion
2019-06-19 08:02:37 -04:00
Kaloyan Raev
ac65d19753
Avoid reporting audit failures for files deleted during audit (#2233) 2019-06-19 12:02:25 +03:00
Jess G
f0f59a5577 Always encrypt inline segments without padding (#2183) 2019-06-19 11:11:27 +03:00
Maximillian von Briesen
ad8cad4909
Expand the inspector tool to provide node id's for each segment, rather than just numeric totals (#2205) 2019-06-18 18:22:14 -04:00
Natalie Villasana
b30c35d306
change ReputationAuditOmega (et al.) to AuditReputationWeight (#2232) 2019-06-18 14:17:25 -04:00
Bill Thorp
119a8fd3cc removed fields (#2234)
* removed fields

* sql tweaks
2019-06-18 13:40:28 -04:00
aligeti
09940d4e0b
value attribution DB interface Insert & Get method support (#2200)
* value attribution DB interface methods support
2019-06-18 09:06:33 -04:00
Maximillian von Briesen
8398fae9b5
Add node churn and containment/reverify monkit stats (#2217)
* add counters for nodes that have/have not been seen in the past 24 hours/week

* add additional uptime counters

* add monkit stats for containment mode
2019-06-18 08:54:52 -04:00
Ivan Fraixedes
35c8648330
[v3-1914] Storage node disqualification: Change type from bool to timestamp (#2212)
* satellite/satellitedb: Alter nodes disqualification column
  Change the type of the 'disqualification' column of the nodes table from
  boolean to timestamp.
* overlay/cache: Change Disqualified field type
  Change the Disqualified field type the NodeDossier struct type from bool
  to time.Time to match with the disqualified type used by the DB layer.
* satellite/satellitedb: Update queries uses disqualified
  Update the queries which uses the disqualified column due to the column
  type has been changed from boolean to nullable timestamp.
* docs/design: Update disqualification due impl changes
  Update the disqualification design document to contain the architectural
  change required to be able to restore unfair disqualified nodes in case
  of an unexpected cause (bug, mistake, hard network disconnection, etc.).
2019-06-18 11:14:31 +02:00
Jennifer Li Johnson
d177761f51
Adds test coverage to CalculateBucketAtRestData (#2203) 2019-06-17 22:20:40 -04:00
Egon Elbre
f4f776d092 Use mail.test as domain in emails (#2224) 2019-06-18 02:28:40 +02:00
Kaloyan Raev
8e29ef8a6b Use zap.Stringer instead of zap.String (#2223) 2019-06-18 01:37:43 +02:00
Bill Thorp
f378125c8b
draft of removing stats from inspector (#2226) 2019-06-17 16:48:04 -04:00
JT Olio
e58a06bd0c config: update release values to match prod (#2192) 2019-06-15 18:19:19 +02:00
littleskunk
319cc77a34
increase audit timeout (#2208) 2019-06-14 13:53:49 +02:00
Kaloyan Raev
ebd9b375fc
Repair should not corrupt files (#2194) 2019-06-14 12:16:31 +03:00
Bill Thorp
6932dc92f3
NodeSelectionConfig inits and test cleanup (#2202)
* NodeSelectionConfig inits and test cleanup
2019-06-13 17:51:18 -04:00
ethanadams
4e1cc37a4c Fix bucket_storage_tally data (#2165) 2019-06-13 12:58:40 -04:00
JT Olio
df2fad15d8
pkg/process/logging: different defaults for release/dev (#2191)
* pkg/process/logging: different defaults for release/dev

Change-Id: I55be80430a31668fededf479b052e106ab18d9ce

* linting

Change-Id: I4e50d4c9569b7324c4704c14df7dd3228dbb7dd5

* Trigger Jenkins

* fix lock file

* use dev=debug and prod=info
2019-06-13 10:43:39 -06:00
JT Olio
fa09a7894b pkg/process: prometheus support (#2176) 2019-06-13 18:29:35 +02:00
Bill Thorp
703e5fee49
mostly removed things (#2199) 2019-06-13 12:06:37 -04:00
Bryan White
bc33964729 Uplink C bindings part 1 (#2196)
* uplink cbindings tooling
2019-06-13 11:09:05 -04:00
JT Olio
1ae5654eba kademlia/routing: add contexts to more places so monkit works (#2188) 2019-06-13 16:51:50 +02:00
Egon Elbre
1a1a084477
testcontext: sanitize folder name (#2195) 2019-06-13 15:46:08 +03:00
aligeti
8e8a5eb89c
initial value attribution check development (#2091)
* initial value attribution check development

* code reveiew updates
2019-06-12 21:35:37 -04:00
Natalie Villasana
5314c950cc
pkg/audit: add more reverify tests (#2144) 2019-06-11 16:14:40 -04:00
Jess G
d84cd719ff
add const stripesPerBlock const to calc blocksize (#2163)
* add const stripesPerBlock, update comments in Pad, add speed to progressbar

* change size of erasure share size, update comments

* missing copyright

* update tests with stripesPerBlock
2019-06-11 10:14:05 -07:00
Kaloyan Raev
252c8ac189
Add email to self node info (#2171) 2019-06-11 16:30:28 +03:00
Ivan Fraixedes
f624b213a3
reputation: Add configuration parameters (#2150)
* reputation: Add configuration parameters
 Add the configuration parameters which will be used by the algorithm
 which will calculate the storage node reputation.
 Because the reputation calculation is based on audit and uptime check
 results some configuration parameters are in pkg/audit, others in
 pkg/discovery and other in the satellite which will combine the both
 reputation results to obtain the storage node reputation for repair and
 uplink.
* satellite-config: Refresh lock file with new params
  Refresh the Satellite configuration yaml lock file with the new
  parameters added in this branch.
2019-06-11 12:14:01 +02:00
Kaloyan Raev
f0880b9b3c
Audit service should download erasure shares in parallel (#2164) 2019-06-11 11:00:59 +03:00
Egon Elbre
749846b42b
Update golangci-lint (#2159) 2019-06-10 11:52:09 +03:00
ethanadams
8f2dca8437 Re-enabling and fixing repairer tests (#2099)
* Disabled discovery service by changiing from Stop() to Pause()

Paused to solve race condition.  If discovery is running, it may mark a node "up" after they've been manually marked "down" in this test.

* Extend to the repair timeout

Fixes intermittent test failures when repairs were taking more than 2 seconds.

* Re-enabled test. Disabled discovery service by changiing from Stop() to Pause()

* Changed back to Stop.

* Revert "Changed back to Stop."

This reverts commit 46d410e72dfae63e0c44915be42784cc9a7b5abf.

* re-enabling TestIdentifyInjuredSegments

* Changed Pause to Stop.  Commented on timeout change

* testing...

* temporarily skipping audit tests

* changing back to discover Stop for testing via jenkins

* Revert "changing back to discover Stop for testing via jenkins"

This reverts commit 6aa8558b11a0053c30e0c8b2dbf0d6c0cb34ee6c.

* Changing back to Stop().  Depends on PR 2137

* Revert "temporarily skipping audit tests"

This reverts commit 1940ed9b315d663a0eb6c95521780cbcb48cb121.

* Removed reference to Graveyard since its been removed
2019-06-10 09:06:21 +02:00
Simon Guindon
3a7ebea76e
Moved monkit tracing after the limit nil check & fix node id formatting. (#2156)
* Moved monkit tracing after the limit nil check & fix node id formatting.

* Fixed to keep function traced even if limits is nil.
2019-06-07 18:34:16 -04:00
Natalie Villasana
16acb58546 rename verifier test files (#2154)
Despite the confusing git trail, this is two pure file renames, no other changes.
2019-06-07 16:02:36 -05:00
Ivan Fraixedes
44359e8bb3
pkg/kademlia: Fix bug using ref of var range loop (#2149)
Fix a bug in a range loop which used the reference of the variable assigned in the same loop to be appended in a slice, turning out that the slice will contain the same reference rather the a reference for each value of the ranged items.
2019-06-07 15:08:09 +02:00
Kaloyan Raev
8ddb1dee0d
Improve error analysis in audit service (#2125)
Co-authored-by: Natalie Ventura Villasana <nat@storj.io>
2019-06-07 15:38:41 +03:00
JT Olio
43d4f3daf5 discovery: remove graveyard (#2145) 2019-06-07 08:40:51 +03:00
Simon Guindon
a55df84bf7
Adding RS values to the info log and NodeID to tracing. (#2130) 2019-06-06 15:51:00 -04:00
Jess G
bf3d168cf6
makes sure all uplink cli configs get passed to libuplink, add stripeSize (#2103)
* makes sure all uplink cli configs get passed to libuplink, add stripSize

* update comment

* update defaults for uplink config blocksize

* changes per CR, update uplink config defaults

* pass shareSize from uplink config

* move block size validation to kvmeta pkg

* fix tests

* shareSize default 1k, rm config option blocksize

* rm printing err to stdout
2019-06-06 11:55:10 -07:00
Egon Elbre
91a0ece959
Remove bwagreement leftovers (#2140) 2019-06-06 18:16:14 +03:00
Egon Elbre
28a1201590 pkg/bwagreements: remove service (#2138) 2019-06-06 15:57:58 +02:00
Ivan Fraixedes
503fe0e5ec pkg/audit: Fix code style named result params (#2136) 2019-06-06 15:55:06 +03:00
JT Olio
2d19475d61 monitor macaroon parse/serialization perf (#2132)
Change-Id: I3c9556194e23af011db77b6b743abf3e5799f3f0
2019-06-06 11:35:03 +02:00
Natalie Villasana
25d7dda135 add disqualified check to node selection queries (#2102)
add disqualified check to queries, skip TestStatDB
2019-06-05 20:21:32 -04:00
Kaloyan Raev
099cf921d3 Fix flaky audit test (#2093)
* Show full unexpected error when assertion fails

* restart jenkins

* restart jenkins

* restart jenkins

* restart jenkins

* Even better display of the unexpected error

* Fix flakiness using kaloyan-raev/errs fork

* Add more logs

* restart jenkins

* restart jenkins

* Better print for unexpected errors

* "error" instead of "err"

* restart jenkins

* Switch to zeebo/errs v1.2.0

* restart jenkins

* restart jenkins

* restart jenkins

* restart jenkins

* restart jenkins

* restart jenkins

* Remove some extra logs

* restart jenkins
2019-06-05 12:33:35 -06:00
JT Olio
f1641af802 storage: add monkit task to missing places (#2122)
* storage: add monkit task to missing places

Change-Id: I9e17a6b14f7c25bbf698eeecf32785e9add3f26e

* fix tests

Change-Id: Id078276fa3de61a28eb3d01d4e751732ecbb173f

* import order

Change-Id: I814e33755b9f10b5219af37cd828cd75eb3da1a4

* remove part of other commit

Change-Id: Idaa4c95cd65e97567fb466de49718db8203cfbe1
2019-06-05 16:23:10 +02:00
JT Olio
ccb158c99b
pkg/auth: add monkit task to missing places (#2123)
What: add monkit.Task to a bunch of functions that are missing it

Why: this will significantly help our instrumentation, data collection, and tracing about what's going on in the network
2019-06-05 07:47:01 -06:00
paul cannon
d15eaed588 add capability of logging all GRPC calls/payloads (#2067) 2019-06-04 14:55:24 +02:00
JT Olio
3fe8343b6c repairer: fix config comments (#2105) 2019-06-04 14:13:31 +02:00
JT Olio
9c5708da32 pkg/*: add monkit task to missing places (#2109) 2019-06-04 13:36:27 +02:00
Cameron
e077b0d380
rename VetNode to IsVetted (#2097)
* rename VetNode to IsVetted
2019-06-03 10:53:30 -04:00
Egon Elbre
8c0c518621 pkg/storj: use proper unquoting of json data in NodeID and PieceID (#2094) 2019-06-03 15:29:07 +02:00
Kaloyan Raev
2ab95b533e
Check errors for possible outcomes from audit's DownloadShares (#2072) 2019-06-03 12:17:09 +03:00
JT Olio
e60ff9dcbb
process/metrics: have metrics suffix default to dev/release status (#2073)
What: this will make it so release binaries default to whatever-release instead of whatever-dev in metrics collection

Why: So we can monitor release binaries with default configuration without getting drowned out by dev binaries
2019-05-31 16:47:48 -06:00
Jeff Wendling
140251882e
fix bug for setting flag only values in process setup (#2089)
* fix bug for setting flag only values in process setup

when the code was changed to directly load values into the config
structs, it was missed that some configuration is only defined
through flags, but can be loaded from config files still.

so, we need to propogate the settings to the flag only values.

* add test for setting propagation

* fix linting error
2019-05-31 21:15:50 +00:00
Maximillian von Briesen
294fddcece
Add audit monkit stats (#2087) 2019-05-31 16:46:25 -04:00
Fadila
5b730e3073
Make maxReverifyCount configurable (#2071)
* make max reverify count configurable
2019-05-31 17:23:00 +02:00
aligeti
4ad5120923
Checker service refactor (v3-1871) (#2082)
*  refactor the checker service

* monkit update
2019-05-31 10:12:49 -04:00
Natalie Villasana
6db9388082 add disqualified column to nodes table (#2086)
* add disqualified column to nodes table, update migrate script and testdata

* fix crazy formatting of postgres.v25.sql
2019-05-30 17:38:23 -04:00
Cameron
590b1a5a1d
Satellite voucher service (#2043)
* set up voucher service skeleton, basic test

* add VetNode db method

* basic test for VetNode

* encode and sign voucher functions

* fill out and sign vouchers

* test pass/fail voucher request

* match EncodeVoucher to other Encode functions
2019-05-30 15:52:33 -04:00
Jennifer Li Johnson
24c8132975
rewrites encode methods for readability (#2084) 2019-05-30 15:01:55 -04:00
Stefan Benten
4b75752d6b
Relocate IP Lookup to ensure it is always set (#2061) 2019-05-30 19:35:04 +02:00
aligeti
934ebf9cbf
Added the irreparable repair functionality (#1955)
* Added the irreparable repair functionality
2019-05-30 11:18:20 -04:00
Jeff Wendling
e74cac52ab
Command line flags features and cleanup (#2068)
* change BindSetup to be an option to Bind
* add process.Bind to allow composite structures
* hack fix for noprefix flags
* used tagged version of structs

Before this PR, some flags were created by calling `cfgstruct.Bind` and having their fields create a flag. Once the flags were parsed, `viper` was used to acquire all the values from them and config files, and the fields in the struct were set through the flag interface.

This doesn't work for slices of things on config structs very well, since it can only set strings, and for a string slice, it turns out that the implementation in `pflag` appends an entry rather than setting it.

This changes three things:

1. Only have a `Bind` call instead of `Bind` and `BindSetup`, and make `BindSetup` an option instead.
2. Add a `process.Bind` call that takes in a `*cobra.Cmd`, binds the struct to the command's flags, and keeps track of that struct in a global map keyed by the command.
3. Use `viper` to get the values and load them into the bound configuration structs instead of using the flags to propagate the changes.

In this way, we can support whatever rich configuration we want in the config yaml files, while still getting command like flags when important.
2019-05-29 17:56:22 +00:00
ethanadams
16e3b77cf5
Enable Scopelint Linter (#2049)
* added scopelint and correcte issues found

* corrected scopelint issue

* made updates based on Ivan's suggestions

Most were around naming conventions
Some were false positives, but I kept them since the test.Run could eventually be changed to run in parallel, which could cause a bug
Others were false positives.  Added // nolint: scopelint
2019-05-29 09:30:16 -04:00
ethanadams
268dc6b7e4
Enable gocritic linter (#2051)
* first round cleanup based on go-critic

* more issues resolved for ifelsechain and unlambda checks

* updated from master and gocritic found a new ifElseChain issue

* disable appendAssign. i reports false positives

* re-enabled go-critic appendAssign and disabled lint check at code line level

* fixed go-critic lint error

* fixed // nolint add gocritic specifically
2019-05-29 09:14:25 -04:00
Michal Niewrzal
f731267e8c Per-project usage limiting (#2036)
What: Changes to support custom usage limit for the project. With this implementation by default project usage limit is taken from configuration flag. If project DB field usage_limit will be set to value larger than 0 it will become custom usage limit and we will be used to verify is limit was exceeded.

Whats changed:

usage_limit (bigint) field added to projects table (with migration)
things related to project usage moved from metainfo endpoint to project usage type
accounting.ProjectAccounting extended with GetProjectUsageLimits() method
Why: We need to have different usage limits per project. https://storjlabs.atlassian.net/browse/V3-1814
2019-05-28 09:36:52 -06:00
Maximillian von Briesen
c07162beef address potential divide by 0` (#2065) 2019-05-28 08:54:30 -06:00
Maximillian von Briesen
5a4ff2c855 add repair monkit stats (#2045)
* add repair monkit stats

* rename values, use meter instead of counter, use success threshold instead of repair threshold

* Counter -> Meter

* add repair segment size

* update names and use ratios for healthy before/after repair

* restart jenkins
2019-05-28 16:10:26 +02:00
Natalie Villasana
aa6ff17b70 add Reverify to auditing (#2041)
Co-authored-by: Maximillian von Briesen <mobyvb@gmail.com>
Co-authored-by: Kaloyan Raev <kaloyan@storj.io>
2019-05-27 14:13:47 +03:00
Michal Niewrzal
4e8b5cb972 Fix flaky TestProjectUsageBandwidth (#2047) 2019-05-25 21:24:12 +02:00
Ivan Fraixedes
5c50aac661 pkg/audit: Readdress ignored Verifier tests (#2048) 2019-05-24 19:57:07 +03:00
Jeff Wendling
1bd52b9f90 server side macaroons (#1945)
What: Adds macaroon support to the server side

Why: So that api keys are now macaroons
2019-05-24 10:51:27 -06:00
Michal Niewrzal
7193b16e92
Java/Android libuplink bindings (#1918) 2019-05-24 10:13:01 +02:00
Natalie Villasana
e862382a6f rename audit.RecordAuditsInfo to audit.Report (#2044)
* rename RecordAuditsInfo to Report

* change comment
2019-05-24 00:32:19 +02:00
Kaloyan Raev
de8070730a Save hash of expected share from missing audited node (#2025) 2019-05-23 16:07:19 -04:00
Maximillian von Briesen
da91d22376 properly check last iteration of checker (#2040) 2019-05-23 18:14:08 +02:00
Natalie Villasana
8b31c4b91f
remove unnecessary containmentDB wrapper (#2027) 2019-05-23 10:37:23 -04:00
Natalie Villasana
ef4849b53b
skip flaky TestProjectUsageBandwidth (#2035) 2019-05-23 10:27:57 -04:00
Maximillian von Briesen
b4f18226db
Send number of files as part of durability stats (#2030) 2019-05-22 18:50:43 -04:00
Maximillian von Briesen
45a2253628 Send durability stats after iterating over all segments (#2028) 2019-05-22 17:17:52 -04:00
Cameron
4058c29ca4
filter duplicate node IPs (#1890)
* add last_ip field to dbx model node, generate dbx

* add last_ip to node proto, generate pb

* migrate

* resolve address in transport.DialNode, update lastIp in cache.UpdateAddress

* use net.SplitHostPort to isolate host address from port

* define DistinctIPs flag

* add test for GetIP

* select last_ip when querying for nodes

* if distinctIPs flag == true, query for nodes with distinct IPs

* some basic tests

* change last_ip to field 14 in proto

* remove comments

* check err

* change distinctIPs to distinctIP

* exclude IPs from newNodes in query for reputable nodes

* add index on last_ip

* only add to excludedIPs if flag is true

* test half new nodes returns distinct IPs

* fix alignment

* add test

* rework ip filter query, add retry logic, add switch for database driver

* add retry to SelectNewNodes

* change discovery intervals so IPs don't get overwritten

* remove TestGetIP

* edit updating node stats in test

* split exclude into nodeIDs and IPs

* separate non-distinct IP query into other function

* trigger checks

* remove else block
2019-05-22 16:06:27 -04:00