JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
2,348 Commits 5 Branches 0 Tags 94 MiB
587be8f206
Commit Graph

1208 Commits

Author SHA1 Message Date
Egon Elbre
5fe95fe8fa kademlia: restrict maximum find near query (#2598) 2019-07-19 18:46:08 +02:00
aligeti
29b576961f
value attribution merge fix and more test cases (#2588) 
* value attribution merge fix and more test cases
 2019-07-19 11:17:34 -04:00
Bill Thorp
a7cc940776
Nodes should not be able to fail the same audit multiple times (#2404) 
update pointer on audit failure
 2019-07-18 14:08:15 -04:00
Jess G
3af9250659
update irreparableDB.GetLimited query to use where instead of offset (#2585) 
* update query to use where instead of offset, update tests

* update cmd/inspector irreparable

* add comment for offset
 2019-07-18 09:21:21 -07:00
Simon Guindon
91f0adef10
Add the ability to set dial and request timeouts from the cmd/uplink CLI to libuplink. (#2439) 
* Added the ability to pass timeout settings from cmd/uplink to libuplink.

* Removed commented out code.

* Updated 2min timeouts for the uplink CLI.

* Removed comment.

* Made transport defaultDialTimeout and defaultRequestTimeout public

* Added comments to describe where these defaults apply.

* Added a new defaults to libuplink and added tests.

* Added a new defaults to libuplink and added tests.
 2019-07-18 11:13:59 -04:00
Andrew Harding
416fa80e85
Link Sharing Service (#2431) 
Link sharing service. See `docs/design/link-sharing-service.md` for the design and `cmd/linksharing/README.md` for operational instructions.
 2019-07-18 06:26:09 -06:00
Kaloyan Raev
0931f0e71b Log fatal errors with respective severity (#2581) 2019-07-17 14:39:10 -04:00
Egon Elbre
f6f65a80d7
storagenode/trust: implement fetching peer identity without kademlia and endpoint (#2584) 2019-07-17 21:14:44 +03:00
Michal Niewrzal
260d9c49a8
Metainfo RPC objects methods (#2534) 2019-07-16 12:39:23 +02:00
Cameron
8b2d46a974
add vouchers to QueryRequest (#2559) 2019-07-15 18:02:22 -04:00
Jeff Wendling
3a34a0df7b repair: fix data race in reliability cache (#2561) 2019-07-15 22:58:39 +03:00
Egon Elbre
002d9748ec
signing: ensure we don't break signatures (#2542) 2019-07-12 16:41:19 -04:00
aligeti
daa3b32ee2
Add Attribution Columns to appropriate tables for OSPP referral link (#2516) 
* adds "partner_id" column to user, project, api_key & bucket_metainfo tables
 2019-07-12 13:59:19 -04:00
Jess G
07b23b7cc3
update tally to rm bucket count (#2539) 
* update tally to rm bucket count

* rm old comment
 2019-07-12 12:19:45 -04:00
Alexander Leitner
64b2769de3
discovery: parallelize refresh (#2535) 
* parallelize discovery refresh

* add paginateQualifiedtest, address pr comments

* Remove duplicate uptime update

* Lower concurrency in Testplanet for discovery
 2019-07-12 10:35:48 -04:00
Jess G
f11bf46a11
Jg/1967 mv bucket metadata uplink (#2505) 
* add bucketstore, add init uplink bucket

* update uplink to use bucket rpc

* fix tests

* wrap metainfo client errors

* add allowedBucket struct, fix tests

* update comment

* add paging

* updates per CR

* add test for pagination

* fix lint

* fix uplink test so its easier tyo understand

* fix gateway pagination bug

* changes per cr

* fix bug w/allowedBuckets, add test to catch
 2019-07-12 08:57:02 -04:00
Ivan Fraixedes
f420b29d35
[V3-1927] Repairer uploads to max threshold instead of success… (#2423) 
* pkg/datarepair: Add test to check num upload pieces
  Add a new test for ensuring the number of pieces that the repair process
  upload when a segment is injured.
* satellite/orders: Don't create "put order limits" over total
  Repair must not create "put order limits" more than the total count.
* pkg/datarepair: Update upload repair pieces test
  Update the test which checks the number of pieces which are uploaded
  during a repair for using the same excess over the success threshold
  value than the implementation.
* satellites/orders: Limit repair put order for not being total
  Limit the number of put orders to be used by repair for only uploading
  pieces to a % excess over the successful threshold.
* pkg/datarepair: Change DataRepair test to pass again
  Make some changes in the DataRepair test to make pass again after the
  repair upload repaired pieces only until a % excess over success
  threshold.
  Also update the steps description of the DataRepair test after it has been
  changed, to match on what's now, besides to leave it more generic for
  avoiding having to update it on minimal future refactorings.
* satellite: Make repair excess optimal threshold configurable
  Add a new configuration parameter to the satellite for being able to
  configure the percentage excess over the optimal threshold, used for
  determining how many pieces should be repaired/uploaded, rather than
  having the value hard coded.
* repairer: Add configurable param to segments/repairer
  Add a new parameters to the segment/repairer to calculate the maximum
  number of excess nodes, based on the optimal threshold, that repaired
  pieces can be uploaded.
  This new parameter has been added for not returning more nodes than the
  number of upload orders for data repair satellite service calculate for
  repairing pieces.
* pkg/storage/ec: Update log message in clien.Repair
* satellite: Update configuration lock file
 2019-07-12 00:44:47 +02:00
Michal Niewrzal
8a9c63809b tests for PiecePublicKey/PiecePrivateKey (#2526) 2019-07-11 17:11:04 -04:00
Egon Elbre
d52f764e54
protocol: implement new piece signing and verification (#2525) 2019-07-11 16:51:40 -04:00
Maximillian von Briesen
8b507f3d73 Address concerns with storagenode Retain endpoint (#2527) 2019-07-11 16:04:21 -04:00
Michal Niewrzal
268c629ba8
Replace base64 encoding for path segments (#2345) 2019-07-11 13:26:07 -04:00
Kaloyan Raev
0f36b160fc Higher dial timeout for TestDownloadSharesDownloadTimeout (#2500) 2019-07-10 17:45:09 -04:00
Maximillian von Briesen
de85d17069
Add checker metrics (#2487) 
checker_segment_total_count - Number of total segments in pointer during checker iteration
checker_segment_healthy_count - Number of healthy segments in pointer during checker iterationn
time_since_checker_queue - Seconds elapsed between checker queue and beginning repair
time_for_repair - Seconds elapsed between beginning repair and ending repair/dequeueing
 2019-07-10 17:27:46 -04:00
Jeff Wendling
10547cc1ea
segments: send in the object path to the initial CreateSegment… (#2518) 
otherwise, api key restictions will fail because we look like we're
asking to put to the bucket metadata path.
 2019-07-10 11:33:55 -04:00
Cameron
c29f033e7d
Move kademlia dialer into separate package (#2466) 
* move kademlia.Dialer into kademliaclient package
 2019-07-10 10:36:37 -04:00
JT Olio
a79c7d77f3 overlay cache: slight modification of node-is-online rules (#2490) 2019-07-09 22:36:09 -04:00
Alexander Leitner
1c5db71faf
Change protobuf expirations to use time.Time (#2509) 
* Change protobuf expirations to use time.Time instead of timestamp.Timestamp
 2019-07-09 17:54:00 -04:00
Jeff Wendling
671648012a Update password based key derivation design doc (#2367) 2019-07-09 00:07:03 -04:00
Ivan Fraixedes
50d601ab07 pkg/identity: Use identity error class (#2488) 2019-07-08 22:52:52 -04:00
JT Olio
887b784f54 discovery: use fetch info to ping (#2491) 2019-07-08 22:10:17 -04:00
Jess G
f9696d6c5e
satellite/metainfo: add buckets RPC and database (#2460) 
* add db interface and methods, add sa metainfo endpoints and svc

* add bucket metainfo svc funcs

* add sadb bucekts

* bucket list gets all buckets

* filter buckets list on macaroon restrictions

* update pb cipher suite to be enum

* add conversion funcs

* updates per comments

* bucket settings should say default

* add direction to list buckets, add tests

* fix test bucket names

* lint err

* only support forward direction

* add comments

* minor refactoring

* make sure list up to limit

* update test

* update protolock file

* fix lint

* change per PR
 2019-07-08 15:32:18 -07:00
Alexander Leitner
3587e1a579 Change pointerdb pointer to use time.Time for Creation date (#2483) 2019-07-09 00:16:50 +02:00
Egon Elbre
674742d1a7
satellite/datarepair: use reliability cache (#1976) 2019-07-09 01:04:35 +03:00
Ivan Fraixedes
a786e4249c pkg: Align errs Class messages (#2485) 
Align 2 errs Class messages with the rest of them.
 2019-07-08 17:37:12 -04:00
Alexander Leitner
19ab9852f2
Update node.proto to use time.Time instead of timestamp (#2482) 2019-07-08 14:24:42 -04:00
Alexander Leitner
dcf8e2936b
Update vouchers to use time.Time instead of timestamp (#2478) 
* Update vouchers to use time.Time instead of timestamp
 2019-07-08 13:07:30 -04:00
Simon Guindon
8d15d774b6
Removing trace of encodedPiece.Read() because it's fast. (#2477) 
* Removing trace of encodedPiece.Read() because it's fast.

* Added comment about why tracing was removed.
 2019-07-08 11:16:13 -04:00
Stefan Benten
ccef5eee46
Add proper Version Handling to Identity, Gateway and Uplink Binary (#2471) 2019-07-08 10:45:20 -04:00
Yaroslav Vorobiov
524eb24c83 storagenode/nodestats: combine stats into single RPC call (#2455) 
* change satellite nodestats endpoint
 2019-07-08 17:33:43 +03:00
Alexander Leitner
88732188cb
Update inspector timestamp to time.time (#2464) 
* Update inspector timestamp
 2019-07-08 10:06:12 -04:00
Michal Niewrzal
094e1b8b90
Add 'Old' suffix to some metainfo methods/messages (#2462) 2019-07-08 15:33:15 +02:00
Yaroslav Vorobiov
7aca0eb284 storagenode/dashboard: show console address (#2456) 2019-07-06 15:40:58 +02:00
Ivan Fraixedes
ea9a8b37bc pkg/datarepair/checker: Fix typo in doc comment (#2461) 2019-07-05 22:20:58 +02:00
Kaloyan Raev
f9ed0dc1a8
Improve stability of TestDownloadSharesDownloadTimeout (#2210) 2019-07-05 19:04:15 +03:00
Ivan Fraixedes
e0a8937c6c
pkg/storage/ec: Make minor improvements in client.Repair (#2454) 
* Fix some log message to actually report the number of pieces needed to
  repaired for reaching the successful/optimal threshold.
* Remove some unneeded `nil` check conditional.
 2019-07-05 12:39:10 +02:00
Jeff Wendling
8b07df37f5 refactor some store encryption stuff (#2434) 2019-07-05 11:36:35 +03:00
Yaroslav Vorobiov
42c125d697
Satellite nodestats endpoint add audit check (#2426) 2019-07-04 13:41:40 +03:00
Michal Niewrzal
5176e4f096 Rename BucketItem into Bucket (#2442) 
* Rename BucketItem into Bucket
 2019-07-03 15:44:02 -04:00
aligeti
ae8b9698f9
Rename/remove EncryptionScheme -> EncryptionParame… (#2363) 
* rename/remove EncryptionScheme -> EncryptionParameters
 2019-07-03 14:07:44 -04:00
Cameron
d499d162f4
implement storj.NodeURL in trusted satellites (#2388) 
* implement storj.NodeURL in trusted satellites
 2019-07-03 13:29:18 -04:00
Michal Niewrzal
2ee5bada2c
Add pieceNum to PieceID derivation function (#2193) 2019-07-03 18:53:15 +02:00
Michal Niewrzal
61dfa61e3a
Add timestamp and piece size to piece hash (#2198) 2019-07-03 18:14:37 +02:00
Maximillian von Briesen
2f6e193c32
Remove long tail timer in ecclient (#2433) 
* remove long tail timer in ecclient
 2019-07-03 11:00:24 -04:00
Michal Niewrzal
46b5c30f35
Fix test-sim debug-addr error (#2420) 2019-07-03 16:10:51 +02:00
Jeff Wendling
42e124cd08
monitor optimal wait fraction (#2435) 
* monitor optimal wait fraction

Change-Id: I1c76da5e8031237cf78ce5a0774732dd5e558ea1

* monitor other times about the upload

Change-Id: Iae81c80fb1446fbf4b3dd04fc6b238f2ede96545
 2019-07-02 20:49:35 +00:00
Egon Elbre
9e26149a2d pkg/cfgstruct: add pflag support to bind (#2425) 2019-07-02 13:53:01 -04:00
Alexander Leitner
6d55bbdb57
OrderLimit creation date time limit (#2412) 
* Limit by order creation
 2019-07-02 12:06:12 -04:00
Andrew Harding
0de4cf31cf
libuplink scope type (#2382) 2019-07-02 09:45:23 -06:00
Kaloyan Raev
d32c907440
overlay.UpdateStats removes node from containment mode (#2419) 2019-07-02 18:16:25 +03:00
Yaroslav Vorobiov
f4dfb8fb95
Add daily node space usage to satellite nodestats endpoint (#2361) 2019-07-02 13:42:09 +03:00
Maximillian von Briesen
52e5a4eee3 pass logger into repairer and ecclient (#2365) 2019-07-02 13:08:02 +03:00
Michal Niewrzal
3337087a94
Add storj.StreamID (#2407) 2019-07-02 11:16:06 +02:00
littleskunk
a2362f92dc
Rollback uptime disqualification (#2417) 2019-07-02 10:39:36 +02:00
Jennifer Li Johnson
a7aa854df8
Routing table antechamber should expire old nodes that failed connection (#2387) 2019-07-01 23:20:12 -04:00
littleskunk
9e62423f47
reduce vetting requirement (#2416) 2019-07-02 01:02:23 +02:00
Jess G
ea55ae63e3
add bucket metainfo rpc (#2383) 
* add bucket metainfo rpc

* updates per design doc

* rm bucketmeta rpc

* add changes per comments

* update Bucket name since same name already exists

* add placeholder methods to fulfill pb.MetainfoServer interface

* update proto lock

* add bucketListItem message

* why failing lint?

* add back ciphersuite but with correct types

* fix proto errs

* update time type

* add proto lock

* rm unneeded import

* update protolock package
 2019-07-01 15:17:30 -07:00
Jennifer Li Johnson
699ccea19f
Creates Routing Table Antechamber (#2318) 2019-07-01 17:20:19 -04:00
littleskunk
fb66867856 fix repair queue (#2405) 2019-07-01 19:00:53 +02:00
Natalie Villasana
4ba212107b
add back testrand (#2408) 2019-07-01 12:39:04 -04:00
Ivan Fraixedes
b6ae6a4e2c
pkg/datarepair/checker: Fix doc comments (#2401) 2019-07-01 18:13:18 +02:00
Egon Elbre
385c046723
pkg/pb: rename Order2 to Order, OrderLimit2 to OrderLimit (#2406) 2019-07-01 18:54:11 +03:00
Natalie Villasana
3ffb42483f
account for disqualified nodes in data repair tests (#2315) 2019-07-01 11:34:42 -04:00
Natalie Villasana
3f643551e7 remove flakiness in TestDataRepair and TestSegmentStoreRepair (#2335) 
* stop audit loop in repair tests to prevent possible timeout
 2019-07-01 11:15:45 -04:00
Kaloyan Raev
4f0e437965
Audit tests cleanup (#2402) 2019-07-01 17:02:00 +03:00
Ivan Fraixedes
fd2e708587 pkg/storage/segments: Abort repaired test unmet requirements (#2403) 
Abort the repairer test when asserting results which are required later
on by the test for being able to continue.
 2019-07-01 08:49:40 -04:00
Maximillian von Briesen
b750bc2d0f
Keep some unhealthy pieces in the event of a partial repair (#2380) 
* if repair is partial, keep saving "unhealthy" pieces that are not duplicates
 2019-06-28 15:48:51 -04:00
Egon Elbre
7e0c5d51d0
pkg/storj: add NodeURL type for verified url-s (#2379) 2019-06-28 21:43:39 +03:00
Egon Elbre
827fb92b47
satellite/console: nicer error handling in tests (#2378) 2019-06-28 16:04:50 +03:00
Jeff Wendling
1a65e42d33 rename EncryptionCtx back to EncryptionAccess (#2368) 
* rename EncryptionCtx back to EncryptionAccess

Change-Id: I5e58915a59979ad6f2e83d36e191b2bbf3ba2ba2

* missed some stuff

Change-Id: Ib65fb186d7f854c0406e5fd7d11498e8941da59e

* oops protolock

Change-Id: I85a5ab4bafb9dd6a804d3dcd17a70811702f07e4

* retrigger

Change-Id: I2d0e3f02b4dbae0299b090e9617662b4437980b0
 2019-06-27 23:18:24 -06:00
aligeti
811168e2c4
Uplink bucket attribution extension (#2353) 
Attribution info added to the bucket info
 2019-06-27 19:25:36 -04:00
Egon Elbre
e83ebd7cde
jenkins: avoid using goimports and distribute load better (#2359) 2019-06-27 21:52:50 +03:00
Jeff Wendling
efcdaa43a3
lib/uplink: encryption context (#2349) 
* lib/uplink: encryption context

Change-Id: I5c23dca3286a46b713b30c4997e9ae6e630b2280

* lib/uplink: bucket operation examples

Change-Id: Ia0f6e69f365dcff0cf11c731f51b30842bce053b

* lib/uplink: encryption key sharing test cases

Change-Id: I3a172d565f33f4e591402cdcb9460664a7cc7fbe

* fix encrypted path prefix restriction issue

Change-Id: I8f3921f9d52aaf4b84039de608b8cbbc88769554

* implement panics in libuplink encryption code

todo on cipher suite selection as well as an api concern

Change-Id: Ifa39eb3cc4b3443f7d96f9304df9b2ac4ec4085d

* implement GetProjectInfo api call to get salt

Change-Id: Ic5f6b3be9ea35df48c1aa214ab5d355fb328e2cf

* some fixes and accessors for encryption store

Change-Id: I3bb61f6712a037900e2a96e72ad4029ec1d3f718

* general fixes to builds/tests/etc

Change-Id: I9930fa96acb3b221d9a001f8e274af5729cc8a47

* java bindings changes

Change-Id: Ia2bd4c9c69739c8d3154d79616cff1f36fb403b6

* get libuplink examples passing

Change-Id: I828f09a144160e0a5dd932324f78491ae2ec8a07

* fix proto.lock file

Change-Id: I2fbbf4d0976a7d0473c2645e6dcb21aaa3be7651

* fix proto.lock again

Change-Id: I92702cf49e1a340eef6379c2be4f7c4a268112a9

* fix golint issues

Change-Id: I631ff9f43307a58e3b25a58cbb4a4cc2495f5eb6

* more linting fixes

Change-Id: I51f8f30b367b5bca14c94b15417b9a4c9e7aa0ce

* bug fixed by structs bump

Change-Id: Ibb03c691fce7606c35c08721b3ef0781ab48a38a

* retrigger

Change-Id: Ieee0470b6a2d07168a1578552e8e7f271ae93a13

* retrigger

Change-Id: I753d63853171e6a436c104ce176048892eb974c5

* semantic merge conflict

Change-Id: I9419448496de90340569047a6a16a1b858a7978a

* update total to match prod defaults

Change-Id: I693d55c1ebb28b5803ee1d26e9e198decf82308b

* retrigger

Change-Id: I28b74d5d6202f61aa3866fe407d423f6a0a14b9e

* retrigger

Change-Id: I6fd054885c715f602e2cef623fd464c42e88742c

* retrigger

Change-Id: I6a01bae88c72406d4ed5a8f13bf8a2b3c650bd2d
 2019-06-27 17:36:51 +00:00
Egon Elbre
27c92ffc10 jenkins: Stop docker container after 18m (#2358) 
* remove unnecessary argument

* remove kademlia lookup short circuiting

* fix ident

* add timeout to docker container

* add comments to dockerfile
 2019-06-27 10:09:37 -06:00
Egon Elbre
2128b460b4
cmd/uplink/cmd: don't create benchmark data on init (#2351) 2019-06-26 22:08:14 +03:00
Egon Elbre
615bfca135 Fix TestGetSignee flakiness (#2350) 
* add IsCanceled

* fixes to error handling

* fix imports

* retrigger jenkins
 2019-06-26 09:30:37 -06:00
JT Olio
fbe9696e92 pkg/kademlia: clean up peer discovery (#2252) 2019-06-26 16:16:46 +03:00
JT Olio
3925e84580 pkg/eestream: plumb contexts through (#2187) 2019-06-26 16:05:58 +03:00
Egon Elbre
7b66e0cd7c Use dial to clarify that it's internally closing the connection. (#2347) 2019-06-26 15:14:48 +03:00
Egon Elbre
b6ad3e9c9f
internal/testrand: new package for random data (#2282) 2019-06-26 13:38:51 +03:00
Philip Hutchins
11016b5067 Moving verbose info logging to debug (#2346) 2019-06-26 12:14:25 +03:00
Kaloyan Raev
ae36979804 Take advantage of Is and IsFunc from zeebo/errs (#2310) 2019-06-26 09:38:07 +02:00
Egon Elbre
caa2fcf62b satellite/orders: don't panic (#2331) 2019-06-26 09:26:33 +02:00
Egon Elbre
c7679b9b30
Fix some leaks and add notes about close handling (#2334) 2019-06-25 23:00:51 +03:00
Stefan Benten
57ef352b3c
Update Wizard to allow more easily addition of satellites and move package to cmd folder (#2340) 2019-06-25 20:25:31 +02:00
Stefan Benten
2cc01c7899 Rename Satellites in the Wizard (#2339) 
* Rename Satellites

* Fixing Typo and changing names for gateway

* Unify Setup wizard across gateway and uplink
 2019-06-25 11:29:52 -06:00
Fadila
20de18feec [v3-1952 test 6] Disqualification is a one way ticket. Storage Node can't get around it. (#2328) 2019-06-25 13:10:22 -04:00
Egon Elbre
c28f800098
Skip TestDataRepair and TestUplinksParallel, because they are flaky (#2337) 2019-06-25 19:30:39 +03:00
Jess G
e5c48fab74
fix ordersDB methods to take correct args (#2314) 
* fix orderdDB methods to take correct args

* update tally to save projectID in correct format

* update var names in splitBucket test

* changes per CR comments
 2019-06-25 08:58:42 -07:00
Egon Elbre
414648d660
Fix some metainfo.Client leaks (#2327) 2019-06-25 18:36:23 +03:00
Maximillian von Briesen
35f2ab5ded
Enable node selection tests (#2316) 2019-06-25 11:26:47 -04:00