Kaloyan Raev
0931f0e71b
Log fatal errors with respective severity ( #2581 )
2019-07-17 14:39:10 -04:00
Egon Elbre
f6f65a80d7
storagenode/trust: implement fetching peer identity without kademlia and endpoint ( #2584 )
2019-07-17 21:14:44 +03:00
Michal Niewrzal
260d9c49a8
Metainfo RPC objects methods ( #2534 )
2019-07-16 12:39:23 +02:00
Cameron
8b2d46a974
add vouchers to QueryRequest ( #2559 )
2019-07-15 18:02:22 -04:00
Jeff Wendling
3a34a0df7b
repair: fix data race in reliability cache ( #2561 )
2019-07-15 22:58:39 +03:00
Egon Elbre
002d9748ec
signing: ensure we don't break signatures ( #2542 )
2019-07-12 16:41:19 -04:00
aligeti
daa3b32ee2
Add Attribution Columns to appropriate tables for OSPP referral link ( #2516 )
...
* adds "partner_id" column to user, project, api_key & bucket_metainfo tables
2019-07-12 13:59:19 -04:00
Jess G
07b23b7cc3
update tally to rm bucket count ( #2539 )
...
* update tally to rm bucket count
* rm old comment
2019-07-12 12:19:45 -04:00
Alexander Leitner
64b2769de3
discovery: parallelize refresh ( #2535 )
...
* parallelize discovery refresh
* add paginateQualifiedtest, address pr comments
* Remove duplicate uptime update
* Lower concurrency in Testplanet for discovery
2019-07-12 10:35:48 -04:00
Jess G
f11bf46a11
Jg/1967 mv bucket metadata uplink ( #2505 )
...
* add bucketstore, add init uplink bucket
* update uplink to use bucket rpc
* fix tests
* wrap metainfo client errors
* add allowedBucket struct, fix tests
* update comment
* add paging
* updates per CR
* add test for pagination
* fix lint
* fix uplink test so its easier tyo understand
* fix gateway pagination bug
* changes per cr
* fix bug w/allowedBuckets, add test to catch
2019-07-12 08:57:02 -04:00
Ivan Fraixedes
f420b29d35
[V3-1927] Repairer uploads to max threshold instead of success… ( #2423 )
...
* pkg/datarepair: Add test to check num upload pieces
Add a new test for ensuring the number of pieces that the repair process
upload when a segment is injured.
* satellite/orders: Don't create "put order limits" over total
Repair must not create "put order limits" more than the total count.
* pkg/datarepair: Update upload repair pieces test
Update the test which checks the number of pieces which are uploaded
during a repair for using the same excess over the success threshold
value than the implementation.
* satellites/orders: Limit repair put order for not being total
Limit the number of put orders to be used by repair for only uploading
pieces to a % excess over the successful threshold.
* pkg/datarepair: Change DataRepair test to pass again
Make some changes in the DataRepair test to make pass again after the
repair upload repaired pieces only until a % excess over success
threshold.
Also update the steps description of the DataRepair test after it has been
changed, to match on what's now, besides to leave it more generic for
avoiding having to update it on minimal future refactorings.
* satellite: Make repair excess optimal threshold configurable
Add a new configuration parameter to the satellite for being able to
configure the percentage excess over the optimal threshold, used for
determining how many pieces should be repaired/uploaded, rather than
having the value hard coded.
* repairer: Add configurable param to segments/repairer
Add a new parameters to the segment/repairer to calculate the maximum
number of excess nodes, based on the optimal threshold, that repaired
pieces can be uploaded.
This new parameter has been added for not returning more nodes than the
number of upload orders for data repair satellite service calculate for
repairing pieces.
* pkg/storage/ec: Update log message in clien.Repair
* satellite: Update configuration lock file
2019-07-12 00:44:47 +02:00
Michal Niewrzal
8a9c63809b
tests for PiecePublicKey/PiecePrivateKey ( #2526 )
2019-07-11 17:11:04 -04:00
Egon Elbre
d52f764e54
protocol: implement new piece signing and verification ( #2525 )
2019-07-11 16:51:40 -04:00
Maximillian von Briesen
8b507f3d73
Address concerns with storagenode Retain endpoint ( #2527 )
2019-07-11 16:04:21 -04:00
Michal Niewrzal
268c629ba8
Replace base64 encoding for path segments ( #2345 )
2019-07-11 13:26:07 -04:00
Kaloyan Raev
0f36b160fc
Higher dial timeout for TestDownloadSharesDownloadTimeout ( #2500 )
2019-07-10 17:45:09 -04:00
Maximillian von Briesen
de85d17069
Add checker metrics ( #2487 )
...
checker_segment_total_count - Number of total segments in pointer during checker iteration
checker_segment_healthy_count - Number of healthy segments in pointer during checker iterationn
time_since_checker_queue - Seconds elapsed between checker queue and beginning repair
time_for_repair - Seconds elapsed between beginning repair and ending repair/dequeueing
2019-07-10 17:27:46 -04:00
Jeff Wendling
10547cc1ea
segments: send in the object path to the initial CreateSegment… ( #2518 )
...
otherwise, api key restictions will fail because we look like we're
asking to put to the bucket metadata path.
2019-07-10 11:33:55 -04:00
Cameron
c29f033e7d
Move kademlia dialer into separate package ( #2466 )
...
* move kademlia.Dialer into kademliaclient package
2019-07-10 10:36:37 -04:00
JT Olio
a79c7d77f3
overlay cache: slight modification of node-is-online rules ( #2490 )
2019-07-09 22:36:09 -04:00
Alexander Leitner
1c5db71faf
Change protobuf expirations to use time.Time ( #2509 )
...
* Change protobuf expirations to use time.Time instead of timestamp.Timestamp
2019-07-09 17:54:00 -04:00
Jeff Wendling
671648012a
Update password based key derivation design doc ( #2367 )
2019-07-09 00:07:03 -04:00
Ivan Fraixedes
50d601ab07
pkg/identity: Use identity error class ( #2488 )
2019-07-08 22:52:52 -04:00
JT Olio
887b784f54
discovery: use fetch info to ping ( #2491 )
2019-07-08 22:10:17 -04:00
Jess G
f9696d6c5e
satellite/metainfo: add buckets RPC and database ( #2460 )
...
* add db interface and methods, add sa metainfo endpoints and svc
* add bucket metainfo svc funcs
* add sadb bucekts
* bucket list gets all buckets
* filter buckets list on macaroon restrictions
* update pb cipher suite to be enum
* add conversion funcs
* updates per comments
* bucket settings should say default
* add direction to list buckets, add tests
* fix test bucket names
* lint err
* only support forward direction
* add comments
* minor refactoring
* make sure list up to limit
* update test
* update protolock file
* fix lint
* change per PR
2019-07-08 15:32:18 -07:00
Alexander Leitner
3587e1a579
Change pointerdb pointer to use time.Time for Creation date ( #2483 )
2019-07-09 00:16:50 +02:00
Egon Elbre
674742d1a7
satellite/datarepair: use reliability cache ( #1976 )
2019-07-09 01:04:35 +03:00
Ivan Fraixedes
a786e4249c
pkg: Align errs Class messages ( #2485 )
...
Align 2 errs Class messages with the rest of them.
2019-07-08 17:37:12 -04:00
Alexander Leitner
19ab9852f2
Update node.proto to use time.Time instead of timestamp ( #2482 )
2019-07-08 14:24:42 -04:00
Alexander Leitner
dcf8e2936b
Update vouchers to use time.Time instead of timestamp ( #2478 )
...
* Update vouchers to use time.Time instead of timestamp
2019-07-08 13:07:30 -04:00
Simon Guindon
8d15d774b6
Removing trace of encodedPiece.Read() because it's fast. ( #2477 )
...
* Removing trace of encodedPiece.Read() because it's fast.
* Added comment about why tracing was removed.
2019-07-08 11:16:13 -04:00
Stefan Benten
ccef5eee46
Add proper Version Handling to Identity, Gateway and Uplink Binary ( #2471 )
2019-07-08 10:45:20 -04:00
Yaroslav Vorobiov
524eb24c83
storagenode/nodestats: combine stats into single RPC call ( #2455 )
...
* change satellite nodestats endpoint
2019-07-08 17:33:43 +03:00
Alexander Leitner
88732188cb
Update inspector timestamp to time.time ( #2464 )
...
* Update inspector timestamp
2019-07-08 10:06:12 -04:00
Michal Niewrzal
094e1b8b90
Add 'Old' suffix to some metainfo methods/messages ( #2462 )
2019-07-08 15:33:15 +02:00
Yaroslav Vorobiov
7aca0eb284
storagenode/dashboard: show console address ( #2456 )
2019-07-06 15:40:58 +02:00
Ivan Fraixedes
ea9a8b37bc
pkg/datarepair/checker: Fix typo in doc comment ( #2461 )
2019-07-05 22:20:58 +02:00
Kaloyan Raev
f9ed0dc1a8
Improve stability of TestDownloadSharesDownloadTimeout ( #2210 )
2019-07-05 19:04:15 +03:00
Ivan Fraixedes
e0a8937c6c
pkg/storage/ec: Make minor improvements in client.Repair ( #2454 )
...
* Fix some log message to actually report the number of pieces needed to
repaired for reaching the successful/optimal threshold.
* Remove some unneeded `nil` check conditional.
2019-07-05 12:39:10 +02:00
Jeff Wendling
8b07df37f5
refactor some store encryption stuff ( #2434 )
2019-07-05 11:36:35 +03:00
Yaroslav Vorobiov
42c125d697
Satellite nodestats endpoint add audit check ( #2426 )
2019-07-04 13:41:40 +03:00
Michal Niewrzal
5176e4f096
Rename BucketItem into Bucket ( #2442 )
...
* Rename BucketItem into Bucket
2019-07-03 15:44:02 -04:00
aligeti
ae8b9698f9
Rename/remove EncryptionScheme -> EncryptionParame… ( #2363 )
...
* rename/remove EncryptionScheme -> EncryptionParameters
2019-07-03 14:07:44 -04:00
Cameron
d499d162f4
implement storj.NodeURL in trusted satellites ( #2388 )
...
* implement storj.NodeURL in trusted satellites
2019-07-03 13:29:18 -04:00
Michal Niewrzal
2ee5bada2c
Add pieceNum to PieceID derivation function ( #2193 )
2019-07-03 18:53:15 +02:00
Michal Niewrzal
61dfa61e3a
Add timestamp and piece size to piece hash ( #2198 )
2019-07-03 18:14:37 +02:00
Maximillian von Briesen
2f6e193c32
Remove long tail timer in ecclient ( #2433 )
...
* remove long tail timer in ecclient
2019-07-03 11:00:24 -04:00
Michal Niewrzal
46b5c30f35
Fix test-sim debug-addr error ( #2420 )
2019-07-03 16:10:51 +02:00
Jeff Wendling
42e124cd08
monitor optimal wait fraction ( #2435 )
...
* monitor optimal wait fraction
Change-Id: I1c76da5e8031237cf78ce5a0774732dd5e558ea1
* monitor other times about the upload
Change-Id: Iae81c80fb1446fbf4b3dd04fc6b238f2ede96545
2019-07-02 20:49:35 +00:00
Egon Elbre
9e26149a2d
pkg/cfgstruct: add pflag support to bind ( #2425 )
2019-07-02 13:53:01 -04:00
Alexander Leitner
6d55bbdb57
OrderLimit creation date time limit ( #2412 )
...
* Limit by order creation
2019-07-02 12:06:12 -04:00
Andrew Harding
0de4cf31cf
libuplink scope type ( #2382 )
2019-07-02 09:45:23 -06:00
Kaloyan Raev
d32c907440
overlay.UpdateStats removes node from containment mode ( #2419 )
2019-07-02 18:16:25 +03:00
Yaroslav Vorobiov
f4dfb8fb95
Add daily node space usage to satellite nodestats endpoint ( #2361 )
2019-07-02 13:42:09 +03:00
Maximillian von Briesen
52e5a4eee3
pass logger into repairer and ecclient ( #2365 )
2019-07-02 13:08:02 +03:00
Michal Niewrzal
3337087a94
Add storj.StreamID ( #2407 )
2019-07-02 11:16:06 +02:00
littleskunk
a2362f92dc
Rollback uptime disqualification ( #2417 )
2019-07-02 10:39:36 +02:00
Jennifer Li Johnson
a7aa854df8
Routing table antechamber should expire old nodes that failed connection ( #2387 )
2019-07-01 23:20:12 -04:00
littleskunk
9e62423f47
reduce vetting requirement ( #2416 )
2019-07-02 01:02:23 +02:00
Jess G
ea55ae63e3
add bucket metainfo rpc ( #2383 )
...
* add bucket metainfo rpc
* updates per design doc
* rm bucketmeta rpc
* add changes per comments
* update Bucket name since same name already exists
* add placeholder methods to fulfill pb.MetainfoServer interface
* update proto lock
* add bucketListItem message
* why failing lint?
* add back ciphersuite but with correct types
* fix proto errs
* update time type
* add proto lock
* rm unneeded import
* update protolock package
2019-07-01 15:17:30 -07:00
Jennifer Li Johnson
699ccea19f
Creates Routing Table Antechamber ( #2318 )
2019-07-01 17:20:19 -04:00
littleskunk
fb66867856
fix repair queue ( #2405 )
2019-07-01 19:00:53 +02:00
Natalie Villasana
4ba212107b
add back testrand ( #2408 )
2019-07-01 12:39:04 -04:00
Ivan Fraixedes
b6ae6a4e2c
pkg/datarepair/checker: Fix doc comments ( #2401 )
2019-07-01 18:13:18 +02:00
Egon Elbre
385c046723
pkg/pb: rename Order2 to Order, OrderLimit2 to OrderLimit ( #2406 )
2019-07-01 18:54:11 +03:00
Natalie Villasana
3ffb42483f
account for disqualified nodes in data repair tests ( #2315 )
2019-07-01 11:34:42 -04:00
Natalie Villasana
3f643551e7
remove flakiness in TestDataRepair and TestSegmentStoreRepair ( #2335 )
...
* stop audit loop in repair tests to prevent possible timeout
2019-07-01 11:15:45 -04:00
Kaloyan Raev
4f0e437965
Audit tests cleanup ( #2402 )
2019-07-01 17:02:00 +03:00
Ivan Fraixedes
fd2e708587
pkg/storage/segments: Abort repaired test unmet requirements ( #2403 )
...
Abort the repairer test when asserting results which are required later
on by the test for being able to continue.
2019-07-01 08:49:40 -04:00
Maximillian von Briesen
b750bc2d0f
Keep some unhealthy pieces in the event of a partial repair ( #2380 )
...
* if repair is partial, keep saving "unhealthy" pieces that are not duplicates
2019-06-28 15:48:51 -04:00
Egon Elbre
7e0c5d51d0
pkg/storj: add NodeURL type for verified url-s ( #2379 )
2019-06-28 21:43:39 +03:00
Egon Elbre
827fb92b47
satellite/console: nicer error handling in tests ( #2378 )
2019-06-28 16:04:50 +03:00
Jeff Wendling
1a65e42d33
rename EncryptionCtx back to EncryptionAccess ( #2368 )
...
* rename EncryptionCtx back to EncryptionAccess
Change-Id: I5e58915a59979ad6f2e83d36e191b2bbf3ba2ba2
* missed some stuff
Change-Id: Ib65fb186d7f854c0406e5fd7d11498e8941da59e
* oops protolock
Change-Id: I85a5ab4bafb9dd6a804d3dcd17a70811702f07e4
* retrigger
Change-Id: I2d0e3f02b4dbae0299b090e9617662b4437980b0
2019-06-27 23:18:24 -06:00
aligeti
811168e2c4
Uplink bucket attribution extension ( #2353 )
...
Attribution info added to the bucket info
2019-06-27 19:25:36 -04:00
Egon Elbre
e83ebd7cde
jenkins: avoid using goimports and distribute load better ( #2359 )
2019-06-27 21:52:50 +03:00
Jeff Wendling
efcdaa43a3
lib/uplink: encryption context ( #2349 )
...
* lib/uplink: encryption context
Change-Id: I5c23dca3286a46b713b30c4997e9ae6e630b2280
* lib/uplink: bucket operation examples
Change-Id: Ia0f6e69f365dcff0cf11c731f51b30842bce053b
* lib/uplink: encryption key sharing test cases
Change-Id: I3a172d565f33f4e591402cdcb9460664a7cc7fbe
* fix encrypted path prefix restriction issue
Change-Id: I8f3921f9d52aaf4b84039de608b8cbbc88769554
* implement panics in libuplink encryption code
todo on cipher suite selection as well as an api concern
Change-Id: Ifa39eb3cc4b3443f7d96f9304df9b2ac4ec4085d
* implement GetProjectInfo api call to get salt
Change-Id: Ic5f6b3be9ea35df48c1aa214ab5d355fb328e2cf
* some fixes and accessors for encryption store
Change-Id: I3bb61f6712a037900e2a96e72ad4029ec1d3f718
* general fixes to builds/tests/etc
Change-Id: I9930fa96acb3b221d9a001f8e274af5729cc8a47
* java bindings changes
Change-Id: Ia2bd4c9c69739c8d3154d79616cff1f36fb403b6
* get libuplink examples passing
Change-Id: I828f09a144160e0a5dd932324f78491ae2ec8a07
* fix proto.lock file
Change-Id: I2fbbf4d0976a7d0473c2645e6dcb21aaa3be7651
* fix proto.lock again
Change-Id: I92702cf49e1a340eef6379c2be4f7c4a268112a9
* fix golint issues
Change-Id: I631ff9f43307a58e3b25a58cbb4a4cc2495f5eb6
* more linting fixes
Change-Id: I51f8f30b367b5bca14c94b15417b9a4c9e7aa0ce
* bug fixed by structs bump
Change-Id: Ibb03c691fce7606c35c08721b3ef0781ab48a38a
* retrigger
Change-Id: Ieee0470b6a2d07168a1578552e8e7f271ae93a13
* retrigger
Change-Id: I753d63853171e6a436c104ce176048892eb974c5
* semantic merge conflict
Change-Id: I9419448496de90340569047a6a16a1b858a7978a
* update total to match prod defaults
Change-Id: I693d55c1ebb28b5803ee1d26e9e198decf82308b
* retrigger
Change-Id: I28b74d5d6202f61aa3866fe407d423f6a0a14b9e
* retrigger
Change-Id: I6fd054885c715f602e2cef623fd464c42e88742c
* retrigger
Change-Id: I6a01bae88c72406d4ed5a8f13bf8a2b3c650bd2d
2019-06-27 17:36:51 +00:00
Egon Elbre
27c92ffc10
jenkins: Stop docker container after 18m ( #2358 )
...
* remove unnecessary argument
* remove kademlia lookup short circuiting
* fix ident
* add timeout to docker container
* add comments to dockerfile
2019-06-27 10:09:37 -06:00
Egon Elbre
2128b460b4
cmd/uplink/cmd: don't create benchmark data on init ( #2351 )
2019-06-26 22:08:14 +03:00
Egon Elbre
615bfca135
Fix TestGetSignee flakiness ( #2350 )
...
* add IsCanceled
* fixes to error handling
* fix imports
* retrigger jenkins
2019-06-26 09:30:37 -06:00
JT Olio
fbe9696e92
pkg/kademlia: clean up peer discovery ( #2252 )
2019-06-26 16:16:46 +03:00
JT Olio
3925e84580
pkg/eestream: plumb contexts through ( #2187 )
2019-06-26 16:05:58 +03:00
Egon Elbre
7b66e0cd7c
Use dial to clarify that it's internally closing the connection. ( #2347 )
2019-06-26 15:14:48 +03:00
Egon Elbre
b6ad3e9c9f
internal/testrand: new package for random data ( #2282 )
2019-06-26 13:38:51 +03:00
Philip Hutchins
11016b5067
Moving verbose info logging to debug ( #2346 )
2019-06-26 12:14:25 +03:00
Kaloyan Raev
ae36979804
Take advantage of Is and IsFunc from zeebo/errs ( #2310 )
2019-06-26 09:38:07 +02:00
Egon Elbre
caa2fcf62b
satellite/orders: don't panic ( #2331 )
2019-06-26 09:26:33 +02:00
Egon Elbre
c7679b9b30
Fix some leaks and add notes about close handling ( #2334 )
2019-06-25 23:00:51 +03:00
Stefan Benten
57ef352b3c
Update Wizard to allow more easily addition of satellites and move package to cmd folder ( #2340 )
2019-06-25 20:25:31 +02:00
Stefan Benten
2cc01c7899
Rename Satellites in the Wizard ( #2339 )
...
* Rename Satellites
* Fixing Typo and changing names for gateway
* Unify Setup wizard across gateway and uplink
2019-06-25 11:29:52 -06:00
Fadila
20de18feec
[v3-1952 test 6] Disqualification is a one way ticket. Storage Node can't get around it. ( #2328 )
2019-06-25 13:10:22 -04:00
Egon Elbre
c28f800098
Skip TestDataRepair and TestUplinksParallel, because they are flaky ( #2337 )
2019-06-25 19:30:39 +03:00
Jess G
e5c48fab74
fix ordersDB methods to take correct args ( #2314 )
...
* fix orderdDB methods to take correct args
* update tally to save projectID in correct format
* update var names in splitBucket test
* changes per CR comments
2019-06-25 08:58:42 -07:00
Egon Elbre
414648d660
Fix some metainfo.Client leaks ( #2327 )
2019-06-25 18:36:23 +03:00
Maximillian von Briesen
35f2ab5ded
Enable node selection tests ( #2316 )
2019-06-25 11:26:47 -04:00
Yaroslav Vorobiov
eb5dc4ba2b
Add node stats GRPC endpoint to the satellite ( #2281 )
2019-06-25 13:40:32 +03:00
Egon Elbre
6502143e79
fix import ordering ( #2322 )
2019-06-25 12:46:29 +03:00
Ivan Fraixedes
fa802dc429
pkg/audit: Improve Reporter.ReporterAudits doc ( #2266 )
...
Add what the method returns because from the method name and the
signature name, it wasn't clear why and when a report is returned.
2019-06-25 11:23:41 +02:00
Jeff Wendling
c35c8e4c24
allow reading bucket metadata with restricted keys ( #2321 )
...
Change-Id: I47d3a2f5f02744ae6c51d54963cdf2dff24134e2
2019-06-24 17:36:09 -06:00
JT Olio
8c57434ded
pkg/process/metrics: add an instance prefix ( #2190 )
...
* pkg/process/metrics: add an instance prefix
the distinction between which satellite is sending which
data should go in the instance field, not the suffix or application
fields. (un)fortunately, the instance id is deliberately not
configurable because we don't want it to be easy to accidentally
have multiple applications collide with the same instance id.
so we're currently stuffing the human readable instance in the
suffix. :(
perhaps a reasonable tradeoff would be an optional instance
prefix that allows operators to put their domain name in
the instance
Change-Id: I6fcc8498be908c5740439cc00f77474ad151febd
* linting
Change-Id: I9f9a44fa9a2634ef5e4f89548d42d57ce9e4450e
2019-06-24 16:45:37 -06:00
Fadila
70f28ae41b
[v3-1952 tests 4 & 5] DQ nodes should not be used for download or upload ( #2272 )
2019-06-25 00:04:06 +02:00
