Commit Graph

1091 Commits

Author SHA1 Message Date
littleskunk
fb66867856 fix repair queue (#2405) 2019-07-01 19:00:53 +02:00
Natalie Villasana
4ba212107b
add back testrand (#2408) 2019-07-01 12:39:04 -04:00
Ivan Fraixedes
b6ae6a4e2c
pkg/datarepair/checker: Fix doc comments (#2401) 2019-07-01 18:13:18 +02:00
Egon Elbre
385c046723
pkg/pb: rename Order2 to Order, OrderLimit2 to OrderLimit (#2406) 2019-07-01 18:54:11 +03:00
Natalie Villasana
3ffb42483f
account for disqualified nodes in data repair tests (#2315) 2019-07-01 11:34:42 -04:00
Natalie Villasana
3f643551e7 remove flakiness in TestDataRepair and TestSegmentStoreRepair (#2335)
* stop audit loop in repair tests to prevent possible timeout
2019-07-01 11:15:45 -04:00
Kaloyan Raev
4f0e437965
Audit tests cleanup (#2402) 2019-07-01 17:02:00 +03:00
Ivan Fraixedes
fd2e708587 pkg/storage/segments: Abort repaired test unmet requirements (#2403)
Abort the repairer test when asserting results which are required later
on by the test for being able to continue.
2019-07-01 08:49:40 -04:00
Maximillian von Briesen
b750bc2d0f
Keep some unhealthy pieces in the event of a partial repair (#2380)
* if repair is partial, keep saving "unhealthy" pieces that are not duplicates
2019-06-28 15:48:51 -04:00
Egon Elbre
7e0c5d51d0
pkg/storj: add NodeURL type for verified url-s (#2379) 2019-06-28 21:43:39 +03:00
Egon Elbre
827fb92b47
satellite/console: nicer error handling in tests (#2378) 2019-06-28 16:04:50 +03:00
Jeff Wendling
1a65e42d33 rename EncryptionCtx back to EncryptionAccess (#2368)
* rename EncryptionCtx back to EncryptionAccess

Change-Id: I5e58915a59979ad6f2e83d36e191b2bbf3ba2ba2

* missed some stuff

Change-Id: Ib65fb186d7f854c0406e5fd7d11498e8941da59e

* oops protolock

Change-Id: I85a5ab4bafb9dd6a804d3dcd17a70811702f07e4

* retrigger

Change-Id: I2d0e3f02b4dbae0299b090e9617662b4437980b0
2019-06-27 23:18:24 -06:00
aligeti
811168e2c4
Uplink bucket attribution extension (#2353)
Attribution info added to the bucket info
2019-06-27 19:25:36 -04:00
Egon Elbre
e83ebd7cde
jenkins: avoid using goimports and distribute load better (#2359) 2019-06-27 21:52:50 +03:00
Jeff Wendling
efcdaa43a3
lib/uplink: encryption context (#2349)
* lib/uplink: encryption context

Change-Id: I5c23dca3286a46b713b30c4997e9ae6e630b2280

* lib/uplink: bucket operation examples

Change-Id: Ia0f6e69f365dcff0cf11c731f51b30842bce053b

* lib/uplink: encryption key sharing test cases

Change-Id: I3a172d565f33f4e591402cdcb9460664a7cc7fbe

* fix encrypted path prefix restriction issue

Change-Id: I8f3921f9d52aaf4b84039de608b8cbbc88769554

* implement panics in libuplink encryption code

todo on cipher suite selection as well as an api concern

Change-Id: Ifa39eb3cc4b3443f7d96f9304df9b2ac4ec4085d

* implement GetProjectInfo api call to get salt

Change-Id: Ic5f6b3be9ea35df48c1aa214ab5d355fb328e2cf

* some fixes and accessors for encryption store

Change-Id: I3bb61f6712a037900e2a96e72ad4029ec1d3f718

* general fixes to builds/tests/etc

Change-Id: I9930fa96acb3b221d9a001f8e274af5729cc8a47

* java bindings changes

Change-Id: Ia2bd4c9c69739c8d3154d79616cff1f36fb403b6

* get libuplink examples passing

Change-Id: I828f09a144160e0a5dd932324f78491ae2ec8a07

* fix proto.lock file

Change-Id: I2fbbf4d0976a7d0473c2645e6dcb21aaa3be7651

* fix proto.lock again

Change-Id: I92702cf49e1a340eef6379c2be4f7c4a268112a9

* fix golint issues

Change-Id: I631ff9f43307a58e3b25a58cbb4a4cc2495f5eb6

* more linting fixes

Change-Id: I51f8f30b367b5bca14c94b15417b9a4c9e7aa0ce

* bug fixed by structs bump

Change-Id: Ibb03c691fce7606c35c08721b3ef0781ab48a38a

* retrigger

Change-Id: Ieee0470b6a2d07168a1578552e8e7f271ae93a13

* retrigger

Change-Id: I753d63853171e6a436c104ce176048892eb974c5

* semantic merge conflict

Change-Id: I9419448496de90340569047a6a16a1b858a7978a

* update total to match prod defaults

Change-Id: I693d55c1ebb28b5803ee1d26e9e198decf82308b

* retrigger

Change-Id: I28b74d5d6202f61aa3866fe407d423f6a0a14b9e

* retrigger

Change-Id: I6fd054885c715f602e2cef623fd464c42e88742c

* retrigger

Change-Id: I6a01bae88c72406d4ed5a8f13bf8a2b3c650bd2d
2019-06-27 17:36:51 +00:00
Egon Elbre
27c92ffc10 jenkins: Stop docker container after 18m (#2358)
* remove unnecessary argument

* remove kademlia lookup short circuiting

* fix ident

* add timeout to docker container

* add comments to dockerfile
2019-06-27 10:09:37 -06:00
Egon Elbre
2128b460b4
cmd/uplink/cmd: don't create benchmark data on init (#2351) 2019-06-26 22:08:14 +03:00
Egon Elbre
615bfca135 Fix TestGetSignee flakiness (#2350)
* add IsCanceled

* fixes to error handling

* fix imports

* retrigger jenkins
2019-06-26 09:30:37 -06:00
JT Olio
fbe9696e92 pkg/kademlia: clean up peer discovery (#2252) 2019-06-26 16:16:46 +03:00
JT Olio
3925e84580 pkg/eestream: plumb contexts through (#2187) 2019-06-26 16:05:58 +03:00
Egon Elbre
7b66e0cd7c Use dial to clarify that it's internally closing the connection. (#2347) 2019-06-26 15:14:48 +03:00
Egon Elbre
b6ad3e9c9f
internal/testrand: new package for random data (#2282) 2019-06-26 13:38:51 +03:00
Philip Hutchins
11016b5067 Moving verbose info logging to debug (#2346) 2019-06-26 12:14:25 +03:00
Kaloyan Raev
ae36979804 Take advantage of Is and IsFunc from zeebo/errs (#2310) 2019-06-26 09:38:07 +02:00
Egon Elbre
caa2fcf62b satellite/orders: don't panic (#2331) 2019-06-26 09:26:33 +02:00
Egon Elbre
c7679b9b30
Fix some leaks and add notes about close handling (#2334) 2019-06-25 23:00:51 +03:00
Stefan Benten
57ef352b3c
Update Wizard to allow more easily addition of satellites and move package to cmd folder (#2340) 2019-06-25 20:25:31 +02:00
Stefan Benten
2cc01c7899 Rename Satellites in the Wizard (#2339)
* Rename Satellites

* Fixing Typo and changing names for gateway

* Unify Setup wizard across gateway and uplink
2019-06-25 11:29:52 -06:00
Fadila
20de18feec [v3-1952 test 6] Disqualification is a one way ticket. Storage Node can't get around it. (#2328) 2019-06-25 13:10:22 -04:00
Egon Elbre
c28f800098
Skip TestDataRepair and TestUplinksParallel, because they are flaky (#2337) 2019-06-25 19:30:39 +03:00
Jess G
e5c48fab74
fix ordersDB methods to take correct args (#2314)
* fix orderdDB methods to take correct args

* update tally to save projectID in correct format

* update var names in splitBucket test

* changes per CR comments
2019-06-25 08:58:42 -07:00
Egon Elbre
414648d660
Fix some metainfo.Client leaks (#2327) 2019-06-25 18:36:23 +03:00
Maximillian von Briesen
35f2ab5ded
Enable node selection tests (#2316) 2019-06-25 11:26:47 -04:00
Yaroslav Vorobiov
eb5dc4ba2b
Add node stats GRPC endpoint to the satellite (#2281) 2019-06-25 13:40:32 +03:00
Egon Elbre
6502143e79
fix import ordering (#2322) 2019-06-25 12:46:29 +03:00
Ivan Fraixedes
fa802dc429
pkg/audit: Improve Reporter.ReporterAudits doc (#2266)
Add what the method returns because from the method name and the
signature name, it wasn't clear why and when a report is returned.
2019-06-25 11:23:41 +02:00
Jeff Wendling
c35c8e4c24 allow reading bucket metadata with restricted keys (#2321)
Change-Id: I47d3a2f5f02744ae6c51d54963cdf2dff24134e2
2019-06-24 17:36:09 -06:00
JT Olio
8c57434ded
pkg/process/metrics: add an instance prefix (#2190)
* pkg/process/metrics: add an instance prefix

the distinction between which satellite is sending which
data should go in the instance field, not the suffix or application
fields. (un)fortunately, the instance id is deliberately not
configurable because we don't want it to be easy to accidentally
have multiple applications collide with the same instance id.

so we're currently stuffing the human readable instance in the
suffix. :(

perhaps a reasonable tradeoff would be an optional instance
prefix that allows operators to put their domain name in
the instance

Change-Id: I6fcc8498be908c5740439cc00f77474ad151febd

* linting

Change-Id: I9f9a44fa9a2634ef5e4f89548d42d57ce9e4450e
2019-06-24 16:45:37 -06:00
Fadila
70f28ae41b
[v3-1952 tests 4 & 5] DQ nodes should not be used for download or upload (#2272) 2019-06-25 00:04:06 +02:00
Jeff Wendling
30f790a040
Create and use an encryption.Store (#2293)
* add path implementation

This commit adds a pkg/paths package which contains two types,
Encrypted and Unencrypted, to statically enforce what is contained
in a path. It's part of a refactoring of the code base to be more
clear about what is contained in a storj.Path at all the layers.

Change-Id: Ifc4d4932da26a97ea99749b8356b4543496a8864

* add encryption store

This change adds an encryption.Store type to keep a collection
of root keys for arbitrary locations in some buckets. It allows
one to look up all of the necessary information to encrypt paths,
decrypt paths and decrypt list operations.

It adds some exported functions to perform encryption on paths
using a Store.

Change-Id: I1a3d230c521d65f0ede727f93e1cb389f8be9497

* add shim around streams store

This commit changes no functionality, but just reorganizes the code
so that changes can be made directly to the streams store
implementation without affecting callers.

It also adds a Path type that will be used at the interface boundary
for the streams store so that it can be sure that it's getting well
formed paths that it expects.

Change-Id: I50bd682995b185beb653b00562fab62ef11f1ab5

* refactor streams to use encryption store

This commit changes the streams store to use the path type as
well as the encryption store to handle all of it's encryption
and decryption.

Some changes were made to how the default key is returned in
the encryption store to have it include the case when the bucket
exists but no paths matched. The path iterator could also be
simplified to not report if a consume was valid: that information
is no longer necessary.

The kvmetainfo tests were changed to appropriately pass the
subtests *testing.T rather than having the closure it executes
use the parent one. The test framework now correctly reports
which test did the failing.

There are still some latent issues with listing in that listing
for "a/" and listing for "a" are not the same operation, but we
treat them as such. I suspect that there are also issues with
paths like "/" or "//foo", but that's for another time.

Change-Id: I81cad4ba2850c3d14ba7e632777c4cac93db9472

* use an encryption store at the upper layers

Change-Id: Id9b4dd5f27b3ecac863de586e9ae076f4f927f6f

* fix linting failures

Change-Id: Ifb8378879ad308d4d047a0483850156371a41280

* fix linting in encryption test

Change-Id: Ia35647dfe18b0f20fe13763b28e53294f75c38fa

* get rid of kvmetainfo rootKey

Change-Id: Id795ca03d9417e3fe9634365a121430eb678d6d5

* Fix linting failure for return with else

Change-Id: I0b9ffd92be42ffcd8fef7ea735c5fc114a55d3b5

* fix some bugs adding enc store to kvmetainfo

Change-Id: I8e765970ba817289c65ec62971ae3bfa2c53a1ba

* respond to review feedback

Change-Id: I43e2ce29ce2fb6677b1cd6b9469838d80ec92c86
2019-06-24 19:23:07 +00:00
JT Olio
e17908a266
metainfo: add project info rpc for getting a project salt using an api key (#2311)
The API exposed in #2302 needs to be able to get a project-level salt (https://github.com/storj/storj/pull/2302/files#diff-72b734f4a2e408a544fef071d2565fc8R200), see https://github.com/storj/storj/blob/master/docs/design/password-key-derivation.md also

I realized this morning that if we need it later this week, we need to get it into today's release.

Please describe the tests: tests will come as part of #2302

Please describe the performance impact: none
2019-06-24 11:15:45 -06:00
Maximillian von Briesen
fd6a4d96f2
change uptime dq threshold to 0.4 (#2313)
* change uptime dq threshold to 0.4

* update config lock
2019-06-24 12:18:32 -04:00
Stefan Benten
01beaa289a
Mask IP Addresses to subnets (#2305) 2019-06-24 17:33:18 +02:00
Fadila
8226024ca8
Do not use disqualified nodes when asking for get order limits (#2303)
Add checks about disqualification when creating order limits
2019-06-24 16:46:10 +02:00
Kaloyan Raev
75d71f70c6 Stop pinging disqualified nodes (#2306) 2019-06-24 14:39:47 +02:00
Michal Niewrzal
fdeb834801
Bucket name validation (#2244) 2019-06-24 11:52:25 +02:00
Jennifer Li Johnson
e285fe1997
Don't require encryption keys for project or bucket management (#2291) 2019-06-23 22:06:14 -04:00
Cameron
1283036e37
add storage node voucher request service (#2158)
* add voucher service on storage node

* config field tag syntax, go routines for requests

* hook up voucher service in storagenode/peer.go

* add voucher config to testplanet

* add voucher config to testplanet

* add voucher response status INVALID, ACCEPTED, REJECTED

* add a test for vouchers service

* handle no row from GetValid, test it

* add trust pool to voucher service

* use trusted list to get satellites

* verify vouchers upon receipt

* test VerifyVoucher
2019-06-21 18:48:52 -04:00
Jeff Wendling
81f1bc19dc add functions for password/root key derivation (#2294)
This commit adds two functions that implement the algorithms
described in the password key derivation design document. They
will be used during setup to derive bucket level root keys or
default passwords to use when buckets do not have their own
independent key.

Change-Id: Ie7fb2d8d549ba7465d0722716a2c1ac0ad907286
2019-06-21 18:21:16 -04:00
Bill Thorp
8f47fca5d3
Remove audit / uptime ratio fields (#2247)
* removed ratios
2019-06-21 13:14:53 -04:00