Commit Graph

83 Commits

Author SHA1 Message Date
Egon Elbre
c8edeb0257
satellite/overlay: rename overlay.Cache to overlay.Service (#2717) 2019-08-06 19:35:59 +03:00
Michal Niewrzal
de7dddbe59
metainfo: Batch request (#2694) 2019-08-06 16:56:23 +02:00
Michal Niewrzal
688d932d93
Make one implementation for SetAttribution/SetBucketAttribution (#2683) 2019-08-05 09:07:40 +02:00
Michal Niewrzal
287fdf9936
Integrate new Metainfo calls (server side) (#2682) 2019-08-01 11:04:31 +02:00
Natalie Villasana
238e264a8f
satellite confirms success threshold reached (#2657) 2019-07-31 14:28:43 -04:00
Egon Elbre
5d0816430f
rename all the things (#2531)
* rename pkg/linksharing to linksharing
* rename pkg/httpserver to linksharing/httpserver
* rename pkg/eestream to uplink/eestream
* rename pkg/stream to uplink/stream
* rename pkg/metainfo/kvmetainfo to uplink/metainfo/kvmetainfo
* rename pkg/auth/signing to pkg/signing
* rename pkg/storage to uplink/storage
* rename pkg/accounting to satellite/accounting
* rename pkg/audit to satellite/audit
* rename pkg/certdb to satellite/certdb
* rename pkg/discovery to satellite/discovery
* rename pkg/overlay to satellite/overlay
* rename pkg/datarepair to satellite/repair
2019-07-28 08:55:36 +03:00
Kaloyan Raev
734c793deb
Add UpdatePieces method to Metainfo service (#2572) 2019-07-25 19:59:46 +03:00
Michal Niewrzal
5710dc3a32
Metainfo RPC segment methods (part 2) (#2616) 2019-07-24 13:33:23 +02:00
Jess G
353b089927
update testplanet with libuplink (#2618)
* update testplanet uplink upload with libuplink

* add libuplink to testplanet download

* update createbucket and delete obj with libuplink

* update downloadStream, fix tests

* fix test

* updates for CR comments
2019-07-23 07:58:45 -07:00
Ivan Fraixedes
3c8f1370d2
[v3 2137] - Add more info to find out repair failures (#2623)
* pkg/datarepair/repairer: Track always time for repair
  Make a minor change in the worker function of the repairer, that when
  successful, always track the metric time for repair independently if the
  time since checker queue metric can be tracked.

* storage/postgreskv: Wrap error in Get func
  Wrap the returned error of the Get function as it is done when the
  query doesn't return any row.

* satellite/metainfo: Move debug msg to the right place
  NewStore function was writing a debug log message when the DB was
  connected, however it was always writing it out despite if an error
  happened when getting the connection.

* pkg/datarepair/repairer: Wrap error before logging it
  Wrap the error returned by process which is executed by the Run method
  of the repairer service to add context to the error log message.

* pkg/datarepair/repairer: Make errors more specific in worker
  Make the error messages of the "worker" method of the Service more
  specific and the logged message for such errors.

* pkg/storage/repair: Improve error reporting Repair
  In order of improving the error reporting by the
  pkg/storage/repair.Repair method, several errors of this method and
  functions/methods which this one relies one have been updated to be
  wrapper into their corresponding classes.

* pkg/storage/segments: Track path param of Repair method
  Track in monkit the path parameter passed to the Repair method.

* satellite/satellitedb: Wrap Error returned by Delete
  Wrap the error returned by repairQueue.Delete method to enhance the
  error with a class and stack and the
  pkg/storage/segments.Repairer.Repair method get a more contextualized
  error from it.
2019-07-23 16:28:06 +02:00
Michal Niewrzal
cba008d7df
Add GetObject method to Metainfo (#2611) 2019-07-23 13:09:12 +02:00
Michal Niewrzal
6f2b85603d
Metainfo RPC segment methods (part 1) (#2567) 2019-07-22 16:45:18 +02:00
Maximillian von Briesen
6c1c3fb4a7
Add metainfo loop service (#2563)
Add a metainfo loop service on the satellite that can be subscribed to by various services that need to make use of metainfo information
2019-07-22 09:34:12 -04:00
aligeti
29b576961f
value attribution merge fix and more test cases (#2588)
* value attribution merge fix and more test cases
2019-07-19 11:17:34 -04:00
Michal Niewrzal
0b827f1375
Speedup metainfo tests (#2556) 2019-07-16 21:16:41 +02:00
Kaloyan Raev
d453cd148e Consider encryption overhead when validating max segment size (#2569) 2019-07-16 15:30:10 +02:00
Michal Niewrzal
260d9c49a8
Metainfo RPC objects methods (#2534) 2019-07-16 12:39:23 +02:00
Jess G
aa99482fa7
Jg/add tests (#2547)
* add a test to confirm only objects in pointerDB

* fix lint

* fix link
2019-07-15 09:30:26 -07:00
Jess G
f11bf46a11
Jg/1967 mv bucket metadata uplink (#2505)
* add bucketstore, add init uplink bucket

* update uplink to use bucket rpc

* fix tests

* wrap metainfo client errors

* add allowedBucket struct, fix tests

* update comment

* add paging

* updates per CR

* add test for pagination

* fix lint

* fix uplink test so its easier tyo understand

* fix gateway pagination bug

* changes per cr

* fix bug w/allowedBuckets, add test to catch
2019-07-12 08:57:02 -04:00
Egon Elbre
d52f764e54
protocol: implement new piece signing and verification (#2525) 2019-07-11 16:51:40 -04:00
Bill Thorp
0e463dccfd
7 day validity window for order limits (#2520)
* 7 day limit
2019-07-10 17:17:00 -04:00
Alexander Leitner
1c5db71faf
Change protobuf expirations to use time.Time (#2509)
* Change protobuf expirations to use time.Time instead of timestamp.Timestamp
2019-07-09 17:54:00 -04:00
Egon Elbre
910e4744ad
satellite/metainfo: optimize pointerdb size (#2506) 2019-07-09 16:36:18 -04:00
Jess G
f9696d6c5e
satellite/metainfo: add buckets RPC and database (#2460)
* add db interface and methods, add sa metainfo endpoints and svc

* add bucket metainfo svc funcs

* add sadb bucekts

* bucket list gets all buckets

* filter buckets list on macaroon restrictions

* update pb cipher suite to be enum

* add conversion funcs

* updates per comments

* bucket settings should say default

* add direction to list buckets, add tests

* fix test bucket names

* lint err

* only support forward direction

* add comments

* minor refactoring

* make sure list up to limit

* update test

* update protolock file

* fix lint

* change per PR
2019-07-08 15:32:18 -07:00
Stefan Benten
16156e3b3d
Ensure we force a segment size and account storage before committing them (#2473) 2019-07-08 18:24:38 -04:00
Alexander Leitner
3587e1a579 Change pointerdb pointer to use time.Time for Creation date (#2483) 2019-07-09 00:16:50 +02:00
aligeti
a554752ce0
adding expiration date check on satellite's CreateSegment (#2476)
* adding expiration check on satellite createsegment
2019-07-08 14:10:16 -04:00
Michal Niewrzal
094e1b8b90
Add 'Old' suffix to some metainfo methods/messages (#2462) 2019-07-08 15:33:15 +02:00
Michal Niewrzal
56fcf228fb Enable bucket name validation (#2449) 2019-07-03 14:56:17 -04:00
aligeti
ae8b9698f9
Rename/remove EncryptionScheme -> EncryptionParame… (#2363)
* rename/remove EncryptionScheme -> EncryptionParameters
2019-07-03 14:07:44 -04:00
Michal Niewrzal
2ee5bada2c
Add pieceNum to PieceID derivation function (#2193) 2019-07-03 18:53:15 +02:00
Michal Niewrzal
61dfa61e3a
Add timestamp and piece size to piece hash (#2198) 2019-07-03 18:14:37 +02:00
Maximillian von Briesen
52e5a4eee3 pass logger into repairer and ecclient (#2365) 2019-07-02 13:08:02 +03:00
Jess G
ea55ae63e3
add bucket metainfo rpc (#2383)
* add bucket metainfo rpc

* updates per design doc

* rm bucketmeta rpc

* add changes per comments

* update Bucket name since same name already exists

* add placeholder methods to fulfill pb.MetainfoServer interface

* update proto lock

* add bucketListItem message

* why failing lint?

* add back ciphersuite but with correct types

* fix proto errs

* update time type

* add proto lock

* rm unneeded import

* update protolock package
2019-07-01 15:17:30 -07:00
Egon Elbre
385c046723
pkg/pb: rename Order2 to Order, OrderLimit2 to OrderLimit (#2406) 2019-07-01 18:54:11 +03:00
Egon Elbre
2b68a72428
internal/testplanet: ensure that metainfo connections get closed (#2381) 2019-07-01 17:35:10 +03:00
Michal Niewrzal
ff92a5bcf6 Improve RS validation message (#2377) 2019-06-29 17:05:46 +02:00
Egon Elbre
e83ebd7cde
jenkins: avoid using goimports and distribute load better (#2359) 2019-06-27 21:52:50 +03:00
Jeff Wendling
efcdaa43a3
lib/uplink: encryption context (#2349)
* lib/uplink: encryption context

Change-Id: I5c23dca3286a46b713b30c4997e9ae6e630b2280

* lib/uplink: bucket operation examples

Change-Id: Ia0f6e69f365dcff0cf11c731f51b30842bce053b

* lib/uplink: encryption key sharing test cases

Change-Id: I3a172d565f33f4e591402cdcb9460664a7cc7fbe

* fix encrypted path prefix restriction issue

Change-Id: I8f3921f9d52aaf4b84039de608b8cbbc88769554

* implement panics in libuplink encryption code

todo on cipher suite selection as well as an api concern

Change-Id: Ifa39eb3cc4b3443f7d96f9304df9b2ac4ec4085d

* implement GetProjectInfo api call to get salt

Change-Id: Ic5f6b3be9ea35df48c1aa214ab5d355fb328e2cf

* some fixes and accessors for encryption store

Change-Id: I3bb61f6712a037900e2a96e72ad4029ec1d3f718

* general fixes to builds/tests/etc

Change-Id: I9930fa96acb3b221d9a001f8e274af5729cc8a47

* java bindings changes

Change-Id: Ia2bd4c9c69739c8d3154d79616cff1f36fb403b6

* get libuplink examples passing

Change-Id: I828f09a144160e0a5dd932324f78491ae2ec8a07

* fix proto.lock file

Change-Id: I2fbbf4d0976a7d0473c2645e6dcb21aaa3be7651

* fix proto.lock again

Change-Id: I92702cf49e1a340eef6379c2be4f7c4a268112a9

* fix golint issues

Change-Id: I631ff9f43307a58e3b25a58cbb4a4cc2495f5eb6

* more linting fixes

Change-Id: I51f8f30b367b5bca14c94b15417b9a4c9e7aa0ce

* bug fixed by structs bump

Change-Id: Ibb03c691fce7606c35c08721b3ef0781ab48a38a

* retrigger

Change-Id: Ieee0470b6a2d07168a1578552e8e7f271ae93a13

* retrigger

Change-Id: I753d63853171e6a436c104ce176048892eb974c5

* semantic merge conflict

Change-Id: I9419448496de90340569047a6a16a1b858a7978a

* update total to match prod defaults

Change-Id: I693d55c1ebb28b5803ee1d26e9e198decf82308b

* retrigger

Change-Id: I28b74d5d6202f61aa3866fe407d423f6a0a14b9e

* retrigger

Change-Id: I6fd054885c715f602e2cef623fd464c42e88742c

* retrigger

Change-Id: I6a01bae88c72406d4ed5a8f13bf8a2b3c650bd2d
2019-06-27 17:36:51 +00:00
aligeti
7facda4422
value attribution integration with libuplink (#2297)
* integrates the value attribution with libuplink
2019-06-26 12:22:01 -04:00
Egon Elbre
b6ad3e9c9f
internal/testrand: new package for random data (#2282) 2019-06-26 13:38:51 +03:00
Michal Niewrzal
6b23380d6b
Make RS validation always enabled (#2336) 2019-06-26 12:27:36 +02:00
Jess G
e5c48fab74
fix ordersDB methods to take correct args (#2314)
* fix orderdDB methods to take correct args

* update tally to save projectID in correct format

* update var names in splitBucket test

* changes per CR comments
2019-06-25 08:58:42 -07:00
Egon Elbre
414648d660
Fix some metainfo.Client leaks (#2327) 2019-06-25 18:36:23 +03:00
Jeff Wendling
c35c8e4c24 allow reading bucket metadata with restricted keys (#2321)
Change-Id: I47d3a2f5f02744ae6c51d54963cdf2dff24134e2
2019-06-24 17:36:09 -06:00
JT Olio
e17908a266
metainfo: add project info rpc for getting a project salt using an api key (#2311)
The API exposed in #2302 needs to be able to get a project-level salt (https://github.com/storj/storj/pull/2302/files#diff-72b734f4a2e408a544fef071d2565fc8R200), see https://github.com/storj/storj/blob/master/docs/design/password-key-derivation.md also

I realized this morning that if we need it later this week, we need to get it into today's release.

Please describe the tests: tests will come as part of #2302

Please describe the performance impact: none
2019-06-24 11:15:45 -06:00
Michal Niewrzal
fdeb834801
Bucket name validation (#2244) 2019-06-24 11:52:25 +02:00
aligeti
d8e62bc06e
support value attribution endpoint (#2231)
Support Value attribution Endpoint
2019-06-21 15:14:34 -04:00
aligeti
043d603cbe
satellite rs config check with validation check set to false default (#2229)
* satellite rs config check with validation check
2019-06-21 14:15:58 -04:00
Kaloyan Raev
964c87c476 Fix checks around repair threshold (#2246) 2019-06-19 22:13:11 +02:00