Commit Graph

43 Commits

Author SHA1 Message Date
Egon Elbre
218f436f6b
uplink: remove dependency to storage/ (#3720) 2019-12-10 22:07:35 +02:00
Michal Niewrzal
ffd570eb04 uplink/metainfo: remove additional GetObject from download
This change removes one additional metainfo.GetObject call during
download.

Change-Id: I86b4e5165f299069e3e18944fe79c697c6a514d3
2019-12-09 11:48:48 +00:00
Ivan Fraixedes
8d49a99ad8 uplink/metainfo: Fix doc comments
Fix some documentation comments in a few methods and apply our code
style conventions to all of them in this source file of the package.

Change-Id: I78186cb1e7d45020d6cf7bbea8a39d146ada1cb6
2019-12-09 08:47:52 +00:00
Egon Elbre
36fead0093 satellite/metainfo: add UserAgent support to endpoints (#3548) 2019-11-26 03:12:37 -08:00
Michal Niewrzal
5964502ce0
uplink/metainfo: remove GetObject from download Batch (#3596) 2019-11-19 04:58:26 -08:00
Michal Niewrzal
bc16cb5d24
libuplink: remove additional GetBucket for upload/download (#3568) 2019-11-15 02:06:17 -08:00
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00
paul cannon
0c025fa937 storage/: remove reverse-key-listing feature
We don't use reverse listing in any of our code, outside of tests, and
it is only exposed through libuplink in the
lib/uplink.(*Project).ListBuckets() API. We also don't know of any users
who might have a need for reverse listing through ListBuckets().

Since one of our prospective pointerdb backends can not support
backwards iteration, and because of the above considerations, we are
going to remove the reverse listing feature.

Change-Id: I8d2a1f33d01ee70b79918d584b8c671f57eef2a0
2019-11-12 18:47:51 +00:00
Ivan Fraixedes
e4a220347a
uplink: Suppress one metainfo call on delete (#3511)
Change signature of metainfo DeleteObject to get rid of an extra call to
kvmetainfo GetBucket method and eliminate one round trip to the
satellite when deleting objects.
2019-11-07 10:39:40 +01:00
Michal Niewrzal
8786a37f89
uplink/storage: use Batch to optimize upload requests (#3408) 2019-10-29 08:49:16 -07:00
Michal Niewrzal
56f8b2d626
uplink/storage: remove bucket store (#3376) 2019-10-28 09:40:46 -07:00
Ivan Fraixedes
4f281ef348
uplink: Refactor segments Store Get for metainfo Batch (#3362)
Refactoring of the segments Store interface Get method signature to
force the implementations to not use metainfo Client and be able for the
callers to batch requests.
2019-10-28 17:23:20 +01:00
Ivan Fraixedes
d9d82b0336 uplink: Reduce satellite request using Batch when possible (#3351)
* uplink/metainfo: Return classified Not Found error

Metainfo client Batch method must return the Storj Not Found error class
when the RCP server response with a not found status code as any other
metainfo Client method does.

Also if the error isn't Not Found one, it must wrap the error.

* uplink/storage/streams: Use Batch request in Delete

Change the 2 individual metainfo Client calls that streamStore Delete
method does by a single Batch one.
2019-10-24 14:18:48 -07:00
Michal Niewrzal
521c39bda0
uplink/metainfo: cleanup method names (#3315) 2019-10-22 23:59:56 -07:00
JT Olio
6ede140df1
pkg/rpc: defeat MITM attacks in most cases (#3215)
This change adds a trusted registry (via the source code) of node address to node id mappings (currently only for well known Satellites) to defeat MITM attacks to Satellites. It also extends the uplink UI such that when entering a satellite address by hand, a node id prefix can also be added to defeat MITM attacks with unknown satellites.

When running uplink setup, satellite addresses can now be of the form 12EayRS2V1k@us-central-1.tardigrade.io (not even using a full node id) to ensure that the peer contacted is the peer that was expected. When using a known satellite address, the known node ids are used if no override is provided.
2019-10-12 14:34:41 -06:00
Jeff Wendling
098cbc9c67 all: use pkg/rpc instead of pkg/transport
all of the packages and tests work with both grpc and
drpc. we'll probably need to do some jenkins pipelines
to run the tests with drpc as well.

most of the changes are really due to a bit of cleanup
of the pkg/transport.Client api into an rpc.Dialer in
the spirit of a net.Dialer. now that we don't need
observers, we can pass around stateless configuration
to everything rather than stateful things that issue
observations. it also adds a DialAddressID for the
case where we don't have a pb.Node, but we do have an
address and want to assert some ID. this happened
pretty frequently, and now there's no more weird
contortions creating custom tls options, etc.

a lot of the other changes are being consistent/using
the abstractions in the rpc package to do rpc style
things like finding peer information, or checking
status codes.

Change-Id: Ief62875e21d80a21b3c56a5a37f45887679f9412
2019-09-25 15:37:06 -06:00
JT Olio
946ec201e2
metainfo: move api keys to part of the request (#3069)
What: we move api keys out of the grpc connection-level metadata on the client side and into the request protobufs directly. the server side still supports both mechanisms for backwards compatibility.

Why: dRPC won't support connection-level metadata. the only thing we currently use connection-level metadata for is api keys. we need to move all information needed by a request into the request protobuf itself for drpc support. check out the .proto changes for the main details.

One fun side-fact: Did you know that protobuf fields 1-15 are special and only use one byte for both the field number and type? Additionally did you know we don't use field 15 anywhere yet? So the new request header will use field 15, and should use field 15 on all protobufs going forward.

Please describe the tests: all existing tests should pass

Please describe the performance impact: none
2019-09-19 10:19:29 -06:00
Michal Niewrzal
64c467ffe7
uplink: integrate new Metainfo calls (#2640) 2019-09-10 08:39:47 -07:00
Michal Niewrzal
a6721ba92f
satellite/metainfo: Improve metainfo ListSegments (#2882) 2019-08-30 23:30:18 +02:00
JT Olio
12d50ebb99
streams: don't encrypt segment count (#2859)
What: this change makes sure the count of segments is not encrypted.

Why: having the segment count encrypted just makes things hard for no reason - a satellite operator can figure out how many segments an object has by looking at the other segments in the database. but if a user has access but has lost their encryption key, they now can't clean up or delete old segments because they can't know how many there are without just guessing until they get errors. :(

Backwards compatibility: clients will still understand old pointers and will still write old pointers. at some point in the future perhaps we can do a migration for remaining old pointers so we can delete the old code.

Please describe the tests: covered by existing tests

Please describe the performance impact: none
2019-08-22 15:15:58 -06:00
Egon Elbre
c8edeb0257
satellite/overlay: rename overlay.Cache to overlay.Service (#2717) 2019-08-06 19:35:59 +03:00
Michal Niewrzal
de7dddbe59
metainfo: Batch request (#2694) 2019-08-06 16:56:23 +02:00
Michal Niewrzal
688d932d93
Make one implementation for SetAttribution/SetBucketAttribution (#2683) 2019-08-05 09:07:40 +02:00
Michal Niewrzal
287fdf9936
Integrate new Metainfo calls (server side) (#2682) 2019-08-01 11:04:31 +02:00
Egon Elbre
9ba8b53ed5 pkg/auth: use grpc.WithPerRPCCredentials (#2670) 2019-07-31 13:57:13 +02:00
Egon Elbre
dd7c8610bb
satellite/repair: move test files (#2649) 2019-07-28 12:15:34 +03:00
Egon Elbre
5d0816430f
rename all the things (#2531)
* rename pkg/linksharing to linksharing
* rename pkg/httpserver to linksharing/httpserver
* rename pkg/eestream to uplink/eestream
* rename pkg/stream to uplink/stream
* rename pkg/metainfo/kvmetainfo to uplink/metainfo/kvmetainfo
* rename pkg/auth/signing to pkg/signing
* rename pkg/storage to uplink/storage
* rename pkg/accounting to satellite/accounting
* rename pkg/audit to satellite/audit
* rename pkg/certdb to satellite/certdb
* rename pkg/discovery to satellite/discovery
* rename pkg/overlay to satellite/overlay
* rename pkg/datarepair to satellite/repair
2019-07-28 08:55:36 +03:00
Michal Niewrzal
5710dc3a32
Metainfo RPC segment methods (part 2) (#2616) 2019-07-24 13:33:23 +02:00
Michal Niewrzal
cba008d7df
Add GetObject method to Metainfo (#2611) 2019-07-23 13:09:12 +02:00
aligeti
29b576961f
value attribution merge fix and more test cases (#2588)
* value attribution merge fix and more test cases
2019-07-19 11:17:34 -04:00
Michal Niewrzal
260d9c49a8
Metainfo RPC objects methods (#2534) 2019-07-16 12:39:23 +02:00
Jess G
f11bf46a11
Jg/1967 mv bucket metadata uplink (#2505)
* add bucketstore, add init uplink bucket

* update uplink to use bucket rpc

* fix tests

* wrap metainfo client errors

* add allowedBucket struct, fix tests

* update comment

* add paging

* updates per CR

* add test for pagination

* fix lint

* fix uplink test so its easier tyo understand

* fix gateway pagination bug

* changes per cr

* fix bug w/allowedBuckets, add test to catch
2019-07-12 08:57:02 -04:00
Egon Elbre
d52f764e54
protocol: implement new piece signing and verification (#2525) 2019-07-11 16:51:40 -04:00
Alexander Leitner
1c5db71faf
Change protobuf expirations to use time.Time (#2509)
* Change protobuf expirations to use time.Time instead of timestamp.Timestamp
2019-07-09 17:54:00 -04:00
Michal Niewrzal
094e1b8b90
Add 'Old' suffix to some metainfo methods/messages (#2462) 2019-07-08 15:33:15 +02:00
Egon Elbre
385c046723
pkg/pb: rename Order2 to Order, OrderLimit2 to OrderLimit (#2406) 2019-07-01 18:54:11 +03:00
Jeff Wendling
efcdaa43a3
lib/uplink: encryption context (#2349)
* lib/uplink: encryption context

Change-Id: I5c23dca3286a46b713b30c4997e9ae6e630b2280

* lib/uplink: bucket operation examples

Change-Id: Ia0f6e69f365dcff0cf11c731f51b30842bce053b

* lib/uplink: encryption key sharing test cases

Change-Id: I3a172d565f33f4e591402cdcb9460664a7cc7fbe

* fix encrypted path prefix restriction issue

Change-Id: I8f3921f9d52aaf4b84039de608b8cbbc88769554

* implement panics in libuplink encryption code

todo on cipher suite selection as well as an api concern

Change-Id: Ifa39eb3cc4b3443f7d96f9304df9b2ac4ec4085d

* implement GetProjectInfo api call to get salt

Change-Id: Ic5f6b3be9ea35df48c1aa214ab5d355fb328e2cf

* some fixes and accessors for encryption store

Change-Id: I3bb61f6712a037900e2a96e72ad4029ec1d3f718

* general fixes to builds/tests/etc

Change-Id: I9930fa96acb3b221d9a001f8e274af5729cc8a47

* java bindings changes

Change-Id: Ia2bd4c9c69739c8d3154d79616cff1f36fb403b6

* get libuplink examples passing

Change-Id: I828f09a144160e0a5dd932324f78491ae2ec8a07

* fix proto.lock file

Change-Id: I2fbbf4d0976a7d0473c2645e6dcb21aaa3be7651

* fix proto.lock again

Change-Id: I92702cf49e1a340eef6379c2be4f7c4a268112a9

* fix golint issues

Change-Id: I631ff9f43307a58e3b25a58cbb4a4cc2495f5eb6

* more linting fixes

Change-Id: I51f8f30b367b5bca14c94b15417b9a4c9e7aa0ce

* bug fixed by structs bump

Change-Id: Ibb03c691fce7606c35c08721b3ef0781ab48a38a

* retrigger

Change-Id: Ieee0470b6a2d07168a1578552e8e7f271ae93a13

* retrigger

Change-Id: I753d63853171e6a436c104ce176048892eb974c5

* semantic merge conflict

Change-Id: I9419448496de90340569047a6a16a1b858a7978a

* update total to match prod defaults

Change-Id: I693d55c1ebb28b5803ee1d26e9e198decf82308b

* retrigger

Change-Id: I28b74d5d6202f61aa3866fe407d423f6a0a14b9e

* retrigger

Change-Id: I6fd054885c715f602e2cef623fd464c42e88742c

* retrigger

Change-Id: I6a01bae88c72406d4ed5a8f13bf8a2b3c650bd2d
2019-06-27 17:36:51 +00:00
Egon Elbre
414648d660
Fix some metainfo.Client leaks (#2327) 2019-06-25 18:36:23 +03:00
Egon Elbre
8f4a6afc8a Fixes for value attribution (#2238)
* Move value attribution to satellite/

* remove unnecessary conversion
2019-06-19 08:02:37 -04:00
aligeti
8e8a5eb89c
initial value attribution check development (#2091)
* initial value attribution check development

* code reveiew updates
2019-06-12 21:35:37 -04:00
ethanadams
268dc6b7e4
Enable gocritic linter (#2051)
* first round cleanup based on go-critic

* more issues resolved for ifelsechain and unlambda checks

* updated from master and gocritic found a new ifElseChain issue

* disable appendAssign. i reports false positives

* re-enabled go-critic appendAssign and disabled lint check at code line level

* fixed go-critic lint error

* fixed // nolint add gocritic specifically
2019-05-29 09:14:25 -04:00
Michal Niewrzal
d7feafe56b Move psserver tests (#1522) 2019-03-20 23:12:00 +02:00
Egon Elbre
05d148aeb5
Storage node and upload/download protocol refactor (#1422)
refactor storage node server
refactor upload and download protocol
2019-03-18 12:55:06 +02:00