Commit Graph

22 Commits

Author SHA1 Message Date
JT Olio
389d1821ea uplink/paths/encryption: support commandline argument to override path cipher to be urlsafe base64 for lists and deletes (#2855) 2019-12-19 12:29:00 +01:00
Michal Niewrzal
45f8bb5084 uplink/storage: remove unused Meta methods
Change-Id: Ia18609646efd6238dc5b9142146839ee2abf388e
2019-12-16 14:46:41 +01:00
Egon Elbre
218f436f6b
uplink: remove dependency to storage/ (#3720) 2019-12-10 22:07:35 +02:00
Michal Niewrzal
5964502ce0
uplink/metainfo: remove GetObject from download Batch (#3596) 2019-11-19 04:58:26 -08:00
Ivan Fraixedes
c193dee9ae
uplink/storage/streams: Fix upload error clean up (#3555)
The number of the last segment uploaded returned by upload wasn't
correct in all the return statements. The Put method calls upload but
with the returned values wasn't even certain of cleaning up correctly
the segments uploaded when an error happens.

This commit moves the logic of cleaning up inside of the upload method
because it's easier to understand than only doing inside of it the clean
up when a context cancellation happens. It also fixes the number of
segments to be deleted by cancelHandler when an error happens.

These changes should avoid trying to delete segments which haven't been
uploaded because of the wrong index value or because of being inline.
2019-11-15 10:37:20 +01:00
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00
paul cannon
0c025fa937 storage/: remove reverse-key-listing feature
We don't use reverse listing in any of our code, outside of tests, and
it is only exposed through libuplink in the
lib/uplink.(*Project).ListBuckets() API. We also don't know of any users
who might have a need for reverse listing through ListBuckets().

Since one of our prospective pointerdb backends can not support
backwards iteration, and because of the above considerations, we are
going to remove the reverse listing feature.

Change-Id: I8d2a1f33d01ee70b79918d584b8c671f57eef2a0
2019-11-12 18:47:51 +00:00
Ivan Fraixedes
6516471cbc
uplink/storage/streams: Upload loop operations reorganization (#3429)
* uplink/storage/streams: Upload loop ops reorganization

  Reorganize the operations of the loop run by streamsStore.upload method
  for not doing unneeded computations on each iteration.

* uplink/storage/streams: Move out returns values declaration

  Move out return values declarations for those which aren't strictly
  needed due to defer statements nor documentation purpose.
2019-11-12 08:30:18 +01:00
littleskunk
7eb6724c92
logging: unify logging around satellite ID, node ID and piece ID (#3491)
* logging: unify logging around satellite ID, node ID and piece ID

* unify segment index
2019-11-05 22:04:07 +01:00
Michal Niewrzal
8786a37f89
uplink/storage: use Batch to optimize upload requests (#3408) 2019-10-29 08:49:16 -07:00
Ivan Fraixedes
14e661a1b0
uplink/storage/segments: non-functional improvements (#3400)
* uplink/storage/streams: Remove unused field of struct

  Remove an unused field from a struct type because it isn't used.

* uplink/storage/streams: Add end period in func doc

  Add missing end period in some functions documentation comments for
  following conventions.

* uplink/storage/segments: Replace switch by if
  
  Replace the switch statement of 2 branches one with a condition which
  returns and the second is the default by a if conditional because it's
  easy to read.
2019-10-29 15:39:17 +01:00
Ivan Fraixedes
4f281ef348
uplink: Refactor segments Store Get for metainfo Batch (#3362)
Refactoring of the segments Store interface Get method signature to
force the implementations to not use metainfo Client and be able for the
callers to batch requests.
2019-10-28 17:23:20 +01:00
Ivan Fraixedes
d9d82b0336 uplink: Reduce satellite request using Batch when possible (#3351)
* uplink/metainfo: Return classified Not Found error

Metainfo client Batch method must return the Storj Not Found error class
when the RCP server response with a not found status code as any other
metainfo Client method does.

Also if the error isn't Not Found one, it must wrap the error.

* uplink/storage/streams: Use Batch request in Delete

Change the 2 individual metainfo Client calls that streamStore Delete
method does by a single Batch one.
2019-10-24 14:18:48 -07:00
Michal Niewrzal
521c39bda0
uplink/metainfo: cleanup method names (#3315) 2019-10-22 23:59:56 -07:00
Jeff Wendling
098cbc9c67 all: use pkg/rpc instead of pkg/transport
all of the packages and tests work with both grpc and
drpc. we'll probably need to do some jenkins pipelines
to run the tests with drpc as well.

most of the changes are really due to a bit of cleanup
of the pkg/transport.Client api into an rpc.Dialer in
the spirit of a net.Dialer. now that we don't need
observers, we can pass around stateless configuration
to everything rather than stateful things that issue
observations. it also adds a DialAddressID for the
case where we don't have a pb.Node, but we do have an
address and want to assert some ID. this happened
pretty frequently, and now there's no more weird
contortions creating custom tls options, etc.

a lot of the other changes are being consistent/using
the abstractions in the rpc package to do rpc style
things like finding peer information, or checking
status codes.

Change-Id: Ief62875e21d80a21b3c56a5a37f45887679f9412
2019-09-25 15:37:06 -06:00
JT Olio
946ec201e2
metainfo: move api keys to part of the request (#3069)
What: we move api keys out of the grpc connection-level metadata on the client side and into the request protobufs directly. the server side still supports both mechanisms for backwards compatibility.

Why: dRPC won't support connection-level metadata. the only thing we currently use connection-level metadata for is api keys. we need to move all information needed by a request into the request protobuf itself for drpc support. check out the .proto changes for the main details.

One fun side-fact: Did you know that protobuf fields 1-15 are special and only use one byte for both the field number and type? Additionally did you know we don't use field 15 anywhere yet? So the new request header will use field 15, and should use field 15 on all protobufs going forward.

Please describe the tests: all existing tests should pass

Please describe the performance impact: none
2019-09-19 10:19:29 -06:00
Isaac Hess
5a50042c77
uplink/storage/streams: Add test for interrupted deletes (#3040)
* uplink/storage/streams: Add test for interrupted deletes

* uplink/storage/streams: Fix linting errors
2019-09-13 13:08:15 -06:00
Michal Niewrzal
64c467ffe7
uplink: integrate new Metainfo calls (#2640) 2019-09-10 08:39:47 -07:00
Michal Niewrzal
61168493dc
uplink: don't stop deleting segments on first error (#2943) 2019-09-05 14:25:30 +02:00
JT Olio
12d50ebb99
streams: don't encrypt segment count (#2859)
What: this change makes sure the count of segments is not encrypted.

Why: having the segment count encrypted just makes things hard for no reason - a satellite operator can figure out how many segments an object has by looking at the other segments in the database. but if a user has access but has lost their encryption key, they now can't clean up or delete old segments because they can't know how many there are without just guessing until they get errors. :(

Backwards compatibility: clients will still understand old pointers and will still write old pointers. at some point in the future perhaps we can do a migration for remaining old pointers so we can delete the old code.

Please describe the tests: covered by existing tests

Please describe the performance impact: none
2019-08-22 15:15:58 -06:00
Egon Elbre
ebbf0e1462
uplink/storage: don't import mock in production code (#2687) 2019-08-02 11:01:36 +03:00
Egon Elbre
5d0816430f
rename all the things (#2531)
* rename pkg/linksharing to linksharing
* rename pkg/httpserver to linksharing/httpserver
* rename pkg/eestream to uplink/eestream
* rename pkg/stream to uplink/stream
* rename pkg/metainfo/kvmetainfo to uplink/metainfo/kvmetainfo
* rename pkg/auth/signing to pkg/signing
* rename pkg/storage to uplink/storage
* rename pkg/accounting to satellite/accounting
* rename pkg/audit to satellite/audit
* rename pkg/certdb to satellite/certdb
* rename pkg/discovery to satellite/discovery
* rename pkg/overlay to satellite/overlay
* rename pkg/datarepair to satellite/repair
2019-07-28 08:55:36 +03:00