Commit Graph

24 Commits

Author SHA1 Message Date
Egon Elbre
b6ad3e9c9f
internal/testrand: new package for random data (#2282) 2019-06-26 13:38:51 +03:00
Jeff Wendling
30f790a040
Create and use an encryption.Store (#2293)
* add path implementation

This commit adds a pkg/paths package which contains two types,
Encrypted and Unencrypted, to statically enforce what is contained
in a path. It's part of a refactoring of the code base to be more
clear about what is contained in a storj.Path at all the layers.

Change-Id: Ifc4d4932da26a97ea99749b8356b4543496a8864

* add encryption store

This change adds an encryption.Store type to keep a collection
of root keys for arbitrary locations in some buckets. It allows
one to look up all of the necessary information to encrypt paths,
decrypt paths and decrypt list operations.

It adds some exported functions to perform encryption on paths
using a Store.

Change-Id: I1a3d230c521d65f0ede727f93e1cb389f8be9497

* add shim around streams store

This commit changes no functionality, but just reorganizes the code
so that changes can be made directly to the streams store
implementation without affecting callers.

It also adds a Path type that will be used at the interface boundary
for the streams store so that it can be sure that it's getting well
formed paths that it expects.

Change-Id: I50bd682995b185beb653b00562fab62ef11f1ab5

* refactor streams to use encryption store

This commit changes the streams store to use the path type as
well as the encryption store to handle all of it's encryption
and decryption.

Some changes were made to how the default key is returned in
the encryption store to have it include the case when the bucket
exists but no paths matched. The path iterator could also be
simplified to not report if a consume was valid: that information
is no longer necessary.

The kvmetainfo tests were changed to appropriately pass the
subtests *testing.T rather than having the closure it executes
use the parent one. The test framework now correctly reports
which test did the failing.

There are still some latent issues with listing in that listing
for "a/" and listing for "a" are not the same operation, but we
treat them as such. I suspect that there are also issues with
paths like "/" or "//foo", but that's for another time.

Change-Id: I81cad4ba2850c3d14ba7e632777c4cac93db9472

* use an encryption store at the upper layers

Change-Id: Id9b4dd5f27b3ecac863de586e9ae076f4f927f6f

* fix linting failures

Change-Id: Ifb8378879ad308d4d047a0483850156371a41280

* fix linting in encryption test

Change-Id: Ia35647dfe18b0f20fe13763b28e53294f75c38fa

* get rid of kvmetainfo rootKey

Change-Id: Id795ca03d9417e3fe9634365a121430eb678d6d5

* Fix linting failure for return with else

Change-Id: I0b9ffd92be42ffcd8fef7ea735c5fc114a55d3b5

* fix some bugs adding enc store to kvmetainfo

Change-Id: I8e765970ba817289c65ec62971ae3bfa2c53a1ba

* respond to review feedback

Change-Id: I43e2ce29ce2fb6677b1cd6b9469838d80ec92c86
2019-06-24 19:23:07 +00:00
JT Olio
aa25c4458f kvmetainfo: merge with storage/buckets (#2277) 2019-06-21 13:29:31 +02:00
Jess G
d84cd719ff
add const stripesPerBlock const to calc blocksize (#2163)
* add const stripesPerBlock, update comments in Pad, add speed to progressbar

* change size of erasure share size, update comments

* missing copyright

* update tests with stripesPerBlock
2019-06-11 10:14:05 -07:00
Jess G
bf3d168cf6
makes sure all uplink cli configs get passed to libuplink, add stripeSize (#2103)
* makes sure all uplink cli configs get passed to libuplink, add stripSize

* update comment

* update defaults for uplink config blocksize

* changes per CR, update uplink config defaults

* pass shareSize from uplink config

* move block size validation to kvmeta pkg

* fix tests

* shareSize default 1k, rm config option blocksize

* rm printing err to stdout
2019-06-06 11:55:10 -07:00
Kaloyan Raev
bfdee78f05
Introduce NodeDossier type and cleanup overlay.DB interface (#1626)
Co-authored-by: Natalie Villasana <navillasa@gmail.com>
Co-authored-by: Bill Thorp <bill3000@hotmail.com>
2019-04-04 19:34:36 +03:00
Egon Elbre
05d148aeb5
Storage node and upload/download protocol refactor (#1422)
refactor storage node server
refactor upload and download protocol
2019-03-18 12:55:06 +02:00
Egon Elbre
17a7f9d002
Enable planet Merging test (#1281) 2019-02-08 22:35:59 +02:00
Kaloyan Raev
4e738bbd44
Fix randomness in stopping nodes in TestGetObjectStream (#1277) 2019-02-08 15:20:12 +02:00
Egon Elbre
9c1e299f3c
Ensure everyone sees everyone else (#1275) 2019-02-08 11:25:13 +02:00
paul cannon
331f0cc845 don't put nil nodes in GetResponse (#1218) 2019-02-05 23:12:27 +02:00
Kaloyan Raev
dd76829d10
Improve logic for cutting the long tail during upload (#909) 2019-02-05 12:54:25 +02:00
Michal Niewrzal
2f8b3da1af
Upload/download for testplanet Uplink (#1183)
* Upload/download for testplanet Uplink

* check error in tests

* cleanup

* refactor node -> uplink

* add missing test file

* rest of refactoring

* workaround to resolve cycles in tests

* rename method

* add missing comments

* review comments

* use KiB
2019-02-04 17:56:10 +01:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Kaloyan Raev
72a20af569
Migrate Minio Gateway to the new Metainfo API (#775) 2018-12-07 20:31:29 +02:00
Kaloyan Raev
b7c45c7c16
Migrate Uplink CLI to Metainfo API (#745) 2018-12-04 18:46:02 +02:00
Kaloyan Raev
0018ebf63e
Object.Bucket as storj.Bucket instead of string (#747) 2018-12-03 16:38:03 +02:00
Kaloyan Raev
66476da1ae
metainfo: basic upload (#726) 2018-11-30 15:50:52 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Kaloyan Raev
67db26b36e
Metainfo stream download (#697) 2018-11-26 09:39:05 +02:00
Kaloyan Raev
aec2c41911
Metainfo remote segment test (#696) 2018-11-21 16:35:53 +02:00
Kaloyan Raev
8c56198090
Use designated error classes for common bucket and object errors (#691) 2018-11-21 13:17:28 +02:00
Kaloyan Raev
bb1cf151e3
Metainfo handle inline segments in ReadOnlyStream (#674) 2018-11-19 13:08:28 +02:00
Kaloyan Raev
0357e61bbd
metainfo objects tests (#662) 2018-11-15 17:31:33 +02:00