Commit Graph

38 Commits

Author SHA1 Message Date
Jeff Wendling
8b07df37f5 refactor some store encryption stuff (#2434) 2019-07-05 11:36:35 +03:00
aligeti
ae8b9698f9
Rename/remove EncryptionScheme -> EncryptionParame… (#2363)
* rename/remove EncryptionScheme -> EncryptionParameters
2019-07-03 14:07:44 -04:00
Jeff Wendling
c35c8e4c24 allow reading bucket metadata with restricted keys (#2321)
Change-Id: I47d3a2f5f02744ae6c51d54963cdf2dff24134e2
2019-06-24 17:36:09 -06:00
Jeff Wendling
30f790a040
Create and use an encryption.Store (#2293)
* add path implementation

This commit adds a pkg/paths package which contains two types,
Encrypted and Unencrypted, to statically enforce what is contained
in a path. It's part of a refactoring of the code base to be more
clear about what is contained in a storj.Path at all the layers.

Change-Id: Ifc4d4932da26a97ea99749b8356b4543496a8864

* add encryption store

This change adds an encryption.Store type to keep a collection
of root keys for arbitrary locations in some buckets. It allows
one to look up all of the necessary information to encrypt paths,
decrypt paths and decrypt list operations.

It adds some exported functions to perform encryption on paths
using a Store.

Change-Id: I1a3d230c521d65f0ede727f93e1cb389f8be9497

* add shim around streams store

This commit changes no functionality, but just reorganizes the code
so that changes can be made directly to the streams store
implementation without affecting callers.

It also adds a Path type that will be used at the interface boundary
for the streams store so that it can be sure that it's getting well
formed paths that it expects.

Change-Id: I50bd682995b185beb653b00562fab62ef11f1ab5

* refactor streams to use encryption store

This commit changes the streams store to use the path type as
well as the encryption store to handle all of it's encryption
and decryption.

Some changes were made to how the default key is returned in
the encryption store to have it include the case when the bucket
exists but no paths matched. The path iterator could also be
simplified to not report if a consume was valid: that information
is no longer necessary.

The kvmetainfo tests were changed to appropriately pass the
subtests *testing.T rather than having the closure it executes
use the parent one. The test framework now correctly reports
which test did the failing.

There are still some latent issues with listing in that listing
for "a/" and listing for "a" are not the same operation, but we
treat them as such. I suspect that there are also issues with
paths like "/" or "//foo", but that's for another time.

Change-Id: I81cad4ba2850c3d14ba7e632777c4cac93db9472

* use an encryption store at the upper layers

Change-Id: Id9b4dd5f27b3ecac863de586e9ae076f4f927f6f

* fix linting failures

Change-Id: Ifb8378879ad308d4d047a0483850156371a41280

* fix linting in encryption test

Change-Id: Ia35647dfe18b0f20fe13763b28e53294f75c38fa

* get rid of kvmetainfo rootKey

Change-Id: Id795ca03d9417e3fe9634365a121430eb678d6d5

* Fix linting failure for return with else

Change-Id: I0b9ffd92be42ffcd8fef7ea735c5fc114a55d3b5

* fix some bugs adding enc store to kvmetainfo

Change-Id: I8e765970ba817289c65ec62971ae3bfa2c53a1ba

* respond to review feedback

Change-Id: I43e2ce29ce2fb6677b1cd6b9469838d80ec92c86
2019-06-24 19:23:07 +00:00
Jess G
f0f59a5577 Always encrypt inline segments without padding (#2183) 2019-06-19 11:11:27 +03:00
JT Olio
9c5708da32 pkg/*: add monkit task to missing places (#2109) 2019-06-04 13:36:27 +02:00
paul cannon
0ae0de75bc use SerializableMeta to store bucket attributes (#1658) 2019-04-10 18:27:04 -04:00
paul cannon
e4a70e3fac
plumb EncryptionScheme, RedundancyScheme through to buckets (#1638)
We want to use those fields in the bucket-level Pointer objects as
bucket defaults, but we need to be able to get at them first.

I don't see any strong reason not to make these available, except
that it was kind of a pain.
2019-04-02 15:15:31 -06:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Bryan White
dabd87cb36
swap golang/protobuf out for gogo/protobuf (#692)
* swap golang/protobuf out for gogo/protobuf

* revert go.mod and go.sum changes

* linter fixes
2018-11-20 19:29:07 +01:00
Kaloyan Raev
514ca50611
Configurable encryption type for paths (#637) 2018-11-13 14:21:52 +02:00
Egon Elbre
eb5e46e336
metainfo readonly stream (#580) 2018-11-06 13:40:06 +02:00
Kaloyan Raev
99640225fd
Refactor Path type (#522)
The old paths.Path type is now replaced with the new storj.Path.

storj.Path is simply an alias to the built-in string type. As such it can be used just as any string, which simplifies a lot working with paths. No more conversions paths.New and path.String().

As an alias storj.Path does not define any methods. However, any functions applying to strings (like those from the strings package) gracefully apply to storj.Path too. In addition we have a few more functions defined:

    storj.SplitPath
    storj.JoinPaths
    encryption.EncryptPath
    encryption.DecryptPath
    encryption.DerivePathKey
    encryption.DeriveContentKey

All code in master is migrated to the new storj.Path type.

The Path example is also updated and is good for reference: /pkg/encryption/examples_test.go

This PR also resolve a nonce misuse issue in path encryption: https://storjlabs.atlassian.net/browse/V3-545
2018-10-25 23:28:16 +03:00
aligeti
a61de403d7
aws CLI cancel handling development (#511)
Handle CTRL+C while using AWS CLI #511 (Storj V3/V3-487)
2018-10-19 10:36:43 -04:00
Kaloyan Raev
ac4805a841
Adapt storj and encryption packages (#513) 2018-10-19 16:38:13 +03:00
Kaloyan Raev
b9e473a5eb
Encapsulate key encryption in the Key and EncryptedPrivateKey types (#499) 2018-10-18 17:21:08 +03:00
Kaloyan Raev
b34adc310b
Extract encryption package from eestream (#497) 2018-10-18 14:10:29 +03:00
nfarah86
92a8a825c4 Encrypt metadata (#462) 2018-10-17 14:34:50 +03:00
Egon Elbre
2dbada73ab
Separate storj.io packages in imports (#481)
Format with goimports -w -local storj.io
Add flag to golang lint
Remove commented out imports as they mess with goimports
2018-10-16 14:43:44 +03:00
Kaloyan Raev
6eafd60945
Fix nonce reuse for encrypting the encryption key (#474) 2018-10-15 21:58:57 +03:00
Maximillian von Briesen
3551b34ea7
Path Encryption (#448)
* begin adding path encryption

* do not encrypt/decrypt first element of path (bucket)

* add path encryption for delete and list

* use encrypted paths in streamstore.Meta

* fix listing with encrypted paths

* move encrypt/decryptAfterBucket to streamstore

* fix listing with no prefix

* remove duplicate logic for listing with no prefix
2018-10-15 11:39:09 -04:00
aligeti
0b4a6188e3
multiple uploads of same file. handled the logic in stream store by deleting first before starting uploading (#412) 2018-10-08 10:19:54 -04:00
aligeti
cf329b1f37
Handle CTRL+C and clean up the partial segments and pieces (#381)
* merged the lasted master changes

* debug working of handling ctrl+c

* Handling of clean up of partially uploaded segments and pieces

* code cleanup per code comment

* updates based on code review comments
2018-10-04 09:52:12 -04:00
Kaloyan Raev
e2d745fe8f
Clean up last segment handling (#408)
* Clean up last segment handling

* Fix increment for AES-GCM nonce

* Fix stream size calculation

* Adapt stream store tests

* Fix Delete method

* Rename info callback to segmentInfo

* Clearer calculation for offset in Nonce.AESGCMNonce()

* Adapt to the new little-endian nonce increment
2018-10-03 16:05:40 +03:00
James Hagans
19ec28f700
Streams Store mock test (#373)
* add basic test for Stream Store
2018-10-01 21:48:11 -04:00
Maximillian von Briesen
821d0b6f1d
Stream encryption (#302)
* begin adding encryption for remote pieces

* begin adding decryption

* add encryption key as arg to Put and Get

* move encryption/decryption to object store

* Add encryption key to object store constructor

* Add the erasure scheme to object store constructor

* Ensure decrypter is initialized with the stripe size used by encrypter

* Revert "Ensure decrypter is initialized with the stripe size used by encrypter"

This reverts commit 07272333f461606edfb43ad106cc152f37a3bd46.

* Revert "Add the erasure scheme to object store constructor"

This reverts commit ea5e793b536159d993b96e3db69a37c1656a193c.

* move encryption to stream store

* move decryption stuff to stream store

* revert changes in object store

* add encryptedBlockSize and close rangers on error during Get

* calculate padding sizes correctly

* encryptedBlockSize -> encryptionBlockSize

* pass encryption key and block size into stream store

* remove encryption key and block size from object store constructor

* move encrypter/decrypter initialization

* remove unnecessary cast

* Fix padding issue

* Fix linter

* add todos

* use random encryption key for data encryption. Store an encrypted copy of this key in segment metadata

* use different encryption key for each segment

* encrypt data in one step if it is small enough

* refactor and move encryption stuff

* fix errors related to nil slices passed to copy

* fix encrypter vs. decrypter bug

* put encryption stuff in eestream

* get captplanet test to pass

* fix linting errors

* add types for encryption keys/nonces and clean up

* fix tests

* more review changes

* add Cipher type for encryption stuff

* fix rs_test

* Simplify type casting of key and nonce

* Init starting nonce to the segment index

* don't copy derived key

* remove default encryption key; force user to explicitly set it

* move getSegmentPath to streams package

* dont require user to specify encryption key for captplanet

* rename GenericKey and GenericNonce to Key and Nonce

* review changes

* fix linting error

* Download uses the encryption type from metadata

* Store enc block size in metadata and use it for download
2018-09-26 09:32:23 -04:00
Egon Elbre
b6b6111173
Flatten proto definitions into a single package (#360)
* protos: move streams to pb
* protos: move overlay to pb
* protos: move pointerdb to pb
* protos: move piecestore to pb
* fix statdb import naming
2018-09-18 07:39:06 +03:00
Kaloyan Raev
ee5f32b0f7
ConcatRanger closes segment readers as soon as data is read (#349)
* ConcatRanger closes a segment readers as soon as data is read

* Fix linter errors

* Fix data race

* Nicer call to CombineErrors
2018-09-14 10:10:43 -04:00
Kaloyan Raev
8f4b29abd0
Lazy segment ranger (#343) 2018-09-12 09:06:44 -04:00
Michal Niewrzal
e90c2140b8 Show file size for uplink ls command (#344)
* Show file size for uplink ls command

* align size to right

* Fix stream store list method to handle meta.Size flag
2018-09-12 09:03:31 -04:00
Kaloyan Raev
f62a3c383c
Filter prefix from list results (#328) 2018-09-09 09:34:23 +03:00
Egon Elbre
4671ac6c5b Abort trying to upload segments when reader fails (#326)
* abort trying to upload segments when reader fails

* also return the error when it fails
2018-09-08 09:41:40 -06:00
Egon Elbre
0a2839b2d7
Add IsPrefix support to piecestore (#313) 2018-09-07 17:20:15 +03:00
JT Olio
a4d1070d68
storage/ec: fix some Get socket leaks (#308)
* storage/ec: fix some Get socket leaks

* fix linters
2018-09-05 10:08:39 -07:00
James Hagans
fdfa6e85c8 Streamstore impl of Stor interface (#191)
* first stab at PUT

* only PUT

* working on PUT

* Put with LimitReader

* start of Get

* reorder of files and proto meta

* working on Meta

* working on Meta

* add aware limit reader

* add size from segment put

* rm if for eof

* update to proto meta

* update gen proto file

* working on get

* working on get

* working on get

* working on list

* working on delete

* working on list

* working on meta method

* fix merge error and working on feedback from PR

* update to proto file

* rm size tuple

* mv eof limit reader to new file

* add toMeta

* rm varible names

* add updates from PR feedback

* updates from PR feedback

* updates from PR feedback

* add toMeta size based on total size

* update toMeta size calculation

* rm passthrough

* add default to config for segment size

* fix get method ranger bug

* add object support for nested stream proto

* rm nested stream meta data

* rm test for another PR
2018-08-23 21:56:38 -06:00
Kaloyan Raev
7e136db9cf
Cleanup metadata across layers (#180)
* Cleanup metadata across layers

* Fix pointer db tests
2018-07-30 21:57:50 +03:00
JT Olio
d986c337aa streamstore: add passthrough implementation (#176)
this doesn't implement streamstore, this just allows us to try and
get the june demo working again in the meantime
2018-07-27 12:13:08 -06:00
Kaloyan Raev
8ce889ed1b
ObjectStore (#128)
* WIP ObjectStore

* Remove methods for extended attributes

* List returns metadata too

* No real need to prepend "object" in path

* Serialize metadata

* List retuns []ListItem instead of []Path
2018-07-16 23:44:28 +03:00