Commit Graph

53 Commits

Author SHA1 Message Date
Egon Elbre
6502143e79
fix import ordering (#2322) 2019-06-25 12:46:29 +03:00
Jeff Wendling
30f790a040
Create and use an encryption.Store (#2293)
* add path implementation

This commit adds a pkg/paths package which contains two types,
Encrypted and Unencrypted, to statically enforce what is contained
in a path. It's part of a refactoring of the code base to be more
clear about what is contained in a storj.Path at all the layers.

Change-Id: Ifc4d4932da26a97ea99749b8356b4543496a8864

* add encryption store

This change adds an encryption.Store type to keep a collection
of root keys for arbitrary locations in some buckets. It allows
one to look up all of the necessary information to encrypt paths,
decrypt paths and decrypt list operations.

It adds some exported functions to perform encryption on paths
using a Store.

Change-Id: I1a3d230c521d65f0ede727f93e1cb389f8be9497

* add shim around streams store

This commit changes no functionality, but just reorganizes the code
so that changes can be made directly to the streams store
implementation without affecting callers.

It also adds a Path type that will be used at the interface boundary
for the streams store so that it can be sure that it's getting well
formed paths that it expects.

Change-Id: I50bd682995b185beb653b00562fab62ef11f1ab5

* refactor streams to use encryption store

This commit changes the streams store to use the path type as
well as the encryption store to handle all of it's encryption
and decryption.

Some changes were made to how the default key is returned in
the encryption store to have it include the case when the bucket
exists but no paths matched. The path iterator could also be
simplified to not report if a consume was valid: that information
is no longer necessary.

The kvmetainfo tests were changed to appropriately pass the
subtests *testing.T rather than having the closure it executes
use the parent one. The test framework now correctly reports
which test did the failing.

There are still some latent issues with listing in that listing
for "a/" and listing for "a" are not the same operation, but we
treat them as such. I suspect that there are also issues with
paths like "/" or "//foo", but that's for another time.

Change-Id: I81cad4ba2850c3d14ba7e632777c4cac93db9472

* use an encryption store at the upper layers

Change-Id: Id9b4dd5f27b3ecac863de586e9ae076f4f927f6f

* fix linting failures

Change-Id: Ifb8378879ad308d4d047a0483850156371a41280

* fix linting in encryption test

Change-Id: Ia35647dfe18b0f20fe13763b28e53294f75c38fa

* get rid of kvmetainfo rootKey

Change-Id: Id795ca03d9417e3fe9634365a121430eb678d6d5

* Fix linting failure for return with else

Change-Id: I0b9ffd92be42ffcd8fef7ea735c5fc114a55d3b5

* fix some bugs adding enc store to kvmetainfo

Change-Id: I8e765970ba817289c65ec62971ae3bfa2c53a1ba

* respond to review feedback

Change-Id: I43e2ce29ce2fb6677b1cd6b9469838d80ec92c86
2019-06-24 19:23:07 +00:00
Michal Niewrzal
fdeb834801
Bucket name validation (#2244) 2019-06-24 11:52:25 +02:00
Jennifer Li Johnson
e285fe1997
Don't require encryption keys for project or bucket management (#2291) 2019-06-23 22:06:14 -04:00
JT Olio
aa25c4458f kvmetainfo: merge with storage/buckets (#2277) 2019-06-21 13:29:31 +02:00
JT Olio
06006effb3 lib/uplink: minor fixes (#2257) 2019-06-20 23:50:13 +02:00
Jess G
f0f59a5577 Always encrypt inline segments without padding (#2183) 2019-06-19 11:11:27 +03:00
Bryan White
bc33964729 Uplink C bindings part 1 (#2196)
* uplink cbindings tooling
2019-06-13 11:09:05 -04:00
Jess G
d84cd719ff
add const stripesPerBlock const to calc blocksize (#2163)
* add const stripesPerBlock, update comments in Pad, add speed to progressbar

* change size of erasure share size, update comments

* missing copyright

* update tests with stripesPerBlock
2019-06-11 10:14:05 -07:00
Jess G
bf3d168cf6
makes sure all uplink cli configs get passed to libuplink, add stripeSize (#2103)
* makes sure all uplink cli configs get passed to libuplink, add stripSize

* update comment

* update defaults for uplink config blocksize

* changes per CR, update uplink config defaults

* pass shareSize from uplink config

* move block size validation to kvmeta pkg

* fix tests

* shareSize default 1k, rm config option blocksize

* rm printing err to stdout
2019-06-06 11:55:10 -07:00
JT Olio
9c5708da32 pkg/*: add monkit task to missing places (#2109) 2019-06-04 13:36:27 +02:00
Jeff Wendling
1bd52b9f90 server side macaroons (#1945)
What: Adds macaroon support to the server side

Why: So that api keys are now macaroons
2019-05-24 10:51:27 -06:00
Kaloyan Raev
8fc5fe1d6f
Refactor pb.Node protobuf (#1785) 2019-04-22 12:07:50 +03:00
paul cannon
0ae0de75bc use SerializableMeta to store bucket attributes (#1658) 2019-04-10 18:27:04 -04:00
Kaloyan Raev
bfdee78f05
Introduce NodeDossier type and cleanup overlay.DB interface (#1626)
Co-authored-by: Natalie Villasana <navillasa@gmail.com>
Co-authored-by: Bill Thorp <bill3000@hotmail.com>
2019-04-04 19:34:36 +03:00
JT Olio
9af4f26d43 libuplink changes for public usage (#1568)
Co-authored-by: paul cannon <thepaul@users.noreply.github.com>
Co-authored-by: Kaloyan Raev <kaloyan-raev@users.noreply.github.com>
2019-04-03 11:46:21 +03:00
paul cannon
e4a70e3fac
plumb EncryptionScheme, RedundancyScheme through to buckets (#1638)
We want to use those fields in the bucket-level Pointer objects as
bucket defaults, but we need to be able to get at them first.

I don't see any strong reason not to make these available, except
that it was kind of a pain.
2019-04-02 15:15:31 -06:00
Kaloyan Raev
30dfc2b20c
Remove PointerDB client (#1520) 2019-03-22 11:01:49 +02:00
Egon Elbre
05d148aeb5
Storage node and upload/download protocol refactor (#1422)
refactor storage node server
refactor upload and download protocol
2019-03-18 12:55:06 +02:00
JT Olio
2a59679766 pkg/transport: require tls configuration for dialing (#1286)
* separate TLS options from server options (because we need them for dialing too)
* stop creating transports in multiple places
* ensure that we actually check revocation, whitelists, certificate signing, etc, for all connections.
2019-02-11 13:17:32 +02:00
Egon Elbre
17a7f9d002
Enable planet Merging test (#1281) 2019-02-08 22:35:59 +02:00
Kaloyan Raev
4e738bbd44
Fix randomness in stopping nodes in TestGetObjectStream (#1277) 2019-02-08 15:20:12 +02:00
Egon Elbre
9c1e299f3c
Ensure everyone sees everyone else (#1275) 2019-02-08 11:25:13 +02:00
paul cannon
331f0cc845 don't put nil nodes in GetResponse (#1218) 2019-02-05 23:12:27 +02:00
Yaroslav Vorobiov
2ff0d9d435
Add bucket to project relationship on satellite (#1143)
* integrate console api keys with buckets in pointerdb

* fix test

* fix tests kvmetainfo

* linter fix

* disable account activation

* fix test

* review fixes

* fix comments

* little refactoring

* remove debug println

* fix typo

* disable activation in a propper way

* fix test

* fix imports

* fix uplink count in testplanet

* move key creation to planet.newUplink
2019-02-05 19:22:17 +02:00
Kaloyan Raev
dd76829d10
Improve logic for cutting the long tail during upload (#909) 2019-02-05 12:54:25 +02:00
Michal Niewrzal
2f8b3da1af
Upload/download for testplanet Uplink (#1183)
* Upload/download for testplanet Uplink

* check error in tests

* cleanup

* refactor node -> uplink

* add missing test file

* rest of refactoring

* workaround to resolve cycles in tests

* rename method

* add missing comments

* review comments

* use KiB
2019-02-04 17:56:10 +01:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Egon Elbre
99d3b7a3c8
Fix import grouping (#1111) 2019-01-22 17:48:23 +02:00
Natalie Villasana
17c60c1f06
moves node selection config setup from uplink to satellite (#891) 2018-12-17 16:05:05 -05:00
Natalie Villasana
9e1ec97b31
adds node selection config (#782) 2018-12-11 12:30:14 -05:00
Kaloyan Raev
72a20af569
Migrate Minio Gateway to the new Metainfo API (#775) 2018-12-07 20:31:29 +02:00
Kaloyan Raev
b7c45c7c16
Migrate Uplink CLI to Metainfo API (#745) 2018-12-04 18:46:02 +02:00
Kaloyan Raev
0018ebf63e
Object.Bucket as storj.Bucket instead of string (#747) 2018-12-03 16:38:03 +02:00
Kaloyan Raev
66476da1ae
metainfo: basic upload (#726) 2018-11-30 15:50:52 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Michal Niewrzal
8dd669b37f
Bandwidth allocations need method designation (GET/PUT) (#708)
* Bandwidth allocations need method designation (GET/PUT)

* add method comment

* goimports
2018-11-26 19:21:44 +01:00
Kaloyan Raev
67db26b36e
Metainfo stream download (#697) 2018-11-26 09:39:05 +02:00
Kaloyan Raev
aec2c41911
Metainfo remote segment test (#696) 2018-11-21 16:35:53 +02:00
Kaloyan Raev
8c56198090
Use designated error classes for common bucket and object errors (#691) 2018-11-21 13:17:28 +02:00
Bryan White
dabd87cb36
swap golang/protobuf out for gogo/protobuf (#692)
* swap golang/protobuf out for gogo/protobuf

* revert go.mod and go.sum changes

* linter fixes
2018-11-20 19:29:07 +01:00
Bryan White
54ccb460a9
rename size fields in protobuf messages for gogo compatibility (#690)
* rename `size` fields in protobuf messages for gogo compatibility

* linter fixes
2018-11-20 18:09:35 +01:00
Kaloyan Raev
bb1cf151e3
Metainfo handle inline segments in ReadOnlyStream (#674) 2018-11-19 13:08:28 +02:00
Kaloyan Raev
f730ce451a
Telemetry in metainfo implementation (#670) 2018-11-16 15:59:27 +02:00
Kaloyan Raev
0357e61bbd
metainfo objects tests (#662) 2018-11-15 17:31:33 +02:00
Egon Elbre
21a102a05f
testplanet: add logger (#649) 2018-11-15 10:57:47 +02:00
Egon Elbre
17519a7532 metainfo: Fix listing paths (#645) 2018-11-14 12:40:53 +02:00
Kaloyan Raev
5cdb03515f
Configure path encryption in metainfo API (#644) 2018-11-14 11:26:18 +02:00
Kaloyan Raev
514ca50611
Configurable encryption type for paths (#637) 2018-11-13 14:21:52 +02:00
Kaloyan Raev
9afa9b6b52
Metainfo buckets tests (#607) 2018-11-12 15:23:19 +02:00