Commit Graph

129 Commits

Author SHA1 Message Date
Michal Niewrzal
ee55d14f35
mobile: create release script (#3197) 2019-10-17 08:39:34 -07:00
Egon Elbre
edf8318c0e
jenkins: update to golangci-lint 1.21.0 (#3289)
* jenkins: update to golint-1.21.0
* don't use underscores in var
* make handles private to satisfy linter

lib\uplinkc\universe.go:32:39: exported method Add returns unexported type storj.io/storj/lib/uplinkc._Ctype_long, which can be annoying to use (golint)
   func (m *Universe) Add(x interface{}) Handle {

* disable wsl explicitly
2019-10-17 10:42:22 +03:00
Michal Niewrzal
57ff0af6ba
mobile: add EncryptionRestrictions (#3260) 2019-10-15 01:26:09 -07:00
Jennifer Li Johnson
b185dbbee2
satellite/discovery: remove discovery related code (#3175) 2019-10-14 10:57:01 -04:00
Caleb Case
e82e61ad61
Fix typo in libuplink docs. (#3222) 2019-10-09 11:23:26 -04:00
TopperDEL
79f3ef4968 Add the tempDir as parameter (#3065) 2019-10-07 14:03:58 +02:00
Michal Niewrzal
6770a9ce1a
mobile: move to lib/uplink-gomobile (#3177) 2019-10-06 15:34:12 -07:00
Jeff Wendling
098cbc9c67 all: use pkg/rpc instead of pkg/transport
all of the packages and tests work with both grpc and
drpc. we'll probably need to do some jenkins pipelines
to run the tests with drpc as well.

most of the changes are really due to a bit of cleanup
of the pkg/transport.Client api into an rpc.Dialer in
the spirit of a net.Dialer. now that we don't need
observers, we can pass around stateless configuration
to everything rather than stateful things that issue
observations. it also adds a DialAddressID for the
case where we don't have a pb.Node, but we do have an
address and want to assert some ID. this happened
pretty frequently, and now there's no more weird
contortions creating custom tls options, etc.

a lot of the other changes are being consistent/using
the abstractions in the rpc package to do rpc style
things like finding peer information, or checking
status codes.

Change-Id: Ief62875e21d80a21b3c56a5a37f45887679f9412
2019-09-25 15:37:06 -06:00
JT Olio
946ec201e2
metainfo: move api keys to part of the request (#3069)
What: we move api keys out of the grpc connection-level metadata on the client side and into the request protobufs directly. the server side still supports both mechanisms for backwards compatibility.

Why: dRPC won't support connection-level metadata. the only thing we currently use connection-level metadata for is api keys. we need to move all information needed by a request into the request protobuf itself for drpc support. check out the .proto changes for the main details.

One fun side-fact: Did you know that protobuf fields 1-15 are special and only use one byte for both the field number and type? Additionally did you know we don't use field 15 anywhere yet? So the new request header will use field 15, and should use field 15 on all protobufs going forward.

Please describe the tests: all existing tests should pass

Please describe the performance impact: none
2019-09-19 10:19:29 -06:00
Michal Niewrzal
64c467ffe7
uplink: integrate new Metainfo calls (#2640) 2019-09-10 08:39:47 -07:00
Bryan White
4bcaa59122 add list direction option to bucket_test.c (#2961) 2019-09-05 14:34:35 +02:00
TopperDEL
e47f08afa0 lib/uplinkc: use ListDirection in struct BucketListOptions (#2946) 2019-09-05 09:25:27 +03:00
Egon Elbre
977472ed32 all: use fewer storage nodes to improve test memory usage (#2875)
* storagenode/inspector: use less storage nodes

* lib/uplinkc: use fewer storage nodes
2019-08-26 14:40:44 -04:00
Egon Elbre
d5667fbe35
lib/uplinkc: ensure it compiles on 32bit arch (#2835) 2019-08-26 16:12:26 +03:00
Isaac Hess
25154720bd
lib/uplink: remove redis and bolt dependencies (#2812)
* identity: remove redis and bolt dependencies

* identity: move revDB creation to main files
2019-08-19 16:10:38 -06:00
Jeff Wendling
e22c0bff94 lib/uplinkc: fix flaky download test (#2791)
The download driver code loops through the downloaded_data buffer
but doesn't ensure that it always passes a valid pointer to the
Go side. In particular, if the malloc'd memory ends against an
unmapped page, and the test passes a pointer one past the end
of the memory region, and since the Go side always dereferences
the pointer when creating a slice, it will attempt to read
unmapped memory, causing a segfault.

This bug doesn't always present. Indeed, it depends on the details
of your system's memory allocator. I validated that this could be
a cause of observed crashes on OS X by using mmap and mprotect
to do the allocations ensuring that the page directly after the
memory we use was unmapped/protected. The crash happened exactly
as seen, and was fixed by changing this condition in the while
loop.

Change-Id: I685dac07ff9b904097375dbf850f387450858753
2019-08-15 11:32:29 +03:00
Egon Elbre
9eba5ac631
lib/uplink: remove Seek method (#2768) 2019-08-13 20:29:02 +03:00
Bryan White
8e60ba269f
libuplink upload/download err handling improvements (#2725) 2019-08-07 16:28:27 +02:00
Bryan White
e5379d4f67
libuplink: Add size to ObjectInfo (#2714) 2019-08-06 11:36:35 +02:00
Jeff Wendling
21a3bf89ee cmd/uplink: use scopes to open (#2501)
What: Change cmd/uplink to use scopes

It moves the fields that will be subsumed by scopes into an explicit legacy section and hides their configuration flags.

Why: So that it can read scopes in from files and stuff
2019-08-05 11:01:20 -06:00
Bryan White
25823d433b add new line to end of require macro print statements (#2710) 2019-08-05 08:02:10 -04:00
Egon Elbre
ebbf0e1462
uplink/storage: don't import mock in production code (#2687) 2019-08-02 11:01:36 +03:00
Egon Elbre
369a51ed00 lib/uplink: ensure it's silent by default (#2676) 2019-08-01 07:14:09 -04:00
Egon Elbre
4cc6972396
lib/uplinkc: avoid allocating too much on stack and nicer names (#2659) 2019-07-30 19:51:37 +03:00
Bryan White
6b40866ed8
Re-implement libstorj API (V2) using libuplink (V3) (#2573) 2019-07-30 13:40:05 +02:00
Egon Elbre
5d0816430f
rename all the things (#2531)
* rename pkg/linksharing to linksharing
* rename pkg/httpserver to linksharing/httpserver
* rename pkg/eestream to uplink/eestream
* rename pkg/stream to uplink/stream
* rename pkg/metainfo/kvmetainfo to uplink/metainfo/kvmetainfo
* rename pkg/auth/signing to pkg/signing
* rename pkg/storage to uplink/storage
* rename pkg/accounting to satellite/accounting
* rename pkg/audit to satellite/audit
* rename pkg/certdb to satellite/certdb
* rename pkg/discovery to satellite/discovery
* rename pkg/overlay to satellite/overlay
* rename pkg/datarepair to satellite/repair
2019-07-28 08:55:36 +03:00
Dennis Coyle
5b6f3020fa
swapping noop to standard zap logger (#2634) 2019-07-25 12:45:22 -04:00
Isaac Hess
b8fe349816
Support empty path components (#2574) 2019-07-24 08:40:22 -06:00
Jess G
665a1e386f
close test project/uplink (#2605)
* close test project/uplink

* check err
2019-07-19 10:24:49 -07:00
aligeti
29b576961f
value attribution merge fix and more test cases (#2588)
* value attribution merge fix and more test cases
2019-07-19 11:17:34 -04:00
Simon Guindon
91f0adef10
Add the ability to set dial and request timeouts from the cmd/uplink CLI to libuplink. (#2439)
* Added the ability to pass timeout settings from cmd/uplink to libuplink.

* Removed commented out code.

* Updated 2min timeouts for the uplink CLI.

* Removed comment.

* Made transport defaultDialTimeout and defaultRequestTimeout public

* Added comments to describe where these defaults apply.

* Added a new defaults to libuplink and added tests.

* Added a new defaults to libuplink and added tests.
2019-07-18 11:13:59 -04:00
Jess G
f11bf46a11
Jg/1967 mv bucket metadata uplink (#2505)
* add bucketstore, add init uplink bucket

* update uplink to use bucket rpc

* fix tests

* wrap metainfo client errors

* add allowedBucket struct, fix tests

* update comment

* add paging

* updates per CR

* add test for pagination

* fix lint

* fix uplink test so its easier tyo understand

* fix gateway pagination bug

* changes per cr

* fix bug w/allowedBuckets, add test to catch
2019-07-12 08:57:02 -04:00
Jeff Wendling
671648012a Update password based key derivation design doc (#2367) 2019-07-09 00:07:03 -04:00
Jess G
f9696d6c5e
satellite/metainfo: add buckets RPC and database (#2460)
* add db interface and methods, add sa metainfo endpoints and svc

* add bucket metainfo svc funcs

* add sadb bucekts

* bucket list gets all buckets

* filter buckets list on macaroon restrictions

* update pb cipher suite to be enum

* add conversion funcs

* updates per comments

* bucket settings should say default

* add direction to list buckets, add tests

* fix test bucket names

* lint err

* only support forward direction

* add comments

* minor refactoring

* make sure list up to limit

* update test

* update protolock file

* fix lint

* change per PR
2019-07-08 15:32:18 -07:00
Egon Elbre
681f502b0e lib/uplink: ensure tests are compiled separately (#2440)
* fix uplink tests

* use consistent receiver names

Change-Id: Ifd5e86aa0a922538374718abfcb605049f5fb3ba

* ugh and actually use the receiver

Change-Id: Iae000ca5549c881ac025432c7722c58868c98e76

* avoid review comments

Change-Id: Id96370e5f2de8a9602d422d470027003ec89b8df
2019-07-05 15:02:49 -05:00
Jeff Wendling
8b07df37f5 refactor some store encryption stuff (#2434) 2019-07-05 11:36:35 +03:00
aligeti
ae8b9698f9
Rename/remove EncryptionScheme -> EncryptionParame… (#2363)
* rename/remove EncryptionScheme -> EncryptionParameters
2019-07-03 14:07:44 -04:00
Andrew Harding
0de4cf31cf
libuplink scope type (#2382) 2019-07-02 09:45:23 -06:00
Maximillian von Briesen
52e5a4eee3 pass logger into repairer and ecclient (#2365) 2019-07-02 13:08:02 +03:00
JT Olio
53e550f7d8
uplink docs (#2372)
* uplink docs

Change-Id: I67414c9e91b158ba1c120188aeb41b2907282431

* review

Change-Id: I34b2185e261e3425beb5099f0a161f988d920966
2019-06-28 07:40:37 -06:00
Kaloyan Raev
f3dc866889
Set correct default erasure share size for libuplink (#2376) 2019-06-28 13:23:59 +03:00
Jeff Wendling
1a65e42d33 rename EncryptionCtx back to EncryptionAccess (#2368)
* rename EncryptionCtx back to EncryptionAccess

Change-Id: I5e58915a59979ad6f2e83d36e191b2bbf3ba2ba2

* missed some stuff

Change-Id: Ib65fb186d7f854c0406e5fd7d11498e8941da59e

* oops protolock

Change-Id: I85a5ab4bafb9dd6a804d3dcd17a70811702f07e4

* retrigger

Change-Id: I2d0e3f02b4dbae0299b090e9617662b4437980b0
2019-06-27 23:18:24 -06:00
aligeti
811168e2c4
Uplink bucket attribution extension (#2353)
Attribution info added to the bucket info
2019-06-27 19:25:36 -04:00
Jeff Wendling
efcdaa43a3
lib/uplink: encryption context (#2349)
* lib/uplink: encryption context

Change-Id: I5c23dca3286a46b713b30c4997e9ae6e630b2280

* lib/uplink: bucket operation examples

Change-Id: Ia0f6e69f365dcff0cf11c731f51b30842bce053b

* lib/uplink: encryption key sharing test cases

Change-Id: I3a172d565f33f4e591402cdcb9460664a7cc7fbe

* fix encrypted path prefix restriction issue

Change-Id: I8f3921f9d52aaf4b84039de608b8cbbc88769554

* implement panics in libuplink encryption code

todo on cipher suite selection as well as an api concern

Change-Id: Ifa39eb3cc4b3443f7d96f9304df9b2ac4ec4085d

* implement GetProjectInfo api call to get salt

Change-Id: Ic5f6b3be9ea35df48c1aa214ab5d355fb328e2cf

* some fixes and accessors for encryption store

Change-Id: I3bb61f6712a037900e2a96e72ad4029ec1d3f718

* general fixes to builds/tests/etc

Change-Id: I9930fa96acb3b221d9a001f8e274af5729cc8a47

* java bindings changes

Change-Id: Ia2bd4c9c69739c8d3154d79616cff1f36fb403b6

* get libuplink examples passing

Change-Id: I828f09a144160e0a5dd932324f78491ae2ec8a07

* fix proto.lock file

Change-Id: I2fbbf4d0976a7d0473c2645e6dcb21aaa3be7651

* fix proto.lock again

Change-Id: I92702cf49e1a340eef6379c2be4f7c4a268112a9

* fix golint issues

Change-Id: I631ff9f43307a58e3b25a58cbb4a4cc2495f5eb6

* more linting fixes

Change-Id: I51f8f30b367b5bca14c94b15417b9a4c9e7aa0ce

* bug fixed by structs bump

Change-Id: Ibb03c691fce7606c35c08721b3ef0781ab48a38a

* retrigger

Change-Id: Ieee0470b6a2d07168a1578552e8e7f271ae93a13

* retrigger

Change-Id: I753d63853171e6a436c104ce176048892eb974c5

* semantic merge conflict

Change-Id: I9419448496de90340569047a6a16a1b858a7978a

* update total to match prod defaults

Change-Id: I693d55c1ebb28b5803ee1d26e9e198decf82308b

* retrigger

Change-Id: I28b74d5d6202f61aa3866fe407d423f6a0a14b9e

* retrigger

Change-Id: I6fd054885c715f602e2cef623fd464c42e88742c

* retrigger

Change-Id: I6a01bae88c72406d4ed5a8f13bf8a2b3c650bd2d
2019-06-27 17:36:51 +00:00
JT Olio
043a5b6fab lib/uplink: unexport bucket attribution check (#2352)
* lib/uplink: unexport bucket attribution check
2019-06-26 13:24:50 -04:00
aligeti
7facda4422
value attribution integration with libuplink (#2297)
* integrates the value attribution with libuplink
2019-06-26 12:22:01 -04:00
Egon Elbre
414648d660
Fix some metainfo.Client leaks (#2327) 2019-06-25 18:36:23 +03:00
Jeff Wendling
30f790a040
Create and use an encryption.Store (#2293)
* add path implementation

This commit adds a pkg/paths package which contains two types,
Encrypted and Unencrypted, to statically enforce what is contained
in a path. It's part of a refactoring of the code base to be more
clear about what is contained in a storj.Path at all the layers.

Change-Id: Ifc4d4932da26a97ea99749b8356b4543496a8864

* add encryption store

This change adds an encryption.Store type to keep a collection
of root keys for arbitrary locations in some buckets. It allows
one to look up all of the necessary information to encrypt paths,
decrypt paths and decrypt list operations.

It adds some exported functions to perform encryption on paths
using a Store.

Change-Id: I1a3d230c521d65f0ede727f93e1cb389f8be9497

* add shim around streams store

This commit changes no functionality, but just reorganizes the code
so that changes can be made directly to the streams store
implementation without affecting callers.

It also adds a Path type that will be used at the interface boundary
for the streams store so that it can be sure that it's getting well
formed paths that it expects.

Change-Id: I50bd682995b185beb653b00562fab62ef11f1ab5

* refactor streams to use encryption store

This commit changes the streams store to use the path type as
well as the encryption store to handle all of it's encryption
and decryption.

Some changes were made to how the default key is returned in
the encryption store to have it include the case when the bucket
exists but no paths matched. The path iterator could also be
simplified to not report if a consume was valid: that information
is no longer necessary.

The kvmetainfo tests were changed to appropriately pass the
subtests *testing.T rather than having the closure it executes
use the parent one. The test framework now correctly reports
which test did the failing.

There are still some latent issues with listing in that listing
for "a/" and listing for "a" are not the same operation, but we
treat them as such. I suspect that there are also issues with
paths like "/" or "//foo", but that's for another time.

Change-Id: I81cad4ba2850c3d14ba7e632777c4cac93db9472

* use an encryption store at the upper layers

Change-Id: Id9b4dd5f27b3ecac863de586e9ae076f4f927f6f

* fix linting failures

Change-Id: Ifb8378879ad308d4d047a0483850156371a41280

* fix linting in encryption test

Change-Id: Ia35647dfe18b0f20fe13763b28e53294f75c38fa

* get rid of kvmetainfo rootKey

Change-Id: Id795ca03d9417e3fe9634365a121430eb678d6d5

* Fix linting failure for return with else

Change-Id: I0b9ffd92be42ffcd8fef7ea735c5fc114a55d3b5

* fix some bugs adding enc store to kvmetainfo

Change-Id: I8e765970ba817289c65ec62971ae3bfa2c53a1ba

* respond to review feedback

Change-Id: I43e2ce29ce2fb6677b1cd6b9469838d80ec92c86
2019-06-24 19:23:07 +00:00
Jennifer Li Johnson
e285fe1997
Don't require encryption keys for project or bucket management (#2291) 2019-06-23 22:06:14 -04:00
Bryan White
9304817927
Uplink C bindings part 4 (#2260) 2019-06-21 20:44:00 +02:00
Bryan White
09e55ca28e
jenkins timeout workaround (#2285) 2019-06-21 16:04:34 +02:00
Bryan White
5f47b7028d Uplink C bindings part 3 (#2258)
* add object upload and download
2019-06-21 08:24:06 -04:00
JT Olio
aa25c4458f kvmetainfo: merge with storage/buckets (#2277) 2019-06-21 13:29:31 +02:00
JT Olio
06006effb3 lib/uplink: minor fixes (#2257) 2019-06-20 23:50:13 +02:00
Bryan White
d5f89afac8
fix merge resolution fail (#2248) 2019-06-19 16:51:06 +02:00
Jess G
f0f59a5577 Always encrypt inline segments without padding (#2183) 2019-06-19 11:11:27 +03:00
Bryan White
76061c50c5
Uplink C bindings part 2 (#2209) 2019-06-15 13:23:12 +02:00
Bryan White
bc33964729 Uplink C bindings part 1 (#2196)
* uplink cbindings tooling
2019-06-13 11:09:05 -04:00
JT Olio
ff7a9030eb lib/uplink: expose restrict on api keys (#2189) 2019-06-12 23:35:57 +02:00
Jess G
d84cd719ff
add const stripesPerBlock const to calc blocksize (#2163)
* add const stripesPerBlock, update comments in Pad, add speed to progressbar

* change size of erasure share size, update comments

* missing copyright

* update tests with stripesPerBlock
2019-06-11 10:14:05 -07:00
Michal Niewrzal
7a28decfca Always use difficulty larger than 8 for identites (#2160)
Set difficulty for libuplink and testplanet larger than 8, because we use last byte in the node id to store the version number, so the difficulty must be larger than 8.
2019-06-10 14:00:53 +02:00
Jess G
bf3d168cf6
makes sure all uplink cli configs get passed to libuplink, add stripeSize (#2103)
* makes sure all uplink cli configs get passed to libuplink, add stripSize

* update comment

* update defaults for uplink config blocksize

* changes per CR, update uplink config defaults

* pass shareSize from uplink config

* move block size validation to kvmeta pkg

* fix tests

* shareSize default 1k, rm config option blocksize

* rm printing err to stdout
2019-06-06 11:55:10 -07:00
JT Olio
d7f3a5f811
internal,lib,uplink: add monkit task to missing places (#2118)
* internal,lib,uplink: add monkit task to missing places

Change-Id: I490053eee4ed517502f9fe00c6394f0095bd13d0

* Include Monkit

* Add missing context

* Another missing ctx

* More ctx missing

* Linting

* go imports

Change-Id: Ibf0ed072eba339f027727ed8039f7bce1f223fa7

* fix semantic merge conflict

Change-Id: I67fb1f4e7b6cd5e89d69987ed7b3966b7d30ee37
2019-06-05 09:03:11 -06:00
Michal Niewrzal
7193b16e92
Java/Android libuplink bindings (#1918) 2019-05-24 10:13:01 +02:00
Jeff Wendling
15e74c8c3d uplink share subcommand (#1924)
* cmd/uplink: add share command to restrict an api key

This commit is an early bit of work to just implement restricting
macaroon api keys from the command line. It does not convert
api keys to be macaroons in general.

It also does not apply the path restriction caveats appropriately
yet because it does not encrypt them.

* cmd/uplink: fix path encryption for shares

It should now properly encrypt the path prefixes when adding
caveats to a macaroon.

* fix up linting problems

* print summary of caveat and require iso8601

* make clone part more clear
2019-05-14 12:15:12 -06:00
Kaloyan Raev
5add0a6b1b
Remove UseIdentity and IdentityVersion from libuplink volatile config (#1935) 2019-05-10 14:17:58 +03:00
aligeti
a83bb17417
updates the libuplink documentation with examples (#1801)
* updates the libuplink documentation with examples

* updated code review comments

* updated code review comments

* mods

* exmaple interface
2019-05-07 11:44:01 -04:00
Egon Elbre
d170ddc14f
avoid race in lib/uplink (#1852) 2019-04-26 20:15:41 +03:00
Michal Niewrzal
fe3dfc1587
Move pointerdb.Service to satellite (#1826) 2019-04-25 10:46:32 +02:00
Kaloyan Raev
8fc5fe1d6f
Refactor pb.Node protobuf (#1785) 2019-04-22 12:07:50 +03:00
JT Olio
d70f6e3760
libuplink: remove encryption key from project opening (#1761)
What: This change moves project-level bucket metadata encryption information to the volatile section, because it is unlikely to remain in future releases

Why: Ultimately, the web user interface will allow bucket management (creation, removal, etc), but not object management as that requires an encryption key for sure and we don't want to have users give the satellite their encryption keys.

At a high level, a (*Project) type should map to all of the things you can do inside the web user interface within a project, which by necessity cannot have an encryption key. So, we really don't want an encryption key in the non-volatile section of this library.
2019-04-16 11:29:33 -04:00
paul cannon
0ae0de75bc use SerializableMeta to store bucket attributes (#1658) 2019-04-10 18:27:04 -04:00
Egon Elbre
1330070209
fix potential truncation of int64 (#1718) 2019-04-10 06:36:27 -04:00
Bryan White
08b8d84248
Identity versioning fix (#1721) 2019-04-09 13:01:45 -04:00
JT Olio
9af4f26d43 libuplink changes for public usage (#1568)
Co-authored-by: paul cannon <thepaul@users.noreply.github.com>
Co-authored-by: Kaloyan Raev <kaloyan-raev@users.noreply.github.com>
2019-04-03 11:46:21 +03:00
Egon Elbre
de15a4fdcf
remove utils.CombineErrors and utils.ErrorGroup (#1603) 2019-03-29 14:30:23 +02:00
Egon Elbre
694b6dc1da
make tests run faster (#1553) 2019-03-22 15:14:17 +02:00
Kaloyan Raev
30dfc2b20c
Remove PointerDB client (#1520) 2019-03-22 11:01:49 +02:00
Dylan Lott
6bf46e80ee
Adds Libuplink (#1452)
* Merge in upstream

* Some initial wireup

* Added common.go file, more misc. work

* WIP adding identity in

* Get FullIdentity combined into Uplink

* Structure libuplink a little better

* Update some types and add some comments

* WIP uplink stuff

* Get uplink types and configs figured out

* add initial setup for tests, happy path is working

* Remove dependency from miniogw

* Adds miniogw code and wires it up correctly

* WIP working on getting test suite setup

* Uplink client now returns successfully and passes some initial happy path tets

* WIP trying to get v2 draft ready

* WIP

* WIP wiring up bucket methods and adjusting to some review feedback

* Getting closer to v2 libuplink draft

* CreateBucket now works and has tests to prove it

* Bucket tests are passing now

* removing some code

* Updates error handling and linter fixes

* Removes main_test

* Uploads and downloads are now working

* Rename BucketOpts to Encryption

* updates

* added test file back to git that was being ignored for some reason

* more test conditions

* changes Checksum in ObjectMeta struct to be type []byte

* linter fix

* Updates how encryption is passed through to bucket opts

* Updates encryption handling at bucket and access level

* Fixes imports

* Updates per code review
2019-03-20 09:43:53 -06:00