Commit Graph

47 Commits

Author SHA1 Message Date
Egon Elbre
0a69da4ff1 all: switch to storj.io/common/uuid
Change-Id: I178a0a8dac691e57bce317b91411292fb3c40c9f
2020-03-31 19:16:41 +03:00
Egon Elbre
8822e98c1f cmd/gateway: simplify module handling
Change-Id: If6ed158a6c9568fa33f69ca2d52e231ee4fcb0cb
2020-02-26 17:59:45 +00:00
Michal Niewrzal
cea4c25f53 mod: bump common and uplink version
Change-Id: Ia063d33c087dd91a46c008e154b078f11fa21527
2020-02-12 14:33:54 +00:00
Michal Niewrzal
2472554826 uplinkc: add ability to not set encryption restrictions with
restrict_scope

Change-Id: I445a945c9dad9d6b9ac5d973619914f50d973185
2020-02-12 11:56:13 +00:00
Michal Niewrzal
90fc1922d0 satellite/metainfo: override bucket RS values with satellite config
Satellite now is keeping RS values for uplink but old uplinks were using
default bucket settings. Because of that we need to override buckets
settings with satellite settings to avoid breaking older uplinks.

Change-Id: Ia1068db70e4adbf741c5e81d27d9e39799049c22
2020-01-28 15:51:04 +00:00
Bryan White
71ea2b0dc0 uplinkc: object test fix
require object creation time to be less than 60 seconds ago instead of less than or equal to 2

Change-Id: I3ec348e12ead1144524509092265b2b4d15109e3
2020-01-02 14:34:28 +00:00
Jeff Wendling
a35eb8027e lib/uplinkc: do some checks around conversions and allocations
ensure that every integer conversion fits into the destination
type, and for any unchecked ones, annotate why they are safe.
additionally, any integers we pass into slice headers need to
check that they are not negative.

all of our allocations should check for allocation failure and
return an error if there was a problem. even though an allocation
just failed, we don't pre-allocate the failure because we don't
want the callers to free it like they have to with the other
errors we return. maybe we should just panic.

Change-Id: Id4dfad802d312d35e565041efc9872453c90d793
2020-01-01 17:36:34 +00:00
Egon Elbre
6615ecc9b6 common: separate repository
Change-Id: Ibb89c42060450e3839481a7e495bbe3ad940610a
2019-12-27 14:11:15 +02:00
Egon Elbre
858f349a1a lib/uplinkc: add user agent
Change-Id: I6627851112823a72e4001b45d9a1b80ba76b8346
2019-12-20 13:11:25 +00:00
Michal Niewrzal
0e12c23431
uplinkc: add restrict_scope function (#3724) 2019-12-17 04:35:17 -08:00
Egon Elbre
fdd86d17ac lib/uplinkc: use SliceHeader instead of [1<<30]
Conversions such as will be invalid with Go 1.14

    (*[1<<30]byte)(unsafe.Pointer(data))

The recommended way is to use:

    *(*[]byte)(unsafe.Pointer(
        &reflect.SliceHeader{
            Data: uintptr(unsafe.Pointer(data)),
            Len: int(length),
            Cap: int(length),
        },
    ))

Also fixed a memory leak.

Change-Id: I1768b7b85505e6b57b49deb62a510474f1bf84c1
2019-12-13 16:49:41 +00:00
Cameron Ayer
6fae361c31 replace planet.Start in tests with planet.Run
planet.Start starts a testplanet system, whereas planet.Run starts a testplanet
and runs a test against it with each DB backend (cockroach compat).

Change-Id: I39c9da26d9619ee69a2b718d24ab00271f9e9bc2
2019-12-10 16:55:54 +00:00
Michal Niewrzal
f56107fc61
uplinkc: add download_range function (#3704) 2019-12-10 03:51:56 -08:00
Michal Niewrzal
de30d232cc uplinkc: add more functions to manage Scope (#3694) 2019-12-10 10:13:25 +01:00
Michal Niewrzal
b779826194 uplinkc: basic support for Scope (#3689) 2019-12-04 18:36:01 +01:00
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00
TopperDEL
39021b9641 lib/uplinkc: add additional fields to UplinkConfig (#3526) 2019-11-12 08:45:40 -08:00
Egon Elbre
f3e803203b
lib/uplinkc: add clarifying comments to download_read (#3525) 2019-11-11 16:11:37 +02:00
Egon Elbre
edf8318c0e
jenkins: update to golangci-lint 1.21.0 (#3289)
* jenkins: update to golint-1.21.0
* don't use underscores in var
* make handles private to satisfy linter

lib\uplinkc\universe.go:32:39: exported method Add returns unexported type storj.io/storj/lib/uplinkc._Ctype_long, which can be annoying to use (golint)
   func (m *Universe) Add(x interface{}) Handle {

* disable wsl explicitly
2019-10-17 10:42:22 +03:00
Jennifer Li Johnson
b185dbbee2
satellite/discovery: remove discovery related code (#3175) 2019-10-14 10:57:01 -04:00
TopperDEL
79f3ef4968 Add the tempDir as parameter (#3065) 2019-10-07 14:03:58 +02:00
JT Olio
946ec201e2
metainfo: move api keys to part of the request (#3069)
What: we move api keys out of the grpc connection-level metadata on the client side and into the request protobufs directly. the server side still supports both mechanisms for backwards compatibility.

Why: dRPC won't support connection-level metadata. the only thing we currently use connection-level metadata for is api keys. we need to move all information needed by a request into the request protobuf itself for drpc support. check out the .proto changes for the main details.

One fun side-fact: Did you know that protobuf fields 1-15 are special and only use one byte for both the field number and type? Additionally did you know we don't use field 15 anywhere yet? So the new request header will use field 15, and should use field 15 on all protobufs going forward.

Please describe the tests: all existing tests should pass

Please describe the performance impact: none
2019-09-19 10:19:29 -06:00
Bryan White
4bcaa59122 add list direction option to bucket_test.c (#2961) 2019-09-05 14:34:35 +02:00
TopperDEL
e47f08afa0 lib/uplinkc: use ListDirection in struct BucketListOptions (#2946) 2019-09-05 09:25:27 +03:00
Egon Elbre
977472ed32 all: use fewer storage nodes to improve test memory usage (#2875)
* storagenode/inspector: use less storage nodes

* lib/uplinkc: use fewer storage nodes
2019-08-26 14:40:44 -04:00
Egon Elbre
d5667fbe35
lib/uplinkc: ensure it compiles on 32bit arch (#2835) 2019-08-26 16:12:26 +03:00
Jeff Wendling
e22c0bff94 lib/uplinkc: fix flaky download test (#2791)
The download driver code loops through the downloaded_data buffer
but doesn't ensure that it always passes a valid pointer to the
Go side. In particular, if the malloc'd memory ends against an
unmapped page, and the test passes a pointer one past the end
of the memory region, and since the Go side always dereferences
the pointer when creating a slice, it will attempt to read
unmapped memory, causing a segfault.

This bug doesn't always present. Indeed, it depends on the details
of your system's memory allocator. I validated that this could be
a cause of observed crashes on OS X by using mmap and mprotect
to do the allocations ensuring that the page directly after the
memory we use was unmapped/protected. The crash happened exactly
as seen, and was fixed by changing this condition in the while
loop.

Change-Id: I685dac07ff9b904097375dbf850f387450858753
2019-08-15 11:32:29 +03:00
Egon Elbre
9eba5ac631
lib/uplink: remove Seek method (#2768) 2019-08-13 20:29:02 +03:00
Bryan White
8e60ba269f
libuplink upload/download err handling improvements (#2725) 2019-08-07 16:28:27 +02:00
Bryan White
e5379d4f67
libuplink: Add size to ObjectInfo (#2714) 2019-08-06 11:36:35 +02:00
Bryan White
25823d433b add new line to end of require macro print statements (#2710) 2019-08-05 08:02:10 -04:00
Egon Elbre
ebbf0e1462
uplink/storage: don't import mock in production code (#2687) 2019-08-02 11:01:36 +03:00
Egon Elbre
369a51ed00 lib/uplink: ensure it's silent by default (#2676) 2019-08-01 07:14:09 -04:00
Egon Elbre
4cc6972396
lib/uplinkc: avoid allocating too much on stack and nicer names (#2659) 2019-07-30 19:51:37 +03:00
Bryan White
6b40866ed8
Re-implement libstorj API (V2) using libuplink (V3) (#2573) 2019-07-30 13:40:05 +02:00
Jess G
f11bf46a11
Jg/1967 mv bucket metadata uplink (#2505)
* add bucketstore, add init uplink bucket

* update uplink to use bucket rpc

* fix tests

* wrap metainfo client errors

* add allowedBucket struct, fix tests

* update comment

* add paging

* updates per CR

* add test for pagination

* fix lint

* fix uplink test so its easier tyo understand

* fix gateway pagination bug

* changes per cr

* fix bug w/allowedBuckets, add test to catch
2019-07-12 08:57:02 -04:00
Jess G
f9696d6c5e
satellite/metainfo: add buckets RPC and database (#2460)
* add db interface and methods, add sa metainfo endpoints and svc

* add bucket metainfo svc funcs

* add sadb bucekts

* bucket list gets all buckets

* filter buckets list on macaroon restrictions

* update pb cipher suite to be enum

* add conversion funcs

* updates per comments

* bucket settings should say default

* add direction to list buckets, add tests

* fix test bucket names

* lint err

* only support forward direction

* add comments

* minor refactoring

* make sure list up to limit

* update test

* update protolock file

* fix lint

* change per PR
2019-07-08 15:32:18 -07:00
aligeti
ae8b9698f9
Rename/remove EncryptionScheme -> EncryptionParame… (#2363)
* rename/remove EncryptionScheme -> EncryptionParameters
2019-07-03 14:07:44 -04:00
Jeff Wendling
1a65e42d33 rename EncryptionCtx back to EncryptionAccess (#2368)
* rename EncryptionCtx back to EncryptionAccess

Change-Id: I5e58915a59979ad6f2e83d36e191b2bbf3ba2ba2

* missed some stuff

Change-Id: Ib65fb186d7f854c0406e5fd7d11498e8941da59e

* oops protolock

Change-Id: I85a5ab4bafb9dd6a804d3dcd17a70811702f07e4

* retrigger

Change-Id: I2d0e3f02b4dbae0299b090e9617662b4437980b0
2019-06-27 23:18:24 -06:00
Jeff Wendling
efcdaa43a3
lib/uplink: encryption context (#2349)
* lib/uplink: encryption context

Change-Id: I5c23dca3286a46b713b30c4997e9ae6e630b2280

* lib/uplink: bucket operation examples

Change-Id: Ia0f6e69f365dcff0cf11c731f51b30842bce053b

* lib/uplink: encryption key sharing test cases

Change-Id: I3a172d565f33f4e591402cdcb9460664a7cc7fbe

* fix encrypted path prefix restriction issue

Change-Id: I8f3921f9d52aaf4b84039de608b8cbbc88769554

* implement panics in libuplink encryption code

todo on cipher suite selection as well as an api concern

Change-Id: Ifa39eb3cc4b3443f7d96f9304df9b2ac4ec4085d

* implement GetProjectInfo api call to get salt

Change-Id: Ic5f6b3be9ea35df48c1aa214ab5d355fb328e2cf

* some fixes and accessors for encryption store

Change-Id: I3bb61f6712a037900e2a96e72ad4029ec1d3f718

* general fixes to builds/tests/etc

Change-Id: I9930fa96acb3b221d9a001f8e274af5729cc8a47

* java bindings changes

Change-Id: Ia2bd4c9c69739c8d3154d79616cff1f36fb403b6

* get libuplink examples passing

Change-Id: I828f09a144160e0a5dd932324f78491ae2ec8a07

* fix proto.lock file

Change-Id: I2fbbf4d0976a7d0473c2645e6dcb21aaa3be7651

* fix proto.lock again

Change-Id: I92702cf49e1a340eef6379c2be4f7c4a268112a9

* fix golint issues

Change-Id: I631ff9f43307a58e3b25a58cbb4a4cc2495f5eb6

* more linting fixes

Change-Id: I51f8f30b367b5bca14c94b15417b9a4c9e7aa0ce

* bug fixed by structs bump

Change-Id: Ibb03c691fce7606c35c08721b3ef0781ab48a38a

* retrigger

Change-Id: Ieee0470b6a2d07168a1578552e8e7f271ae93a13

* retrigger

Change-Id: I753d63853171e6a436c104ce176048892eb974c5

* semantic merge conflict

Change-Id: I9419448496de90340569047a6a16a1b858a7978a

* update total to match prod defaults

Change-Id: I693d55c1ebb28b5803ee1d26e9e198decf82308b

* retrigger

Change-Id: I28b74d5d6202f61aa3866fe407d423f6a0a14b9e

* retrigger

Change-Id: I6fd054885c715f602e2cef623fd464c42e88742c

* retrigger

Change-Id: I6a01bae88c72406d4ed5a8f13bf8a2b3c650bd2d
2019-06-27 17:36:51 +00:00
Jennifer Li Johnson
e285fe1997
Don't require encryption keys for project or bucket management (#2291) 2019-06-23 22:06:14 -04:00
Bryan White
9304817927
Uplink C bindings part 4 (#2260) 2019-06-21 20:44:00 +02:00
Bryan White
09e55ca28e
jenkins timeout workaround (#2285) 2019-06-21 16:04:34 +02:00
Bryan White
5f47b7028d Uplink C bindings part 3 (#2258)
* add object upload and download
2019-06-21 08:24:06 -04:00
Bryan White
d5f89afac8
fix merge resolution fail (#2248) 2019-06-19 16:51:06 +02:00
Bryan White
76061c50c5
Uplink C bindings part 2 (#2209) 2019-06-15 13:23:12 +02:00
Bryan White
bc33964729 Uplink C bindings part 1 (#2196)
* uplink cbindings tooling
2019-06-13 11:09:05 -04:00