Commit Graph

117 Commits

Author SHA1 Message Date
Egon Elbre
5a4745eddb all: remove usages of testplanet.New
Ensure that tests use testplanet.Run, so we always require running
against all database backends.

Change-Id: I6b0209e6a4912cf3328bd35b2c31bb8598930acb
2020-01-22 22:42:57 +02:00
Michal Niewrzal
6502454947 satellite/metainfo: move RS configuration to satellite
With this change RS configuration will be set on satellite. Uplink with
get RS values with BeginObject request and will use it. For backward
compatibility and to avoid super large change redundancy scheme stored
with bucket is not touched. This can be done in future.

Change-Id: Ia5f76fc10c37e2c44e4f7b8754f28eafe1f97eff
2020-01-22 09:33:53 +00:00
Michal Niewrzal
86f194769f uplink: adjust to changes in storj/uplink
This change is adjusting code base to changes in storj/uplink.

https://review.dev.storj.io/c/storj/uplink/+/643

Change-Id: Ieca87f9f5983e391bf4b4fec8b9d5491fd32bfa1
2020-01-20 22:06:19 +00:00
Michal Niewrzal
b579c260ab cmd: rename "scope" flag to "access"
We decided that better name for "scope" will be "access". This change
refactors cmd part of code but don't touch libuplink. For backward
compatibility old configs with "scope" field will be loaded without any
issue. Old flag "scope" won't be supported directly from command line.

https://storjlabs.atlassian.net/browse/V3-3488

Change-Id: I349d6971c798380d147937c91e887edb5e9ae4aa
2020-01-10 15:27:53 +00:00
Natalie Ventura Villasana
131c4d94f0 {cmd/uplink, lib/uplink}: change RS total to 110
Bump the uplink's RS Total value from 95 to 110.

Change-Id: I208beed17b5f53fe359847e0469a29fec80a4f7e
2020-01-10 14:27:32 +00:00
Egon Elbre
082ec81714
uplink: move to storj.io/uplink (#3746) 2020-01-08 15:40:19 +02:00
Egon Elbre
3f38a7ed1e lib/uplink: revert library move
Moving lib/uplink actually makes implementing the new library harder.
Move the package back where it was.

Change-Id: If758e897e3e56155a48ae82b23904cd8cdd72c13
2020-01-07 18:39:15 +00:00
Egon Elbre
6098f606a2 lib/uplink: move to uplink
Setup aliases in lib/uplink and set uplink as the default.

Change-Id: Ic06b5f3d59fe402faaed2143fdf4b2314e3d06b9
2020-01-02 17:56:16 +00:00
Bryan White
71ea2b0dc0 uplinkc: object test fix
require object creation time to be less than 60 seconds ago instead of less than or equal to 2

Change-Id: I3ec348e12ead1144524509092265b2b4d15109e3
2020-01-02 14:34:28 +00:00
Jeff Wendling
a35eb8027e lib/uplinkc: do some checks around conversions and allocations
ensure that every integer conversion fits into the destination
type, and for any unchecked ones, annotate why they are safe.
additionally, any integers we pass into slice headers need to
check that they are not negative.

all of our allocations should check for allocation failure and
return an error if there was a problem. even though an allocation
just failed, we don't pre-allocate the failure because we don't
want the callers to free it like they have to with the other
errors we return. maybe we should just panic.

Change-Id: Id4dfad802d312d35e565041efc9872453c90d793
2020-01-01 17:36:34 +00:00
Egon Elbre
6615ecc9b6 common: separate repository
Change-Id: Ibb89c42060450e3839481a7e495bbe3ad940610a
2019-12-27 14:11:15 +02:00
Egon Elbre
858f349a1a lib/uplinkc: add user agent
Change-Id: I6627851112823a72e4001b45d9a1b80ba76b8346
2019-12-20 13:11:25 +00:00
Egon Elbre
1eaf9e9ed7 pkg/storj: move non-common types
Change-Id: I2dd15c95cd334660f29a528dfb3dec9c30b29cab
2019-12-18 14:14:53 +00:00
Egon Elbre
69dc514fcb lib/uplink: fix test data race
cfg was being modified concurrently.

Change-Id: I431f152e7e3f72582f1a7d39967c2496e0e0dc1e
2019-12-17 15:41:46 +00:00
Michal Niewrzal
0e12c23431
uplinkc: add restrict_scope function (#3724) 2019-12-17 04:35:17 -08:00
Egon Elbre
fdd86d17ac lib/uplinkc: use SliceHeader instead of [1<<30]
Conversions such as will be invalid with Go 1.14

    (*[1<<30]byte)(unsafe.Pointer(data))

The recommended way is to use:

    *(*[]byte)(unsafe.Pointer(
        &reflect.SliceHeader{
            Data: uintptr(unsafe.Pointer(data)),
            Len: int(length),
            Cap: int(length),
        },
    ))

Also fixed a memory leak.

Change-Id: I1768b7b85505e6b57b49deb62a510474f1bf84c1
2019-12-13 16:49:41 +00:00
Cameron Ayer
6fae361c31 replace planet.Start in tests with planet.Run
planet.Start starts a testplanet system, whereas planet.Run starts a testplanet
and runs a test against it with each DB backend (cockroach compat).

Change-Id: I39c9da26d9619ee69a2b718d24ab00271f9e9bc2
2019-12-10 16:55:54 +00:00
Michal Niewrzal
f56107fc61
uplinkc: add download_range function (#3704) 2019-12-10 03:51:56 -08:00
Michal Niewrzal
de30d232cc uplinkc: add more functions to manage Scope (#3694) 2019-12-10 10:13:25 +01:00
Michal Niewrzal
ffd570eb04 uplink/metainfo: remove additional GetObject from download
This change removes one additional metainfo.GetObject call during
download.

Change-Id: I86b4e5165f299069e3e18944fe79c697c6a514d3
2019-12-09 11:48:48 +00:00
Michal Niewrzal
b779826194 uplinkc: basic support for Scope (#3689) 2019-12-04 18:36:01 +01:00
Egon Elbre
36fead0093 satellite/metainfo: add UserAgent support to endpoints (#3548) 2019-11-26 03:12:37 -08:00
JT Olio
01895d8bdd lib/uplink: explain safe versions (#3644)
Change-Id: Idf7391474ab10b0dab6ce97c773cabee59d5b664
2019-11-25 16:55:54 +01:00
JT Olio
031ba86de5
argon2: choose a steady parallelism value (#3630)
* argon2: choose a steady parallelism value

Change-Id: I6006da7d7980cda88f5f08ee759612df23a8132d

* whoops, not cruft

Change-Id: Ied9039f9a9be1d0f6ff3c7d5c4839a83fc7b4b1f

* fix broken test file

Change-Id: I07288cd6cef32ba387f2f003febff5c297e50997

* fix linting error

Change-Id: Icdbda8b709cc100a86f3859303c40edb8dff1e0f
2019-11-22 14:00:04 -07:00
Michal Niewrzal
bc16cb5d24
libuplink: remove additional GetBucket for upload/download (#3568) 2019-11-15 02:06:17 -08:00
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00
Egon Elbre
1e64006e32 lint: add staticcheck as a separate step (#3569) 2019-11-14 10:31:30 +02:00
TopperDEL
39021b9641 lib/uplinkc: add additional fields to UplinkConfig (#3526) 2019-11-12 08:45:40 -08:00
Egon Elbre
f3e803203b
lib/uplinkc: add clarifying comments to download_read (#3525) 2019-11-11 16:11:37 +02:00
Ivan Fraixedes
e4a220347a
uplink: Suppress one metainfo call on delete (#3511)
Change signature of metainfo DeleteObject to get rid of an extra call to
kvmetainfo GetBucket method and eliminate one round trip to the
satellite when deleting objects.
2019-11-07 10:39:40 +01:00
Maximillian von Briesen
d9bb25b4b9 satellite/metainfo: support a wider range of values for RS.Total in satellite metainfo validation (#3431)
change uplink RS default configuration from 130 to 95
2019-10-31 15:04:33 -04:00
Michal Niewrzal
acc7b116aa
scripts: use postgres script with all tests (#3404) 2019-10-31 07:03:54 -07:00
Michal Niewrzal
8786a37f89
uplink/storage: use Batch to optimize upload requests (#3408) 2019-10-29 08:49:16 -07:00
Michal Niewrzal
7f6893ea86
satellite/metainfo: fix broken object listing (#3348) 2019-10-24 14:05:08 -07:00
Michal Niewrzal
521c39bda0
uplink/metainfo: cleanup method names (#3315) 2019-10-22 23:59:56 -07:00
JT Olio
2c6fa3c5f8
pkg/rpc: remove read/write deadlines as a mechanism for request timeouts (#3335)
libuplink was incorrectly setting timeouts to 10 seconds still, but
should have been at least 10 minutes. the order sender was setting them
to 1 hour. we don't want timeouts in uplink-side logic as it establishes
a minimum rate on tcp streams.

instead of all of this, just use tcp keep alive. tcp keep alive packets are
sent every 15 seconds and if the peer stops responding the connection
dies. this is enabled by default with go. this will kill tcp connections
when they stop working.

Change-Id: I3d7ad49f71950b3eb43044eedf4b17993116045b
2019-10-22 17:57:24 -06:00
Kaloyan Raev
97018d6d64
Expose range download in gomobile (#3336) 2019-10-22 14:51:01 +03:00
Kaloyan Raev
244fea774b
gomobile Reader.Read should take offset and length params (#3326) 2019-10-21 18:17:31 +03:00
Michal Niewrzal
ee55d14f35
mobile: create release script (#3197) 2019-10-17 08:39:34 -07:00
Egon Elbre
edf8318c0e
jenkins: update to golangci-lint 1.21.0 (#3289)
* jenkins: update to golint-1.21.0
* don't use underscores in var
* make handles private to satisfy linter

lib\uplinkc\universe.go:32:39: exported method Add returns unexported type storj.io/storj/lib/uplinkc._Ctype_long, which can be annoying to use (golint)
   func (m *Universe) Add(x interface{}) Handle {

* disable wsl explicitly
2019-10-17 10:42:22 +03:00
Michal Niewrzal
57ff0af6ba
mobile: add EncryptionRestrictions (#3260) 2019-10-15 01:26:09 -07:00
Jennifer Li Johnson
b185dbbee2
satellite/discovery: remove discovery related code (#3175) 2019-10-14 10:57:01 -04:00
Caleb Case
e82e61ad61
Fix typo in libuplink docs. (#3222) 2019-10-09 11:23:26 -04:00
TopperDEL
79f3ef4968 Add the tempDir as parameter (#3065) 2019-10-07 14:03:58 +02:00
Michal Niewrzal
6770a9ce1a
mobile: move to lib/uplink-gomobile (#3177) 2019-10-06 15:34:12 -07:00
Jeff Wendling
098cbc9c67 all: use pkg/rpc instead of pkg/transport
all of the packages and tests work with both grpc and
drpc. we'll probably need to do some jenkins pipelines
to run the tests with drpc as well.

most of the changes are really due to a bit of cleanup
of the pkg/transport.Client api into an rpc.Dialer in
the spirit of a net.Dialer. now that we don't need
observers, we can pass around stateless configuration
to everything rather than stateful things that issue
observations. it also adds a DialAddressID for the
case where we don't have a pb.Node, but we do have an
address and want to assert some ID. this happened
pretty frequently, and now there's no more weird
contortions creating custom tls options, etc.

a lot of the other changes are being consistent/using
the abstractions in the rpc package to do rpc style
things like finding peer information, or checking
status codes.

Change-Id: Ief62875e21d80a21b3c56a5a37f45887679f9412
2019-09-25 15:37:06 -06:00
JT Olio
946ec201e2
metainfo: move api keys to part of the request (#3069)
What: we move api keys out of the grpc connection-level metadata on the client side and into the request protobufs directly. the server side still supports both mechanisms for backwards compatibility.

Why: dRPC won't support connection-level metadata. the only thing we currently use connection-level metadata for is api keys. we need to move all information needed by a request into the request protobuf itself for drpc support. check out the .proto changes for the main details.

One fun side-fact: Did you know that protobuf fields 1-15 are special and only use one byte for both the field number and type? Additionally did you know we don't use field 15 anywhere yet? So the new request header will use field 15, and should use field 15 on all protobufs going forward.

Please describe the tests: all existing tests should pass

Please describe the performance impact: none
2019-09-19 10:19:29 -06:00
Michal Niewrzal
64c467ffe7
uplink: integrate new Metainfo calls (#2640) 2019-09-10 08:39:47 -07:00
Bryan White
4bcaa59122 add list direction option to bucket_test.c (#2961) 2019-09-05 14:34:35 +02:00
TopperDEL
e47f08afa0 lib/uplinkc: use ListDirection in struct BucketListOptions (#2946) 2019-09-05 09:25:27 +03:00