Commit Graph

141 Commits

Author SHA1 Message Date
Egon Elbre
ed627144ed all: use DialNodeURL throughout the codebase
Change-Id: Iaf9ae3aeef7305c937f2660c929744db2d88776c
2020-05-20 10:36:30 +00:00
Jeff Wendling
13baacc479 lib/uplink: better diagnostics with invalid encryption access
Fixes #3841

Change-Id: I965b8fbb8fd221c740c2fc1421d66721c046f038
2020-05-05 09:45:57 +00:00
Egon Elbre
d225e2de48 all: add missing ctx.Cleanup calls in tests
Change-Id: Iaa65f90b9731d721691322bb92fc3da736aa10fe
2020-04-29 17:58:40 +00:00
Egon Elbre
9052085f70 private/testplanet: simplify uplink usage
Change-Id: I3e488dc296f1094ce95e6d6597ca6d3f8da90a76
2020-04-16 16:45:55 +00:00
JT Olio
d08b2de558 lib/uplink: deprecate docs
guess what docs you get if you google "storj uplink"?

Change-Id: I600f1811f74a516d224189872f97727d57a153b1
2020-04-13 21:00:31 -06:00
Egon Elbre
11a44cdd88 all: don't depend on gogo/proto directly
Change-Id: I8822dea0d1b7b99e0b828e0373a0308a42dde2be
2020-04-08 17:32:15 +00:00
Michal Niewrzal
d23e03f8a2 lib/uplink-gomobile: extend encryption key capabilities
Change-Id: Id10b14cc34ff983ec451440fe4b0305651115022
2020-04-07 06:58:08 +00:00
Michal Niewrzal
b17dc656b5 lib/uplink-gomobile: update to latest master
Change-Id: I1be7991ee73cc2a6e911ae024dce3fba97c02071
2020-04-04 19:08:48 +00:00
Egon Elbre
6492b13d81 all: remove old uuid
Change-Id: I3a137f73456f010c37d3933dbe12cbbb840b809f
2020-04-02 19:30:36 +03:00
Egon Elbre
1024bf9ce1 all: simplify uuid usage
Instead of uuid.Parse, use uuid.FromString.
This removes a bunch of pointer management logic.

Change-Id: Id25bd174eb43c71d00b450158a198abafd8958f2
2020-04-02 13:45:19 +00:00
Egon Elbre
0a69da4ff1 all: switch to storj.io/common/uuid
Change-Id: I178a0a8dac691e57bce317b91411292fb3c40c9f
2020-03-31 19:16:41 +03:00
Egon Elbre
1d79228ed0 satellite/metainfo: support uplink useragent
Adds support for parsing user agent and specifying uplink version and
it's library dependencies.

Change-Id: Ibaddde4deb93e153ac05c91b676c5b5f1ae1aa37
2020-03-31 15:11:31 +00:00
Kaloyan Raev
10b032e484 libuplink: return deleted bucket/object (step 4)
Switch back to the original DeleteBucket and DeleteObject methods.

Next step: remove the DeleteBucketReturnDeleted and
DeleteObjectReturnDeleted from storj.io/uplink.

Change-Id: I273a305326d411e51ce354ce72fcc6ecadf4dd5f
2020-03-19 13:32:07 +02:00
Kaloyan Raev
78b253c774 libuplink: return deleted bucket/object (step 2)
step 1 in https://review.dev.storj.io/c/storj/uplink/+/1236

Now the old libuplink uses the temporary DeleteBucketReturnDeleted and
DeleteObjectReturnDeleted methods. This way, in the next step, we will
be able to change the DeleteBucket and DeleteObject methods to return
the deleted bucket/object.

Change-Id: I2e638be1960bca6ce1456c92849fcdd6d93e5252
2020-03-18 17:26:23 +00:00
Bill Thorp
94c11c5212 satellite: remove some unnecessary UTC() calls
Fixes some easy cases of extraneous UTC() calls

Change-Id: I3f4c287ae622a455b9a492a8892a699e0710ca9a
2020-03-13 13:49:44 +00:00
Michal Niewrzal
9f390f37da satellite/metainfo: return default ciphers (path and encryption) for old
uplinks

New libuplink is not storing encryption values in with bucket but old
uplinks are using those values for configuration. If bucket was created
with new libuplink we will send back satellite defaults.

Change-Id: Ie1bf3682847e07b302270b4c4bf1a7219f4bf011
2020-03-05 10:04:50 +00:00
Egon Elbre
8822e98c1f cmd/gateway: simplify module handling
Change-Id: If6ed158a6c9568fa33f69ca2d52e231ee4fcb0cb
2020-02-26 17:59:45 +00:00
Egon Elbre
5342dd9fe6 go.mod: update uplink
Change-Id: I867a6a1eef8aa5d60bb676e5112b98c4192ce811
2020-02-21 16:08:12 +02:00
Michal Niewrzal
4fb70b4383 lib/uplink: use empty encryption access for restricting
Change-Id: Idc0c5ceb577bc2321e419b90dee83c4264f324e3
2020-02-19 11:30:51 +00:00
Michal Niewrzal
cea4c25f53 mod: bump common and uplink version
Change-Id: Ia063d33c087dd91a46c008e154b078f11fa21527
2020-02-12 14:33:54 +00:00
Michal Niewrzal
2472554826 uplinkc: add ability to not set encryption restrictions with
restrict_scope

Change-Id: I445a945c9dad9d6b9ac5d973619914f50d973185
2020-02-12 11:56:13 +00:00
Jeff Wendling
7999d24f81 all: use monkit v3
this commit updates our monkit dependency to the v3 version where
it outputs in an influx style. this makes discovery much easier
as many tools are built to look at it this way.

graphite and rothko will suffer some due to no longer being a tree
based on dots. hopefully time will exist to update rothko to
index based on the new metric format.

it adds an influx output for the statreceiver so that we can
write to influxdb v1 or v2 directly.

Change-Id: Iae9f9494a6d29cfbd1f932a5e71a891b490415ff
2020-02-05 23:53:17 +00:00
Michal Niewrzal
a181e0b627 libuplink: adjust tests to changes in encryption store
We move PathCipher to encryption.Store and we need to adjust
storj/uplink for those changes. Uplink repo is also using libuplink to
run tests so we need first adjust storj/storj libuplink and later
storj/uplink.

Change-Id: I84f23e6bad18ac139f72c19939dc526f9f46d88b
2020-01-30 22:00:24 +00:00
Michal Niewrzal
90fc1922d0 satellite/metainfo: override bucket RS values with satellite config
Satellite now is keeping RS values for uplink but old uplinks were using
default bucket settings. Because of that we need to override buckets
settings with satellite settings to avoid breaking older uplinks.

Change-Id: Ia1068db70e4adbf741c5e81d27d9e39799049c22
2020-01-28 15:51:04 +00:00
Egon Elbre
5a4745eddb all: remove usages of testplanet.New
Ensure that tests use testplanet.Run, so we always require running
against all database backends.

Change-Id: I6b0209e6a4912cf3328bd35b2c31bb8598930acb
2020-01-22 22:42:57 +02:00
Michal Niewrzal
6502454947 satellite/metainfo: move RS configuration to satellite
With this change RS configuration will be set on satellite. Uplink with
get RS values with BeginObject request and will use it. For backward
compatibility and to avoid super large change redundancy scheme stored
with bucket is not touched. This can be done in future.

Change-Id: Ia5f76fc10c37e2c44e4f7b8754f28eafe1f97eff
2020-01-22 09:33:53 +00:00
Michal Niewrzal
86f194769f uplink: adjust to changes in storj/uplink
This change is adjusting code base to changes in storj/uplink.

https://review.dev.storj.io/c/storj/uplink/+/643

Change-Id: Ieca87f9f5983e391bf4b4fec8b9d5491fd32bfa1
2020-01-20 22:06:19 +00:00
Michal Niewrzal
b579c260ab cmd: rename "scope" flag to "access"
We decided that better name for "scope" will be "access". This change
refactors cmd part of code but don't touch libuplink. For backward
compatibility old configs with "scope" field will be loaded without any
issue. Old flag "scope" won't be supported directly from command line.

https://storjlabs.atlassian.net/browse/V3-3488

Change-Id: I349d6971c798380d147937c91e887edb5e9ae4aa
2020-01-10 15:27:53 +00:00
Natalie Ventura Villasana
131c4d94f0 {cmd/uplink, lib/uplink}: change RS total to 110
Bump the uplink's RS Total value from 95 to 110.

Change-Id: I208beed17b5f53fe359847e0469a29fec80a4f7e
2020-01-10 14:27:32 +00:00
Egon Elbre
082ec81714
uplink: move to storj.io/uplink (#3746) 2020-01-08 15:40:19 +02:00
Egon Elbre
3f38a7ed1e lib/uplink: revert library move
Moving lib/uplink actually makes implementing the new library harder.
Move the package back where it was.

Change-Id: If758e897e3e56155a48ae82b23904cd8cdd72c13
2020-01-07 18:39:15 +00:00
Egon Elbre
6098f606a2 lib/uplink: move to uplink
Setup aliases in lib/uplink and set uplink as the default.

Change-Id: Ic06b5f3d59fe402faaed2143fdf4b2314e3d06b9
2020-01-02 17:56:16 +00:00
Bryan White
71ea2b0dc0 uplinkc: object test fix
require object creation time to be less than 60 seconds ago instead of less than or equal to 2

Change-Id: I3ec348e12ead1144524509092265b2b4d15109e3
2020-01-02 14:34:28 +00:00
Jeff Wendling
a35eb8027e lib/uplinkc: do some checks around conversions and allocations
ensure that every integer conversion fits into the destination
type, and for any unchecked ones, annotate why they are safe.
additionally, any integers we pass into slice headers need to
check that they are not negative.

all of our allocations should check for allocation failure and
return an error if there was a problem. even though an allocation
just failed, we don't pre-allocate the failure because we don't
want the callers to free it like they have to with the other
errors we return. maybe we should just panic.

Change-Id: Id4dfad802d312d35e565041efc9872453c90d793
2020-01-01 17:36:34 +00:00
Egon Elbre
6615ecc9b6 common: separate repository
Change-Id: Ibb89c42060450e3839481a7e495bbe3ad940610a
2019-12-27 14:11:15 +02:00
Egon Elbre
858f349a1a lib/uplinkc: add user agent
Change-Id: I6627851112823a72e4001b45d9a1b80ba76b8346
2019-12-20 13:11:25 +00:00
Egon Elbre
1eaf9e9ed7 pkg/storj: move non-common types
Change-Id: I2dd15c95cd334660f29a528dfb3dec9c30b29cab
2019-12-18 14:14:53 +00:00
Egon Elbre
69dc514fcb lib/uplink: fix test data race
cfg was being modified concurrently.

Change-Id: I431f152e7e3f72582f1a7d39967c2496e0e0dc1e
2019-12-17 15:41:46 +00:00
Michal Niewrzal
0e12c23431
uplinkc: add restrict_scope function (#3724) 2019-12-17 04:35:17 -08:00
Egon Elbre
fdd86d17ac lib/uplinkc: use SliceHeader instead of [1<<30]
Conversions such as will be invalid with Go 1.14

    (*[1<<30]byte)(unsafe.Pointer(data))

The recommended way is to use:

    *(*[]byte)(unsafe.Pointer(
        &reflect.SliceHeader{
            Data: uintptr(unsafe.Pointer(data)),
            Len: int(length),
            Cap: int(length),
        },
    ))

Also fixed a memory leak.

Change-Id: I1768b7b85505e6b57b49deb62a510474f1bf84c1
2019-12-13 16:49:41 +00:00
Cameron Ayer
6fae361c31 replace planet.Start in tests with planet.Run
planet.Start starts a testplanet system, whereas planet.Run starts a testplanet
and runs a test against it with each DB backend (cockroach compat).

Change-Id: I39c9da26d9619ee69a2b718d24ab00271f9e9bc2
2019-12-10 16:55:54 +00:00
Michal Niewrzal
f56107fc61
uplinkc: add download_range function (#3704) 2019-12-10 03:51:56 -08:00
Michal Niewrzal
de30d232cc uplinkc: add more functions to manage Scope (#3694) 2019-12-10 10:13:25 +01:00
Michal Niewrzal
ffd570eb04 uplink/metainfo: remove additional GetObject from download
This change removes one additional metainfo.GetObject call during
download.

Change-Id: I86b4e5165f299069e3e18944fe79c697c6a514d3
2019-12-09 11:48:48 +00:00
Michal Niewrzal
b779826194 uplinkc: basic support for Scope (#3689) 2019-12-04 18:36:01 +01:00
Egon Elbre
36fead0093 satellite/metainfo: add UserAgent support to endpoints (#3548) 2019-11-26 03:12:37 -08:00
JT Olio
01895d8bdd lib/uplink: explain safe versions (#3644)
Change-Id: Idf7391474ab10b0dab6ce97c773cabee59d5b664
2019-11-25 16:55:54 +01:00
JT Olio
031ba86de5
argon2: choose a steady parallelism value (#3630)
* argon2: choose a steady parallelism value

Change-Id: I6006da7d7980cda88f5f08ee759612df23a8132d

* whoops, not cruft

Change-Id: Ied9039f9a9be1d0f6ff3c7d5c4839a83fc7b4b1f

* fix broken test file

Change-Id: I07288cd6cef32ba387f2f003febff5c297e50997

* fix linting error

Change-Id: Icdbda8b709cc100a86f3859303c40edb8dff1e0f
2019-11-22 14:00:04 -07:00
Michal Niewrzal
bc16cb5d24
libuplink: remove additional GetBucket for upload/download (#3568) 2019-11-15 02:06:17 -08:00
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00