Commit Graph

437 Commits

Author SHA1 Message Date
Márton Elek
d23cfa6445 go.mod: bump storj.io/common and storj.io/uplink
Change-Id: Ie78b69f2ee855e9e45ef241d4b8f8c225502b8a7
2022-08-23 17:47:08 +00:00
Sean Harvey
38d2d0f2fc
mod: bump storj/private and storj/common dependencies
Change-Id: I73ce29b1a6c28d6eec1570dc6568052b16502c61
2022-08-19 15:08:50 +12:00
Márton Elek
0698b1ef88 go.mod: bump storj/private version
Change-Id: I72ff0a6d8b8d929f0753988e447bf2b41552199c
2022-08-11 09:08:32 +00:00
Jeff Wendling
f343b856dd go.mod: bump common/private/monkit/monkit-jaeger
this includes all of the fixes identified for end to end
tracing to finally work.

Change-Id: Ia433a2cf9130dc43b6810379af32f5c10f9d3888
2022-07-27 16:17:00 +00:00
Qweder93
cf12802a12 satellite/metainfo: ListBuckets extended with value attribution assigned to bucket
We need to provide the ability to see bucket attribution on the gateway side
so customers can validate if bucket is attributed to them. Extendet metainfo.ListBuckets
request with UserAgent.

Fixes https://github.com/storj/storj/issues/4965

Change-Id: I5624874a7faa14cda06183ad44013e9ebb385b63
2022-07-18 18:23:50 +00:00
Clement Sam
ebd463b57c go.mod: update dependencies
Change-Id: I5360ea7b6487295cb7def759f417b31b133c34da
2022-07-05 18:41:39 +03:00
Artur M. Wolff
0f626a59c1 mod: bump storj.io/uplink
Change-Id: I6d51fd330ebb882564b6c0cd02d829de08082b46
2022-07-05 10:27:27 +00:00
JT Olio
bb1eb1bae5 storj: bump private
Change-Id: I3dd80fc19a1412ac1d8264080cbf20bc4f040f13
2022-06-14 11:54:28 -04:00
Márton Elek
84d02f7fbf metabase: use predefined snapshot for unit tests avoid migration
Change-Id: I6225ae32c312030e3559d8ed0faa137ffc3cc5b7
2022-06-01 12:07:00 +00:00
Michał Niewrzał
456aea727e satellite: use PieceIDDeriver for derivation
We can use PieceIDDeriver in all places where we are deriving id from
the same id multiple times. We have serveral such places: gc, segment
deletion, segment validation, order limit creation. Using it should
save some resources.

Change-Id: I24668d516c0f7cea4aec6470614067734149501d
2022-05-19 06:31:42 +00:00
Michał Niewrzał
9b49ffe48a mod: bump storj/private dependency
Change-Id: I5067c0cb3f1d33d9704ecdbcbb0f36cda9dc0bb0
2022-05-12 14:39:51 +00:00
Qweder93
384118ee1e satellite/metabase: use pgutil.NullTimestampTZArray
This code cleanup to use helper DB method.

Change-Id: Idc438234cb4f7da23d6ade2d69f5d889f15e9009
2022-05-03 19:30:21 +00:00
Jeff Wendling
4f196dd39e cmd/uplink: bump clingy
this causes an error code to be returned for missing commands

Change-Id: I1ac33ec7c0de506e964fb823e0ed80832cb80d3b
2022-04-27 10:25:44 +00:00
Egon Elbre
175d1e694c mod: bump storj.io/private
This updates the database drivers and sqlite implementation.

Change-Id: I384d81d7beca7267f7c1d6115b3e5e33629f8dc7
2022-04-26 19:18:52 +00:00
Michał Niewrzał
e66beb2429 satellite/metainfo: set metadata with BeginObject
This change makes possible to set metadata at the beginning
of upload.

https://github.com/storj/team-metainfo/issues/105

Change-Id: If5b6b90243d9c9318a8b72c2af3fbde400c5d8e7
2022-04-26 08:04:27 +00:00
Clement Sam
583d48c432 mod: bump storj/private
Change-Id: Ia8299a082347b8f4ecc2f36988387b3a48701da5
2022-04-12 11:09:10 +00:00
Michał Niewrzał
a3d90167c7 mod: bump to latest uplink
Change-Id: I790be139cdb4c52395507c01d4c4d36eab6a5915
2022-04-11 08:52:03 +00:00
paul cannon
985ccbe721 satellite/repair: in dns redial, don't retry if CloseError
To save load on DNS servers, the repair code first tries to dial the
last known good ip and port for a node, and then falls back to a DNS
lookup only if we fail to connect to the last known good ip and port.

However, it looks like we are seeing errors during the client stream
Close() call (probably due to quic-go code), and those are classified
the same as errors encountered during Dial. The repairer code sees this
error, assumes that we failed to contact the node, and retries- but
since we did actually succeed in connecting the first time around, this
results in submitting the same order limit (with the same serial number)
to the storage node, which (rightfully) rejects it.

So together with change I055c186d5fd4e79560f67763175bc3130b9bc7d2 in
storj/uplink, this should avoid the double submission and avoid dinging
nodes' suspension scores unfairly.

See https://github.com/storj/storj/issues/4687.

Also, moving the testsuite directory check up above check-monkit in the
Jenkins Lint task, so that a non-tidy testsuite/go.mod can be recognized
and handled before everything breaks weirdly and seemingly randomly
later on.

Change-Id: Icb2b05aaff921d0af6aba10e450ac7e0a7bb2655
2022-04-04 17:01:09 +00:00
Michał Niewrzał
2014e4655a satellite/metabase: bring back copy/move validation for key and nonce
Uplink is fixed and now we should always get both key and nonce
or both empty.

Fixes https://github.com/storj/storj/issues/4646

Change-Id: I65dca2d4d5a10787c2fecad39e301121f1ae242a
2022-04-01 07:23:22 +00:00
Egon Elbre
e5972d8920 all: fix linting errors
strings.Title is deprecated in Go 1.18, replace it with
golang.org/x/text/cases.

Change-Id: I6185b97d37309dbe4a6715f794383ab259a0658b
2022-03-30 18:45:23 +03:00
Sean Harvey
53de525721 mod: bump storj/common, storj/private, and storj/uplink deps
Change-Id: Iac8528ba793e6bba04d8e47a1254fcef665cd342
2022-03-28 22:06:16 +00:00
Mya
98f4fae02c satellite/oidc: add integration test
This change adds an integration test that performs an OAuth
workflow and verifies the OIDC endpoints are functioning as
expected.

Change-Id: I18a8968b4f0385a1e4de6784dee68e1b51df86f7
2022-03-18 16:14:18 +00:00
JT Olio
0a298778be storj/storj: bump storj/common
Change-Id: I2b18f166f55b7d2b46dce7f09c0ecd27ebcf7d89
2022-03-18 17:07:36 +02:00
Egon Elbre
466832e4bc storagenode/piecestore: check for remote closing
Remote closing during upload or download is entirely expected and
it shouldn't lead to an error in the log.

Bump drpc to get the version that contains correct error code
for it. Also bump errs, which contains a fix for .Has.

Fixes https://github.com/storj/storj/issues/4609

Change-Id: I9297cabcfdc4b3a2c19d478dc729f779a2aef0c3
2022-03-17 19:27:42 +02:00
Mya
4a110b266e satellite/console: added oidc endpoints
This change adds endpoints for supporting OpenID Connect (OIDC) and
OAuth requests. This allows application developers to easily
develop apps with Storj using common mechanisms for authentication
and authorization.

Change-Id: I2a76d48bd1241367aa2d1e3309f6f65d6d6ea4dc
2022-03-16 12:01:26 +00:00
Michał Niewrzał
ccd16bbef6 satellite/metabase: handle NewEncryptedMetadata while coping object
Copy object functionality should support setting new metadata for
copy. This change is adjusting FinishCopyObject method to set new
metadata when OverrideMetadata field is set to true.

Fixes https://github.com/storj/storj/issues/4483

Change-Id: Ica37cb57e8edae301cdc483fbda4f3ddba5d2702
2022-03-09 12:31:33 +01:00
Erik van Velzen
53e851bfb2 satellite/metabase/delete: adjust delete for copy feature
Adjust deletion of committed exact version so that it takes into account
copied objects.

Change-Id: I5daaa94ff8ad87417d52c79f305bc3275b683efa
2022-03-04 13:50:58 +00:00
Mya
05a17ef42d deps: upgrade storj.io/common
In addition to upgrading the storj.io/common library, this change
moves off the TCPConnector in favor of the HybridConnector per
the deprecation warning.

Change-Id: I7e7e1e7568e8b95e4a99ad9caa158a799e68e1e3
2022-02-16 18:59:19 +00:00
Egon Elbre
11a35520c3 mod: bump common, quic, golang/x packages
Change-Id: I3b1f8ec69bac59b60665f8e59e4183dd4a0a221b
2022-02-03 12:20:43 +00:00
Jeff Wendling
b7ca2289e0 cmd/uplinkng: improvements to prepare for doc update
* improve setup wizard
* added access grant file support
* improved consistency across commands
* a couple bug fixes
* added access import and export

Change-Id: I30ad9d4771f15430904a503a4d465bc40be471b5
2022-01-31 17:37:54 +00:00
Egon Elbre
64c8de6ea5 mod: use vendored base58
Change-Id: I5aa29515928848c862500330218cc094618638d7
2022-01-31 15:54:33 +02:00
Clement Sam
9f3c1f9cda satellite/contact: add PingMe endpoint
Change-Id: I832a72fafeacf76ad64a0129bcc6582cc4f9290d
2022-01-19 17:52:33 +00:00
Michał Niewrzał
eb0d08d59b satellite/metainfo: collect versions of user tools
We want to know usage statistics for our main tools
like uplink-cli or rclone. Initially we will collect
only usage stats without relation to specific process
e.g. download or upload.

Change-Id: I203b1a6c07ae014e710368f77163f13fdf10763c
2022-01-12 13:49:15 +00:00
Qweder93
b6625cadea satellite/metainfo: uncomment tests, renamed EncryptedPath into EncryptedObjectKey
moved tests back and renamed params that were renamed on uplink side

Change-Id: I4a9592bd955f2217c5809c235c394a4e78d337ea
2022-01-11 17:18:18 +02:00
dlamarmorgan
ab37b65cfc satellite/{accounting,orders,satellitedb}: group bucket bandwidth rollups by time window
Batching of the order submissions can lead to combining the allocated
traffic totals for two completely different time windows, resulting
in incorrect customer accounting. This change will group the batched
order submissions by projectID as well as time window, leading to
distinct updates of a buckets bandwidth rollup based on the hour
window in which the order was created.

Change-Id: Ifb4d67923eec8a533b9758379914f17ff7abea32
2022-01-05 20:24:48 +00:00
Jeff Wendling
4426470173 cmd/uplinkng: rename parallelism flags, small fixes
this makes the flags match rclone nomenclature

fixes test-uplinkng to use the temporary config dir
instead of the machine default, and clean up some.

bumps clingy so that the command errors when an unknown
command is specified.

also fixes some printfs in share to use clingy stdout.
it still does some external actions that should be
passed through a ulext.External for mocking, but
that's ok for now.

Change-Id: Icc231e7e26393541c312396fec907b640b97718e
2021-12-22 13:13:30 +00:00
Michał Niewrzał
db7985ca90 satellite/metainfo: set EncryptedMetadataEncryptedKey while listing objects
At some point we missed to add metadata key to list objects
response. Because of that uplink is taking key from pb.SteamMeta.
We need to clean it up.

Tests will be added on uplink side.

https://github.com/storj/uplink/issues/71

Change-Id: I3328e2f1b86bca15aeaf89f8e59cdca3c8e97742
2021-12-21 13:09:11 +00:00
Jeff Wendling
9615bd191f cmd/uplinkng: update clingy for better boolean support
this allows commands like

    uplinkng cp -r sj://foo sj://bar

to work correctly, rather than complain that sj://foo is
not a boolean.

Change-Id: I003e47aabb85566bc2b454851cf55043b17ee7ea
2021-12-14 09:41:38 +00:00
Michał Niewrzał
ace90fd844 mod: bump clingy version
Latest version have a fix for:
https://github.com/storj/storj/issues/4257

Change-Id: I0e3a67fe632cd5c209b24e1ee3080563cafbb626
2021-12-08 11:10:25 +01:00
Kaloyan Raev
f773bb80c8 mod: bump common to fetch latest placement type changes
Change-Id: I3d0813f05622e706c9be1a578b5e4d4159d16dfc
2021-11-16 12:42:25 +00:00
Mya
bf51c286d9
satellite/geoip: update node check-in to associate a country code
Resolves https://github.com/storj/storj/issues/4247

Change-Id: Idfd71bf1795d48ca3c686066bbdb95b9c6594f00
2021-11-10 16:44:41 +01:00
Márton Elek
9bdcc415bc satellite/nodeselection: add geofencing constraints to the node selection criteria
Closes https://github.com/storj/storj/issues/4242
Change-Id: Ieda59a4f37c673e4e81abb4c89c09daf3199bbc7
2021-11-08 17:04:31 +00:00
dlamarmorgan
1dd537953e satellite/metainfo: strip the uplink version from the UserAgent and bound its size
The UserAgent should be stored as is, with the exception of removing the trailing version from any libuplink user agents

Change-Id: If17ef2fc4b59480a3477300f2585a07d64cc2bf4
2021-11-04 10:09:54 -07:00
Jeff Wendling
4d023f773f cmd/storagenode-updater: update to correct version
Currently if the nodes is below the minimum version it will immediately
update to the suggested version, regardless if its eligible from the seed
or not. This change corrects the behaviour to update to the minimum
version only and then properly respect/wait for the rollout to include it.

Updates based on logic here: https://review.dev.storj.io/c/storj/private/+/6187

Change-Id: Ic6c91c48ae9b8a116378b2573fbfca7e7bd5cc32
2021-10-29 20:38:30 +00:00
Stefan Benten
8c1a149de2 build: bump private dependency
Enables proper version printing.

Change-Id: I37e12d6823f86f9275ac9f64abf324003019bd18
2021-10-29 12:25:33 +13:00
Egon Elbre
1409288c49 mod: bump semantic version to go 1.17
This enables module graph pruning and lazy module loading.

Change-Id: I5d63a686d4f4eda93b23576169bb03847292f8c7
2021-10-26 11:58:13 +00:00
Stefan Benten
df6cf60ffe
go.mod: update to minimum supported go version (#4239) 2021-10-22 21:12:13 +02:00
Egon Elbre
56f35e9ea9 mod: bump private repo
Contains a fix to ignore cockroach DROP error while testing.

Change-Id: I74b02240f570c30a66f6f0b938ce036392c05e06
2021-10-22 15:02:48 +03:00
Clement Sam
f82b696858 mod: bump common dependency
Change-Id: I24ba0ef8d0ca25a34e28de1064a2f1213787ccc5
2021-10-21 16:33:48 +00:00
Egon Elbre
bdadc4b460 private/testplanet: add stack traces related to the planet
It's useful to see only the goroutines that are related to the current
testplanet.

Change-Id: Id77c62dec42c09d44c523b019f7bdc940905fcf1
2021-10-20 23:12:07 +03:00