Commit Graph

528 Commits

Author SHA1 Message Date
Michal Niewrzal
20dfcedd97 satellite/metainfo: return full object on commit
We never extended metainfo protocol to return committed object
detailed into and this change is doing it now. Main motivation to
do this now is need for providing object version after upload.

Change-Id: Ib59bdfd9485e4a0091ac02458cc63427cb7159de
2023-10-26 09:46:18 +00:00
Clement Sam
cfbb5dac14 go.mod: update storj.io/common
Change-Id: Ibdfd962b8be8407a687753d0648149a5f98a44b2
2023-10-05 15:52:06 +00:00
Egon Elbre
800bba7a0e go.mod: bump uplink
This brings in 32bit arm bug fix.

Change-Id: I61b72abf2366643f58b1e670d2e02f4072aee0b5
2023-10-05 08:51:11 +00:00
Jeremy Wharton
fd835859d5 go.mod: bump stripe-go from v73 to v75
This change updates the version of the stripe-go dependency from v73 to
v75 in order to improve performance (see stripe/stripe-go#1728).

Resolves #6287

Change-Id: I727ac08dcaa90d7138ba30d907711cc44daf0b7a
2023-10-03 10:12:29 +00:00
Jeremy Wharton
2f19636eb3 go.mod,satellite/payments/stripe: bump stripe-go from v72 to v73
This change updates the version of the stripe-go dependency from v72 to
v73. This is part of a process to reach v75, which contains performance
improvements.

References #6287

Change-Id: I95f132378fe05e506f0388a0fe039cb1c2db58e2
2023-10-03 07:38:07 +00:00
Márton Elek
58b98bc335 satellite/repair: repair is configurable to work only on included/excluded placements
This patch finishes the placement aware repair.

We already introduced the parameters to select only the jobs for specific placements, the remaining part is just to configure the exclude/include rules. + a full e2e unit test.

Change-Id: I223ba84e8ab7481a53e5a444596c7a5ae51573c5
2023-09-27 14:54:06 +00:00
Márton Elek
5bbd477a58 go.mod: bump dependencies (uplink,common,monkit)
Change-Id: I72e17c3acbabd36ffb0a0adbcccec4c252ba710c
2023-09-19 00:35:07 +00:00
Egon Elbre
bc517cae2f go.mod: bump common
This brings in common/grant that doesn't depend on protobuf anymore.
This ends up causing the console wasm bundle from ~11MB to ~4.7MB.

Change-Id: I145dcb9239952a7a9e352c8793c111acb61ff0cc
2023-09-15 17:43:44 +00:00
Michal Niewrzal
8f1682941e mod: bump storj/uplink to v1.12.0
Change-Id: Ia58de16d4ac43c7b888000ad8ca85a979496a2cb
2023-09-14 09:20:38 +02:00
Kaloyan Raev
3119b614ae cmd/uplink: --max-object-ttl flag for share and access restrict commands
Context: https://github.com/storj/storj/issues/6249

Change-Id: Ic65a1d8aef61f1a88752a7b12a23fb854dac8f6d
2023-09-11 08:14:55 +00:00
Kaloyan Raev
4e499fb9bf satellite/metainfo: respect MaxObjectTTL in BeginObject
If MaxObjectTTL is set in the API key, BeginObject will use it for the
object expiration time, unless an explicit ExpireAt is available in the
request.

Context: https://github.com/storj/storj/issues/6249

Change-Id: I2adf57d979a9c68eec3a787f3739d2f1dbec1f7e
2023-09-08 09:30:48 +00:00
Moby von Briesen
8d1a765fd6 satellite/console: Partially revert change to remove graphql
This partially reverts commit 516241e406.

Endpoints are added to the backend, as there are some customers who may
use these endpoints, even though they are no longer necessary for the
satellite UI.

Change-Id: I52a99912d9eacf269fbb2ddca603e53c4af6d6bf
2023-09-07 20:50:24 +00:00
Márton Elek
f40baf8629
go.mod: bump dependencies (private,uplink,common)
Change-Id: I6c55735b45cadaf36697eff53e78b5b09afe9dea
2023-09-06 13:28:22 +02:00
Artur M. Wolff
37d6df23fa satellite: implement metainfo.GetBucketLocation endpoint
Updates storj/storj-private#408
Updates storj/storj-private#409

Change-Id: Idaaca74b4a5c9c7907d095e0a3a5f29e52843ce6
2023-08-28 13:48:07 +02:00
Wilfred Asomani
516241e406 cmd,satellite: remove Graphql code and dependencies
This change removes unused GraphQL code. It also updates storj sim code
to use the GraphQL replacement HTTP endpoints and removes the GraphQL
dependency.

Issue: https://github.com/storj/storj/issues/6142

Change-Id: Ie502553706c4b1282cd883a9275ea7332b8fc92d
2023-08-22 12:23:14 +00:00
Egon Elbre
1f261bcc70 go.mod: bump lang to 1.19 and common
Change-Id: I8d91f97d786456da29ebe89a78412c50efbb8ccc
2023-08-10 18:41:15 +03:00
JT Olio
e78658d174 go.mod: use latest maxmind-golang library
Change-Id: I6f06785a096d7d05b175d468482db658b7436a87
2023-08-01 15:22:35 +00:00
Michal Niewrzal
16c5b3c340 mod: bump uplink to v1.11.0
Change-Id: I88922421f250a261b78227a409f4879ae5221140
2023-07-26 15:01:18 +00:00
paul cannon
9a871bf3bc go.mod: bump storj.io/common
In particular, to get commit 9db74ed9 in for the next storagenode build.

Change-Id: If07283bc72d66dfbd0ee91bc1cc8b2ce015871cd
2023-07-19 09:50:06 -05:00
Egon Elbre
9370bc4580 satellite/{nodeselection,overlay}: bump common and fix some potential issues
* Handle failed country code conversion.
* Avoid potential issues with a data-race due to shared slice.

Updates #6028

Change-Id: If7beef2619abd084e1f4109de2d323f834a6090a
2023-07-11 11:13:41 +00:00
Márton Elek
97a89c3476 satellite: switch to use nodefilters instead of old placement.AllowedCountry
placement.AllowedCountry is the old way to specify placement, with the new approach we can use a more generic (dynamic method), which can check full node information instead of just the country code.

The 90% of this patch is just search and replace:

 * we need to use NodeFilters instead of placement.AllowedCountry
 * which means, we need an initialized PlacementRules available everywhere
 * which means we need to configure the placement rules

The remaining 10% is the placement.go, where we introduced a new type of configuration (lightweight expression language) to define any kind of placement without code change.

Change-Id: Ie644b0b1840871b0e6bbcf80c6b50a947503d7df
2023-07-07 16:55:45 +00:00
Clement Sam
1851d103f9 go.mod: bump storj.io/private
Updates https://github.com/storj/storj/issues/6006

Change-Id: I1f549d5642213e420f3e5d0df4ef972c77add713
2023-07-03 23:59:15 +00:00
Márton Elek
8482b37c14
go.mod: bump to use latest common
Change-Id: Ie31e7779e86d13ca3c8acaacfe6ed1e23f2c9740
2023-07-03 13:32:35 +02:00
paul cannon
d06e4589ae bump storj.io/private
Change-Id: I0b773fe140bad485fc323701928eaf246638032c
2023-06-27 13:29:54 -05:00
Egon Elbre
049953a7ce go.mod: bump storj.io/uplink
Change-Id: Ib535cb50195c5a0e6197e83e2e05865ba95bcaf8
2023-06-26 13:51:13 +00:00
Vitalii
1eee2fad69 satellite/{db, admin}: added endpoints to update user's and project's user_agent
Added backend (for now) implementation for updating user's and projects's user_agent using admin API.
Updating both user and project also updates bucket_metainfo and value_attribution tables.

Issue:
https://github.com/storj/storj-private/issues/297

Change-Id: I40244bbaa08b46834c1b1d0720e7d84d0c2a0330
2023-06-16 19:41:05 +00:00
Egon Elbre
edbea5efe1 go.mod: bump to pgx/v5
Change-Id: I31cf3bec1d7db94f0f612f6ed04b782f8b04d876
2023-06-14 18:32:54 +03:00
Márton Elek
981dd4cbdc go.mod: bump dependencies (storj.io/private, storj.io/uplink)
Change-Id: Ie5a6a14a39786dd745116c3b6e4fdbc8e890d2c3
2023-06-14 16:36:45 +03:00
Jeff Wendling
f3c58174c4 cmd/uplink: only use new code path for uploads
downloads still need the old copy code because they aren't
parallel in the same way uploads are. revert all the code
that removed the parallel copy, only use the non-parallel
copy for uploads, and add back the parallelism and chunk
size flags and have them set the maximum concurrent pieces
flags to values based on each other when only one is set
for backwards compatibility.

mostly reverts 54ef1c8ca2

Change-Id: I8b5f62bf18a6548fa60865c6c61b5f34fbcec14c
2023-06-09 23:45:30 +00:00
Jeff Wendling
c559df6a1b all: bump storj.io/uplink
this fixes a memory leak in the upload codepath

Change-Id: I6bf6214d331cbe4e8e8912942035ef01a285e3b4
2023-06-08 19:52:24 +00:00
Márton Elek
974b4f938a go.mod: bump storj.io/private
Change-Id: I2d130cf2e794c8e5e0a978433cb3d504e6747af0
2023-06-08 12:01:55 +00:00
JT Olio
8c8340f1bc go.mod: bump eventkit
Change-Id: I2ab54bf640fc8a2aeafab77418c63bcd9cabd1aa
2023-06-07 15:35:24 +00:00
JT Olio
08a63fa5d1 go.mod: bump storj.io/common and storj.io/uplink
Change-Id: Ifc33429ab3e6f5b52b685c800b7c96dff6b904f2
2023-06-02 18:01:16 -04:00
JT Olio
36038af3d1 go.mod: bump storj.io/common
Change-Id: I18d4d491cdca99f6694a20f3ead5fba3f75bb658
2023-06-02 11:23:02 -04:00
Jeff Wendling
a0fbc87b31 cmd/uplink: add upload log file flag
Change-Id: I740eaf431cfe5bb3dcb746e4128dea5efc935257
2023-05-25 14:14:00 +00:00
Michal Niewrzal
36e046375c satellite/repair/checker: remove segments loop parts
We are switching completely to ranged loop.

https://github.com/storj/storj/issues/5368

Change-Id: I8583549973cd36aa0e0c482c20d7a75cb7568ab3
2023-05-08 12:19:13 +00:00
Jeff Wendling
6ac72e42cd all: bump storj.io/drpc
this includes a fix to cancellation when soft cancel
is enabled.

Change-Id: I1b9ab071f623153a2bd715888f97ca20277a76d7
2023-05-06 01:02:50 +00:00
Egon Elbre
8b82dba602 storagenode/blobstore/filestore: add tracking of blobs
We've had issues with forgetting to close readers and writers.
Add leak tracking to find those pesky issues.

Change-Id: If6b0ad6e9958318a7e0affee9c6d0a1ece412b6d
2023-05-05 15:40:15 +03:00
Jeff Wendling
80b3edf1d1 storagenode/piecestore: respect maximum chunk size requests
See https://review.dev.storj.io/c/storj/common/+/10297 for
more details.

Change-Id: Id5d19f029ae872780a554874592679191c1b5b2f
2023-04-28 16:31:41 -04:00
Márton Elek
1588bf9606 go.mod: bump monkit
Change-Id: Icc286378a8b01518510a4b4dd4d213d0c5d3b5b6
2023-04-25 14:26:11 +00:00
Egon Elbre
2405bc8f3b satellite/metabase: stop using the common error type
Updates https://github.com/storj/storj/issues/5291

Change-Id: I7b57a4b454d3619cb5d8ae4cd92f818ad2839c8b
2023-04-19 19:18:18 +03:00
Egon Elbre
d94207048a go.mod: bump vbauerster/mpb/v8 for fixes
Change-Id: I2d24eec4389ca8e5effdbfc25d7012ce083d46f5
2023-04-18 15:53:35 +03:00
Jeff Wendling
df9cc4d10f all: bump storj.io/{common,uplink,drpc}
Change-Id: Ie98a7eacc11a492a20c76636f8044c81b7ed580d
2023-04-17 19:59:30 +00:00
Jeff Wendling
54ef1c8ca2 cmd/uplink: use new upload code path
the parallelism and parallelism-chunk-size flags
which used to control how many parts to split a
segment into and many to perform in parallel
are now deprecated and replaced by
maximum-concurrent-pieces and long-tail-margin.

now, for an individual transfer, the total number
of piece uploads that transfer will perform is
controlled by maximum-concurrent-pieces, and
segments within that transfer will automatically
be performed in parallel. so if you used to set
your parallelism to n, a good value for the pieces
might be something approximately like 130*n, and
the parallelism-chunk-size is unnecessary.

Change-Id: Ibe724ca70b07eba89dad551eb612a1db988b18b9
2023-04-13 16:52:38 -04:00
Andrew Harding
e676b5c893 cmd/uplink: progress bars for recursive copy
```
$ uplink cp -r -t=3 files/ sj://files
uploading 6 files...
files/bar/buz   (2 of 6) 134.22 MB / 134.22 MB [============================================] 100.00% 36.43 MiB/s
files/bar/baz   (1 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 18.39 MiB/s
files/boo       (3 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 20.42 MiB/s
files/foo       (4 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 57.83 MiB/s
files/glue/flew (5 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 55.01 MiB/s
files/stew      (6 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 91.43 MiB/s
```

Change-Id: Ibd9d07a1291f7a599bd27fba93c1b2e0f17dc787
2023-04-10 15:13:22 +00:00
Márton Elek
e5995947d0 go.mod: bump storj.io/private and storj.io/uplink
Change-Id: I01790c1af8d6302b3c9ffee0482c952bb9a5d8da
2023-04-06 13:44:34 +00:00
Michal Niewrzal
c4c391e154 satellite/accounting/live: replace address parsing with redis util
With this change we are replacing parsing code with existing go-redis
util.

We also switch redis client to version 9.

Change-Id: Ie4a651e3ae6960e68958c690873925d319b70e10
2023-04-05 13:20:11 +00:00
Clement Sam
5ab2e6ac19 go.mod: bump storj.io/private
Updates https://github.com/storj/storj/issues/5349

Change-Id: I5b91a8e9c28d39c6ec432992a352de08aff886d9
2023-04-05 14:46:59 +03:00
Michal Niewrzal
4c05293d8b satellite: set names for lrucache metrics
Fixes https://github.com/storj/storj/issues/5685

Change-Id: I9c19b7644b0cd3297448982b920c97abcdce9119
2023-04-04 11:55:20 +00:00
JT Olio
bd238819b6 go.mod: bump storj.io/drpc
this includes improved connection reset error handling

Change-Id: I211221493b5dd3f4b95d4db44bf022302352f050
2023-03-29 19:27:57 +00:00