Commit Graph

136 Commits

Author SHA1 Message Date
Egon Elbre
1f261bcc70 go.mod: bump lang to 1.19 and common
Change-Id: I8d91f97d786456da29ebe89a78412c50efbb8ccc
2023-08-10 18:41:15 +03:00
JT Olio
e78658d174 go.mod: use latest maxmind-golang library
Change-Id: I6f06785a096d7d05b175d468482db658b7436a87
2023-08-01 15:22:35 +00:00
Michal Niewrzal
16c5b3c340 mod: bump uplink to v1.11.0
Change-Id: I88922421f250a261b78227a409f4879ae5221140
2023-07-26 15:01:18 +00:00
Clement Sam
4cb85186b2 storagenode/pieces: enable lazyfilewalker by default
Resolves https://github.com/storj/storj/issues/5861

Change-Id: I20e0a5b8a15ca966cbccd71369322a517a2c2130
2023-07-19 15:25:20 +00:00
paul cannon
9a871bf3bc go.mod: bump storj.io/common
In particular, to get commit 9db74ed9 in for the next storagenode build.

Change-Id: If07283bc72d66dfbd0ee91bc1cc8b2ce015871cd
2023-07-19 09:50:06 -05:00
Egon Elbre
9370bc4580 satellite/{nodeselection,overlay}: bump common and fix some potential issues
* Handle failed country code conversion.
* Avoid potential issues with a data-race due to shared slice.

Updates #6028

Change-Id: If7beef2619abd084e1f4109de2d323f834a6090a
2023-07-11 11:13:41 +00:00
Márton Elek
97a89c3476 satellite: switch to use nodefilters instead of old placement.AllowedCountry
placement.AllowedCountry is the old way to specify placement, with the new approach we can use a more generic (dynamic method), which can check full node information instead of just the country code.

The 90% of this patch is just search and replace:

 * we need to use NodeFilters instead of placement.AllowedCountry
 * which means, we need an initialized PlacementRules available everywhere
 * which means we need to configure the placement rules

The remaining 10% is the placement.go, where we introduced a new type of configuration (lightweight expression language) to define any kind of placement without code change.

Change-Id: Ie644b0b1840871b0e6bbcf80c6b50a947503d7df
2023-07-07 16:55:45 +00:00
Clement Sam
1851d103f9 go.mod: bump storj.io/private
Updates https://github.com/storj/storj/issues/6006

Change-Id: I1f549d5642213e420f3e5d0df4ef972c77add713
2023-07-03 23:59:15 +00:00
Márton Elek
8482b37c14
go.mod: bump to use latest common
Change-Id: Ie31e7779e86d13ca3c8acaacfe6ed1e23f2c9740
2023-07-03 13:32:35 +02:00
paul cannon
d06e4589ae bump storj.io/private
Change-Id: I0b773fe140bad485fc323701928eaf246638032c
2023-06-27 13:29:54 -05:00
Egon Elbre
049953a7ce go.mod: bump storj.io/uplink
Change-Id: Ib535cb50195c5a0e6197e83e2e05865ba95bcaf8
2023-06-26 13:51:13 +00:00
Vitalii
1eee2fad69 satellite/{db, admin}: added endpoints to update user's and project's user_agent
Added backend (for now) implementation for updating user's and projects's user_agent using admin API.
Updating both user and project also updates bucket_metainfo and value_attribution tables.

Issue:
https://github.com/storj/storj-private/issues/297

Change-Id: I40244bbaa08b46834c1b1d0720e7d84d0c2a0330
2023-06-16 19:41:05 +00:00
Egon Elbre
edbea5efe1 go.mod: bump to pgx/v5
Change-Id: I31cf3bec1d7db94f0f612f6ed04b782f8b04d876
2023-06-14 18:32:54 +03:00
Márton Elek
981dd4cbdc go.mod: bump dependencies (storj.io/private, storj.io/uplink)
Change-Id: Ie5a6a14a39786dd745116c3b6e4fdbc8e890d2c3
2023-06-14 16:36:45 +03:00
Jeff Wendling
c559df6a1b all: bump storj.io/uplink
this fixes a memory leak in the upload codepath

Change-Id: I6bf6214d331cbe4e8e8912942035ef01a285e3b4
2023-06-08 19:52:24 +00:00
Márton Elek
974b4f938a go.mod: bump storj.io/private
Change-Id: I2d130cf2e794c8e5e0a978433cb3d504e6747af0
2023-06-08 12:01:55 +00:00
JT Olio
8c8340f1bc go.mod: bump eventkit
Change-Id: I2ab54bf640fc8a2aeafab77418c63bcd9cabd1aa
2023-06-07 15:35:24 +00:00
JT Olio
08a63fa5d1 go.mod: bump storj.io/common and storj.io/uplink
Change-Id: Ifc33429ab3e6f5b52b685c800b7c96dff6b904f2
2023-06-02 18:01:16 -04:00
JT Olio
36038af3d1 go.mod: bump storj.io/common
Change-Id: I18d4d491cdca99f6694a20f3ead5fba3f75bb658
2023-06-02 11:23:02 -04:00
Jeff Wendling
a0fbc87b31 cmd/uplink: add upload log file flag
Change-Id: I740eaf431cfe5bb3dcb746e4128dea5efc935257
2023-05-25 14:14:00 +00:00
Clement Sam
018b6eeeaf storagenode: add tests for lazyfilewalker
Updates https://github.com/storj/storj/issues/5349

Change-Id: I9544c14ba2acacd5b304f151ab29c70ff61adc5b
2023-05-08 21:50:40 +00:00
Jeff Wendling
6ac72e42cd all: bump storj.io/drpc
this includes a fix to cancellation when soft cancel
is enabled.

Change-Id: I1b9ab071f623153a2bd715888f97ca20277a76d7
2023-05-06 01:02:50 +00:00
Egon Elbre
8b82dba602 storagenode/blobstore/filestore: add tracking of blobs
We've had issues with forgetting to close readers and writers.
Add leak tracking to find those pesky issues.

Change-Id: If6b0ad6e9958318a7e0affee9c6d0a1ece412b6d
2023-05-05 15:40:15 +03:00
Jeff Wendling
80b3edf1d1 storagenode/piecestore: respect maximum chunk size requests
See https://review.dev.storj.io/c/storj/common/+/10297 for
more details.

Change-Id: Id5d19f029ae872780a554874592679191c1b5b2f
2023-04-28 16:31:41 -04:00
Márton Elek
1588bf9606 go.mod: bump monkit
Change-Id: Icc286378a8b01518510a4b4dd4d213d0c5d3b5b6
2023-04-25 14:26:11 +00:00
Egon Elbre
2405bc8f3b satellite/metabase: stop using the common error type
Updates https://github.com/storj/storj/issues/5291

Change-Id: I7b57a4b454d3619cb5d8ae4cd92f818ad2839c8b
2023-04-19 19:18:18 +03:00
Egon Elbre
d94207048a go.mod: bump vbauerster/mpb/v8 for fixes
Change-Id: I2d24eec4389ca8e5effdbfc25d7012ce083d46f5
2023-04-18 15:53:35 +03:00
Jeff Wendling
f52ea2744b testsuite/ui: bump gateway-mt and enable vet
go vet was disabled while gateway-mt was not yet
updated to build with some breaking changes in
common. now it's updated, so enable.

Change-Id: Idbb0ab0e56b5bd6545273dd3d41b47e7c79eb4b6
2023-04-17 22:07:57 +00:00
Jeff Wendling
df9cc4d10f all: bump storj.io/{common,uplink,drpc}
Change-Id: Ie98a7eacc11a492a20c76636f8044c81b7ed580d
2023-04-17 19:59:30 +00:00
Andrew Harding
e676b5c893 cmd/uplink: progress bars for recursive copy
```
$ uplink cp -r -t=3 files/ sj://files
uploading 6 files...
files/bar/buz   (2 of 6) 134.22 MB / 134.22 MB [============================================] 100.00% 36.43 MiB/s
files/bar/baz   (1 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 18.39 MiB/s
files/boo       (3 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 20.42 MiB/s
files/foo       (4 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 57.83 MiB/s
files/glue/flew (5 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 55.01 MiB/s
files/stew      (6 of 6) 67.11 MB / 67.11 MB [==============================================] 100.00% 91.43 MiB/s
```

Change-Id: Ibd9d07a1291f7a599bd27fba93c1b2e0f17dc787
2023-04-10 15:13:22 +00:00
Márton Elek
e5995947d0 go.mod: bump storj.io/private and storj.io/uplink
Change-Id: I01790c1af8d6302b3c9ffee0482c952bb9a5d8da
2023-04-06 13:44:34 +00:00
Michal Niewrzal
c4c391e154 satellite/accounting/live: replace address parsing with redis util
With this change we are replacing parsing code with existing go-redis
util.

We also switch redis client to version 9.

Change-Id: Ie4a651e3ae6960e68958c690873925d319b70e10
2023-04-05 13:20:11 +00:00
Clement Sam
5ab2e6ac19 go.mod: bump storj.io/private
Updates https://github.com/storj/storj/issues/5349

Change-Id: I5b91a8e9c28d39c6ec432992a352de08aff886d9
2023-04-05 14:46:59 +03:00
Michal Niewrzal
4c05293d8b satellite: set names for lrucache metrics
Fixes https://github.com/storj/storj/issues/5685

Change-Id: I9c19b7644b0cd3297448982b920c97abcdce9119
2023-04-04 11:55:20 +00:00
JT Olio
bd238819b6 go.mod: bump storj.io/drpc
this includes improved connection reset error handling

Change-Id: I211221493b5dd3f4b95d4db44bf022302352f050
2023-03-29 19:27:57 +00:00
Márton Elek
5e6d1cc00a
go.mod: bump storj.io/{common,private,uplink}
Change-Id: I7af875b6ac73af3a9d77928f69a05b80a5321696
2023-03-27 16:13:52 +02:00
JT Olio
0b4b04900a private/server: debounce noise and tls connections
to support TCP_FAST_OPEN, we're considering just using
two TCP connections in parallel per request, one with
and one without. this allows us to safely fire both
concurrently without stressing out the node too much.

see https://review.dev.storj.io/c/storj/storj/+/9933

Change-Id: I9aa8a0252350db5ace04ee125bfe469203e980ec
2023-03-21 16:51:31 +00:00
Kaloyan Raev
2c28a0e599 cmd/uplink: ability to opt-out from object key encryption
Option added to `uplink access setup` and `uplink access create`
commands to disable object key encryption.

Related to https://github.com/storj/storj/issues/5678

Change-Id: I4789a94143742ff4b232fd60decc029ad2883c2a
2023-03-20 13:56:15 +00:00
Vitalii
a01d47b3ed satellite:{console, web}: remove old project dashboard
Removed old project dashboard along with a feature flag.

Change-Id: I9292b19caf101a84042d8c3006c27bc44db8d6b5
2023-03-07 13:34:59 +02:00
JT Olio
8befcaab27 go.mod: bump storj.io/{common,drpc,uplink}
this set of changes re-enables noise with fixes

Change-Id: I122142d202dd76a83b678ac8d709407b0791ed0b
2023-02-24 23:17:16 +00:00
Michal Niewrzal
16b7901fde satellite/metabase: add piece size calculation to segment
This code is essentially replacement for eestream.CalcPieceSize. To call
eestream.CalcPieceSize we need eestream.RedundancyStrategy which is not
trivial to get as it requires infectious.FEC. For example infectious.FEC
creation is visible on GE loop observer CPU profile because we were
doing this for each segment in DB.

New method was added to storj.Redundancy and here we are just wiring it
with metabase Segment.

BenchmarkSegmentPieceSize
BenchmarkSegmentPieceSize/eestream.CalcPieceSize
BenchmarkSegmentPieceSize/eestream.CalcPieceSize-8         	    5822	    189189 ns/op	    9776 B/op	       8 allocs/op
BenchmarkSegmentPieceSize/segment.PieceSize
BenchmarkSegmentPieceSize/segment.PieceSize-8              	94721329	        11.49 ns/op	       0 B/op	       0 allocs/op

Change-Id: I5a8b4237aedd1424c54ed0af448061a236b00295
2023-02-22 11:04:02 +00:00
JT Olio
f905878ff9 go.mod: bump storj.io/uplink
this change brings in:

 * private/piecestore: disable noise
 * uplink: fix noise version report
 * private/piecestore: fix noise
 * private/metaclient: honor context cancellation on retries
 * private/metaclient: fix multi-passphrase ListObjects paging
 * private/metaclient: fix broken ListUploads method

it also adds:

 * private/storage/streams/batchaggregator: introduce Aggregator
 * private/storage/streams/streambatcher: introduce Batcher
 * private/storage/streams: pieceupload package
 * private/storage/streams: segmenttracker package

but these commits provably change no existing behavior, since they
only add unused, new packages and types.

Change-Id: Id9efc65117b7e1ddcb3ffc8cc15d3e665ed11d7d
2023-02-21 16:09:19 +00:00
Cameron
dd0013bdda bump stripe-go version to v72.90.0
v72.90.0 grants access to InvoiceParams.PendingInvoiceItemsBehavior
which allows the caller to specify whether the invoice should apply all
pending invoice items.

Change-Id: I7c8295d38791ed7daa3d5b1fce15438eb42a962f
2023-02-15 21:16:56 +00:00
JT Olio
ca13eca718 go.mod: bump libuplink to include noise
Change-Id: If5bceb139ce6fdf6c0792b4bb536bc61b54e32bb
2023-02-10 17:03:51 +00:00
JT Olio
522aed083d private/server,satellite/contact,misc: use new storj/common noise helpers
this change uses the new storj/common noise helpers, which:
 * add a security fix (require an expected node id for validating
   noise key attestations)
 * stops doing an unnecessary order signature validation (it's
   already been done inside of PutPiece)
 * removes some duplicate code

Change-Id: I5e67a08ff216cd9c5b0b82e40b4d9de664b6b0fc
2023-02-07 09:53:45 -05:00
Egon Elbre
873a202530 mod: bump storj.io/common
This bumps common, such that things build with Go 1.20.

Also, adds `go vet` checks for testsuite/storjscan and testsuite/ui.

The latest golang.org/x/bcrypt has a check that the new password is less
than 72 bytes, because bcrypt silently discarded them. This means our
own password validation has the same limitation. Old passwords should
still work fine.

Change-Id: Ibb8735b15eeb91460145906b81ae4e365e9ac418
2023-02-03 16:49:41 +02:00
Egon Elbre
3976a2fd1d go.mod: bump common to latest
Temporarily allow larger wasm size, while we migrate to picobuf.

Change-Id: I55916f47486465e4223cbd79f412ab2876fee042
2023-02-01 12:33:37 +02:00
JT Olio
3e01e9c07a storj/private/server: listen for noiseconn requests
Change-Id: Ia74dcc576fa0c97460207d93d129aa7d88cd2fba
2023-01-30 13:23:08 -05:00
Michal Niewrzal
bd8867cd09 satellite: adjust code to handle context cancelation for SQL queries
Our DB support in storj/private was updated to enable basic context
support for executing SQL queries. This change requires some small
adjustments as not all parts were working correctly.

storj/private commit with change:
4bc77107b7acfcc2f7ad65796d5dd3d7c64801e4

Change-Id: I64d7ed92788ea0920d12cecd1aa0e414720e9b9c
2023-01-27 10:07:43 +01:00
Kaloyan Raev
674068a5f2 mod: bump go-redis to v8.11.5
This removes the indirect dependency to go.opentelemetry.io/otel, which
causes a dependency conflict in storj.io/ipfs-go-ds-storj.

Change-Id: I28c1e3637bb23f7949988511051d39c08b3fc852
2023-01-26 12:29:03 +00:00