Commit Graph

7568 Commits

Author SHA1 Message Date
JT Olio
572c5b305b satellite/context: fix pingme test
Change-Id: I30fcfe1dea43d65e457c76dd44f475b5c197cbee
2023-01-30 15:01:32 -05:00
JT Olio
2a641b60d3 private/server: support tcp fastopen
we are not enabling it on client-side code yet, but
it will be hard to test this in the wild without
server-side support.

this is phase 2 of tcp fast open support. see
https://forum.storj.io/t/two-new-blueprints-design-drafts-seeking-feedback-replacing-tls-with-noise-and-tcp-fastopen/21053/12
for more details

Change-Id: I20362be4c49abab90afcc9b6572ef9f79816345b
2023-01-30 13:30:43 -05:00
JT Olio
3e01e9c07a storj/private/server: listen for noiseconn requests
Change-Id: Ia74dcc576fa0c97460207d93d129aa7d88cd2fba
2023-01-30 13:23:08 -05:00
Wilfred Asomani
73ffa0827f cmd/tools: remove nullify-bad-user-agents tool
This tool is being removed because it has served its purpose and was blocking another removal from being verified.

Change-Id: Ie888aa7ae1b153a34210af3a5d5a3682b381ba82
2023-01-30 15:57:33 +00:00
paul cannon
2f04e20627 storage/filestore: better error message on data corruption
A user on the forum was seeing the error "bad message", which was not
very helpful. This case from the ext4 filesystem using the code EBADMSG
to indicate it detected an invalid CRC, suggesting disk corruption.

This change adds some explanatory information about probable disk
corruption to all errors coming from the (*blobInfo).Stat() call, which
is where storagenode fs corruption problems will usually manifest.

Refs: https://github.com/storj/storj/issues/5375
Change-Id: I87f4a800236050415c4191ef1a0fc952f9def315
2023-01-30 08:54:06 -06:00
paul cannon
ed7c82439d storage/filestore: avoid stat() during walkNamespaceInPath
Calling stat() (really, lstat()) on every file during a directory walk
is the step that takes up the most time. Furthermore, not all directory
walk uses _need_ to have a stat done on every file. Therefore, in this
commit we avoid doing the stat at the lowest level of
walkNamespaceInPath. The stat will still be done when it is requested,
with the Stat() method on the blobInfo object.

The major upside of this is that we can avoid the stat call on most
files during a Retain operation. This should speed up garbage collection
considerably.

The major downside is that walkNamespaceInPath will no longer
automatically skip over directories that are named like blob files, or
blob files which are deleted between readdir() and stat(). Callers to
walkNamespaceInPath and its variants (WalkNamespace,
WalkSatellitePieces, etc) are now expected to handle these cases
individually.

Thanks to forum member Toyoo for the insight that this would speed up
garbage collection.

Refs: https://github.com/storj/storj/issues/5454
Change-Id: I72930573d58928fa25057ed89cd4ec474b884199
2023-01-30 13:47:03 +00:00
NickolaiYurchenko
facbd65882 web/satellite: EstimatedCostsAndCredits components migrated to use composition api
Change-Id: I11ee04880a5dbdc5494f7a51c0b4077e88827a11
2023-01-30 09:22:25 +00:00
Lizzy Thomson
1bff41e6b3 cmd/tools: add migration tool to update salt column when null
Add migration tool (and test) to update salt column in projects table
with the SHA-256 hash of the project ID when null

Issue https://github.com/storj/storj-private/issues/66

Change-Id: Ib8d484ac8d6ee25859064d803e2ac8fb46b45921
2023-01-27 22:04:07 +00:00
Andrew Harding
abd0ad92dc satellite/metainfo: RetryBeginSegmentPieces RPC implementation
Part of:
https://github.com/storj/uplink/issues/120

Change-Id: I2a2873455f7498ffd31f50ade16c173fe1d18157
2023-01-27 15:04:59 +00:00
Michal Niewrzal
bd8867cd09 satellite: adjust code to handle context cancelation for SQL queries
Our DB support in storj/private was updated to enable basic context
support for executing SQL queries. This change requires some small
adjustments as not all parts were working correctly.

storj/private commit with change:
4bc77107b7acfcc2f7ad65796d5dd3d7c64801e4

Change-Id: I64d7ed92788ea0920d12cecd1aa0e414720e9b9c
2023-01-27 10:07:43 +01:00
Vitalii
6f11c8b32c web/satellite: show create/enter passphrase modal after login
Show create/enter passphrase modal after login for new project level passphrase flow.
Also fixed buckets view mounted hook to load create bucket modal instead of old flow.

Issue:
https://github.com/storj/storj/issues/5510

Change-Id: If9ea70faaa2987f336d72d55a6ed2bbd02ced592
2023-01-26 18:06:59 +02:00
Egon Elbre
e4b325537e satellite/satellitedb: missing rows.Close in error case
Change-Id: Ib1cae87171c302653771507cdc31d0c11a68f504
2023-01-26 15:21:48 +00:00
Clement Sam
6277d93c82 cmd/storagenode: remove storagenode dockerfiles
Updates https://github.com/storj/storagenode-docker/issues/1

Change-Id: I6b2f42194cb0e32f1b3978072fd6e1cdec753704
2023-01-26 13:49:12 +00:00
Kaloyan Raev
674068a5f2 mod: bump go-redis to v8.11.5
This removes the indirect dependency to go.opentelemetry.io/otel, which
causes a dependency conflict in storj.io/ipfs-go-ds-storj.

Change-Id: I28c1e3637bb23f7949988511051d39c08b3fc852
2023-01-26 12:29:03 +00:00
Michal Niewrzal
b5c5c62d7b satellite/metabase: add missing error check
Change-Id: I6891f0647cb8e4a8dd6534eaa3588bbe76e2721d
2023-01-26 11:18:14 +00:00
Kaloyan Raev
805fcdf341 Revert "mod: bump go-redis to v8.11.5"
This reverts commit b2b564cc13.

Change-Id: I79a6b92758d1095113c85d49d673b243c31e6c82
2023-01-26 13:08:18 +02:00
Kaloyan Raev
b2b564cc13 mod: bump go-redis to v8.11.5
This removes the indirect dependency to go.opentelemetry.io/otel, which
causes a dependency conflict in storj.io/ipfs-go-ds-storj.

Change-Id: I28c1e3637bb23f7949988511051d39c08b3fc852
2023-01-26 13:07:13 +02:00
Michal Niewrzal
bb2ac4279a satellite/metainfo: enable multiple versions fix by default
Change-Id: I6cc7ba928e59cc8b8fa50f2ab19ec5418dc76507
2023-01-26 09:35:20 +00:00
Moby von Briesen
7c152f7ea0 satellite/console: Filter new characters out of user input
HTML and JS escape user input for create and update user.

Change-Id: I91d972f454341a5a7f333d006a87c6f854595490
2023-01-25 22:52:37 +00:00
NickolaiYurchenko
a53849f874 web/satellite: PeriodSelection, SmallDepositHistory, SortingHeader migrated to use composition api
Change-Id: I029b8daa8268aae1fd9a12ab83a368f5e6862ba7
2023-01-25 22:17:52 +00:00
NickolaiYurchenko
15efa1e319 web/satellite: CouponArea, DetailedHistory, PaymentHistoryItemDate, PaymentsItem migrated to use composition api
Change-Id: I3963835053e69dd3b989a417b295abaa0b645284
2023-01-25 21:47:32 +00:00
Andrew Harding
e7b35381f2 mod: bump common and uplink deps
Change-Id: Ie2d87f517b57a8760f50ac5c9f1b8b0448f5b3b5
2023-01-25 13:34:47 -07:00
Egon Elbre
d0686648db certificate/authorization: implement gob to pb migration
Change-Id: I6f33f5802e3f0a3c8a5f0c3cff61ef836a645c41
2023-01-25 11:09:05 +02:00
Egon Elbre
10c552fec4 certificate/authorization: add encode as pb
gob package is not stable across Go version, let's switch to protobuf
for encoding these. We still need backwards compatibility for the
moment.

Change-Id: If1da50658ab39a75d1b2b1f988356b56347cac14
2023-01-25 10:28:36 +02:00
Egon Elbre
e9bc06608e certificate/certificatepb: add definitions for migration
Also fixing the timestamp in .proto which was actually int64.

Change-Id: I65e3172793a49bf8bfd946b199016204ed72f221
2023-01-25 10:28:36 +02:00
JT Olio
e40191afd6 storj: upgrade to use latest storj/common NodeAddress
Change-Id: I5987391bcfe5f6dfd7b525698c337a4cbda9b76e
2023-01-25 01:37:26 +00:00
Vitalii
c98ef89311 web/satellite: fixed multiple passphrases notification for object browser
Fixed notifications link to toggle manage passphrase modal instead of redirecting to buckets screen.

Issue:
https://github.com/storj/storj/issues/5452

Change-Id: I84ce67da7445c6316eb0658606c0129bb4af680e
2023-01-24 16:21:35 +00:00
Erik van Velzen
eab595397f cmd/uplink: add --analytics flag
Add a flag to enable/disable analytics so uplink can be run
non-interactively. Also when run non-interactively for the first time
it will not error any more but instead default to disable analytics.

Part of https://github.com/storj/storj/issues/5126

Change-Id: I07ac8a040664334efcb4e2536f26c330c1751a6f
2023-01-24 15:33:55 +00:00
Yaroslav Vorobiov
3639c5ee1e satellite/payments/stripecoinpayments: skip paying invoices with due date
When running pay invoices command skip invoices that has due date set, so
customers can still pay it themselves until due date.

https://github.com/storj/storj/issues/5453

Change-Id: I8e557062491ab0c8246b28bc5ca57e845eb32e29
2023-01-24 15:12:29 +01:00
Clement Sam
95960572b3 storagenode/piecestore: improve logs for incoming requests
- Adds "Remote Address" field to all INFO logs related to GET,
PUT, and DELETE requests
- Adds Offset and Size fields to all info logs related to GET
requests

Resolves https://github.com/storj/storj/issues/5404

Change-Id: I5dab1867619385362e5f1e0455dfab17d295a37a
2023-01-24 10:23:34 +00:00
Erik van Velzen
cb01aca13d cmd/uplink: add docker image
Add a docker image for uplink-cli and push it to docker hub.
We used to have this before the change to uplinkng. I'm not
sure if the pushing works, we'll see after merge.

To test, build an image with `make uplink-image`, read the tag from the
output and run normal uplink-cli commands using
`docker run -it storjlabs/uplink:df9bbceca-uplink-docker-go1.18.8-amd64 [command]`

Part of https://github.com/storj/uplink/issues/109

Change-Id: I8a10aab2b778951ff42a22ba2f252c581eb66b65
2023-01-24 09:48:02 +00:00
paul cannon
740cb0d9c7 cmd/tools/segment-verify: fix read-csv subcommand
We were reading in a segment's stream ID and position, and assuming that
was enough for the downloader. But of course, the downloader needs
AliasPieces filled in. So now we request each segment record from the
metabase and fill in the VerifySegment records entirely.

Change-Id: If85236388eb99a65e2cb739aa976bd49ee2b2c89
2023-01-24 09:08:03 +00:00
Michal Niewrzal
8850fde9f5 satellite/metabase/metabasetest: detect full scan table queries
This is automated test around metabase tests. It's detecting queries
which were performing full table scan during test execution.

After merging this and checking that this is not problematic in any way
we will enable this also for testplanet tests.

One query was adjusted to avoid full table scan and pass new check.

https://github.com/storj/storj/issues/5471

Change-Id: I2d176f0c25deda801e8a731b75954f83d18cc1ce
2023-01-23 19:40:20 +00:00
NickolaiYurchenko
bf5b378836 web/satellite: CreditCardContainer and billing Overview migrated to use composition api
Change-Id: I8e0a027671aa996f42edf916c9b0434e773de618
2023-01-23 18:53:25 +00:00
Yaroslav Vorobiov
b86ce0d527 satellite/payments/storjscan: return 404 when there is no wallet
When there is no wallet in the database for a particular customer
return 404 http response status code instread of internal server error.
Change web/satellite payments API to return empty wallet on 404 response
code instead of throwing an error.

Change-Id: Ib44914f9ed002382258968fb81846f2b97dee0fe
2023-01-23 18:07:33 +00:00
Vitalii
d1c230fe4d web/satellite: fix Vuex state clearing on session timeout
Fixed Vuex state clearing on session timeout.
Now state is being cleared right after redirect to login and before logout API call.

Change-Id: I3cddb85735899f0913a273a49335730a4097435e
2023-01-23 17:58:26 +02:00
Qweder93
d6a948f59d satellite/repair : implemented ranged loop observer
implemented observer and partial, created new structures to keep mon
metrics remain in same way as in segment loop

Change-Id: I209c126096c84b94d4717332e56238266f6cd004
2023-01-23 14:23:03 +00:00
Egon Elbre
c320835a38 private/migrate: add version to failed migration step
Change-Id: I98984636f2f7df794d54e5c65cf649e7f0beeded
2023-01-23 15:29:35 +02:00
Wilfred Asomani
250f505f4d web/satellite: fix new folder button in file browser
This change opens the "new folder" modal on click of the "new folder"
button in the file browser

Change-Id: I063d644c70ad1eff005eb509d97b95cf30aca00a
2023-01-23 11:19:12 +00:00
NickolaiYurchenko
b9c0bbce2f web/satellite: access page responsiveness added
navigation logo shrink fixed by updating icon;
https://github.com/storj/storj/issues/5461
https://github.com/storj/storj/issues/5458

Change-Id: I6794cbf1c63786978336810f4c8151dc7825c7c8
2023-01-23 12:06:03 +02:00
NickolaiYurchenko
5a4f089750 web/satellite: Billing history and Coupons components migrated to use composition api
Change-Id: I589af5376af5d94a15069c1d6ff4fbe2ce1c6fb0
2023-01-20 16:16:23 +00:00
Michal Niewrzal
984da95543 Jenkinsfile: collect full table scan queries from tests
This change is first attempt to detect queries which are introducing
bad performance with full table scans. CRDB have special query to list
all queries which performs FTS.

With this change we are collecting all full table scan queries executed while unit tests. Results are combined from all started CRDB nodes into single results file. Such output can be later reviewed. This approach is not perfect but it's just one piece what we would like to build to detect FTS before pushing it into production.

part of https://github.com/storj/storj/issues/5471

Change-Id: I615fd9e624a136bb0870f9ff9c10e4eb5918339c
2023-01-20 12:32:47 +00:00
Michal Niewrzal
4cbb1ed296 satellite/orders: log bandwidth values we are dropping
When we have problem with inserting bandwidth amounts into cache
or DB we are logging information about it but log entries are not
very detailed. This change adds bandwidth amounts to the log entry.

https://github.com/storj/storj/issues/5470

Change-Id: I55ccad837d17b141501d3def1dec7ad5f3acdb0b
2023-01-20 09:28:25 +00:00
JT Olio
8d69837f02 storagenode/piecestore: handle order if provided in first message
we may in the future want to accept orders as part of the initial
request message
(e.g. https://review.dev.storj.io/c/storj/uplink/+/9246).

this change is forward compatible but continues to work with
existing clients.

Change-Id: I475ad50d6cbfee8a1f843383230698e4ef9b9e54
2023-01-20 08:53:22 +00:00
Cameron
7ffa9ef914 satellite/payments: add config for partner package plans
Create a config to specify one-time prices and corresponding coupon
ids for partners.

github issue: https://github.com/storj/storj-private/issues/118

Change-Id: I67b26e7208b12ba8f0e6dc1b164dd9545b09cac0
2023-01-20 02:33:36 +00:00
paul cannon
c3b5c18d00 cmd/tools/segment-verify: learn to take a CSV list of segments as input
This will allow us to retry some specific segments from
segments-retry.csv with particularly high counts of "retry" pieces.

Change-Id: I48fd419cc0350a3be4c9e77ce8d28871565b7f97
2023-01-18 20:53:27 +00:00
Wilfred Asomani
d76f764b72 satellite/web: replace cancel with back buttons
This replaces "Cancel" with "Back" on Clear, Create and Switch steps.
It also removes margin-top from the button rows in Clear, Create steps.

Issue: https://github.com/storj/storj/issues/5463

Change-Id: Ib2ef0dba9b91e82a7cf5b042fad0162128fa071a
2023-01-18 17:51:18 +00:00
Vitalii
65a090d111 satellite/console: enable project level passphrase by default
Enable project level passphrase by default.

Issue:
https://github.com/storj/storj/issues/5456

Change-Id: I8c08f2d81a3ce1b5e0134172b7440309b850c5c2
2023-01-18 15:32:29 +02:00
JT Olio
b022f371d2 blueprint: tcp fastopen
Change-Id: I20fc843b916b5ae0b72acfa5a03cf75f211739c0
2023-01-17 20:35:47 +00:00
JT Olio
14c91aa05d blueprint: noise over tcp simplification
the noise over tcp blueprint had two unresolved questions:

 * what do we do for IPv6?
 * what do we do for UDP support?

this change punts those two questions for later and explicitly
rejects solving them as part of noise over TCP for now.

this change also punts renaming NodeURL to reduce codebase
churn.

Change-Id: Ib209942bae61716b2ff6cea852c336af072d8bc5
2023-01-17 16:06:05 +00:00