Commit Graph

551 Commits

Author SHA1 Message Date
Michal Niewrzal
3abe7ac0da private/testplanet: add uplink config to use different path cipher
From time to time it would be handy to test uplink operations
without encrypting paths.

Change-Id: I00c64fafa133cb711fcdcfe915c3bfbe5eb75d81
2023-02-22 10:28:25 +00:00
Michal Niewrzal
94d341bcf3 satellite: use ranged loop with GC-GF peer
Peer for generating bloom filters will be able to use ranged loop.

As an addition some cleanup were made:
* remove unused parts of GC BF peer (identity, version control)
* added missing Close method for ranged loop service
* some additional tests added

https://github.com/storj/storj/issues/5545

Change-Id: I9a3d85f5fffd2ebc7f2bf7ed024220117ab2be29
2023-02-13 18:32:21 +00:00
paul cannon
362d73f783 satellite/audit: add tests for concurrent audits
This commit introduces tests that perform multiple concurrent audits
against the same storage node, to make sure that doing so does not
create incorrect outcomes.

Refs: https://github.com/storj/storj/issues/5495
Change-Id: Iaae49e042306bfa59bdf04c1a1540667488e51e5
2023-02-09 19:58:50 +00:00
JT Olio
522aed083d private/server,satellite/contact,misc: use new storj/common noise helpers
this change uses the new storj/common noise helpers, which:
 * add a security fix (require an expected node id for validating
   noise key attestations)
 * stops doing an unnecessary order signature validation (it's
   already been done inside of PutPiece)
 * removes some duplicate code

Change-Id: I5e67a08ff216cd9c5b0b82e40b4d9de664b6b0fc
2023-02-07 09:53:45 -05:00
paul cannon
d6f8be1ec6 satellite/audit: add ContainmentSyncChore
We will be needing an infrequent chore to check which nodes are in the
reverify queue and synchronize that set with the 'contained' field in
the nodes db, since it is easily possible for them to get out of sync.
(We can't require that the reverification queue table be in the same
database as the nodes table, so maintaining consistency with SQL
transactions is out. Plus, even if they were in the same database, using
such SQL transactions to maintain consistency would be slow and
unwieldy.)

This commit adds the actual chore.

Refs: https://github.com/storj/storj/issues/5431
Change-Id: Id78b40bf69fae1ac39010e3b553315db8a1472bd
2023-02-07 01:18:49 +00:00
Egon Elbre
3146ad7f2e satellite/satellitedb: cleanup testing access
Previously we were exposing the testing facilities via interface casting
the necessary parts, however, when things are not part of the main
satellite.DB interface they need to be manually propagated. Rather than
relying on using hidden methods lets expose things as long as they don't
create a direct dependency to the database driver.

Change-Id: I2eb7d8b60f4b64de1320c2d32581f7be267c0f57
2023-02-06 14:36:11 +02:00
Michal Niewrzal
36cbf4f0b8 private/testplanet: enable full table scan detection
Testplanet tests will print into logs (WARN) if full table scan will
be detected. Test won't be failed automatically. That's because
currently we have multiple queries which are doing full table scan and it's not trivial to change.

https://github.com/storj/storj/issues/5471

Change-Id: Ia2fcbfb9102424d58f95e00071329454a8c1066e
2023-02-03 12:11:18 +00:00
Michal Niewrzal
c1fffe881a satellite/satellitedb/satellitedbtest: enable full table scan detection
Satellite DB tests will print into logs (WARN) if full table scan will
be detected. Test won't be failed automatically. That's because currently
we have multiple queries which are doing full table scan and it's not
trivial to change.

We may change that behavior when we will figure out how to skip
specific query from detection or we will fix all problematic queries.

https://github.com/storj/storj/issues/5471

Change-Id: Icafe782257a0d353e8bcdf6fa8a19c20b1091a0b
2023-02-02 12:14:05 +00:00
Egon Elbre
9a6be7f1e7 private/server: avoid relying on timing
The previous test relied on timing, but instead let's try dialing
the server and see whether we can do something with the connection.

We probably should test all the supported protocols instead of just tcp.

Change-Id: I9217494859faea0a7b93515aad706da4fdd8a140
2023-02-01 12:33:56 +02:00
JT Olio
382af95499 storagenode/contact: send noise key and settings as contact info
Change-Id: I1e7a83de36d5cf16eed8874091b15af1e0b73df7
2023-01-31 21:49:20 +00:00
JT Olio
c17ceef093 private/server: tcp_fastopen failures should not kill the server
Change-Id: Iec42524f83619c22d8be26f373978e8abd468ee8
2023-01-31 14:48:14 +00:00
Egon Elbre
053beef8c4 private/server: fix fastopen socket option
Change-Id: I7acbe0562ebfe9d78dae5c57f6057b8364dfa0e7
2023-01-31 14:48:08 +00:00
Egon Elbre
9bb9275acf private/testplanet: add some more error wrapping
Change-Id: I4cbdaafc0f6827e569d9b70acac3ce448359f125
2023-01-31 15:44:59 +02:00
JT Olio
2a641b60d3 private/server: support tcp fastopen
we are not enabling it on client-side code yet, but
it will be hard to test this in the wild without
server-side support.

this is phase 2 of tcp fast open support. see
https://forum.storj.io/t/two-new-blueprints-design-drafts-seeking-feedback-replacing-tls-with-noise-and-tcp-fastopen/21053/12
for more details

Change-Id: I20362be4c49abab90afcc9b6572ef9f79816345b
2023-01-30 13:30:43 -05:00
JT Olio
3e01e9c07a storj/private/server: listen for noiseconn requests
Change-Id: Ia74dcc576fa0c97460207d93d129aa7d88cd2fba
2023-01-30 13:23:08 -05:00
Michal Niewrzal
bd8867cd09 satellite: adjust code to handle context cancelation for SQL queries
Our DB support in storj/private was updated to enable basic context
support for executing SQL queries. This change requires some small
adjustments as not all parts were working correctly.

storj/private commit with change:
4bc77107b7acfcc2f7ad65796d5dd3d7c64801e4

Change-Id: I64d7ed92788ea0920d12cecd1aa0e414720e9b9c
2023-01-27 10:07:43 +01:00
Egon Elbre
c320835a38 private/migrate: add version to failed migration step
Change-Id: I98984636f2f7df794d54e5c65cf649e7f0beeded
2023-01-23 15:29:35 +02:00
Moby von Briesen
3d1007ae18 private/apigen: Change order of operations in api generation
Move the IsAuthenticated check until after initial parameter
parsing/validation. IsAuthenticated will be more expensive than
parsing/validation, so we should fail before auth if possible.

Change-Id: I96a020892eabcb750e8ec9ecc1d8b7d9bf8bf573
2023-01-11 16:54:53 +00:00
Qweder93
8c69ee62fc {cmd/storj-sim, satellite/rangedloop}: added rangedloop to storj-sim, removed identity
added in storj-sim rangedloop for each satellite, to verify it works for metrics oveserver,
removed identity from rangedloop peer as we never use it, added logs on service run, added loop
to service instead of endless for loop, interval value to config

Closes: https://github.com/storj/storj/issues/5414

Change-Id: Ibc3b06071b68feda4a35b45da2bbe36e22a02fc8
2023-01-05 11:29:00 +00:00
Wilfred Asomani
e598c2b3b1 console/userinfo: stub userinfo endpoint
This change stubs userinfo endpoint from storj/common/pb/userinfo.proto.
It also adds config for allowed peers, and a method for verifying peers.

Issue: https://github.com/storj/storj/issues/5358

Change-Id: I057a0e873a9e9b3b9ad0bba69305f0d708bd9b9e
2022-12-20 06:10:08 +00:00
Michal Niewrzal
5110803102 storagenode/piecestore: add Exists endpoint
Adds new method Exists which can be used to verify which
requested piece ids exists on storage node. Will verify only pieces
which belongs to the satellite that used that endpoint.

Minum WASM size was increased a bit.

https://github.com/storj/storj/issues/5415

Change-Id: Ia5f9cadeb526541b2776a8973eb7d50133ad8636
2022-12-17 04:08:26 +00:00
paul cannon
7b851b42f7 satellite/audit: split out auditor process
This change creates a new independent process, the 'auditor', comparable
to the repairer, gc, and api processes. This will allow auditors to be
scaled independently of the core.

Refs: https://github.com/storj/storj/issues/5251
Change-Id: I8a29eeb0a6e35753dfa0eab5c1246048065d1e91
2022-12-16 12:44:32 -06:00
paul cannon
fc905a15f7 satellite/audit: newContainment->containment
Now that all the reverification changes have been made and the old code
is out of the way, this commit renames the new things back to the old
names. Mostly, this involves renaming "newContainment" to "containment"
or "NewContainment" to "Containment", but there are a few other renames
that have been promised and are carried out here.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: I34e2b857ea338acbb8421cdac18b17f2974f233c
2022-12-16 17:59:52 +00:00
paul cannon
0342ca1aa6 satellite/audit: delete now-unused code
Now that we are doing scalable piecewise reverifications, the code for
handling the old way of doing things (containment, pending audits,
reporting, testing) can now be removed.

Refs: https://github.com/storj/storj/issues/5230
Change-Id: Ief1a75f423eff682e8f3d57804e343b3409a6631
2022-12-16 14:53:39 +00:00
Qweder93
9d7e94b57b private/testplanet: extended satellite with rangedloop peer
Change-Id: I3c82f5c970c4fef139fc089937e43c5cb1769815
2022-12-14 16:21:50 +02:00
paul cannon
c575a21509 satellite/audit: add audit.ReverifyWorker
Here we add a worker class comparable to audit.Worker, which will be
responsible for pulling items off of the reverification queue and
calling reverifier.ReverifyPiece on them.

Note that piecewise reverification audits (which this will control) are
not yet being done. That is, nothing is being added to the
reverification queue at this point.

Refs: https://github.com/storj/storj/issues/5251
Change-Id: I94e28830e27caa49f2c8bd4a2336533e187ab69c
2022-12-12 11:57:08 +00:00
Clement Sam
7461ffe148 {storagenode,web/multinode}: fix storage usage db/cache retrieval queries
The query changes we did while fixing the usage graph led to wrong
payout calculations directly linked to disk space.

This change:

- avoids converting from Bh to B directly in the query
- returns the at_rest_total in the original bytes*hour value
- returns at_rest_total_bytes as the calculated disk spaced used in bytes
- uses the at_rest_total_bytes only for the disk space graph
- return summary_bytes as the average disk space used within the specified date
- updates the disk space graph header to "average disk space used this month"

The total disk used in the month is also displayed in B not B*day

Resolves https://github.com/storj/storj/issues/5355

Change-Id: I2cfefb0fe711f9c59de2adb547c4ab50b05c7cbb
2022-12-09 11:07:33 +00:00
paul cannon
378b8915c4 satellite/{satellitedb,audit}: add NewContainment
NewContainment will replace Containment later in this commit chain, but
for now it is not yet being used.

NewContainment will allow a node to be contained for multiple pending
reverify jobs at a time. It is implemented by way of the reverify queue.

Refs: https://github.com/storj/storj/issues/5231
Change-Id: I126eda0b3dfc4710a88fe4a5f41780618ec19101
2022-12-07 18:03:37 +00:00
Clement Sam
746c3b2d83 private/multinodepb: return at_rest_total_bytes and summary_bytes for StorageUsage endpoint
at_rest_total_bytes and summary_bytes are storage usages return as bytes
instead of bytes*hour. This is used for the disk space graph.

Updates https://github.com/storj/storj/issues/5355

Change-Id: I81f77fe9b9069cf3b29ab681586e506363e5b066
2022-12-06 12:27:34 +00:00
paul cannon
2617925f8d satellite/audit: Split Reverifier from Verifier
Adding a new worker comparable to Verifier, called Reverifier; as the
name suggests, it will be used for reverifications, whereas Verifier
will be used for verifications.

This allows distinct logging from the two classes, plus we can add some
configuration that is specific to the Reverifier.

There is a slight modification to GetNextJob that goes along with this.

This should have no impact on operational concerns.

Refs: https://github.com/storj/storj/issues/5251
Change-Id: Ie60d2d833bc5db8660bb463dd93c764bb40fc49c
2022-12-05 09:55:49 -06:00
Egon Elbre
8777523255 private/testplanet: disable WAL for storagenodes
Change-Id: I1be4f7901c830e829118afeb90f04b87df555459
2022-12-05 11:41:06 +00:00
Andrew Harding
b2b8efff7d Makefile: disable postgres fsync in the test container
This change turns off fsync on the postgres container used for tests. This
reduces migration time significantly when initializing new satellite
databases.

The change also includes a new benchmark for satellite initialization in
testplanet.

$ benchstat old.txt new.txt name                       old time/op  new time/op  delta
Run_Satellite/Postgres-16   1.36s ± 0%   0.08s ± 0%   ~     (p=1.000 n=1+1)

Change-Id: Ic954767133864770cf652b0dfdcd6b109a167b5f
2022-12-01 22:03:31 +00:00
Andrew Harding
4fdea51d5c storagenode/storagenodedb: faster test db init
Running all of the migrations necessary to initialize a storage node
database takes a significant amount of time during runs.

The package current supports initializing a database from manually coalesced
migration data (i.e. snapshot) which improves the situation somewhat.

This change takes things a bit further by changing the snapshot code to
instead hydrate the database directory from a pre-generated snapshot zip
file.

name                                old time/op  new time/op  delta
Run_StorageNodeCount_4/Postgres-16   2.50s ± 0%   0.16s ± 0%   ~     (p=1.000 n=1+1)

Change-Id: I213bbba5f9199497fbe8ce889b627e853f8b29a0
2022-12-01 20:45:36 +00:00
Egon Elbre
4df1f5b50f private/testplanet: add benchmark for testplanet creation
Change-Id: I0f44ea417b6b570b2b3ea6dd82945f03119524ce
2022-11-30 17:08:33 +02:00
Jeremy Wharton
7a2be3e6f6 private/web,satellite/console/.../consoleapi: serve rate limiting errors as JSON
This change causes rate limiting errors to be returned to the client
as JSON objects rather than plain text to prevent the satellite UI from
encountering issues when trying to parse them.

Resolves storj/customer-issues#88

Change-Id: I11abd19068927a22f1c28d18fc99e7dad8461834
2022-11-23 17:56:07 +00:00
paul cannon
8b494f3740 satellite/audit: use db for auditor queue
As part of the effort of splitting out the auditor workers to their own
process, we are transitioning the communication between the auditor
chore and the verification workers to a queue implemented in the
database, rather than the sequence of in-memory queues we used to use.

This logical database is safely partitionable from the rest of
satelliteDB.

Refs: https://github.com/storj/storj/issues/5251

Change-Id: I6cd31ac5265423271fbafe6127a86172c5cb53dc
2022-11-22 14:04:00 +00:00
Cameron
87660bd9b3 satellite/overlay/offlinenodes: insert offline nodes into node events
Add a new chore to periodically insert nodes who are offline and
have not gotten an offline email in a certain amount of time into node
events

Change-Id: I658b385bb777b0240c98092946a93d65bee94abc
2022-11-18 12:10:06 -05:00
Cameron
57be07f60a satellite/nodeevents: add Chore
Create NodeEvents Chore on satellite core to read nodeevents DB and
notify node operators on node events. The chore sends notifications
grouped by email and event type: it selects the oldest entry in
nodeevents.DB and also any other event with the same email and event
type no matter how old it is. The oldest entry of a group must exist for
a minimum amount of time before that group can be selected, however.
This minimum amount of time is a configurable value:
--node-events.selection-wait-period. This wait period allows us to
combine events of the same time and same email address into a singular
email.

Change-Id: I8b444aa324d2dae265cc27d9e9e85faef79195d8
2022-11-18 12:00:14 +00:00
Yaroslav Vorobiov
7b13af1184 private/blockchain: address checksum hex
Updates blockchain.Address.Hex to return mixedcase
checksum hash.

Change-Id: I74c830b49eceb30876c8ef20d3adc4816de51a64
2022-11-08 18:19:07 +00:00
Cameron
f06da25c3d satellite/overlay: add nodeevents.DB to satellite overlay service
Add nodeevents.DB to satellite overlay service so we can insert node
events into the nodeevents DB.

Change-Id: I642c0ccc9941ecdb08cb22d5c8cf701959a55156
2022-11-02 15:56:37 +00:00
Egon Elbre
aeb645d32b all: replace deprecated ioutil
Change-Id: I60b0bbf5b68b066e2d44b8b99438594d600a3c2d
2022-10-31 15:50:41 +00:00
JT Olio
58a9c55f36 mod: bump dependencies
- storj.io/common

Change-Id: Ib78154acc253a13683495abfdd96d702625fdce8
2022-10-19 17:01:53 +00:00
Yaroslav Vorobiov
5cfd64579e private/blockchain: fix Hex to add 0x prefix
Change-Id: I47da9cea160fe9bbdf452956836e3d4b5b44646f
2022-10-11 19:15:24 +00:00
Egon Elbre
8b70f969b6 all: fix nolint directives
Change-Id: I261c8b12e4961e6401cc4024fa5abc35b1a5efa6
2022-10-11 18:31:20 +00:00
Egon Elbre
ff22fc7ddd all: fix deprecated ioutil commands
Change-Id: I59db35116ec7215a1b8e2ae7dbd319fa099adfac
2022-10-11 15:27:29 +00:00
Márton Elek
ac6bb1e187 storage: experimental flag to turn off file sync
Full file sync before saving files to piecestore seems to be very expensive.

Bwfore we do any step to eliminate them we are planning to do more measurement which requires a temporary flag to turn it off. (not for production).

Change-Id: I5cb8f8cb348ca3590fb5eae14d02edb3f0424617
2022-10-06 13:10:34 +00:00
Michal Niewrzal
a32619c6e3 private/testplanet: benchmark with testplanet
Currently, its not strightforward how to benchmark with testplanet.
This change add Bench method to make it easy.

Change-Id: I212dfe71a18bb6ddd7a127e5b6d313b1b0c1f824
2022-09-23 16:06:43 +00:00
Michal Niewrzal
4e71dd302a satellite/{metainfo,metabase}: prepare feature flag MultipleVersions
We will introduce new logic for creating new objects (BeginObject).
Instead of using single version internally (1) we will be selecting first
available version during object creation. Because we need to be sure
that everything is wired up correctly we need a feature flag to be
able to control if new feature is enabled.

Change-Id: If0f8496397130811f43bf9db9fdcc2b30cd2e4ca
2022-09-21 13:07:13 +00:00
Erik van Velzen
e6b5501f9b satellite/gc/sender: new service to send retain filters
Implement a new service to read retain filter from a bucket and
send them out to storagenodes.

This allows the retain filters to be generated by a separate command on
a backup of the database.

Paralellism (setting ConcurrentSends) and end-to-end garbage collection
tests will be restored in a subsequent commit.

Solves https://github.com/storj/team-metainfo/issues/121

Change-Id: Iaf8a33fbf6987676cc3cf74a18a8078916fe673d
2022-09-20 11:49:40 +00:00
Egon Elbre
794a4cc365 private/apigen: make code compile with 1.16
Our Test Versions still requires 1.16 to be compatible with our oldest
uplink versions. These changes make the code compile with 1.16.

Also, it makes go generate work in private/apigen/example.

Change-Id: Ib2f7493941a16f361328fe01d2be293f26123719
2022-09-02 16:34:46 +03:00