Commit Graph

80 Commits

Author SHA1 Message Date
Márton Elek
0affe03007 cmd/tools/tag-signer: make the output less noisy
This patch removes the following lines from the output (with disable tracing + set the log level to warn):

```
2023-09-25T15:30:58+02:00	INFO	process/tracing.go:73	Anonymized tracing enabled
2023-09-25T15:30:58+02:00	DEBUG	tracing collector	monkit-jaeger@v0.0.0-20220915074555-d100d7589f41/udp.go:128	started
2023-09-25T15:30:58+02:00	DEBUG	process/debug.go:37	debug server listening on 127.0.0.1:34803
```

Change-Id: Iccbf4fc3bde9436e0571943d0d85c51ebc766ef9
2023-09-26 16:30:48 +00:00
Márton Elek
98921f9faa satellite/overlay: fix placement selection config parsing
When we do `satellite run api --placement '...'`, the placement rules are not parsed well.

The problem is based on `viper.AllSettings()`, and the main logic is sg. like this (from a new unit test):

```
		r := ConfigurablePlacementRule{}
		err := r.Set(p)
		require.NoError(t, err)
		serialized := r.String()

		r2 := ConfigurablePlacementRule{}
		err = r2.Set(serialized)
		require.NoError(t, err)

		require.Equal(t, p, r2.String())
```

All settings evaluates the placement rules in `ConfigurablePlacementRules` and stores the string representation.

The problem is that we don't have proper `String()` implementation (it prints out the structs instead of the original definition.

There are two main solutions for this problem:

 1. We can fix the `String()`. When we parse a placement rule, the `String()` method should print out the original definition
 2. We can switch to use pure string as configuration parameter, and parse the rules only when required.

I feel that 1 is error prone, we can do it (and in this patch I added a lot of `String()` implementations, but it's hard to be sure that our `String()` logic is inline with the parsing logic.

Therefore I decided to make the configuration value of the placements a string (or a wrapper around string).

That's the main reason why this patch seems to be big, as I updated all the usages.

But the main part is in beginning of the `placement.go` (configuration parsing is not a pflag.Value implementation any more, but a separated step).

And `filter.go`, (a few more String implementation for filters.

https://github.com/storj/storj/issues/6248

Change-Id: I47c762d3514342b76a2e85683b1c891502a0756a
2023-09-21 14:31:41 +00:00
Egon Elbre
1f261bcc70 go.mod: bump lang to 1.19 and common
Change-Id: I8d91f97d786456da29ebe89a78412c50efbb8ccc
2023-08-10 18:41:15 +03:00
Michal Niewrzal
7f249ab7ca cmd/tools: remove migrate-segment-copies tool
Migration was done. We can remove tool now.

https://github.com/storj/storj/issues/5891

Change-Id: I5d56bad1ac680cd77dabfcf271788e100a6a435b
2023-08-08 10:00:40 +00:00
Michal Niewrzal
d9525a0f27 cmd/tools/migrate-segment-copies: fix placement scan from DB
Placement can be null in DB and we need adjust scanning this column
from DB.

Additionally this change sets application name for DB connection.

Change-Id: I3c7d6294f4a3e5e441160b2fd4aeafffe705ec76
2023-07-21 13:12:08 +00:00
Michal Niewrzal
31bb6d54c7 cmd/tools: add tool to migrate segment copies metadata
We need migrate all existing segment copies to contain all the same
metadata as original segment. So far we were not duplicating stored
pieces but we are changing this behavior right now. We will use this
tool after enabling new way of doing server side copies.

Fixes https://github.com/storj/storj/issues/5890

Change-Id: Ia9ca12486f3c527abd28949eb438d1c4c7138d55
2023-07-18 15:12:51 +00:00
Márton Elek
97a89c3476 satellite: switch to use nodefilters instead of old placement.AllowedCountry
placement.AllowedCountry is the old way to specify placement, with the new approach we can use a more generic (dynamic method), which can check full node information instead of just the country code.

The 90% of this patch is just search and replace:

 * we need to use NodeFilters instead of placement.AllowedCountry
 * which means, we need an initialized PlacementRules available everywhere
 * which means we need to configure the placement rules

The remaining 10% is the placement.go, where we introduced a new type of configuration (lightweight expression language) to define any kind of placement without code change.

Change-Id: Ie644b0b1840871b0e6bbcf80c6b50a947503d7df
2023-07-07 16:55:45 +00:00
Márton Elek
70cdca5d3c
satellite: move satellite/nodeselection/uploadselection => satellite/nodeselection
All the files in uploadselection are (in fact) related to generic node selection, and used not only for upload,
but for download, repair, etc...

Change-Id: Ie4098318a6f8f0bbf672d432761e87047d3762ab
2023-07-07 10:32:03 +02:00
Márton Elek
d38b8fa2c4 satellite/nodeselection: use the same Node object from overlay and nodeselection
We use two different Node types in `overlay` and `uploadnodeselection` and converting back and forth.

Using the same object would allow us to use a unified node selection interface everywhere.

Change-Id: Ie71e29d60184ee0e5b4547eb54325f09c418f73c
2023-07-03 16:59:33 +00:00
Márton Elek
20a47034a5
cmd/tools: tag-signer utility to create signed node tags
Change-Id: I2983d688a109325a02fcd060ca1a2d4eb8e9e931
2023-07-03 18:10:08 +02:00
Egon Elbre
edbea5efe1 go.mod: bump to pgx/v5
Change-Id: I31cf3bec1d7db94f0f612f6ed04b782f8b04d876
2023-06-14 18:32:54 +03:00
Michal Niewrzal
c0e7f463fe satellite/metabase: remove segmentsloop package
Last change to remove segments loop from codebase.

https://github.com/storj/storj/issues/5237

Change-Id: I77b12911b6b4e390a7385e6e8057c7587e74b70a
2023-05-18 19:08:29 +00:00
Michal Niewrzal
cf5ff537e3 cmd/tools/metabase-verify: switch to ranged loop
We would like to remove segments loop so we need to refactor
our tools to use ranged loop.

To simplify change ranged loop is used with single range only.

https://github.com/storj/storj/issues/5237

Change-Id: I94d96d54f9d0e37b06def4f4fc16b71c5b79baba
2023-05-18 14:27:38 +00:00
Jeremy Wharton
f61230a670 satellite/console/dbcleanup: create console DB cleanup chore
A chore responsible for purging data from the console DB has been
implemented. Currently, it removes old records for unverified user
accounts. We plan to extend this functionality to include expired
project member invitations in the future.

Resolves #5790
References #5816

Change-Id: I1f3ef62fc96c10a42a383804b3b1d2846d7813f7
2023-05-05 19:11:53 +00:00
Egon Elbre
a1e920bbdf cmd/tools/segment-verify: satisfy wastedassign linter
The assignemnt to `err = nil` is not used in the rest of the code,
however, this was a protective err = nil assignment.

Change-Id: Id70fb2a2e68b91e2481952d865334e603ca41188
2023-03-31 08:52:46 +00:00
Lizzy Thomson
7e2d98988b cmd/tools: remove migration tool code for generating missing salt
Remove generate-missing-project-salt migration tool code and related
tests. This migration has already been run and this code is no longer
needed.

Issue https://github.com/storj/storj-private/issues/163

Change-Id: I4e36dcd95a07c5305c597113a7fd08148e100ccc
2023-03-10 16:38:54 +00:00
Márton Elek
ffaf15a3b0 satellite/overlay: remove unused mail service from overlay
It was surprising that `satellite auditor` complained about SMTP mail settings, even if it's not supposed to sending any mail.

Looks like we can remove the mail service dependency, as it's not a hard requirement for overlay.Service.

Change-Id: I29a52eeff3f967ddb2d74a09458dc0ee2f051bd7
2023-03-09 12:17:35 +00:00
Egon Elbre
3146ad7f2e satellite/satellitedb: cleanup testing access
Previously we were exposing the testing facilities via interface casting
the necessary parts, however, when things are not part of the main
satellite.DB interface they need to be manually propagated. Rather than
relying on using hidden methods lets expose things as long as they don't
create a direct dependency to the database driver.

Change-Id: I2eb7d8b60f4b64de1320c2d32581f7be267c0f57
2023-02-06 14:36:11 +02:00
Moby von Briesen
d45b65bcfc cmd/tools/generate-missing-project-salt: Clean up test
Move global variables to be local for each test to reduce the likelihood
of unexpected bugs. Also parallelize the different db tests and clean up
unnecessary lines/checks.

Change-Id: I9dc3894d0945430908b10af5aeeba2f9246caf2a
2023-02-02 16:36:38 +00:00
Michal Niewrzal
c1fffe881a satellite/satellitedb/satellitedbtest: enable full table scan detection
Satellite DB tests will print into logs (WARN) if full table scan will
be detected. Test won't be failed automatically. That's because currently
we have multiple queries which are doing full table scan and it's not
trivial to change.

We may change that behavior when we will figure out how to skip
specific query from detection or we will fix all problematic queries.

https://github.com/storj/storj/issues/5471

Change-Id: Icafe782257a0d353e8bcdf6fa8a19c20b1091a0b
2023-02-02 12:14:05 +00:00
Egon Elbre
6e298cca90 cmd/tools/node-cleanup: include tz information in test
Change-Id: Ic31b585b7df3fcb65fa37055ee62888fa43fa418
2023-02-01 17:21:38 +00:00
Wilfred Asomani
73ffa0827f cmd/tools: remove nullify-bad-user-agents tool
This tool is being removed because it has served its purpose and was blocking another removal from being verified.

Change-Id: Ie888aa7ae1b153a34210af3a5d5a3682b381ba82
2023-01-30 15:57:33 +00:00
Lizzy Thomson
1bff41e6b3 cmd/tools: add migration tool to update salt column when null
Add migration tool (and test) to update salt column in projects table
with the SHA-256 hash of the project ID when null

Issue https://github.com/storj/storj-private/issues/66

Change-Id: Ib8d484ac8d6ee25859064d803e2ac8fb46b45921
2023-01-27 22:04:07 +00:00
JT Olio
e40191afd6 storj: upgrade to use latest storj/common NodeAddress
Change-Id: I5987391bcfe5f6dfd7b525698c337a4cbda9b76e
2023-01-25 01:37:26 +00:00
paul cannon
740cb0d9c7 cmd/tools/segment-verify: fix read-csv subcommand
We were reading in a segment's stream ID and position, and assuming that
was enough for the downloader. But of course, the downloader needs
AliasPieces filled in. So now we request each segment record from the
metabase and fill in the VerifySegment records entirely.

Change-Id: If85236388eb99a65e2cb739aa976bd49ee2b2c89
2023-01-24 09:08:03 +00:00
paul cannon
c3b5c18d00 cmd/tools/segment-verify: learn to take a CSV list of segments as input
This will allow us to retry some specific segments from
segments-retry.csv with particularly high counts of "retry" pieces.

Change-Id: I48fd419cc0350a3be4c9e77ce8d28871565b7f97
2023-01-18 20:53:27 +00:00
Michal Niewrzal
0185bba90a cmd: cleanup segment verify/repair tools
* use the same DB application name for satellite and metabase
* use noop orders DB implementation to avoid storing allocated bandwidth
in DB

Change-Id: I20e88c694d38240fe1a20c45719e210cfb76402c
2023-01-12 15:27:07 +00:00
paul cannon
246c193145 cmd/tools/segment-verify: add timeout to VerifyWithExists
We have to wait until the slowest node is done being tested before we
can move on to the next of segments. Since the slowest node can be
arbitrarily slow, we'll set a timeout and treat too-slow nodes as
temporarily offline.

Change-Id: I80fe865dd4e8f826700430fb0140c2d3aefca381
2023-01-06 17:25:30 +00:00
paul cannon
23acee2df0 cmd/tools/segment-verify: better handle timeouts
When we are verifying pieces by downloading the first byte, if we
encounter a timeout, treat the node as if we failed to connect to it,
and log the error once instead of twice.

Change-Id: I70602d554183c98f1213f3ffb1bfec41100ea0e7
2023-01-06 16:52:28 +00:00
paul cannon
3a9ad48345 cmd/tools/segment-verify: fix problem-pieces.csv
This csv file was being closed as soon as the service was created.
All subsequent writes to the closed file handle produced errors,
which were logged but otherwise ignored.

Instead, we would like the file to remain open and writable, until
the service is destroyed.

Change-Id: Ib29944d25b2f5b2d0f90fdbdcde44fea8d769321
2023-01-06 16:23:34 +00:00
paul cannon
6e1554652a cmd/tools/segment-verify: handle dq'd nodes
Previously, if any pieces are still on disqualified nodes, this tool
would treat those pieces as fine (if the disqualified node is still
online) or temporarily unavailable (if the disqualified node is
offline). Instead, we should treat such pieces as lost.

This also fixes a slight problem with the code that handles a broken
alias. This is not likely to happen, but if we do see an alias that is
not in the alias map, we return an error instead of nil.

Change-Id: Ib4e2e729ef0535dd7bd9ce2f621680d9f959891c
2023-01-04 17:54:03 +00:00
paul cannon
2feb49afc3 cmd/tools/segment-verify: don't cache offline status forever
Because it was originally intended to work on only a few pieces from
each segment at a time, and would frequently have reset its list of
online nodes, segment-verify has been taking nodes out of its
onlineNodes set and never putting them back. This means that over a long
run in Check=0 mode, we end up treating more and more nodes as offline,
permanently. This trend obfuscates the number of missing pieces that
each segment really has, because we don't check pieces on offline nodes.

This commit changes the onlineNodes set to an "offlineNodes" set, with
an expiration time on the offline-ness quality. So nodes are added to
the offlineNodes set when we see they are offline, and then we only
treat them as offline for the next 30 minutes (configurable). After that
point, we will try connecting to them again.

Change-Id: I14f0332de25cdc6ef655f923739bcb4df71e079e
2023-01-03 23:11:42 +00:00
paul cannon
46d99a06d5 cmd/tools/segment-verify: write to pieces csv from WithExists methods
The WithExists methods previously were not writing problematic pieces to
problem-pieces.csv. With this change, they will.

Change-Id: I51eadd3d8f4299e1efa787c9266a7aacfa525eb3
2022-12-28 01:12:46 +00:00
paul cannon
9544936794 cmd/tools/segment-verify: don't double-count notfound
When this branch is followed, `audit.OutcomeFailure` is returned, and
`MarkNotFound()` is immediately called again (in
`(*NodeVerifier).Verify()`). Calling `MarkNotFound()` twice for the same
piece is not correct.

Change-Id: I1a2764bc32ed015628fcd9353ac3307f269b4bbd
2022-12-28 00:37:14 +00:00
paul cannon
aec596bb39 cmd/tools/segment-verify: monkit-ify WithExists methods
It may help to know how much faster these methods are than the
alternative (asking nodes for each piece in turn).

Change-Id: Ieb7c963f62b662f72c84a49de8a09c065c14f782
2022-12-28 00:00:40 +00:00
paul cannon
42e2a14316 cmd/tools/segment-verify: flush after write to pieces csv
It was ok as it was, but since we want to keep a close eye on progress
while the tool is running, it will help to have results written to the
output file immediately instead of after the buffer is full or the
program exits.

Change-Id: Ie027f05771a637afb06969ec775cd32b142b7635
2022-12-27 14:58:54 -06:00
paul cannon
b2422caaef cmd/tools/segment-verify: log less retry segments
When Check == 0 (check all pieces), there is nearly always a piece left
in the retry count, so most segments get logged in segments-retry.csv.
This change makes it so we require retry>5 before adding to
segments-retry.csv (only in the check==0 case).

Change-Id: Iaea523c27eb777e3c248c27c7ef5effe77ae54cf
2022-12-23 14:29:25 +00:00
paul cannon
0b790070a3 cmd/tools/segment-verify: pass over bad segments
Change-Id: I1b4dd9da755c6a2028760723e15219f5821f702f
2022-12-22 18:12:12 -06:00
Michal Niewrzal
4851b4e06d cmd/tools/segment-verify: small improvements
* better error handling when Exists method is not avaialble on SN
* more optimal processing of response from Exists method

Change-Id: I6d61c09473e9f5ab76a4601720e8bd520767f4c2
2022-12-22 15:21:33 +00:00
Clement Sam
cda1d67465 cmd/tools/segment-verify: adjust to SN Exists endpoint
Change-Id: I409aeae29aa87996f2a6047f976d215a69e9d7f5
2022-12-21 19:24:31 +00:00
Fadila Khadar
d23e25ce0f cmd/tools/segment-verify: remove unused test code
Accidentally added some code to a test. As it is unused, this PR removes it.

Change-Id: I7adddc78c5ed747225e365989ab58504a9625ad7
2022-12-19 14:33:08 +00:00
Ethan Adams
1c309a0318 cmd/tools/segment-verify: check for unvetted nodes
this also renames the command from `duplicates` to `node-check`

Change-Id: Idd303b17ec03f5b55fbbb1f4039a7761da37abe6
2022-12-19 09:59:13 +00:00
Egon Elbre
04f16f8768 cmd/tools/segment-verify: tool for checking duplicate net
Change-Id: Ie47c1282e580ffc418bf3b1f3c8820a48973aefc
2022-12-15 22:58:36 +00:00
paul cannon
727136141a satellite/cmd/tools/segment-verify: check all pieces
This adds the capability to the segment-verify tool of checking all
pieces of every indicated segment.

Pieces which could not be accessed (i.e. we couldn't get a single
byte from them) are recorded in a csv file.

I haven't been able to test this in any very meaningful way, yet, but I
am comforted by the fact that the worst things it could possibly do are
(a) download pieces too many times, and (b) miss downloading some
pieces.

Change-Id: I3aba30921572c974993363eb36d0fd5b3ae97907
2022-12-14 19:06:08 +00:00
Fadila Khadar
995f78d579 satellite/cmd: segment-verify verifies segments in given bucket list
Provides the `segment-verify run buckets` command for verifying segments within a list of buckets.

Bucket list is a csv file with `project_id,bucket_name` to be checked.

https://github.com/storj/storj-private/issues/101

Change-Id: I3d25c27b56fcab4a6a1aebb6f87514d6c97de3ff
2022-12-13 20:10:00 +00:00
Cameron
f06da25c3d satellite/overlay: add nodeevents.DB to satellite overlay service
Add nodeevents.DB to satellite overlay service so we can insert node
events into the nodeevents DB.

Change-Id: I642c0ccc9941ecdb08cb22d5c8cf701959a55156
2022-11-02 15:56:37 +00:00
Michal Niewrzal
d21bbab2b2 satellite: fix metabase configuration wiring
New flag 'MultipleVersions' was not correctly passed from metainfo
configuration to metabase configuration. Because configuration was
set correctly for unit tests we didn't catch it and issue was found
while testing on QA satellite.

This change reduce number of places where new metabase flags needs
to be propagated from metainfo configuration to avoid problems with
setting new flags in the future.

Fixes https://github.com/storj/storj/issues/5274

Change-Id: I74bc122649febefd87f665be2fba628f6bfd9044
2022-11-02 15:17:34 +00:00
Jennifer Johnson
1da7520a88 cmd/tools/node-cleanup: swap deletion for atredis emails with last contact and created at timestamps
Change-Id: I4acde09e00ea4e5450dcb0fd26bbb2c3afa767f6
2022-10-31 14:13:56 +00:00
JT Olio
58a9c55f36 mod: bump dependencies
- storj.io/common

Change-Id: Ib78154acc253a13683495abfdd96d702625fdce8
2022-10-19 17:01:53 +00:00
Egon Elbre
22c0b0ac5c cmd/tools/segment-verify: don't mark node immediately offline
Rather than marking node immediately offline, wait for more failures
until removing from the set.

Change-Id: I4363294a75d7d2844afc1f9c0025f664f933c2d7
2022-10-14 08:10:26 +00:00