Commit Graph

35 Commits

Author SHA1 Message Date
Cameron
f06da25c3d satellite/overlay: add nodeevents.DB to satellite overlay service
Add nodeevents.DB to satellite overlay service so we can insert node
events into the nodeevents DB.

Change-Id: I642c0ccc9941ecdb08cb22d5c8cf701959a55156
2022-11-02 15:56:37 +00:00
Michal Niewrzal
d21bbab2b2 satellite: fix metabase configuration wiring
New flag 'MultipleVersions' was not correctly passed from metainfo
configuration to metabase configuration. Because configuration was
set correctly for unit tests we didn't catch it and issue was found
while testing on QA satellite.

This change reduce number of places where new metabase flags needs
to be propagated from metainfo configuration to avoid problems with
setting new flags in the future.

Fixes https://github.com/storj/storj/issues/5274

Change-Id: I74bc122649febefd87f665be2fba628f6bfd9044
2022-11-02 15:17:34 +00:00
Jennifer Johnson
1da7520a88 cmd/tools/node-cleanup: swap deletion for atredis emails with last contact and created at timestamps
Change-Id: I4acde09e00ea4e5450dcb0fd26bbb2c3afa767f6
2022-10-31 14:13:56 +00:00
JT Olio
58a9c55f36 mod: bump dependencies
- storj.io/common

Change-Id: Ib78154acc253a13683495abfdd96d702625fdce8
2022-10-19 17:01:53 +00:00
Egon Elbre
22c0b0ac5c cmd/tools/segment-verify: don't mark node immediately offline
Rather than marking node immediately offline, wait for more failures
until removing from the set.

Change-Id: I4363294a75d7d2844afc1f9c0025f664f933c2d7
2022-10-14 08:10:26 +00:00
Egon Elbre
a80a0ebeae cmd/tools/segment-verify: redial once rather than giving up
It's quite likely to hit some timelimit, rather than giving up
immediately, let's retry after the throttle amount.

Change-Id: I20944b058d771f5d4bfa0eea7a2c26cefcd74739
2022-10-14 01:18:31 +00:00
Cameron
a52f766273 satellite/overlay: add email-sending functionality to overlay service
We want to send emails to SNOs. Node status changes go through the
overlay service, so it's a good place to add the mail service.
Add the mailservice.Service, satellite address, and satellite name to
overlay service. Also add feature flag --overlay.send-node-emails

Change-Id: I3bd2cb3bf22f9724954ce2374f8b651b902b3a24
2022-10-13 18:01:05 +00:00
Egon Elbre
dd60318147 cmd/tools/segment-verify: use resolved ip
Change-Id: I3662aaea3ff8721c415c038b2b5324d165b60975
2022-10-12 12:43:11 +00:00
Egon Elbre
ff22fc7ddd all: fix deprecated ioutil commands
Change-Id: I59db35116ec7215a1b8e2ae7dbd319fa099adfac
2022-10-11 15:27:29 +00:00
Egon Elbre
142a04f208 cmd/tools/segment-verify: add connection pool
Change-Id: If0f85edbf99438ac41c23fc7107fdab926288cc2
2022-10-11 09:06:44 +03:00
Egon Elbre
8916f2ee92 cmd/tools/segment-verify: allow ignoring specific nodes
This adds a new flag that allows to ignore some nodes completely.

Change-Id: I203d25f931262c809037e25e9c37e9a89bf47026
2022-10-10 20:14:38 +03:00
Egon Elbre
9e50d837e3 cmd/tools/segment-verify: add tool for summarizing log
Change-Id: I3177ab71dfd25e11adfedce32a530d83dda63bd6
2022-10-10 20:02:50 +03:00
Egon Elbre
5f01dad3a3 cmd/tools/segment-verify: add total progress indicator
Change-Id: Ib729abf6adbeba8d94e08c7e11497c6d5ddd5ec2
2022-10-10 20:02:30 +03:00
Egon Elbre
ea4b3023d9 cmd/tools/segment-verify: fix piece id derivation
Change-Id: Ib27fd8630e1e5a90060dff2a09c51f488960177f
2022-10-06 13:43:08 +00:00
Egon Elbre
c8506cdda3 satellite/metabase,cmd/tools/segment-verify: simplify interface
Change-Id: Icdd445b1713bc26cee3b3a125b68b0cde0739837
2022-10-06 13:42:00 +00:00
Egon Elbre
c1817ab743 cmd/tools/segment-verify: a few fixes
The flags weren't properly loading from config.

The code assumed that every node that's online for downloading also have
data uploaded to them -- which is not true.

Change-Id: Ifd65a47b9eca5b4841231928244fab17acbde6fb
2022-10-05 15:51:38 +00:00
Egon Elbre
4c374a2357 cmd/tools/segment-verify: add tiny readme
Change-Id: Ia314c615f8b7fdb13e2b5f81c1be82ec686ca819
2022-10-03 16:01:24 +00:00
Michal Niewrzal
a97cd97789 satellite/orders: remove unused service dependency
Orders service doesn't need buckets service anymore.

Change-Id: I27853cda87e82b528f53667e4b4866801f7bfb62
2022-09-28 08:56:36 +00:00
Egon Elbre
8069973dee cmd/tools/segment-verify: add failure tests
* Fix an invalid slice index calculation.

Change-Id: I7f1b85edc46df362697aa132b967d5d23f9d5522
2022-09-26 19:38:16 +00:00
Egon Elbre
f98d551c9b cmd/tools/segment-verify: test service
Change-Id: Ibd83960c18123e8f29e22089007dc32c8d532240
2022-09-22 17:23:02 +00:00
Egon Elbre
0bfaadcc6c cmd/tools/segment-verify: fixes and more tests
* Disallow too large listing limit, which would cause a lot of memory to
  be consumed.
* Fix throttling logic and add a test.
* Fix read error handling; depending on the concurrency it can return
  the NotFound status either in the Read or Close.

Change-Id: I778f04a5961988b2480df5c7faaa22393fc5d760
2022-09-22 10:32:30 +00:00
Egon Elbre
0e99f7a8cf cmd/tools/segment-verify: add loading of priority nodes
Change-Id: Idcc41469ea5f71eab1b9dccbe0f14da537386a17
2022-09-21 14:56:13 +00:00
Egon Elbre
8b527f2d12 cmd/tools/segment-verify: add throttling
Change-Id: Ia0b4ec255adc90d874f4366b80799414a1a94700
2022-09-21 14:52:51 +00:00
Egon Elbre
cf50696745 cmd/tools/segment-verify: wire up overlay logic
Change-Id: I0a4c737a8b0995a1c3e3adeac728fe833d0ce684
2022-09-19 11:32:18 +03:00
Egon Elbre
0809ae73cf cmd/tools/segment-verify: add main
Change-Id: Ib7161a0f44d447f9ddb9be83f6673587a0bd7712
2022-09-19 10:36:57 +03:00
Jennifer Johnson
8529a169ee cmd/tools/segment-verify: add verifier
Change-Id: I4cc1fbcf964c4a9a37cf80322f6f99dd956f3d7b
2022-09-19 10:36:57 +03:00
Egon Elbre
9b520b2114 satellite/metabase: expose ConvertNodesToAliases and ConvertAliasesToNodes
They are needed for segment-verify tool.

Also rename some of the conversion methods to make clear,
which of them have side-effects.

Change-Id: Ie9a0952548e9ed5068c7a30c2fd2134b07139bca
2022-09-15 13:56:10 +00:00
Egon Elbre
cd81c5bd58 cmd/tools/segment-verify: add csv writer
Change-Id: I9306d7a6927f4dacca9623d7bd57f8560404db3e
2022-09-15 13:28:21 +00:00
Egon Elbre
507b099d44 cmd/tools/segment-verify: add monitoring / error
Change-Id: I6fd0369719ddf176a98208348560004a4134f810
2022-09-14 18:20:48 +00:00
Egon Elbre
6127f465dc cmd/tools/segment-verify: add logic for iterating over segments
This adds parts for:
1. iterating over the segments
2. using an interface for writing the segments
3. stubs for handling deleted segments

Change-Id: I76a17cac6deb0b6c042a8ab7c4155a890db9da84
2022-09-14 18:20:31 +00:00
JT Olio
3b7d8e8798 cmd/tools/convert-node-id: tool for node id conversions
Change-Id: I3d18a051131135dfdbfd9c54002efbc853e4004d
2022-09-14 11:13:50 +03:00
Egon Elbre
8459a3faa2 cmd/tools/segment-verify: add processing of a collection of segments
This adds logic to put segments into batches and retry them.

Change-Id: I86450e2f5ab13edb3ae9a5d5b9417e0cced2892b
2022-09-12 18:29:04 +03:00
Egon Elbre
3467fd4b7f cmd/tools/segment-verify: implement batch creation
Implements creating roughly load-balanced set of batched
that can be used to make multiple requests.

Change-Id: I349b276176dcb8ba9163e7e06a94509d73fa5ddc
2022-09-12 18:29:01 +03:00
Egon Elbre
731916ffa4 cmd/inspector: move command back from cmd/tools/
Some of the makefiles require it to be in cmd/inspector.
This was initially moved in 381c1e1257.

Change-Id: I850f3bfa1d66e66fbd1d9069e353984a27a7c58c
2022-09-07 09:35:12 +00:00
Egon Elbre
381c1e1257 cmd/tools: move tooling to a separate folder
This helps to cleanup the cmd folder a bit.

Change-Id: I24025c3dbfd35966325d7d5aaa95cd9a1176a8b7
2022-09-02 18:25:36 +03:00