JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,054 Commits 5 Branches 0 Tags 94 MiB
52ede93086
Commit Graph

31 Commits

Author SHA1 Message Date
Egon Elbre
22c0b0ac5c cmd/tools/segment-verify: don't mark node immediately offline 
Rather than marking node immediately offline, wait for more failures
until removing from the set.

Change-Id: I4363294a75d7d2844afc1f9c0025f664f933c2d7
 2022-10-14 08:10:26 +00:00
Egon Elbre
a80a0ebeae cmd/tools/segment-verify: redial once rather than giving up 
It's quite likely to hit some timelimit, rather than giving up
immediately, let's retry after the throttle amount.

Change-Id: I20944b058d771f5d4bfa0eea7a2c26cefcd74739
 2022-10-14 01:18:31 +00:00
Cameron
a52f766273 satellite/overlay: add email-sending functionality to overlay service 
We want to send emails to SNOs. Node status changes go through the
overlay service, so it's a good place to add the mail service.
Add the mailservice.Service, satellite address, and satellite name to
overlay service. Also add feature flag --overlay.send-node-emails

Change-Id: I3bd2cb3bf22f9724954ce2374f8b651b902b3a24
 2022-10-13 18:01:05 +00:00
Egon Elbre
dd60318147 cmd/tools/segment-verify: use resolved ip 
Change-Id: I3662aaea3ff8721c415c038b2b5324d165b60975
 2022-10-12 12:43:11 +00:00
Egon Elbre
ff22fc7ddd all: fix deprecated ioutil commands 
Change-Id: I59db35116ec7215a1b8e2ae7dbd319fa099adfac
 2022-10-11 15:27:29 +00:00
Egon Elbre
142a04f208 cmd/tools/segment-verify: add connection pool 
Change-Id: If0f85edbf99438ac41c23fc7107fdab926288cc2
 2022-10-11 09:06:44 +03:00
Egon Elbre
8916f2ee92 cmd/tools/segment-verify: allow ignoring specific nodes 
This adds a new flag that allows to ignore some nodes completely.

Change-Id: I203d25f931262c809037e25e9c37e9a89bf47026
 2022-10-10 20:14:38 +03:00
Egon Elbre
9e50d837e3 cmd/tools/segment-verify: add tool for summarizing log 
Change-Id: I3177ab71dfd25e11adfedce32a530d83dda63bd6
 2022-10-10 20:02:50 +03:00
Egon Elbre
5f01dad3a3 cmd/tools/segment-verify: add total progress indicator 
Change-Id: Ib729abf6adbeba8d94e08c7e11497c6d5ddd5ec2
 2022-10-10 20:02:30 +03:00
Egon Elbre
ea4b3023d9 cmd/tools/segment-verify: fix piece id derivation 
Change-Id: Ib27fd8630e1e5a90060dff2a09c51f488960177f
 2022-10-06 13:43:08 +00:00
Egon Elbre
c8506cdda3 satellite/metabase,cmd/tools/segment-verify: simplify interface 
Change-Id: Icdd445b1713bc26cee3b3a125b68b0cde0739837
 2022-10-06 13:42:00 +00:00
Egon Elbre
c1817ab743 cmd/tools/segment-verify: a few fixes 
The flags weren't properly loading from config.

The code assumed that every node that's online for downloading also have
data uploaded to them -- which is not true.

Change-Id: Ifd65a47b9eca5b4841231928244fab17acbde6fb
 2022-10-05 15:51:38 +00:00
Egon Elbre
4c374a2357 cmd/tools/segment-verify: add tiny readme 
Change-Id: Ia314c615f8b7fdb13e2b5f81c1be82ec686ca819
 2022-10-03 16:01:24 +00:00
Michal Niewrzal
a97cd97789 satellite/orders: remove unused service dependency 
Orders service doesn't need buckets service anymore.

Change-Id: I27853cda87e82b528f53667e4b4866801f7bfb62
 2022-09-28 08:56:36 +00:00
Egon Elbre
8069973dee cmd/tools/segment-verify: add failure tests 
* Fix an invalid slice index calculation.

Change-Id: I7f1b85edc46df362697aa132b967d5d23f9d5522
 2022-09-26 19:38:16 +00:00
Egon Elbre
f98d551c9b cmd/tools/segment-verify: test service 
Change-Id: Ibd83960c18123e8f29e22089007dc32c8d532240
 2022-09-22 17:23:02 +00:00
Egon Elbre
0bfaadcc6c cmd/tools/segment-verify: fixes and more tests 
* Disallow too large listing limit, which would cause a lot of memory to
  be consumed.
* Fix throttling logic and add a test.
* Fix read error handling; depending on the concurrency it can return
  the NotFound status either in the Read or Close.

Change-Id: I778f04a5961988b2480df5c7faaa22393fc5d760
 2022-09-22 10:32:30 +00:00
Egon Elbre
0e99f7a8cf cmd/tools/segment-verify: add loading of priority nodes 
Change-Id: Idcc41469ea5f71eab1b9dccbe0f14da537386a17
 2022-09-21 14:56:13 +00:00
Egon Elbre
8b527f2d12 cmd/tools/segment-verify: add throttling 
Change-Id: Ia0b4ec255adc90d874f4366b80799414a1a94700
 2022-09-21 14:52:51 +00:00
Egon Elbre
cf50696745 cmd/tools/segment-verify: wire up overlay logic 
Change-Id: I0a4c737a8b0995a1c3e3adeac728fe833d0ce684
 2022-09-19 11:32:18 +03:00
Egon Elbre
0809ae73cf cmd/tools/segment-verify: add main 
Change-Id: Ib7161a0f44d447f9ddb9be83f6673587a0bd7712
 2022-09-19 10:36:57 +03:00
Jennifer Johnson
8529a169ee cmd/tools/segment-verify: add verifier 
Change-Id: I4cc1fbcf964c4a9a37cf80322f6f99dd956f3d7b
 2022-09-19 10:36:57 +03:00
Egon Elbre
9b520b2114 satellite/metabase: expose ConvertNodesToAliases and ConvertAliasesToNodes 
They are needed for segment-verify tool.

Also rename some of the conversion methods to make clear,
which of them have side-effects.

Change-Id: Ie9a0952548e9ed5068c7a30c2fd2134b07139bca
 2022-09-15 13:56:10 +00:00
Egon Elbre
cd81c5bd58 cmd/tools/segment-verify: add csv writer 
Change-Id: I9306d7a6927f4dacca9623d7bd57f8560404db3e
 2022-09-15 13:28:21 +00:00
Egon Elbre
507b099d44 cmd/tools/segment-verify: add monitoring / error 
Change-Id: I6fd0369719ddf176a98208348560004a4134f810
 2022-09-14 18:20:48 +00:00
Egon Elbre
6127f465dc cmd/tools/segment-verify: add logic for iterating over segments 
This adds parts for:
1. iterating over the segments
2. using an interface for writing the segments
3. stubs for handling deleted segments

Change-Id: I76a17cac6deb0b6c042a8ab7c4155a890db9da84
 2022-09-14 18:20:31 +00:00
JT Olio
3b7d8e8798 cmd/tools/convert-node-id: tool for node id conversions 
Change-Id: I3d18a051131135dfdbfd9c54002efbc853e4004d
 2022-09-14 11:13:50 +03:00
Egon Elbre
8459a3faa2 cmd/tools/segment-verify: add processing of a collection of segments 
This adds logic to put segments into batches and retry them.

Change-Id: I86450e2f5ab13edb3ae9a5d5b9417e0cced2892b
 2022-09-12 18:29:04 +03:00
Egon Elbre
3467fd4b7f cmd/tools/segment-verify: implement batch creation 
Implements creating roughly load-balanced set of batched
that can be used to make multiple requests.

Change-Id: I349b276176dcb8ba9163e7e06a94509d73fa5ddc
 2022-09-12 18:29:01 +03:00
Egon Elbre
731916ffa4 cmd/inspector: move command back from cmd/tools/ 
Some of the makefiles require it to be in cmd/inspector.
This was initially moved in 381c1e1257.

Change-Id: I850f3bfa1d66e66fbd1d9069e353984a27a7c58c
 2022-09-07 09:35:12 +00:00
Egon Elbre
381c1e1257 cmd/tools: move tooling to a separate folder 
This helps to cleanup the cmd folder a bit.

Change-Id: I24025c3dbfd35966325d7d5aaa95cd9a1176a8b7
 2022-09-02 18:25:36 +03:00