JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
95d85c640c
storj/monkit.lock

148 lines
10 KiB
Plaintext
Raw Normal View History

private/lifecycle: monitor unexpected shutdowns Change-Id: I9af3a95a1b60c1572cd57bb5e9539d60aa0337bb
2021-06-25 17:21:30 +01:00
storj.io/storj/private/lifecycle."slow_shutdown" Event
storj.io/storj/private/lifecycle."unexpected_shutdown" Event
monkit: fix monkit lock Previously check-monitoring ignored ScopeNamed. Change-Id: I3d0d472e722cb19a65b471a914f5124cd224fc34
2021-04-22 18:32:42 +01:00
storj.io/storj/satellite/accounting."bucket_bytes" IntVal
storj.io/storj/satellite/accounting."bucket_objects" IntVal
storj.io/storj/satellite/accounting."bucket_segments" IntVal
storj.io/storj/satellite/accounting."total_bytes" IntVal
storj.io/storj/satellite/accounting."total_objects" IntVal
storj.io/storj/satellite/accounting."total_segments" IntVal
tally monkit: change location to monitor piecesize When we observed the value for total piecesizes stored in the network, we were doing it after converting them to byte-hours, rather than using the actual piece sizes. This fixes that issue. Change-Id: I1564d21b519f70eb59f298d97dbd777baf127723
2021-01-25 20:07:24 +00:00
storj.io/storj/satellite/accounting/tally."nodetallies.totalsum" IntVal
satellite/audit: do not contain nodes for unknown errors (#3592) * skip unknown errors (wip) * add tests to make sure nodes that time out are added to containment * add bad blobs store * call "Skipped" "Unknown" * add tests to ensure unknown errors do not trigger containment * add monkit stats to lockfile * typo * add periods to end of bad blobs comments
2019-11-19 16:30:28 +00:00
storj.io/storj/satellite/audit."audit_contained_nodes" IntVal
storj.io/storj/satellite/audit."audit_contained_nodes_global" Meter
storj.io/storj/satellite/audit."audit_contained_percentage" FloatVal
storj.io/storj/satellite/audit."audit_fail_nodes" IntVal
storj.io/storj/satellite/audit."audit_fail_nodes_global" Meter
storj.io/storj/satellite/audit."audit_failed_percentage" FloatVal
storj.io/storj/satellite/audit."audit_offline_nodes" IntVal
storj.io/storj/satellite/audit."audit_offline_nodes_global" Meter
storj.io/storj/satellite/audit."audit_offline_percentage" FloatVal
storj.io/storj/satellite/audit."audit_success_nodes" IntVal
storj.io/storj/satellite/audit."audit_success_nodes_global" Meter
storj.io/storj/satellite/audit."audit_successful_percentage" FloatVal
storj.io/storj/satellite/audit."audit_total_nodes" IntVal
storj.io/storj/satellite/audit."audit_total_nodes_global" Meter
storj.io/storj/satellite/audit."audit_total_pointer_nodes" IntVal
storj.io/storj/satellite/audit."audit_total_pointer_nodes_global" Meter
storj.io/storj/satellite/audit."audit_unknown_nodes" IntVal
storj.io/storj/satellite/audit."audit_unknown_nodes_global" Meter
storj.io/storj/satellite/audit."audit_unknown_percentage" FloatVal
storj.io/storj/satellite/audit."audited_percentage" FloatVal
satellite/audit: log error and increment metric if shares cannot be verified If we encounter an error during the infectious error correction, we just add it to the errlist to be logged at the worker level. We want to make sure we know about this if it happens. Give it its own error log and increment a monkit metric. Change-Id: Ie5946ae3cd97b766e3099af8ce160a686135ee27
2021-08-16 21:11:45 +01:00
storj.io/storj/satellite/audit."could_not_verify_audit_shares" Counter
satellite/audit: do not contain nodes for unknown errors (#3592) * skip unknown errors (wip) * add tests to make sure nodes that time out are added to containment * add bad blobs store * call "Skipped" "Unknown" * add tests to ensure unknown errors do not trigger containment * add monkit stats to lockfile * typo * add periods to end of bad blobs comments
2019-11-19 16:30:28 +00:00
storj.io/storj/satellite/audit."reverify_contained" IntVal
storj.io/storj/satellite/audit."reverify_contained_global" Meter
storj.io/storj/satellite/audit."reverify_contained_in_segment" IntVal
storj.io/storj/satellite/audit."reverify_fails" IntVal
storj.io/storj/satellite/audit."reverify_fails_global" Meter
storj.io/storj/satellite/audit."reverify_offlines" IntVal
storj.io/storj/satellite/audit."reverify_offlines_global" Meter
storj.io/storj/satellite/audit."reverify_successes" IntVal
storj.io/storj/satellite/audit."reverify_successes_global" Meter
storj.io/storj/satellite/audit."reverify_total_in_segment" IntVal
storj.io/storj/satellite/audit."reverify_unknown" IntVal
storj.io/storj/satellite/audit."reverify_unknown_global" Meter
satellite/audit/verifier: add metric for number of successfully downloaded shares Change-Id: Ia4f1dc6e088db802e340aaecf80cc7ef6dc237a4
2020-02-26 21:19:58 +00:00
storj.io/storj/satellite/audit."verify_shares_downloaded_successfully" IntVal
satellite/console: add monkit metrics around user registraion/login github issue: https://github.com/storj/storj/issues/4807 Change-Id: Id56ec73ec91b07b639b8011f0f916b4adbb01be6
2022-05-24 19:18:52 +01:00
storj.io/storj/satellite/console."create_user_attempt" Counter
storj.io/storj/satellite/console."create_user_captcha_error" Counter
storj.io/storj/satellite/console."create_user_captcha_unsuccessful" Counter
storj.io/storj/satellite/console."create_user_duplicate_unverified" Counter
storj.io/storj/satellite/console."create_user_duplicate_verified" Counter
storj.io/storj/satellite/console."create_user_success" Counter
storj.io/storj/satellite/console."login_attempt" Counter
storj.io/storj/satellite/console."login_email_invalid" Counter
storj.io/storj/satellite/console."login_email_unverified" Counter
storj.io/storj/satellite/console."login_failed" Counter
storj.io/storj/satellite/console."login_invalid_password" Counter
storj.io/storj/satellite/console."login_locked_out" Counter
storj.io/storj/satellite/console."login_lockout_initiated" Counter
storj.io/storj/satellite/console."login_lockout_reinitiated" Counter
storj.io/storj/satellite/console."login_mfa_conflict" Counter
storj.io/storj/satellite/console."login_mfa_missing" Counter
storj.io/storj/satellite/console."login_mfa_passcode_failure" Counter
storj.io/storj/satellite/console."login_mfa_passcode_success" Counter
storj.io/storj/satellite/console."login_mfa_recovery_failure" Counter
storj.io/storj/satellite/console."login_mfa_recovery_success" Counter
storj.io/storj/satellite/console."login_success" Counter
storj.io/storj/satellite/console."login_user_failed_count" IntVal
satellite/{accounting, contact}: Remove periods and spaces from metrics. Change-Id: I84179c2931293e3a1eb0ff8050416d25e481ce07
2020-12-02 22:17:59 +00:00
storj.io/storj/satellite/contact."failed_dial" Event
storj.io/storj/satellite/contact."failed_ping_node" Event
satellite/gracefulexit: separate functional code in endpoint (#3476)
2019-11-08 18:57:51 +00:00
storj.io/storj/satellite/gracefulexit."graceful_exit_fail_max_failures_percentage" Meter
storj.io/storj/satellite/gracefulexit."graceful_exit_fail_validation" Meter
storj.io/storj/satellite/gracefulexit."graceful_exit_final_bytes_transferred" IntVal
storj.io/storj/satellite/gracefulexit."graceful_exit_final_pieces_failed" IntVal
storj.io/storj/satellite/gracefulexit."graceful_exit_final_pieces_succeess" IntVal
storj.io/storj/satellite/gracefulexit."graceful_exit_init_node_age_seconds" FloatVal
storj.io/storj/satellite/gracefulexit."graceful_exit_init_node_audit_success_count" IntVal
storj.io/storj/satellite/gracefulexit."graceful_exit_init_node_audit_total_count" IntVal
storj.io/storj/satellite/gracefulexit."graceful_exit_init_node_piece_count" IntVal
storj.io/storj/satellite/gracefulexit."graceful_exit_success" Meter
storj.io/storj/satellite/gracefulexit."graceful_exit_successful_pieces_transfer_ratio" IntVal
storj.io/storj/satellite/gracefulexit."graceful_exit_transfer_piece_fail" Meter
storj.io/storj/satellite/gracefulexit."graceful_exit_transfer_piece_success" Meter
satellite/metabase/segmentloop: don't shutdown satellite on loop error We made decision to avoid satellite shutdown when segment loop will return error. Loop still can reeturn error but it will be logged and we will make monitoring/alert around that error. Change-Id: I6aa8e284406edf644a09d6b1fe00c3155c5430c9
2021-07-29 09:28:23 +01:00
storj.io/storj/satellite/metabase/segmentloop."segmentloop_error" Event
satellite/metabase: add segmentloop service We want to move some of current metainfo loop observers to segment loop. This change adds new service, similar to metainfo loop but which is iterating only over segments. Change-Id: I67f7f461781723a4476e2b83377f31736d7c4870
2021-05-28 10:42:55 +01:00
storj.io/storj/satellite/metabase/segmentloop."segmentsProcessed" IntVal
storj.io/storj/satellite/metabase/segmentloop.*Service.RunOnce Task
Add metrics to track rate limit. Add monkit metric for the rate-limit when the rate limit is hit Logs warning with projectID https://storjlabs.atlassian.net/browse/SM-165 Change-Id: I352dc40006021990d1bc66a999f62bbf8deb54db
2020-01-30 17:43:37 +00:00
storj.io/storj/satellite/metainfo."metainfo_rate_limit_exceeded" Event
satellite/metainfo/piecedeletion Remove spaces from metrics. Change-Id: Iaf1d8a96a43087f2fcc579347f581e8a78a0fb58
2020-12-30 17:03:31 +00:00
storj.io/storj/satellite/metainfo/piecedeletion."delete_batch_size" IntVal
storj.io/storj/satellite/metainfo/piecedeletion."deletion_pieces_unhandled_count" IntVal
satellite/metrics: fix metrics for total inline/remote bytes and segments Change-Id: I567ce127590a4712cab296d28a19838e3a632021
2021-07-30 16:31:20 +01:00
storj.io/storj/satellite/metrics."total_inline_bytes" IntVal
storj.io/storj/satellite/metrics."total_inline_segments" IntVal
storj.io/storj/satellite/metrics."total_remote_bytes" IntVal
storj.io/storj/satellite/metrics."total_remote_segments" IntVal
satellite/metainfo, satellite/repair, uplink/eestream: add metric for download failed due to not enough pieces available (#3665)
2019-12-04 21:24:36 +00:00
storj.io/storj/satellite/orders."download_failed_not_enough_pieces_uplink" Meter
satellite/{satellitedb, repair/{queue, checker}}: Use new column "segmentHealth" instead of "numHealthy" in injured segments queue We plan to add support for a new Reed-Solomon scheme soon, but our repair queue orders segments by least number of healthy pieces first. With a second RS scheme, fewer healthy pieces will not necessarily correlate to lower health. This change just adds the new column in a migration. A separate change will add the new health function. Right now, since we only support one RS scheme, behavior will not change. Number of healthy pieces is being inserted as "segment health" until the new health function is merged. Segment health is calculated with a new priority function created in commit 3e5640359. In order to use the function, a new config value is added, called NodeFailureRate, representing the approximate probability of any individual node going down in the duration of one checker run. Change-Id: I51c4202203faf52528d923befbe886dbf86d02f2
2020-10-21 23:02:54 +01:00
storj.io/storj/satellite/repair/checker."checker_injured_segment_health" FloatVal
scripts: add check-monitoring script to detect changes to monkit calls (#3114)
2019-10-15 18:00:14 +01:00
storj.io/storj/satellite/repair/checker."checker_segment_age" IntVal
satellite/{satellitedb, repair/{queue, checker}}: Use new column "segmentHealth" instead of "numHealthy" in injured segments queue We plan to add support for a new Reed-Solomon scheme soon, but our repair queue orders segments by least number of healthy pieces first. With a second RS scheme, fewer healthy pieces will not necessarily correlate to lower health. This change just adds the new column in a migration. A separate change will add the new health function. Right now, since we only support one RS scheme, behavior will not change. Number of healthy pieces is being inserted as "segment health" until the new health function is merged. Segment health is calculated with a new priority function created in commit 3e5640359. In order to use the function, a new config value is added, called NodeFailureRate, representing the approximate probability of any individual node going down in the duration of one checker run. Change-Id: I51c4202203faf52528d923befbe886dbf86d02f2
2020-10-21 23:02:54 +01:00
storj.io/storj/satellite/repair/checker."checker_segment_health" FloatVal
scripts: add check-monitoring script to detect changes to monkit calls (#3114)
2019-10-15 18:00:14 +01:00
storj.io/storj/satellite/repair/checker."checker_segment_healthy_count" IntVal
storj.io/storj/satellite/repair/checker."checker_segment_time_until_irreparable" IntVal
storj.io/storj/satellite/repair/checker."checker_segment_total_count" IntVal
satellite/repair: add monkit counter for segments below minimum required The current monkit reporting for "remote_segments_lost" is not usable for triggering alerts, as it has reported no data. To allow alerting, two new metrics "checker_segments_below_min_req" and "repairer_segments_below_min_req" will increment by zero on each segment unless it is below the minimum required piece count. The two metrics report what is found by the checker and the repairer respectively. Change-Id: I98a68bb189eaf68a833d25cf5db9e68df535b9d7
2020-11-10 14:49:19 +00:00
storj.io/storj/satellite/repair/checker."checker_segments_below_min_req" Counter
satellite/{repair,satellitedb}: clean up healthy segments from repair queue at end of checker iteration Repair workers prioritize the most unhealthy segments. This has the consequence that when we finally begin to reach the end of the queue, a good portion of the remaining segments are healthy again as their nodes have come back online. This makes it appear that there are more injured segments than there actually are. solution: Any time the checker observes an injured segment it inserts it into the repair queue or updates it if it already exists. Therefore, we can determine which segments are no longer injured if they were not inserted or updated by the last checker iteration. To do this we add a new column to the injured segments table, updated_at, which is set to the current time when a segment is inserted or updated. At the end of the checker iteration, we can delete any items where updated_at < checker start. Change-Id: I76a98487a4a845fab2fbc677638a732a95057a94
2020-09-09 21:52:22 +01:00
storj.io/storj/satellite/repair/checker."healthy_segments_removed_from_queue" IntVal
satellite/repair/{checker,queue}: add metric for new segments added to repair queue * add monkit stat new_remote_segments_needing_repair, which reports the number of new unhealthy segments in the repair queue since the previous checker iteration Change-Id: I2f10266006fdd6406ece50f4759b91382059dcc3
2020-05-22 20:54:05 +01:00
storj.io/storj/satellite/repair/checker."new_remote_segments_needing_repair" IntVal
scripts: add check-monitoring script to detect changes to monkit calls (#3114)
2019-10-15 18:00:14 +01:00
storj.io/storj/satellite/repair/checker."remote_files_checked" IntVal
storj.io/storj/satellite/repair/checker."remote_files_lost" IntVal
storj.io/storj/satellite/repair/checker."remote_segments_checked" IntVal
satellite/repair: add monitoring for remote_segments_healthy_percentage Change-Id: I6ad29fe1a947ac19d15e40ea33164a510eb33d4f
2020-03-10 13:59:29 +00:00
storj.io/storj/satellite/repair/checker."remote_segments_failed_to_check" IntVal
storj.io/storj/satellite/repair/checker."remote_segments_healthy_percentage" FloatVal
scripts: add check-monitoring script to detect changes to monkit calls (#3114)
2019-10-15 18:00:14 +01:00
storj.io/storj/satellite/repair/checker."remote_segments_lost" IntVal
storj.io/storj/satellite/repair/checker."remote_segments_needing_repair" IntVal
satellite/repair/checker: add monkit metrics for segments immediately above repair threshold Record counts for segments at health=rt+1 through health=rt+5 for every checker iteration. Change-Id: I2a00c0bc34d17beb21cacdeab4dac77f755faefe
2020-02-24 23:06:52 +00:00
storj.io/storj/satellite/repair/checker."remote_segments_over_threshold_1" IntVal
storj.io/storj/satellite/repair/checker."remote_segments_over_threshold_2" IntVal
storj.io/storj/satellite/repair/checker."remote_segments_over_threshold_3" IntVal
storj.io/storj/satellite/repair/checker."remote_segments_over_threshold_4" IntVal
storj.io/storj/satellite/repair/checker."remote_segments_over_threshold_5" IntVal
satellite/metainfo, satellite/repair, uplink/eestream: add metric for download failed due to not enough pieces available (#3665)
2019-12-04 21:24:36 +00:00
storj.io/storj/satellite/repair/repairer."download_failed_not_enough_pieces_repair" Meter
satellite/repair: lock monkit stats in checker and repairer Change-Id: Ia10fc8da0177389a500359ce51d21a5806f3f7b1
2020-01-29 22:44:19 +00:00
storj.io/storj/satellite/repair/repairer."healthy_ratio_after_repair" FloatVal
storj.io/storj/satellite/repair/repairer."healthy_ratio_before_repair" FloatVal
storj.io/storj/satellite/repair/repairer."repair_attempts" Meter
satellite/repair/repairer: metrics for repair bytes uploaded and downloaded Change-Id: Icb0850692ecc155f6c8169edf1b045b2b546ff48
2021-07-20 15:41:38 +01:00
storj.io/storj/satellite/repair/repairer."repair_bytes_downloaded" Meter
storj.io/storj/satellite/repair/repairer."repair_bytes_uploaded" Meter
satellite/repair: lock monkit stats in checker and repairer Change-Id: Ia10fc8da0177389a500359ce51d21a5806f3f7b1
2020-01-29 22:44:19 +00:00
storj.io/storj/satellite/repair/repairer."repair_failed" Meter
storj.io/storj/satellite/repair/repairer."repair_nodes_unavailable" Meter
storj.io/storj/satellite/repair/repairer."repair_partial" Meter
storj.io/storj/satellite/repair/repairer."repair_segment_pieces_canceled" IntVal
storj.io/storj/satellite/repair/repairer."repair_segment_pieces_failed" IntVal
storj.io/storj/satellite/repair/repairer."repair_segment_pieces_successful" IntVal
storj.io/storj/satellite/repair/repairer."repair_segment_pieces_total" IntVal
storj.io/storj/satellite/repair/repairer."repair_segment_size" IntVal
storj.io/storj/satellite/repair/repairer."repair_success" Meter
satellite/repair: put irreparable segments in irreparableDB Previously, we were simply discarding rows from the repair queue when they couldn't be repaired (either because the overlay said too many nodes were down, or because we failed to download enough pieces). Now, such segments will be put into the irreparableDB for further and (hopefully) more focused attention. This change also better differentiates some error cases from Repair() for monitoring purposes. Change-Id: I82a52a6da50c948ddd651048e2a39cb4b1e6df5c
2020-02-24 20:13:12 +00:00
storj.io/storj/satellite/repair/repairer."repair_too_many_nodes_failed" Meter
satellite/repair: lock monkit stats in checker and repairer Change-Id: Ia10fc8da0177389a500359ce51d21a5806f3f7b1
2020-01-29 22:44:19 +00:00
storj.io/storj/satellite/repair/repairer."repair_unnecessary" Meter
satellite/repair: add monkit counter for segments below minimum required The current monkit reporting for "remote_segments_lost" is not usable for triggering alerts, as it has reported no data. To allow alerting, two new metrics "checker_segments_below_min_req" and "repairer_segments_below_min_req" will increment by zero on each segment unless it is below the minimum required piece count. The two metrics report what is found by the checker and the repairer respectively. Change-Id: I98a68bb189eaf68a833d25cf5db9e68df535b9d7
2020-11-10 14:49:19 +00:00
storj.io/storj/satellite/repair/repairer."repairer_segments_below_min_req" Counter
satellite/repair: put irreparable segments in irreparableDB Previously, we were simply discarding rows from the repair queue when they couldn't be repaired (either because the overlay said too many nodes were down, or because we failed to download enough pieces). Now, such segments will be put into the irreparableDB for further and (hopefully) more focused attention. This change also better differentiates some error cases from Repair() for monitoring purposes. Change-Id: I82a52a6da50c948ddd651048e2a39cb4b1e6df5c
2020-02-24 20:13:12 +00:00
storj.io/storj/satellite/repair/repairer."segment_deleted_before_repair" Meter
satellite/repair: lock monkit stats in checker and repairer Change-Id: Ia10fc8da0177389a500359ce51d21a5806f3f7b1
2020-01-29 22:44:19 +00:00
storj.io/storj/satellite/repair/repairer."segment_repair_count" IntVal
storj.io/storj/satellite/repair/repairer."segment_time_until_repair" IntVal
storj.io/storj/satellite/repair/repairer."time_for_repair" FloatVal
storj.io/storj/satellite/repair/repairer."time_since_checker_queue" FloatVal
satellite/satellitedb: Update audit history from overlay.UpdateStats and overlay.BatchUpdateStats Change-Id: Ib530b61895ca4a8b12ba022c408a416b237b56d7
2020-08-04 21:28:46 +01:00
storj.io/storj/satellite/satellitedb."audit_online_score" FloatVal
scripts: add check-monitoring script to detect changes to monkit calls (#3114)
2019-10-15 18:00:14 +01:00
storj.io/storj/satellite/satellitedb."audit_reputation_alpha" FloatVal
storj.io/storj/satellite/satellitedb."audit_reputation_beta" FloatVal
satellite/overlay: uncomment dq for offline and disable with feature flag Change-Id: Ib39e2be32e880b822a94eddfb81af99a38843a27
2020-10-15 17:00:08 +01:00
storj.io/storj/satellite/satellitedb."bad_audit_dqs" Meter
storj.io/storj/satellite/satellitedb."offline_dqs" Meter
satellite/{audit,overlay,satellitedb}: implement unknown audit reputation and suspension * change overlay.UpdateStats to allow a third audit outcome. Now it can handle successful, failed, and unknown audits. * when "unknown audit reputation" (unknownAuditAlpha/(unknownAuditAlpha+unknownAuditBeta)) falls below the DQ threshold, put node into suspension. * when unknown audit reputation goes above the DQ threshold, remove node from suspension. * record unknown audits from audit reporter. * add basic tests around unknown audits and suspension. Change-Id: I125f06f3af52e8a29ba48dc19361821a9ff1daa1
2020-03-09 15:35:54 +00:00
storj.io/storj/satellite/satellitedb."unknown_audit_reputation_alpha" FloatVal
storj.io/storj/satellite/satellitedb."unknown_audit_reputation_beta" FloatVal
satellite/overlay: uncomment dq for offline and disable with feature flag Change-Id: Ib39e2be32e880b822a94eddfb81af99a38843a27
2020-10-15 17:00:08 +01:00
storj.io/storj/satellite/satellitedb."unknown_suspension_dqs" Meter
storage/filestore: Monitor when we open files in trash Change-Id: I817bf8349c2e1ba55e1490f06162af1099bebdb0
2019-11-26 18:54:30 +00:00
storj.io/storj/storage/filestore."open_file_in_trash" Meter
storagenode: add new metric satellite.request (#3610) * storagenode: add new metric satellite.request * storagenode: metrics fixed * switch from Counter to Meter
2019-11-19 21:11:31 +00:00
storj.io/storj/storagenode/contact."satellite_contact_request" Meter
storj.io/storj/storagenode/gracefulexit."satellite_gracefulexit_request" Meter
storagenode/piecestore/usedserials: add monkit metric for serials that are randomly deleted This will give storagenode operators a better idea of whether the memory allocated to the usedserials store is sufficient. Change-Id: I5c30f2e39473a573f43409511ad9e2e32680479c
2020-06-09 21:58:48 +01:00
storj.io/storj/storagenode/piecestore/usedserials."delete_random_serial" Meter
Reference in New Issue Copy Permalink