Commit Graph

3939 Commits

Author SHA1 Message Date
littleskunk
1336070fec
storagenode/piecestore: add missing log message about audit errors (#3861)
* storagenode/piecestore: add missing log message about audit errors
* storagenode/piecestore: add monkit data for oder limit verification errors
2020-04-23 13:20:47 -04:00
Moby von Briesen
720e26d235 satellite/satellitedb/overlaycache: update unknown alpha/beta values properly
Update unknown_audit_reputation_alpha and unknown_audit_reputation_beta.
Add test to verify that BatchUpdateStats properly modifies unknown audit
alpha/beta

Change-Id: I0d5f9cac96a99f64905cf575b772402db0756a9d
2020-04-23 10:40:53 -04:00
Ivan Fraixedes
a0692d0db8
private/migrate: enhance docs in some funcs
Enhance the doc comment in some migration methods.

Change-Id: I3d91f7e01f24670fe3d972bd3b022b8a47251bdc
2020-04-23 13:06:06 +02:00
Moby von Briesen
72b93f3120 satellite/satellitedb: disqualify suspended nodes when the grace period passes
If a node is suspended and receives an unknown or failing audit,
disqualify them if the grace period (default 1w in production) has
passed.

Migrate the nodes table so any node that is currently suspended gets
unsuspended when the satellite starts up.

Change-Id: I7b81c68026f823417faa0bf5e5cb5e67c7156b82
2020-04-22 15:45:00 -04:00
Egon Elbre
676f3e8516 satellite/metainfo/piecedeletion: try to make batches larger
Currently it was possible that PopAll returns 1010 items, then
makes one RPC call with 1000 items, then RPC call 10 items. Meanwhile,
there have been added 500 new items added to the queue.

This change ensures that we pull items from the queue early and
try to make rpc batches as large as possible.

Change-Id: I1a30dde9164c2ff7b90c906a9544593c4f1cf0e9
2020-04-22 18:43:29 +00:00
Yingrong Zhao
0bdcf123cf bump monkit, monkit-jaeger, and private to latest
Also bump storj.io/common and sync repo

Change-Id: If8e60db6bdf0af8077b7befcb1da304c3c4dcae4
2020-04-22 12:30:37 -04:00
Ethan Adams
60e07f0a8b Revert "satellite/accounting: Remove unnecessary index bucket_bandwidth_rollups_project_id_action_interval_index"
This reverts commit 105dc7acc6.

Reason for revert: Recent changes to the Postgres query plan seems to want to use this index now.  Reverting until we have time to analyze what's happening.

Change-Id: I74b4b5a8f15c3850d8a958a29f51dbc80e7c282c
2020-04-22 14:49:04 +00:00
Moby von Briesen
178aa8b5e0 satellite/{metainfo,repair}: Delete expired segments from metainfo
* Delete expired segments in expired segments service using metainfo
loop
* Add test to verify expired segments service deletes expired segments
* Ignore expired segments in checker observer
* Modify checker tests to verify that expired segments are ignored
* Ignore expired segments in segment repairer and drop from repair queue
* Add repair test to verify that a segment that expires after being
added to the repair queue is ignored and dropped from the repair queue

Change-Id: Ib2b0934db525fef58325583d2a7ca859b88ea60d
2020-04-22 13:02:31 +00:00
Matt Robinson
e34937317c
Call the right make target for the segment-reaper (#3860) 2020-04-22 08:34:24 +02:00
Yingrong Zhao
0b8699bcb5 cmd: add prompt for enabling tracing during uplink cli setup
We want to make tracing to be opt-in.
For now, we will use `tracing.sample` as the toggle config to enable or
disable tracing and default to sample every traces from uplink cli.
If user wants to change the default sampling rate, they can do so by
using the `--tracing.sample` flag to override the default value

Change-Id: I6f25dac0f43024c50a8aaf6c549e6a514211f834
2020-04-21 20:57:10 +00:00
Qweder93
805e328c47 storagenode/heldamount payments removed
Change-Id: I87cc04f43d182a4190a571ef417be85d02db9d34
2020-04-21 17:15:31 +00:00
Ethan
105dc7acc6 satellite/accounting: Remove unnecessary index bucket_bandwidth_rollups_project_id_action_interval_index
See https://storjlabs.atlassian.net/browse/SM-738

Change-Id: I9ba3cc3fbff9f13fc0b95d25feee5a19e5a5c486
2020-04-21 16:43:09 +00:00
VitaliiShpital
7365fc434e web/satellite: available credits disabled due to coupon expiration bug
Change-Id: I8271640a7f4364e27e5ff570c7e26d5ee4fdbcd4
2020-04-21 15:48:15 +00:00
NickolaiYurchenko
89c877f461 web/storagenode: payout calculation fix
Change-Id: Ibd030d0ef91a28e2cfa94da78c211c27959bb753
2020-04-21 18:11:47 +03:00
Qweder93
3d56efc82d storagenode/console/service: Satellites EarliestJoinDate calculation ignores empty date
Change-Id: Ic528467dbf0a47a7779fd7ae054856744298a39c
2020-04-21 17:50:21 +03:00
Matt Robinson
1e295a48e7
add container image for segment-reaper (#3855) 2020-04-21 17:48:40 +03:00
Egon Elbre
e655e160dc private/testuplink: delete delete
ecclient.Delete is a deprecated func that shouldn't be used anymore.

Change-Id: Ica4d17e334220311c99cea28f1d0e2d854d72896
2020-04-21 13:56:40 +00:00
NickolaiYurchenko
0300076684 web/storagenode: audit checks based on score
Change-Id: I7e9c16ded3165a7da31117412700092de135da1d
2020-04-21 16:18:35 +03:00
NickolaiYurchenko
a237512123 web/storagenode: added division on price multiplier
Change-Id: Ie1146ae6eac1f626753e4bcfaecd3c4919d1e464
2020-04-21 15:56:00 +03:00
NickolaiYurchenko
ed701c196d web/storagenode: disk space displayed by hour
Change-Id: Id52fc9da39e0c38a05b6b343e97d18f5453ea1f5
2020-04-21 15:30:58 +03:00
Qweder93
e999f24e54 storagenode/nodestats/cache: storagenodeDB/heldamount sync with satelliteDB/storagenode_paystub
Change-Id: If894166809bee8a5e036e618005d8141c2a0c594
2020-04-20 19:12:17 +00:00
NickolaiYurchenko
b9dbd80515 web/storagenode: all fetched paystub data treated as list
Change-Id: I536d36bc0edf5c54eaa07b60e55b93f1e2a1f826
2020-04-20 18:33:30 +00:00
Yingrong Zhao
8375a09c89 cmd: remove InitTracing from satellite and storagenode main.go file
Change-Id: I4addbe7d0645f66abfb3e98d74d17035e9624e69
2020-04-20 14:06:26 -04:00
JT Olio
5f38f8f1fe satellite/gc: use hostname for metric instance ids instead of node id
currently production uses a different application suffix for gc
services, so chronograf can distinguish between gc processes and core
processes, but it'd be nice to be a bit more consistent with repairers
and api servers

Change-Id: Icb96fed006c59d7afd730317d35636a6e4573b58
2020-04-20 14:52:44 +00:00
Qweder93
1af70703ef storagenode/console/service: SatellitePayStubMonthly returns array of objects
Change-Id: I06d86087c81acd2eb3acd73c1997bab9734bae9e
2020-04-20 14:26:07 +00:00
Michal Niewrzal
884638a294 scripts: reduce segment size for integration tests
This will speedup integration tests.

Change-Id: Ifb1fadd01692390457971bfe7f9ba73a0796abb2
2020-04-20 15:13:05 +02:00
Michal Niewrzal
c021b35879 private/testplanet: migrate testplanet to new libuplink
Replace most of old libuplink usages in testplanet. 100% migration will
be possible when we will be able to implement UploadWithClientConfig
with new libuplink.

Change-Id: I432d7d4917c7b67d46a058abd0a2a6a13f565ac4
2020-04-20 12:43:34 +00:00
NickolaiYurchenko
ac1cc9eeda web/storagenode: payout fetch error fix, held row added
Change-Id: Ibf5c254033583759ca282aa9791900154785f88a
2020-04-20 14:35:23 +03:00
Qweder93
6e3585e394 satellite/heldamount/endpoint : GetAllPaystubs added
Change-Id: Ic8cdd9db8b2a68796f9579c7fed2d49d9054bd64
2020-04-19 19:21:54 +03:00
NickolaiYurchenko
31c9cc322f web/storagenode: dark theme
Change-Id: I7c987ea19eef4b95832501773e00f592300fc9c5
2020-04-18 23:42:27 +03:00
NickolaiYurchenko
aa6cff405d web/storagenode: payout page bug fix
Change-Id: I1ab8a698101bbe84e1b1cd0f00c18ae079cf27ab
2020-04-18 22:31:49 +03:00
littleskunk
52cd33a633
scripts/deploy-storagenode: tag docker image as latest (#3858)
Co-authored-by: Ivan Fraixedes <ivan@fraixed.es>
2020-04-18 11:59:56 +02:00
Ethan
4cd86ff780 satellite/accounting: Add index on bucket_bandwidth_rollups for action, interval_start, and project_id
See https://storjlabs.atlassian.net/browse/SM-551 for details

Change-Id: I104c4e87d5aef500cc4a3893817763808f76c484
2020-04-17 19:14:45 +00:00
JT Olio
7403e9a8ab satellite web ui: fix spelling mistake
Change-Id: I866c1045a3a487ccb22a336ba34932c7ca9e1fb5
2020-04-17 09:11:19 -06:00
Qweder93
30369b027c storagenode/storagenodedb/reputation: add joined_at
Change-Id: Ic471fac97bf54b537f2c34f24b4069b0641c746d
2020-04-17 12:12:09 +00:00
Egon Elbre
7e0e74c65c satellite/metainfo: attribution based on useragent
Automatically attach attribution information to bucket during
BeginObject or CreateBucket when the UserAgent is set.

Change-Id: I405cb26c5a2f7394b30e3f2cf5d2214c8781eb8b
2020-04-17 10:47:14 +03:00
Cameron Ayer
a6f0be2047 cmd/inspector: decode path
Change-Id: I5c1eb7842ba5fc76c19c339e5be8f108d3f21fc9
2020-04-16 20:12:08 +00:00
Qweder93
c498e52cf2 satellite/nodestats/endoint: reputation stats updated
Change-Id: I03a44c4c3465d4f8f23fcbbc742148f5e49544e2
2020-04-16 22:28:58 +03:00
Egon Elbre
9052085f70 private/testplanet: simplify uplink usage
Change-Id: I3e488dc296f1094ce95e6d6597ca6d3f8da90a76
2020-04-16 16:45:55 +00:00
Egon Elbre
a129a8bd35 all: separate err check for http
We want to avoid net/http dependency in errs2 package, hence we removed
http.ErrServerClosed from IgnoreCanceled and IsCanceled check. Now we
need to add that check explicitly to every http endpoint.

Change-Id: I62b1cc0a0a2d3b43301d713a7951e5022145f88f
2020-04-16 18:50:24 +03:00
VitaliiShpital
2dce4c232c web/satellite: redirect to verification page on sign up if inside iframe
Change-Id: I606b63fd27bef46597697b491970523e8a3a0cae
2020-04-16 13:35:49 +00:00
Egon Elbre
93e39a6b33 satellite/metainfo: support co-joining the loop
This adds support for observers to join the loop together. This allows
to ensure that when multiple observers join, they will be part of the
same loop iteration.

Change-Id: Ie887d4cedfb074b65c782690a2c09c1704f56dfe
2020-04-16 13:31:17 +03:00
Egon Elbre
45d1ca87f5 satellite/metainfo/piecedeletion: handle zero case
During testing it's possible to get into a scenario where all nodes are
offline and list of requests is empty.

Change-Id: I271c0ca2c72009244df13e8bc1441fcd5f3da9e0
2020-04-16 13:00:32 +03:00
Jess G
dc78cd9634
storagenode/piecestore: fix annoying info log for upload canceled (#3857)
* storagenode/piecestore: fix annoying info upload canceled log

Change-Id: I3dd0f44226e7b946a2b30c3d0f30f28749ca6e88

* keep as info

Change-Id: I7ebb5c19e4865e3030a8a6bb7f6279d316853e89

Co-authored-by: Fadila <Fadila82@users.noreply.github.com>
2020-04-15 14:15:07 -07:00
Jess G
75b9a5971e
satellite: update log levels (#3851)
* satellite: update log levels

Change-Id: I86bc32e042d742af6dbc469a294291a2e667e81f

* log version on start up for every service

Change-Id: Ic128bb9c5ac52d4dc6d6c4cb3059fbad73f5d3de

* Use monkit for tracking failed ip resolutions

Change-Id: Ia5aa71d315515e0c5f62c98d9d115ef984cd50c2

* fix compile errors

Change-Id: Ia33c8b6e34e780bd1115120dc347a439d99e83bf

* add request limit value to storage node rpc err

Change-Id: I1ad6706a60237928e29da300d96a1bafa94156e5

* we cant track storage node ids in monkit metrics so lets use logging to track that for expired orders

Change-Id: I1cc1d240b29019ae2f8c774792765df3cbeac887

* fix build errs

Change-Id: I6d0ffe058e9a38b7ed031c85a29440f3d68e8d47
2020-04-15 12:32:22 -07:00
Egon Elbre
d3ce845f82 satellite: revert log lines used to figure out node id
Currently storj-sim relies on the log lines to be exactly the same,
when they change it cannot find the necessary information from log.

Change-Id: Ia039915ef3375a7cf60f107b2c05c958de15b6d5
2020-04-15 17:07:56 +03:00
Michal Niewrzal
90bac51544 cmd/uplink/cmd: improve error message when config.yaml or --access flag
is missing

https://storjlabs.atlassian.net/browse/USR-78

Change-Id: I8221822d164279b85f24573df0ac05fcd6b3881e
2020-04-15 13:22:02 +00:00
Kaloyan Raev
a2ce836761 remove sugar logging
Change-Id: I6b6ca9704837cb3f5f5449ba7f55661487814d9f
2020-04-15 12:37:47 +00:00
VitaliiShpital
158013a866 satellite/console: redirect on account activation
Change-Id: I2506ce0fd3832bf46fbcdcc5a42bb83dc926e99a
2020-04-15 11:49:50 +00:00
Egon Elbre
2c0d61b18e satellite/metainfo: avoid temporary list
Currently ListV2 loaded the whole data into memory, even when all the
data wasn't being used, using up more memory than needed.

Change-Id: I5846d979344729b447c108a6cc9f4227229ec981
2020-04-15 08:01:42 +00:00