Commit Graph

3939 Commits

Author SHA1 Message Date
Caleb Case
921b574554
docs/blueprints: slow down and retry (#3826) 2020-04-15 00:25:08 +02:00
Jess G
5ea1602ca5
satellite/overlay: add selected node cache (#3846)
* init implementation cache

Change-Id: Ia54a1943e0707a77189bc5f4a9aaa8339c98d99a

* one query to init cache

Change-Id: I7c04b3ae104b553ae23fca372351a4328f632c66

* add monit tracking of cache

Change-Id: I7d209e12c8f32d43708b23bf2126c5d5098e0a07

* add first test

Change-Id: I0646a9349d457a9eb3920f7cd2d62fb72ffc3ab5

* add staleness to cache

Change-Id: If002329bfdd53a4b200ad14dbd2ffc8b280aedb8

* add init test

Change-Id: I3a3d0aa74cfac1d125fa93cb749316ed2a74d5b1

* fix comment

Change-Id: I73353d00ccf0952b38c0f8ef7d1755c15cbfe9d9

* mv to nodeselection pkg

Change-Id: I62487f768296c7a7b597fa398a4c42daf6e9c5b7

* add state to cache

Change-Id: I081e77ec0e16706faee1a267de9a7fa643d6ac11

* add refresh concurrent test

Change-Id: Idcba72508291099f280edc65355273c0acc3d3ce

* add a few more tests

Change-Id: I9422e9eaa22bf01c11f14bdb892ebcf7b3e5e5fb

* fix tests, add min version to select allnodes

Change-Id: I926f41d568951ad4ff70c6d4ceb87abb1e3e5009

* update comments

Change-Id: I6ffe33e245ca65fb523c880cd72e63ce35776eb9

* fixes and rm Init

Change-Id: Ifbe09b668978b5d9af09ca38cb080d02a2154cf4

* fix format

Change-Id: I03cc217e28dc1839190c5c6dbdbb602c132a5a38
2020-04-14 13:50:02 -07:00
Moby von Briesen
d7794a4851 satellite/overlay: hardcode default values for audit alpha/beta
Alpha=1 and beta=0 are the expected first values for any alpha/beta
reputation system we are using in the codebase. So we are removing the
configurability of these values.

Change-Id: Ic61861b8ea5047fa1438ea6609b1d0048bf0abc3
2020-04-14 19:12:40 +00:00
JT Olio
e2d5b403e6 cmd/uplink: support --force (like awscli) for rb
Change-Id: If835c6dd08ee95e7c66ba7e4c7451cb3f0f95442
2020-04-14 18:10:54 +00:00
VitaliiShpital
6109fa685d web/satellite: allow users who have balance to create projects
Change-Id: I5c93dea09812fcbfcc1b7a9ca2202de3087b7032
2020-04-14 17:10:35 +00:00
Qweder93
bb4b7a919e storagenode/console/service satellites extends with data of oldest join to satellite
Change-Id: I413d5d649a0f331bda8fb4b72c4d43cbc8152361
2020-04-14 19:30:19 +03:00
Qweder93
743b3fb226 storagenode/nodestats: add pricing model, storagenode/cache: add paystub history storing
Change-Id: I9bc104a1407c8f286a964c796656d89b122bf752
2020-04-14 19:04:00 +03:00
VitaliiShpital
d392c6e658 web/satellite: copy api key container reworked
Change-Id: I51945d260095605327cd36e2f7d6e51f4044a429
2020-04-14 15:27:28 +00:00
Cameron Ayer
3ee6c14f54 satellite/downtime: add concurrency to downtime estimation
We want to increase our throughput for downtime estimation. This commit
adds the ability to reach out to multiple nodes concurrently for downtime
estimation. The number of concurrent routines is determined by a new config
flag, EstimationConcurrencyLimit. It also increases the default
EstimationBatchSize to 1000.

Change-Id: I800ce7ec1035885afa194c3c3f64eedd4f6f61eb
2020-04-14 14:39:13 +00:00
VitaliiShpital
17ccf36c15 web/satellite: payment methods component divided into smaller ones
Change-Id: Ibecc3803878e0902d458c3b28613c4ffbec0c634
2020-04-14 13:44:27 +00:00
Moby von Briesen
68946c86b4 cmd/storagenode: update colored storagenode dashboard to have correct indendation
Change-Id: I8f8fc2322c42e7e1098437abf4aad28bd754e79c
2020-04-14 13:21:17 +00:00
Egon Elbre
c97131ae78 satellite/metainfo: organize attribution methods
Change-Id: I4f35599c3f923861b5b05b128bf904480679f5f9
2020-04-14 11:27:43 +03:00
JT Olio
b9dc72c8b2 update links in README.md
Change-Id: I576f6f8d3f6d0f0530862ea5f0068a5b7017a656
2020-04-13 21:00:31 -06:00
JT Olio
67a19ec958 docs/blueprints: update link to uplink docs
Change-Id: I178d6518d2f7f26ac20b7e1efa7c52a5daa8748d
2020-04-13 21:00:31 -06:00
JT Olio
d08b2de558 lib/uplink: deprecate docs
guess what docs you get if you google "storj uplink"?

Change-Id: I600f1811f74a516d224189872f97727d57a153b1
2020-04-13 21:00:31 -06:00
Kaloyan Raev
e35cbfb6ee bump storj.io/common
Change-Id: I280e7d69973f1508ae09fbe9cf2a8ceea9001dc3
2020-04-13 17:05:13 +00:00
Egon Elbre
1cad686e9b storage: reduce default lookup limit to 500
Change-Id: Ic0adbf2f519babd780237d34c60636c1a1606762
2020-04-13 19:00:23 +03:00
Jeff Wendling
83bbc7a37e satellite/accounting/tally: ignore expired pointers
until we do a good job of cleaning them up, we should at least
not charge or pay people for them. nodes already locally delete
expired segments.

subsumes the tests in 1112.

Change-Id: I5961185764e02f6136b3231b44ecc75a9a8832c9
2020-04-10 11:36:00 -06:00
Cameron Ayer
02613407ae satellite/satellitedb: only suspend node if not already suspended
Whenever the node's reputation is updated, if its unknown audit
reputation is below the suspension threshold, its suspension field
is set to the current time. This could overwrite the previous
"suspendedAt" value resulting a node that never reaches the end of
its suspension.

Also log whenever a node is disqualified or its suspension status
changes

Change-Id: I5e8c8f1c46f66d79cb279b5b16a84fe03f533deb
2020-04-10 09:37:37 +00:00
NickolaiYurchenko
4e2a101fcc web/storagenode: audit checks depending on score
Change-Id: I134c25db4474ffcf0bc7e3183343902f32431650
2020-04-10 09:15:45 +00:00
Jeff Wendling
e33da90879 private/dbutil/cockroachutil: stop checking for jackc/pgx
we do not use that driver, and removing the case from the
type assertion reduces the satellite binary size by 5%.

Change-Id: I1c1b5e1e57dc4a98415103cfddd4f8c091588573
2020-04-10 07:19:02 +00:00
Egon Elbre
d86cce202c satellite/satellitedb: use arrays for arguments in node selection
This simplifies the code and makes queries faster:

name                               old time/op  new time/op  delta
SelectStorageNodes-32              7.72ms ± 6%  7.22ms ± 3%  -6.44%  (p=0.016 n=5+5)
SelectNewStorageNodes-32           7.75ms ± 2%  7.37ms ± 1%  -4.89%  (p=0.008 n=5+5)
SelectStorageNodesExclusion-32     16.9ms ± 0%  16.6ms ± 0%  -2.15%  (p=0.008 n=5+5)
SelectNewStorageNodesExclusion-32  17.2ms ± 0%  16.6ms ± 2%  -3.69%  (p=0.008 n=5+5)
FindStorageNodes-32                45.5ms ± 0%  45.1ms ± 1%    ~     (p=0.056 n=5+5)
FindStorageNodesExclusion-32       77.4ms ± 0%  75.9ms ± 0%  -1.91%  (p=0.008 n=5+5)

Change-Id: I38f77f6282b9738e8416113d42c6acb46c03da7b
2020-04-09 21:16:10 +03:00
Egon Elbre
ccf4f9ed2d satellite/satellitedb: node selection code cleanup
Reduce the number of non-methods to reduce funcs in the namespace also
combine a func to slightly condense the code more.

Change-Id: Ifbe728eb8c8ca4c981df648decd259c2097b6b40
2020-04-09 20:41:29 +03:00
Natalie Villasana
cf80b3caf3
satellite/overlay: combine SelectStorageNodes and SelectNewStorageNodes (#3831) 2020-04-09 11:19:44 -04:00
Jeff Wendling
d658a6a6ec private/dbutil/txutil: fix logic in transaction retries
before this change, any transaction that took longer than 5 minutes
even if it succeeded, would get a retry error included in the
result.

try to make the logic more clear and add comments for the reader.

Change-Id: Ib84a89a33907a24426ecf52c90404be0e0dfa307
2020-04-09 13:58:53 +00:00
Moby von Briesen
14b3704f56 storagenode: add suspended status to storagenode dashboard/api
* Add migration to storagenode reputation table to add suspended
timestamp
* Send suspended info to storagenode from satellite nodestats endpoint
* Add suspended status to storagenode api
* Add an indicator on the storagenode dashboard informing operator of
the satellites the node is suspended on

Change-Id: Ie3669f6069cc0258ba76ec99d17006e1b5fd9c8a
2020-04-09 13:36:23 +00:00
Michal Niewrzal
f36e8548f1 satellite/metainfo: adjust max inline segment size validation to
potential encryption overhead.

This is the same approach we have for validating remote segment size.

https://storjlabs.atlassian.net/browse/USR-619

Change-Id: I2597ee734313a3068fd986001680bbedbf1bed2a
2020-04-09 12:34:10 +00:00
Egon Elbre
11a44cdd88 all: don't depend on gogo/proto directly
Change-Id: I8822dea0d1b7b99e0b828e0373a0308a42dde2be
2020-04-08 17:32:15 +00:00
Natalie Villasana
0a1bbc9824 satellite/overlay: add TestEnsureMinimumRequested
Adds a test to make sure that the correct amount of total nodes,
reputable nodes, and new nodes are returned by SelectStorageNodes
in different cases.

Change-Id: I0939159600afde8a46c35735f1edf0576fcdb4cd
2020-04-08 16:06:25 +00:00
Qweder93
3a9422cc9a satellite/nodestats: add pricing model to endpoint
Change-Id: Iddace8e437216a343458f440b543cee61164f233
2020-04-08 14:29:51 +00:00
Egon Elbre
cf26951a5b satellite/satellitedb/pbold: remove dead code
Change-Id: I7464773c20b8f99a601ca9cc4bee804f1ac14cf9
2020-04-08 15:22:31 +03:00
Fadila Khadar
1eb501b5c8 satellite/accounting: test zombie segments not billed
Check zombie segments are not billed

Change-Id: If998e785dcc82c382613318aee3043eaef7fd9ea
2020-04-08 08:28:49 +00:00
Yingrong Zhao
96e58d21b4 cmd;pkg/server: init tracing collector in all processes
Add tracing handler in drpc server.
Initializing tracing collector in admin, satellite api, garbage
collection, satellite core, repaier, storagenode.
Change-Id: Ie98420e35dfc6913836ebd82b517d9d12877aefc

Change-Id: I91057b6265a4ac8bde033dfde692b8a28acca99f
2020-04-07 17:20:59 -04:00
paul cannon
43cf036113
installer/windows: don't require ca.key file (#3847)
As noted by @sixcorners at
https://github.com/storj/storj/pull/3819#issuecomment-609025487, the
Windows installer code incorrectly fails if the ca.key file is not
present.

The ca.key file should not be necessary for regular storage node usage,
and in fact we recommend that users move the ca.key file to secure
storage rather than keeping it on the node.

I'm not sure how to run the tests, so I'll get a nod from the SNO Growth
team before merging.

Change-Id: Ib2fe236de5c75165644e880caa827a2a1a034c87
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2020-04-07 10:02:41 -05:00
igor gaidaienko
80ee7321cd satellite/accounting: add test billing without expansion factor
Test checking that billing contains the download traffic without the extra pieces

Change-Id: I5c5d6d877116d26a83c98a93da424eecffcb804e
2020-04-07 12:55:43 +00:00
VitaliiShpital
ef6c9d89bb web/satellite: dashboard blur step after adding payment method removed
Change-Id: I926016feca0cf0bf97feef8fd6936fbf466a5a51
2020-04-07 12:12:03 +00:00
Jeff Wendling
2ded64ba2c satellite/compensation: more fixes to get prod running smoothly
Change-Id: I13a76d9d49222fb10796415a015f224d4084fde3
2020-04-07 10:10:27 +00:00
Michal Niewrzal
d23e03f8a2 lib/uplink-gomobile: extend encryption key capabilities
Change-Id: Id10b14cc34ff983ec451440fe4b0305651115022
2020-04-07 06:58:08 +00:00
Egon Elbre
a4c554f2ed satellite/admin: support user query by email
This adds new endpoint /api/user/{user-email} which allows to get the
projects where the user is a member.

It also moves existing endpoint:

    /project/{projectid}/limit -> /api/project/{projectid}/limit

To avoid future conflicts for displaying pages.

Change-Id: I5efe3e1c8f79894c136f92ed815f635a34ba6f98
2020-04-06 18:32:25 +00:00
Qweder93
0f71e60a53 storanode/version/chore notifications temporary disabled
Change-Id: Ifacdede772bc7b678288410d215e665b014f566f
2020-04-06 16:57:12 +00:00
VitaliiShpital
61b6ff5186 web/satellite: available credits amount added on billing page
Change-Id: I9d9c18ad08cf150ceba078356e39132a602558d4
2020-04-06 14:39:05 +00:00
Michal Niewrzal
2d1a6968b0 satellite/metainfo: revert returning segment size and inline segment
size with BeginObject

Such solution will add one round trip to satellite during upload so for
now we are reverting this until we will have solution for this.

Change-Id: Ic2d826448ab7b0318cd6922df05deee9167cf2f0
2020-04-06 13:36:34 +02:00
Jennifer Johnson
1547e791a3 satellitedb: remove free_bandwidth column from nodes table
Change-Id: I9d1d3de9216c6533c1042ef473631721a011d086
2020-04-06 09:30:28 +00:00
Michal Niewrzal
b17dc656b5 lib/uplink-gomobile: update to latest master
Change-Id: I1be7991ee73cc2a6e911ae024dce3fba97c02071
2020-04-04 19:08:48 +00:00
Yingrong Zhao
4ab553cf3f scripts: only latest releases have version file
Change-Id: Ice212ac39b011c18b1d3d48ea9e6f580b4fd7f0c
2020-04-04 18:47:33 +00:00
Cameron Ayer
42be4bdc0f satellite/contact: add timeout to PingBack method
Change-Id: I2ec2f82e2e10d8be16f82e9de13ce42358e47c98
2020-04-04 18:26:30 +00:00
Egon Elbre
9200efc61f satellite/satellitedb: fix selecting a nullable string
Change-Id: I59e645966e09da586512c69101691b47055c1e5a
2020-04-03 21:30:20 +03:00
paul cannon
0c8c11b251 satellite/audit: add not_enough_shares_for_audit counter
We have been using the SQL expression `name='(*Verifier).Verify' AND
error_name='not enough shares for successful audit'` thus far to detect
cases of this problem and alert on them. Unfortunately, since this
rarely (hopefully never) happens, influxdb has no data for most of the
auditor instances, and when it has no data for a time series, it returns
no columns either. This makes Redash upset when it tries to perform a
query for an alert and can't find the column whose value it expects to
check.

This change should make it so zero values are reported when the problem
has not happened, and higher values when it has.

Change-Id: I79e5e000f879678b661dac88caae1e2915b39ab1
2020-04-03 17:00:50 +00:00
VitaliiShpital
dbd036de8b web/satellite: api key and proj member name's length visibility extended
Change-Id: If42dc48849f6414f4224d57ffa7267431426b457
2020-04-03 11:32:41 +00:00
Matt Robinson
4b01a8dd18
Descellate Usage Error to Debug (#3780) 2020-04-03 13:20:20 +02:00