Commit Graph

3900 Commits

Author SHA1 Message Date
Egon Elbre
a129a8bd35 all: separate err check for http
We want to avoid net/http dependency in errs2 package, hence we removed
http.ErrServerClosed from IgnoreCanceled and IsCanceled check. Now we
need to add that check explicitly to every http endpoint.

Change-Id: I62b1cc0a0a2d3b43301d713a7951e5022145f88f
2020-04-16 18:50:24 +03:00
VitaliiShpital
2dce4c232c web/satellite: redirect to verification page on sign up if inside iframe
Change-Id: I606b63fd27bef46597697b491970523e8a3a0cae
2020-04-16 13:35:49 +00:00
Egon Elbre
93e39a6b33 satellite/metainfo: support co-joining the loop
This adds support for observers to join the loop together. This allows
to ensure that when multiple observers join, they will be part of the
same loop iteration.

Change-Id: Ie887d4cedfb074b65c782690a2c09c1704f56dfe
2020-04-16 13:31:17 +03:00
Egon Elbre
45d1ca87f5 satellite/metainfo/piecedeletion: handle zero case
During testing it's possible to get into a scenario where all nodes are
offline and list of requests is empty.

Change-Id: I271c0ca2c72009244df13e8bc1441fcd5f3da9e0
2020-04-16 13:00:32 +03:00
Jess G
dc78cd9634
storagenode/piecestore: fix annoying info log for upload canceled (#3857)
* storagenode/piecestore: fix annoying info upload canceled log

Change-Id: I3dd0f44226e7b946a2b30c3d0f30f28749ca6e88

* keep as info

Change-Id: I7ebb5c19e4865e3030a8a6bb7f6279d316853e89

Co-authored-by: Fadila <Fadila82@users.noreply.github.com>
2020-04-15 14:15:07 -07:00
Jess G
75b9a5971e
satellite: update log levels (#3851)
* satellite: update log levels

Change-Id: I86bc32e042d742af6dbc469a294291a2e667e81f

* log version on start up for every service

Change-Id: Ic128bb9c5ac52d4dc6d6c4cb3059fbad73f5d3de

* Use monkit for tracking failed ip resolutions

Change-Id: Ia5aa71d315515e0c5f62c98d9d115ef984cd50c2

* fix compile errors

Change-Id: Ia33c8b6e34e780bd1115120dc347a439d99e83bf

* add request limit value to storage node rpc err

Change-Id: I1ad6706a60237928e29da300d96a1bafa94156e5

* we cant track storage node ids in monkit metrics so lets use logging to track that for expired orders

Change-Id: I1cc1d240b29019ae2f8c774792765df3cbeac887

* fix build errs

Change-Id: I6d0ffe058e9a38b7ed031c85a29440f3d68e8d47
2020-04-15 12:32:22 -07:00
Egon Elbre
d3ce845f82 satellite: revert log lines used to figure out node id
Currently storj-sim relies on the log lines to be exactly the same,
when they change it cannot find the necessary information from log.

Change-Id: Ia039915ef3375a7cf60f107b2c05c958de15b6d5
2020-04-15 17:07:56 +03:00
Michal Niewrzal
90bac51544 cmd/uplink/cmd: improve error message when config.yaml or --access flag
is missing

https://storjlabs.atlassian.net/browse/USR-78

Change-Id: I8221822d164279b85f24573df0ac05fcd6b3881e
2020-04-15 13:22:02 +00:00
Kaloyan Raev
a2ce836761 remove sugar logging
Change-Id: I6b6ca9704837cb3f5f5449ba7f55661487814d9f
2020-04-15 12:37:47 +00:00
VitaliiShpital
158013a866 satellite/console: redirect on account activation
Change-Id: I2506ce0fd3832bf46fbcdcc5a42bb83dc926e99a
2020-04-15 11:49:50 +00:00
Egon Elbre
2c0d61b18e satellite/metainfo: avoid temporary list
Currently ListV2 loaded the whole data into memory, even when all the
data wasn't being used, using up more memory than needed.

Change-Id: I5846d979344729b447c108a6cc9f4227229ec981
2020-04-15 08:01:42 +00:00
Caleb Case
921b574554
docs/blueprints: slow down and retry (#3826) 2020-04-15 00:25:08 +02:00
Jess G
5ea1602ca5
satellite/overlay: add selected node cache (#3846)
* init implementation cache

Change-Id: Ia54a1943e0707a77189bc5f4a9aaa8339c98d99a

* one query to init cache

Change-Id: I7c04b3ae104b553ae23fca372351a4328f632c66

* add monit tracking of cache

Change-Id: I7d209e12c8f32d43708b23bf2126c5d5098e0a07

* add first test

Change-Id: I0646a9349d457a9eb3920f7cd2d62fb72ffc3ab5

* add staleness to cache

Change-Id: If002329bfdd53a4b200ad14dbd2ffc8b280aedb8

* add init test

Change-Id: I3a3d0aa74cfac1d125fa93cb749316ed2a74d5b1

* fix comment

Change-Id: I73353d00ccf0952b38c0f8ef7d1755c15cbfe9d9

* mv to nodeselection pkg

Change-Id: I62487f768296c7a7b597fa398a4c42daf6e9c5b7

* add state to cache

Change-Id: I081e77ec0e16706faee1a267de9a7fa643d6ac11

* add refresh concurrent test

Change-Id: Idcba72508291099f280edc65355273c0acc3d3ce

* add a few more tests

Change-Id: I9422e9eaa22bf01c11f14bdb892ebcf7b3e5e5fb

* fix tests, add min version to select allnodes

Change-Id: I926f41d568951ad4ff70c6d4ceb87abb1e3e5009

* update comments

Change-Id: I6ffe33e245ca65fb523c880cd72e63ce35776eb9

* fixes and rm Init

Change-Id: Ifbe09b668978b5d9af09ca38cb080d02a2154cf4

* fix format

Change-Id: I03cc217e28dc1839190c5c6dbdbb602c132a5a38
2020-04-14 13:50:02 -07:00
Moby von Briesen
d7794a4851 satellite/overlay: hardcode default values for audit alpha/beta
Alpha=1 and beta=0 are the expected first values for any alpha/beta
reputation system we are using in the codebase. So we are removing the
configurability of these values.

Change-Id: Ic61861b8ea5047fa1438ea6609b1d0048bf0abc3
2020-04-14 19:12:40 +00:00
JT Olio
e2d5b403e6 cmd/uplink: support --force (like awscli) for rb
Change-Id: If835c6dd08ee95e7c66ba7e4c7451cb3f0f95442
2020-04-14 18:10:54 +00:00
VitaliiShpital
6109fa685d web/satellite: allow users who have balance to create projects
Change-Id: I5c93dea09812fcbfcc1b7a9ca2202de3087b7032
2020-04-14 17:10:35 +00:00
Qweder93
bb4b7a919e storagenode/console/service satellites extends with data of oldest join to satellite
Change-Id: I413d5d649a0f331bda8fb4b72c4d43cbc8152361
2020-04-14 19:30:19 +03:00
Qweder93
743b3fb226 storagenode/nodestats: add pricing model, storagenode/cache: add paystub history storing
Change-Id: I9bc104a1407c8f286a964c796656d89b122bf752
2020-04-14 19:04:00 +03:00
VitaliiShpital
d392c6e658 web/satellite: copy api key container reworked
Change-Id: I51945d260095605327cd36e2f7d6e51f4044a429
2020-04-14 15:27:28 +00:00
Cameron Ayer
3ee6c14f54 satellite/downtime: add concurrency to downtime estimation
We want to increase our throughput for downtime estimation. This commit
adds the ability to reach out to multiple nodes concurrently for downtime
estimation. The number of concurrent routines is determined by a new config
flag, EstimationConcurrencyLimit. It also increases the default
EstimationBatchSize to 1000.

Change-Id: I800ce7ec1035885afa194c3c3f64eedd4f6f61eb
2020-04-14 14:39:13 +00:00
VitaliiShpital
17ccf36c15 web/satellite: payment methods component divided into smaller ones
Change-Id: Ibecc3803878e0902d458c3b28613c4ffbec0c634
2020-04-14 13:44:27 +00:00
Moby von Briesen
68946c86b4 cmd/storagenode: update colored storagenode dashboard to have correct indendation
Change-Id: I8f8fc2322c42e7e1098437abf4aad28bd754e79c
2020-04-14 13:21:17 +00:00
Egon Elbre
c97131ae78 satellite/metainfo: organize attribution methods
Change-Id: I4f35599c3f923861b5b05b128bf904480679f5f9
2020-04-14 11:27:43 +03:00
JT Olio
b9dc72c8b2 update links in README.md
Change-Id: I576f6f8d3f6d0f0530862ea5f0068a5b7017a656
2020-04-13 21:00:31 -06:00
JT Olio
67a19ec958 docs/blueprints: update link to uplink docs
Change-Id: I178d6518d2f7f26ac20b7e1efa7c52a5daa8748d
2020-04-13 21:00:31 -06:00
JT Olio
d08b2de558 lib/uplink: deprecate docs
guess what docs you get if you google "storj uplink"?

Change-Id: I600f1811f74a516d224189872f97727d57a153b1
2020-04-13 21:00:31 -06:00
Kaloyan Raev
e35cbfb6ee bump storj.io/common
Change-Id: I280e7d69973f1508ae09fbe9cf2a8ceea9001dc3
2020-04-13 17:05:13 +00:00
Egon Elbre
1cad686e9b storage: reduce default lookup limit to 500
Change-Id: Ic0adbf2f519babd780237d34c60636c1a1606762
2020-04-13 19:00:23 +03:00
Jeff Wendling
83bbc7a37e satellite/accounting/tally: ignore expired pointers
until we do a good job of cleaning them up, we should at least
not charge or pay people for them. nodes already locally delete
expired segments.

subsumes the tests in 1112.

Change-Id: I5961185764e02f6136b3231b44ecc75a9a8832c9
2020-04-10 11:36:00 -06:00
Cameron Ayer
02613407ae satellite/satellitedb: only suspend node if not already suspended
Whenever the node's reputation is updated, if its unknown audit
reputation is below the suspension threshold, its suspension field
is set to the current time. This could overwrite the previous
"suspendedAt" value resulting a node that never reaches the end of
its suspension.

Also log whenever a node is disqualified or its suspension status
changes

Change-Id: I5e8c8f1c46f66d79cb279b5b16a84fe03f533deb
2020-04-10 09:37:37 +00:00
NickolaiYurchenko
4e2a101fcc web/storagenode: audit checks depending on score
Change-Id: I134c25db4474ffcf0bc7e3183343902f32431650
2020-04-10 09:15:45 +00:00
Jeff Wendling
e33da90879 private/dbutil/cockroachutil: stop checking for jackc/pgx
we do not use that driver, and removing the case from the
type assertion reduces the satellite binary size by 5%.

Change-Id: I1c1b5e1e57dc4a98415103cfddd4f8c091588573
2020-04-10 07:19:02 +00:00
Egon Elbre
d86cce202c satellite/satellitedb: use arrays for arguments in node selection
This simplifies the code and makes queries faster:

name                               old time/op  new time/op  delta
SelectStorageNodes-32              7.72ms ± 6%  7.22ms ± 3%  -6.44%  (p=0.016 n=5+5)
SelectNewStorageNodes-32           7.75ms ± 2%  7.37ms ± 1%  -4.89%  (p=0.008 n=5+5)
SelectStorageNodesExclusion-32     16.9ms ± 0%  16.6ms ± 0%  -2.15%  (p=0.008 n=5+5)
SelectNewStorageNodesExclusion-32  17.2ms ± 0%  16.6ms ± 2%  -3.69%  (p=0.008 n=5+5)
FindStorageNodes-32                45.5ms ± 0%  45.1ms ± 1%    ~     (p=0.056 n=5+5)
FindStorageNodesExclusion-32       77.4ms ± 0%  75.9ms ± 0%  -1.91%  (p=0.008 n=5+5)

Change-Id: I38f77f6282b9738e8416113d42c6acb46c03da7b
2020-04-09 21:16:10 +03:00
Egon Elbre
ccf4f9ed2d satellite/satellitedb: node selection code cleanup
Reduce the number of non-methods to reduce funcs in the namespace also
combine a func to slightly condense the code more.

Change-Id: Ifbe728eb8c8ca4c981df648decd259c2097b6b40
2020-04-09 20:41:29 +03:00
Natalie Villasana
cf80b3caf3
satellite/overlay: combine SelectStorageNodes and SelectNewStorageNodes (#3831) 2020-04-09 11:19:44 -04:00
Jeff Wendling
d658a6a6ec private/dbutil/txutil: fix logic in transaction retries
before this change, any transaction that took longer than 5 minutes
even if it succeeded, would get a retry error included in the
result.

try to make the logic more clear and add comments for the reader.

Change-Id: Ib84a89a33907a24426ecf52c90404be0e0dfa307
2020-04-09 13:58:53 +00:00
Moby von Briesen
14b3704f56 storagenode: add suspended status to storagenode dashboard/api
* Add migration to storagenode reputation table to add suspended
timestamp
* Send suspended info to storagenode from satellite nodestats endpoint
* Add suspended status to storagenode api
* Add an indicator on the storagenode dashboard informing operator of
the satellites the node is suspended on

Change-Id: Ie3669f6069cc0258ba76ec99d17006e1b5fd9c8a
2020-04-09 13:36:23 +00:00
Michal Niewrzal
f36e8548f1 satellite/metainfo: adjust max inline segment size validation to
potential encryption overhead.

This is the same approach we have for validating remote segment size.

https://storjlabs.atlassian.net/browse/USR-619

Change-Id: I2597ee734313a3068fd986001680bbedbf1bed2a
2020-04-09 12:34:10 +00:00
Egon Elbre
11a44cdd88 all: don't depend on gogo/proto directly
Change-Id: I8822dea0d1b7b99e0b828e0373a0308a42dde2be
2020-04-08 17:32:15 +00:00
Natalie Villasana
0a1bbc9824 satellite/overlay: add TestEnsureMinimumRequested
Adds a test to make sure that the correct amount of total nodes,
reputable nodes, and new nodes are returned by SelectStorageNodes
in different cases.

Change-Id: I0939159600afde8a46c35735f1edf0576fcdb4cd
2020-04-08 16:06:25 +00:00
Qweder93
3a9422cc9a satellite/nodestats: add pricing model to endpoint
Change-Id: Iddace8e437216a343458f440b543cee61164f233
2020-04-08 14:29:51 +00:00
Egon Elbre
cf26951a5b satellite/satellitedb/pbold: remove dead code
Change-Id: I7464773c20b8f99a601ca9cc4bee804f1ac14cf9
2020-04-08 15:22:31 +03:00
Fadila Khadar
1eb501b5c8 satellite/accounting: test zombie segments not billed
Check zombie segments are not billed

Change-Id: If998e785dcc82c382613318aee3043eaef7fd9ea
2020-04-08 08:28:49 +00:00
Yingrong Zhao
96e58d21b4 cmd;pkg/server: init tracing collector in all processes
Add tracing handler in drpc server.
Initializing tracing collector in admin, satellite api, garbage
collection, satellite core, repaier, storagenode.
Change-Id: Ie98420e35dfc6913836ebd82b517d9d12877aefc

Change-Id: I91057b6265a4ac8bde033dfde692b8a28acca99f
2020-04-07 17:20:59 -04:00
paul cannon
43cf036113
installer/windows: don't require ca.key file (#3847)
As noted by @sixcorners at
https://github.com/storj/storj/pull/3819#issuecomment-609025487, the
Windows installer code incorrectly fails if the ca.key file is not
present.

The ca.key file should not be necessary for regular storage node usage,
and in fact we recommend that users move the ca.key file to secure
storage rather than keeping it on the node.

I'm not sure how to run the tests, so I'll get a nod from the SNO Growth
team before merging.

Change-Id: Ib2fe236de5c75165644e880caa827a2a1a034c87
Co-authored-by: Stefan Benten <mail@stefan-benten.de>
2020-04-07 10:02:41 -05:00
igor gaidaienko
80ee7321cd satellite/accounting: add test billing without expansion factor
Test checking that billing contains the download traffic without the extra pieces

Change-Id: I5c5d6d877116d26a83c98a93da424eecffcb804e
2020-04-07 12:55:43 +00:00
VitaliiShpital
ef6c9d89bb web/satellite: dashboard blur step after adding payment method removed
Change-Id: I926016feca0cf0bf97feef8fd6936fbf466a5a51
2020-04-07 12:12:03 +00:00
Jeff Wendling
2ded64ba2c satellite/compensation: more fixes to get prod running smoothly
Change-Id: I13a76d9d49222fb10796415a015f224d4084fde3
2020-04-07 10:10:27 +00:00
Michal Niewrzal
d23e03f8a2 lib/uplink-gomobile: extend encryption key capabilities
Change-Id: Id10b14cc34ff983ec451440fe4b0305651115022
2020-04-07 06:58:08 +00:00
Egon Elbre
a4c554f2ed satellite/admin: support user query by email
This adds new endpoint /api/user/{user-email} which allows to get the
projects where the user is a member.

It also moves existing endpoint:

    /project/{projectid}/limit -> /api/project/{projectid}/limit

To avoid future conflicts for displaying pages.

Change-Id: I5efe3e1c8f79894c136f92ed815f635a34ba6f98
2020-04-06 18:32:25 +00:00