Commit Graph

4422 Commits

Author SHA1 Message Date
Moby von Briesen
789b07e226 storagenode/orders/store.go: Do not return error from ListUnsentBySatellite when order files are corrupted.
If we see an UnexpectedEOF error when attempting to read orders, return
the orders we have been able to read successfully and do not return an
error. This behavior ensures that the storagenode orders service
attempts to archive corrupted files and does not retry them repeatedly
and get stuck.

Change-Id: I0d00d1e174f968af6e99ca861eddad190f1339e2
2020-09-10 23:36:05 +00:00
JT Olio
1a094fd6da storj: bump storj.io/private
Change-Id: Ia2927010332bb2e1d140603299341a8549093233
2020-09-10 16:21:12 -06:00
VitaliiShpital
df5a6ebe32 web/satellite: navigation sidebar and dashboard header reworked
WHAT:
Navigation sidebar reworked.
Project dropdown moved back to header, resources and settings sections moved to header as dropdowns.
Header reworked to match new design.

WHY:
GUI redesign

Change-Id: I73e1269326d3dbfba2e44d848dba4a415e64e539
2020-09-10 10:38:00 +00:00
Qweder93
ac29d80495 storagenode: heldamount GetPaystub refactored, estimationPayouts logic separated form console to separate service, storagenodeapi tests fixed.
Change-Id: I902823ef40a62861ce32799e9fb7a67a1e14710d
2020-09-09 15:31:16 +00:00
Jeff Wendling
1fecaed7df satellite/orders: don't version check old endpoint
nodes are submitting using both the legacy and windowed endpoints
and thus having their legacy submissions rejected.

it is legal to use both the legacy and windowed endpoints
in phase1 since they use the same backend. the legacy endpoint
is disabled in phase2 and phase3.

therefore, if we wait an order expiration period (2 days) after
we determine enough nodes have started using the windowed
endpoint, we can be sure that any orders they did have to
submit with the legacy endpoint will have expired.

Change-Id: I4418a881bf8bb9377efaef4c651e6103a5dc6ed0
2020-09-09 10:23:48 -04:00
stefanbenten
2cf12b9409 satellite/admin: use correct period end time to get results
Change-Id: Ib0e76f96e61b39c98f0f5f683b39d622d235c4d8
2020-09-09 09:01:42 +00:00
Michal Niewrzal
27a9d14e2a satellite/repair: use metabase.SegmentKey type in repair package
Another change which is a part of refactoring to replace path parameter
(string/[]byte) with key paramter (metabase.SegmentKey)

Change-Id: I617878442442e5d59bbe5c995f913c3c93c16928
2020-09-08 19:35:20 +00:00
Michal Niewrzal
c753d17e8f satellite/metainfo/objectdeletion: replace
objectdeletion.ObjectIdentifier with metabase.ObjectLocation

Another change to use metabase.ObjectLocation across satellite codebase
to avoid duplication and provide better type safety.

Change-Id: I82cb52b94a9107ed3144255a6ef4ad9f3fc1ca63
2020-09-08 21:09:34 +02:00
VitaliiShpital
c4d6f472fc web/satellite: notification bar for reaching projects count limit
WHAT:
notification bar added to project dashboard page. It is shown when projects count limit is reached.
Create project button is removed after creating last available project

WHY:
inform user that their projects count limit was reached

Change-Id: If0d67148003be40cc9eb4d8b25cc17f8204008d4
2020-09-08 15:48:27 +00:00
Egon Elbre
9fd97fa973 ci: add check-monkit
Change-Id: I124d013ffb434fdea8ec315face090de4e572ebf
2020-09-08 17:20:27 +03:00
JT Olio
249f98f4c1 audit logging: log important user activity
Change-Id: Ic423dab4b6c513fa44a48cb6fe2903c70fe916fb
2020-09-07 15:26:34 +00:00
Matt Robinson
9996aeca8d
build: Go 1.15 (#3927) 2020-09-07 16:01:29 +03:00
Stefan Benten
179b5adad4 storagenode/orders: add missing mon.Task parameter
Change-Id: If98cf347a81f29698a6bdb0907520d60f71db433
2020-09-06 00:05:53 +00:00
Stefan Benten
2111d90684 satellite/admin: add get handler for projects
Change-Id: Ibfb761b1c907b83451570c52eea03ecfa75290ca
2020-09-06 01:40:11 +02:00
Stefan Benten
8b4b44d42b private/web: fix ratelimter IP handling
Change-Id: Idab43f15fb5b90d9d831193d0e7119e64513f271
2020-09-05 18:39:49 +02:00
Egon Elbre
b45cad5eed bump common to latest
This makes tests pass on Go 1.15.

Change-Id: I8407f16dcfc0b306aa685da30dcfedea6d3b515d
2020-09-05 10:51:58 +03:00
Jennifer Johnson
4e2413a99d satellite/satellitedb: uses vetted_at field to select for reputable nodes
Additionally, this PR changes NewNodeFraction devDefault and testplanet config from 0.05 to 1.
This is because many tests relied on selecting nodes that were reputable based on audit and uptime
counts of 0, in effect, selecting new nodes as reputable ones.
However, since reputation is now indicated by a vetted_at db field that is explicitly set
rather than implied by audit and uptime counts, it would be more complicated to try to
update all of the nodes' reputations before selecting nodes for tests.
Now we just allow all test nodes to be new if needed.

Change-Id: Ib9531be77408662315b948fd029cee925ed2ca1d
2020-09-04 16:45:32 +00:00
Yingrong Zhao
cccd62fff9 bump uplink version to v1.3.0
Change-Id: I9e7833713ca00f19bfb662eeaf61ef57bdc3bbbd
2020-09-04 09:57:35 -04:00
Michal Niewrzal
8649a00557 satellite/gracefulexit: replace Path []byte to `Key
metabaseSegmentKey` TransferQueueItem

We are unifying which name (and type) we are using for value we are
using to point to segment. We want to use `key` instead of `path`.
Dedicated type `metabase.SegmentKey` was created for this purposes also.
This change is doing refactoring around gracefulexit.

Change-Id: I90d51ff087b206179e61d5f1bc95f4709d76f917
2020-09-04 11:09:48 +00:00
Michal Niewrzal
9202295348 satellite/metainfo: replace ScopedPath with metabase.SegmentLocation
Change-Id: I7e89c9e8eaeae58be828a32ad47ed3028501f4c7
2020-09-04 10:06:52 +00:00
NickolaiYurchenko
93898146f0 web/storagenode: net total payout displaying fix
Change-Id: I2259e835b6c81d41c4fc64031b09c9edf64be321
2020-09-04 12:28:44 +03:00
Yingrong Zhao
2cc3c19d55 satellite/piecedeletion: add test for disproportionate requests/nodes
ratio

Change-Id: I7902e4ffeca39c5a6a5b9170ce6f010d21e7e19e
2020-09-03 21:00:53 +00:00
Michal Niewrzal
aa47e70f03 satellite/metainfo: use metabase.SegmentKey with metainfo.Service
Instead of using string or []byte we will be using dedicated type
SegmentKey.

Change-Id: I6ca8039f0741f6f9837c69a6d070228ed10f2220
2020-09-03 15:11:32 +00:00
Malcolm Bouzi
3f0c21323d
web/satellite: create new project button added to projects dropdown (#3936)
Change-Id: Ib604f24e0c11fee97d22e71a0c7c83b7e8a1e56a

Co-authored-by: VitaliiShpital <takur12345@gmail.com>
2020-09-03 17:58:29 +03:00
Egon Elbre
77b53bd21c private/lifecycle: log fatal ending to a runner
Change-Id: If07b62dad7f4ac235dd51a3a217c2c56d30978ad
2020-09-03 16:54:40 +03:00
VitaliiShpital
c1e089b226 web/satellite: create project popup replaced with separate page
WHAT:
create project popup removed. Separate page implemented instead.

WHY:
to fulfill multiple project state requirements

change-Id: Iab65981aef1d9b0caceca94b1109fa63f8f22891
2020-09-03 12:56:40 +03:00
Egon Elbre
dc48197bd8 satellite/orders: add bucket id to order limit
Change-Id: I9019ec77d692e62ac17b67a1da71dc3535cde50c
2020-09-03 10:50:11 +03:00
NickolaiYurchenko
9247883c64 web/storagenode: list of audits and suspension scores
WHAT:
list of suspension and audit scores by each satellite

WHY:
a way to show audit and suspension scores of all satellite when no satellite selected

Change-Id: I4b3ac55c2da826e50131a2381c004493d82c5d5b
2020-09-02 17:27:23 +00:00
Yingrong Zhao
af773ec8a6 cmd/uplink: use DeleteBucketWithObjects for bucket force deletion
This PR updates `uplink rb --force` command to use the new libuplink API
`DeleteBucketWithObjects`.
It also updates `DeleteBucket` endpoint to return a specific error
message when a given bucket has concurrent writes while being deleted.

Change-Id: Ic9593d55b0c27b26cd8966dd1bc8cd1e02a6666e
2020-09-02 16:39:20 +00:00
Qweder93
36d752e92d storagenode/reputation: offline_under_review_at added
Change-Id: Ia7ec79b2d6f20fe29de0c36223f9485380d2845c
2020-09-02 18:48:28 +03:00
Qweder93
7d9897b7af storagenode/nodestats: online_score added
Change-Id: I84b50a6cace306e5f10d53a2073fe8810d4d2960
2020-09-02 17:45:01 +03:00
NickolaiYurchenko
5ed0038ace web/satellite: estimation table surge added
Change-Id: Ied7d737174d357076f0735091cc0a18c9e95b464
2020-09-02 11:35:56 +00:00
VitaliiShpital
3fbbe1847e web/satellite: scrolling with no paywall banner fixed
WHAT:
dashboard and billing pages' scrolling fixed with no paywall banner

WHY:
content was slightly clipped

Change-Id: I4d663775456980887af7bc41c72bf3a8da87d301
2020-09-02 09:51:04 +00:00
Yingrong Zhao
7c7334e9d5 satellite/metainfo/piecedeletion: fix deadlock when waiting for threshold to reach
This PR fixes a deadlock that can happen when the number of piece
deletion requests is different from the distinct node count from those
requests. The success threshold should be based on the number of nodes
instead of the amount of requests

Change-Id: I83073a22eb1e111be1e27641cebcefecdc16afcb
2020-09-02 07:07:49 +00:00
Yaroslav Vorobiov
4783c3e1d3 storagenode-updater/linux: restart systemd service
Change-Id: I8ac25ecc41323ec0d5abf8ee65276c1d7a59f74d
2020-09-02 06:22:19 +00:00
Egon Elbre
61b17f1214 satellite/orders: add encryption keys flag to Service
Change-Id: Ie96e75bc96241b799d04654ef5e05b82e6a899bb
2020-09-02 05:02:14 +00:00
JT Olio
1f711523d5 satellite/repair: switch to piecestore.UploadReader part 2
Change-Id: I5a91d2960b037c7a3c96d01bc40404316ba028e3
2020-09-01 12:40:54 -06:00
Natalie Villasana
95ff29cce1 satellite/metainfo: reduce lookupLimit default to 2500
Change-Id: I6569c6d1f145b127a9e8e1a65e4344dd62c989bb
2020-09-01 12:04:48 -04:00
JT Olio
b872fe52a1 satellite/repair: switch to piecestore.UploadReader
Change-Id: Ia99ad2cf5422e6ba1d98b32946740f9cadba7b6d
2020-09-01 09:26:54 -06:00
Michal Niewrzal
084e280a6d satellite/admin: fix time issues in tests
With day -1 test was creating tallies for two different months.

Change-Id: I10201f4e787e73bae40bf7120856fb7cfdcd66f7
2020-09-01 16:03:52 +02:00
Michal Niewrzal
0604a672c1 satellite/metainfo: use metabase in loop
Change-Id: I1bb0c6fe0a762895fde950690b06f7dd9d77e178
2020-09-01 10:06:16 +00:00
Isaac Hess
2d54447088 satellite/metainfo: Fix GetObjectIPs loop and test
This change forces the test of GetObjectIPs to use multiple remote
segments (earlier versions of the test were accidentally using inline
segments). This change also revealed a small bug in the for loop code,
which is fixed.

Change-Id: Ic486b079d221952ba13553acf0ca41a8873f3f21
2020-08-31 15:56:12 -06:00
Cameron Ayer
ca0c1a5f0c storagenode/{monitor,pieces}, storage/filestore: add loop to check storage directory writability
periodically create and delete a temp file in the storage directory
to verify writability. If this check fails, shut the node down.

Change-Id: I433e3a8d1d775fc779ae78e7cf3144a05ffd0574
2020-08-31 21:20:49 +00:00
Moby von Briesen
5d21e85529 satellite/audit/queue: Separate audit queue into two separate structs.
* The audit worker wants to get items from the queue and process them.
* The audit chore wants to create new queues and swap them in when the
old queue has been processed.

This change adds a "Queues" struct which handles the concurrency
issues around the worker fetching a queue and the chore swapping a new
queue in. It simplifies the logic of the "Queue" struct to its bare
bones, so that it behaves like a normal queue with no need to understand
the details of swapping and worker/chore interactions.

Change-Id: Ic3689ede97a528e7590e98338cedddfa51794e1b
2020-08-31 20:51:25 +00:00
Isaac Hess
ba1a113e2e satellite/metainfo: Add test for GetObjectIPs
This PR adds the integration test for GetObjectIPs, testing all repos.

Change-Id: Id4d4f19c427a2e320b2a82efa68150fa7f5f86fb
2020-08-31 10:27:56 -06:00
Isaac Hess
351aa70eb7 satellite/metainfo: Implement GetObjectIPs
Change-Id: Ibabbe7c555b790498d28a6ac4c95fcf2f7376978
2020-08-31 10:27:56 -06:00
Moby von Briesen
2d01dd9732 satellite/satellitedb: Add online_score column to nodes table
Add online score used for the new audit history offline tracking system
to the nodes table. This allows us easy access to the node's online
score for the storagenode dashboard as well as for data analysis.

Change-Id: Ie99be1192e5236862a5b3dbed2e5ef03b9169410
2020-08-31 15:07:07 +00:00
Bill Thorp
328004c0ef satellite/accounting: fix build - time rounding
We were seeing error on the last day of the month with TestProjectAllocatedBandwidthRetainTwo.
This is due to AddDate normalizes its result in the same way that Date does, so, for example,
adding one month to October 31 yields December 1, the normalized form for November 31."

I also fixed a minor UTC issue with this test as well.

Change-Id: I0157873e7befa57810e5f264a922b188890fa46a
2020-08-31 09:37:13 -04:00
nerdatwork
e072febbcc
Fixed typo in log for allocated space (#3934) 2020-08-29 16:36:37 +02:00
Egon Elbre
c86c732fc0 satellite: simplify tests
satellite.DB.Console().Projects().GetAll database query
can be replaced with planet.Uplinks[0].Projects[0].ID

Change-Id: I73b82b91afb2dde7b690917345b798f9d81f6831
2020-08-28 22:28:04 +00:00