Commit Graph

4782 Commits

Author SHA1 Message Date
Moby von Briesen
825dc71227 satellite/{overlay, satellitedb}: Refactor audit history
* Separate audit history interface into its own file in the overlay
package
* Add overlay.AuditHistory struct so that internalpb.AuditHistory is
only used from within the database layer
* Add overlay.GetAuditHistory function for features that will require
access to detailed audit history information
* Do not return full audit history from UpdateAuditHistory - callers to
that function only need to know the online score and whether a full
tracking period has been completed
* Move audit history tests out of satellite/satellitedb, since they are
independent of database implementation

Change-Id: I35b0c4ac23bbaabd80624f8a9631c3cb1a1f33bd
2020-12-29 18:50:22 +00:00
Moby von Briesen
85ae13f11d satellite/satellitedb: Drop nodes_offline_times table.
Now that the deprecated downtime tracking service is removed
(3fc76f4ffe), we can safely remove
the nodes_offline_times table.

Change-Id: Ia7c6efe32ba104dff5a830af5f2beee3337eefe5
2020-12-29 18:17:50 +00:00
Moby von Briesen
e24262c2c9 satellite/satellitedb: Do not consider nodes with offline_suspended as reputable.
Nodes which are offline_suspended will no longer be considered for new
uploads. The current threshold that enters a node into offline
suspension is 0.6. Disqualification for offline suspension is still
disabled.

Change-Id: I0da9abf47167dd5bf6bb21e0bc2186e003e38d1a
2020-12-29 17:59:09 +00:00
Cameron Ayer
ca12e98d5d docs: change 'offline score' to 'online score' in downtime tracking doc
Change doc to reflect a change made during the implementation.

Change-Id: I31f6a2c08ae97cba7068975ab6c9dd276726273a
2020-12-29 15:31:20 +00:00
Stefan Benten
02d7638eda
Jenkinsfile.public: specify main branch for docker build
Docker build defaults to the master branch, when specifying a git repo.
With this change, it should be smart enough to use the proper main branch.
2020-12-29 16:56:36 +02:00
Stefan Benten
7f1871b8f1 all: switch from master to main 2020-12-28 22:59:06 +01:00
Ivan Fraixedes
2223877439 storgenode/piecestore: Log upload size on Upload
Add the upload size to the log lines of the storagenode Upload endpoint
to provides the information to Storage node operators.

Change-Id: Ife661d28be72c2bf02579093e21fa811566ac8dd
2020-12-28 10:28:09 +00:00
NickolaiYurchenko
73fb54fed2 web/multinode/app/store: fixed nodes mutation
Change-Id: Id999b309d73770c44e97caa07d901e821b09d05e
2020-12-24 17:43:01 +00:00
crawter
b0258f085f web/multinode/app: initial state design
Change-Id: If182d1c173e99be1df18daf5cc7083a0d188b3cb
2020-12-24 17:58:43 +02:00
NickolaiYurchenko
6e68e8e00e web/storagenode: held progress typo fix
Change-Id: Ifa77928d27815942990a5ce1b32a572f309d1799
2020-12-24 12:33:56 +02:00
Stefan Benten
ad58459198 satellite/admin: allow more than just "paid" invoice status during user deletion
Currently we do not allow anything other than the "paid" status for invoices when
trying to delete a user. However there can be a couple of other states that are
still fine to accept during deletion of a user. This change reverses the order to
check for the status that we do not want to allow.

Change-Id: I78d85af6438015c55100fa201ccffc731c91de1c
2020-12-23 16:40:44 +01:00
VitaliiShpital
7aea6feee3 web/satellite: access grants delete popup blinking bug fixed
WHAT:
when user deletes access grants, an empty list of being deleted grants appears for a second.
It is fixed now

WHY:
better user experience

Change-Id: Ibda50ad809dfc49f95c4c844487d9527c676fd76
2020-12-23 13:22:05 +00:00
VitaliiShpital
95053531fa web/satellite: access grant: result step back click bug fixed, small styling fixes
WHAT:
back click from result step now sends correct key to passphrase step.
Small styling fixes were also made

WHY:
bug fix

Change-Id: I62463d914496a654a45f01dfb45dd704edbaecb9
2020-12-23 13:21:58 +00:00
JT Olio
f787aa4647 go.mod: bump uplink to v1.4.3
Change-Id: Ia6ff4f3a84897aaebe7aef7680f17b94b59cb9de
2020-12-23 11:18:26 +00:00
JT Olio
7faaeed2bf satellite/access grant wizard: don't hardcode the satellites
Change-Id: Id9fbf68882cdb2fce846b7a2604cf965cc53ab1a
2020-12-22 21:24:45 -07:00
JT Olio
efde103dba accounting: rollup test is broken for the hour before midnight UTC
this change isn't the real fix. it's just ignoring the problem.

i don't know what the real fix is. is the problem with the test, or
is there actually a problem with the rollup code?

Change-Id: I552bdd947deadc212cc56efc5f818942b9827126
2020-12-22 14:14:52 -07:00
Ethan Adams
6070018021
satellite/overlay: use AS OF SYSTEM TIME with Cockroach
Query nodes table using AS OF SYSTEM TIME '-10s' (by default) when on CRDB to alleviate contention on the nodes table and minimize CRDB retries. Queries for standard uploads are already cached, and node lookups for graceful exit uploads has retry logic so it isn't necessary for the nodes returned to be current.
2020-12-22 21:07:07 +02:00
Jennifer Johnson
7cccbdb766 cmd/uplink/share: add -- to flags referenced in help statements
Change-Id: I86cd08d51c5306effec14f338f37c53c2743d6b2
2020-12-22 17:03:00 +00:00
Bill Thorp
1b0424cad6 uplink/cmd: Export a RegisterAccess method.
Gateway-MT requires integration tests, which would be aided by having an
exported RegisterAccess() method in uplink/cmd.

To support this change, a little of the Uplink cmd logic was shifted around
and a method was made public.  I also normalized finding the access
between accessInspect and accessRegister.

Change-Id: I29369296521c2cc179e27233f5451b95f46109d8
2020-12-22 14:52:40 +00:00
Jeff Wendling
876e1be3b5 Blueprint: Sparse Order Storage
Change-Id: I1cf6f3bda84c9b9d5ccfbbaf51813d6ea9a65679
2020-12-21 18:04:54 +00:00
Ethan Adams
563197c628
satellite/overlay: Add index on nodes table (#4012)
satellite/accounting: Add index for project_id on bucket_storage_tallies
2020-12-21 12:48:48 -05:00
crawter
c78fdf4727 web/multinode: base router implemented
Change-Id: I831d087a87e05f6055646419b14d15e34d5de6b1
2020-12-21 16:42:36 +00:00
Ethan Adams
9b52283570
satellite/accounting: Add index for project_id on bucket_storage_tallies (#4010)
Change-Id: I47ab2d1e24f94307c3383c497cffe2a150fa8ab7
2020-12-21 11:42:00 -05:00
Ethan Adams
6e501898c3
satellite/accounting: Performance improvements to getNodeIds used by GetBandwidthSince (#4009) 2020-12-21 16:37:01 +01:00
Stefan Benten
d14f2e4164
Makefile,scripts: move from aarch64 to arm64v8 (#4008)
It turns out that alpine dropped support/updates for the aarch64 image.
Instead they have been using the arm64v8 notation for quite a while,
which resulted in breaking our recent aarch64 builds due to missing
dependencies/updates.

Both arches are exactly the same, aarch64 was created originally by Apple
and arm64 by GNU. The backends have been merged by now and the arm64 became
the de facto standard.
2020-12-21 14:50:57 +01:00
Stefan Benten
866ce478bf
build: update node to v14.15.3 (#4007) 2020-12-20 17:20:19 +01:00
Stefan Benten
7eab859030
cmd: ensure proper arch is used for docker container 2020-12-20 09:26:23 +02:00
Jessica Grebenschikov
d961437889 satellite/orders: remove the config IncludeEncryptedMetadata
Since the Satellite now requires the order encryption functionality (since serial_number table is deprecated) to properly function, we can remove the config flag to turn on/off the feature.

Change-Id: Ie973f72a9a05a81cef9e53dc9c99d22c940c2488
2020-12-18 10:39:29 -08:00
Jessica Grebenschikov
da0327c9b7 satellite/dbcleanup: remove expired serial chore
Change-Id: Ib71d41eb6679d6435e5bc10b6244dac66380a74e
2020-12-18 09:36:28 -08:00
Jessica Grebenschikov
97a5e6c814 satellite/orders: stop inserting/reading from serial_numbers table
This PR contains the minimum changes needed to stop inserting into the serial_numbers table. This is the first step in completely deprecating that table.
The next step is to create another PR to remove the expiredSerial chore, fix more tests, and remove any other methods on the serial_number table.

Change-Id: I5f12a56ebf3fa4d1a1976141d2911f25a98d2cc3
2020-12-18 08:35:13 -08:00
NickolaiYurchenko
aaa4a9f31b web/storagenode: overused disk space added to chart
What: diskspace utilization extended with overused value.

Why: to show overused space on chart.

Change-Id: I84adf0d0cc94b19026d97655cb3060aad4560860
2020-12-18 14:58:55 +00:00
VitaliiShpital
f645654d2e web/satellite: access grant CLI step copy token fix
WHAT:
copy button now copies restricted key instead of regular

WHY:
bug fix

Change-Id: I6696dfa4b5d804a64a6d7b49aa443ba16043e466
2020-12-18 16:22:11 +02:00
Stefan Benten
e1456bc53f
Makefile: build wasm files only once (#4005) 2020-12-18 01:59:27 +01:00
littleskunk
2437d5b171
satellite/access-grants: default auth service url (#4002)
* satellite/access-grants: default auth service url
2020-12-17 23:38:16 +01:00
paul cannon
d3604a5e90 satellite/repair: use survivability model for segment health
The chief segment health models we've come up with are the "immediate
danger" model and the "survivability" model. The former calculates the
chance of losing a segment becoming lost in the next time period (using
the CDF of the binomial distribution to estimate the chance of x nodes
failing in that period), while the latter estimates the number of
iterations for which a segment can be expected to survive (using the
mean of the negative binomial distribution). The immediate danger model
was a promising one for comparing segment health across segments with
different RS parameters, as it is more precisely what we want to
prevent, but it turns out that practically all segments in production
have infinite health, as the chance of losing segments with any
reasonable estimate of node failure rate is smaller than DBL_EPSILON,
the smallest possible difference from 1.0 representable in a float64
(about 1e-16).

Leaving aside the wisdom of worrying about the repair of segments that
have less than a 1e-16 chance of being lost, we want to be extremely
conservative and proactive in our repair efforts, and the health of the
segments we have been repairing thus far also evaluates to infinity
under the immediate danger model. Thus, we find ourselves reaching for
an alternative.

Dr. Ben saves the day: the survivability model is a reasonably close
approximation of the immediate danger model, and even better, it is
far simpler to calculate and yields manageable values for real-world
segments. The downside to it is that it requires as input an estimate
of the total number of active nodes.

This change replaces the segment health calculation to use the
survivability model, and reinstates the call to SegmentHealth() where it
was reverted. It gets estimates for the total number of active nodes by
leveraging the reliability cache.

Change-Id: Ia5d9b9031b9f6cf0fa7b9005a7011609415527dc
2020-12-17 21:30:17 +00:00
littleskunk
3feee9f4f8
satellite/accounting: default project limits (#4001) 2020-12-17 22:27:05 +01:00
Cameron Ayer
28eaae66af satellite/satellitedb: drop num_healthy_pieces column from injuredsegments
This column is no longer used as it has been replaced by the segment_health
column.

Change-Id: I6b4df89cd4f994d8418976f88e8c5f57615f8115
2020-12-17 20:17:08 +00:00
VitaliiShpital
79a3a47805 build: added brotli compression for wasm bits
WHAT:
added brotli compression for wasm files and added copying of those files to static/wasm folder in Dockerfile

WHY:
those files are a part of web worker webpack bundle and I didn't find a way to compress them separately using webpack.
I'm open to any other ideas if they come up

Change-Id: I105cc1582e9816fd9b63052ba48358525c85a164
2020-12-17 19:23:53 +00:00
VitaliiShpital
f4bbd0f5df web/satellite: use brotli instead of gzip
WHAT:
we'll use brotli instead of gzip from now on

WHY:
better compression

Change-Id: Ibeadd6bfc783e9c15cf3f62f719af692071a7721
2020-12-17 19:23:44 +00:00
VitaliiShpital
50dd9fb11a web/satellite: move access grant web worker initialization to onlogin loading state
WHAT:
web worker is initialized during onlogin loading screen now

WHY:
removed unnecessary initializations and increased UX experience

Change-Id: I734f194f862c15b3fb08e436a161da32d8d4a8ac
2020-12-17 19:23:36 +00:00
Michal Niewrzal
b1712cc93b cmd/storj-sim: update default storj-sim access with real node id
Currently node id in access grant is '1' and it cannot be parsed to
valid node id. This change update access grant satellite address with
randomly generated node id.

Change-Id: Id1684ac71509bc5a8177b069a914355be3c72d43
2020-12-17 18:31:43 +00:00
crawter
9ea147d234 web/multinode: initial app and configs
Change-Id: I96f8ebcedf982139ff0d263266f25dd63746091c
2020-12-17 19:46:56 +02:00
crawter
a3c2711b2f mnd/nodes: db interface methods updated
Change-Id: I78643f5bdefa7e2f2cbeea06a5203627dbfa92ee
2020-12-17 17:05:18 +02:00
Yaroslav Vorobiov
26e65eeefc multinode/console: node list and get api
Change-Id: Icc3cc9e6997b715c865fee9f96c8a848b694f41f
2020-12-17 16:19:57 +02:00
VitaliiShpital
cd2cdb616a web/satellite: create access grant: restricted key for CLI step, CLI step added to onboarding tour
Change-Id: I2cd0308a61ca724144720eb7f90ba83a1052aee1

WHAT:
CLI step now has restricted key.
CLI step added to onboarding tour

WHY:
bug fixing, extending onboarding flow

Change-Id: I496a23437d602e5dc9d5fdc64bf0e8b97b656e50
2020-12-17 13:11:26 +00:00
Qweder93
2fd7809e54 storagenode/payout: stefanbenten satellite name added to payout history, satellites with no held history removed from list
Change-Id: I96861058ccb9c8ce52698796c91b999eaec1f6e6
2020-12-17 11:01:28 +00:00
Jeff Wendling
0e83233700 storj-sim: add node id to default access
Change-Id: I59874fe8d73a832d04a5597c98d05971a74d2164
2020-12-17 09:38:23 +00:00
Egon Elbre
12055e7864 all: minor cleanups
Change-Id: I4248dbe36a62a223b06135254b32851485a2eec1
2020-12-16 10:47:46 +00:00
Cameron Ayer
8c52bb3a18 satellite/checker: use numHealthy as segment health in repair queue
A few weeks ago it was discovered that the segment health function
was not working as expected with production values. As a bandaid,
we decided to insert the number of healthy pieces into the segment
health column. This should have effectively reverted our means of
prioritizing repair to the previous implementation.

However, it turns out that the bandaid was placed into the code which
removes items from the irreparable db and inserts them into the repair
queue.

This change: insert number of healthy pieces into the repair queue in the
method, RemoteSegment

Change-Id: Iabfc7984df0a928066b69e9aecb6f615253f1ad2
2020-12-15 17:16:59 -05:00
Ivan Fraixedes
187680b0c1
scripts: Fix typo in a comment
Change-Id: If79e778e786db06d2263bcd5393f639a4bb92542
2020-12-15 18:58:54 +01:00