Commit Graph

5042 Commits

Author SHA1 Message Date
Cameron Ayer
ca12e98d5d docs: change 'offline score' to 'online score' in downtime tracking doc
Change doc to reflect a change made during the implementation.

Change-Id: I31f6a2c08ae97cba7068975ab6c9dd276726273a
2020-12-29 15:31:20 +00:00
Stefan Benten
02d7638eda
Jenkinsfile.public: specify main branch for docker build
Docker build defaults to the master branch, when specifying a git repo.
With this change, it should be smart enough to use the proper main branch.
2020-12-29 16:56:36 +02:00
Stefan Benten
7f1871b8f1 all: switch from master to main 2020-12-28 22:59:06 +01:00
Ivan Fraixedes
2223877439 storgenode/piecestore: Log upload size on Upload
Add the upload size to the log lines of the storagenode Upload endpoint
to provides the information to Storage node operators.

Change-Id: Ife661d28be72c2bf02579093e21fa811566ac8dd
2020-12-28 10:28:09 +00:00
NickolaiYurchenko
73fb54fed2 web/multinode/app/store: fixed nodes mutation
Change-Id: Id999b309d73770c44e97caa07d901e821b09d05e
2020-12-24 17:43:01 +00:00
crawter
b0258f085f web/multinode/app: initial state design
Change-Id: If182d1c173e99be1df18daf5cc7083a0d188b3cb
2020-12-24 17:58:43 +02:00
NickolaiYurchenko
6e68e8e00e web/storagenode: held progress typo fix
Change-Id: Ifa77928d27815942990a5ce1b32a572f309d1799
2020-12-24 12:33:56 +02:00
Stefan Benten
ad58459198 satellite/admin: allow more than just "paid" invoice status during user deletion
Currently we do not allow anything other than the "paid" status for invoices when
trying to delete a user. However there can be a couple of other states that are
still fine to accept during deletion of a user. This change reverses the order to
check for the status that we do not want to allow.

Change-Id: I78d85af6438015c55100fa201ccffc731c91de1c
2020-12-23 16:40:44 +01:00
VitaliiShpital
7aea6feee3 web/satellite: access grants delete popup blinking bug fixed
WHAT:
when user deletes access grants, an empty list of being deleted grants appears for a second.
It is fixed now

WHY:
better user experience

Change-Id: Ibda50ad809dfc49f95c4c844487d9527c676fd76
2020-12-23 13:22:05 +00:00
VitaliiShpital
95053531fa web/satellite: access grant: result step back click bug fixed, small styling fixes
WHAT:
back click from result step now sends correct key to passphrase step.
Small styling fixes were also made

WHY:
bug fix

Change-Id: I62463d914496a654a45f01dfb45dd704edbaecb9
2020-12-23 13:21:58 +00:00
JT Olio
f787aa4647 go.mod: bump uplink to v1.4.3
Change-Id: Ia6ff4f3a84897aaebe7aef7680f17b94b59cb9de
2020-12-23 11:18:26 +00:00
JT Olio
7faaeed2bf satellite/access grant wizard: don't hardcode the satellites
Change-Id: Id9fbf68882cdb2fce846b7a2604cf965cc53ab1a
2020-12-22 21:24:45 -07:00
JT Olio
efde103dba accounting: rollup test is broken for the hour before midnight UTC
this change isn't the real fix. it's just ignoring the problem.

i don't know what the real fix is. is the problem with the test, or
is there actually a problem with the rollup code?

Change-Id: I552bdd947deadc212cc56efc5f818942b9827126
2020-12-22 14:14:52 -07:00
Ethan Adams
6070018021
satellite/overlay: use AS OF SYSTEM TIME with Cockroach
Query nodes table using AS OF SYSTEM TIME '-10s' (by default) when on CRDB to alleviate contention on the nodes table and minimize CRDB retries. Queries for standard uploads are already cached, and node lookups for graceful exit uploads has retry logic so it isn't necessary for the nodes returned to be current.
2020-12-22 21:07:07 +02:00
Jennifer Johnson
7cccbdb766 cmd/uplink/share: add -- to flags referenced in help statements
Change-Id: I86cd08d51c5306effec14f338f37c53c2743d6b2
2020-12-22 17:03:00 +00:00
Bill Thorp
1b0424cad6 uplink/cmd: Export a RegisterAccess method.
Gateway-MT requires integration tests, which would be aided by having an
exported RegisterAccess() method in uplink/cmd.

To support this change, a little of the Uplink cmd logic was shifted around
and a method was made public.  I also normalized finding the access
between accessInspect and accessRegister.

Change-Id: I29369296521c2cc179e27233f5451b95f46109d8
2020-12-22 14:52:40 +00:00
Michal Niewrzal
9a8959d429 Merge 'master' branch
Change-Id: Iba69ea73ca4d3f1cd4ae94243eaaae033c5324e8
2020-12-22 14:55:57 +01:00
Kaloyan Raev
5934969dd6 satellite/orders: remove obsolete CreateDeleteOrderLimits method
Not used anywhere.

Change-Id: I878635d2d533ad4b06ba0d07a94908105546cb82
2020-12-22 13:15:11 +02:00
Kaloyan Raev
777f6e583b satellite/metainfo: replace a call to PointerDB with call to Metabase
Change-Id: I10aa89dacf91cb6c1528698031e6b53c52915bd9
2020-12-22 12:44:28 +02:00
Egon Elbre
c4578eb3ec satellite/gc: add test for pending object
Change-Id: Ifb076ab38442f88f94a3e0c2ae1b19528a55f724
2020-12-22 09:42:32 +00:00
Michal Niewrzal
66d4d5eb48 satellite/metainfo/metabase: implement IterateObjectsAllVersions for
pending/committed objects

Change-Id: Ibf390821b6a23919078de4f18c2653e308320904
2020-12-22 10:27:09 +01:00
Jeff Wendling
876e1be3b5 Blueprint: Sparse Order Storage
Change-Id: I1cf6f3bda84c9b9d5ccfbbaf51813d6ea9a65679
2020-12-21 18:04:54 +00:00
Ethan Adams
563197c628
satellite/overlay: Add index on nodes table (#4012)
satellite/accounting: Add index for project_id on bucket_storage_tallies
2020-12-21 12:48:48 -05:00
Michal Niewrzal
dad8360b39 satellite/metainfo/metabase: rename IterateObjectsAllVersions to
IterateObjectsAllVersionsWithStatus

We need different implementation for IterateObjectsAllVersions because
we want to iterate over all object without specifying object status.
Existing method will have new name but implementation details are not
changed.

Change-Id: I01b987996772fa7f8fd73da9910d52db2d1aa0d7
2020-12-21 16:47:32 +00:00
crawter
c78fdf4727 web/multinode: base router implemented
Change-Id: I831d087a87e05f6055646419b14d15e34d5de6b1
2020-12-21 16:42:36 +00:00
Ethan Adams
9b52283570
satellite/accounting: Add index for project_id on bucket_storage_tallies (#4010)
Change-Id: I47ab2d1e24f94307c3383c497cffe2a150fa8ab7
2020-12-21 11:42:00 -05:00
Kaloyan Raev
bafc6af992 ci: remove workaround for failing tests
Change-Id: I3eb673fae6c81bee17d7437cb870d5f5ba6978d5
2020-12-21 18:07:40 +02:00
Ethan Adams
6e501898c3
satellite/accounting: Performance improvements to getNodeIds used by GetBandwidthSince (#4009) 2020-12-21 16:37:01 +01:00
Stefan Benten
d14f2e4164
Makefile,scripts: move from aarch64 to arm64v8 (#4008)
It turns out that alpine dropped support/updates for the aarch64 image.
Instead they have been using the arm64v8 notation for quite a while,
which resulted in breaking our recent aarch64 builds due to missing
dependencies/updates.

Both arches are exactly the same, aarch64 was created originally by Apple
and arm64 by GNU. The backends have been merged by now and the arm64 became
the de facto standard.
2020-12-21 14:50:57 +01:00
Egon Elbre
529bae3052 satellite/accounting: fix TestBilling_AuditRepairTraffic
Change-Id: I1743b2ab9e1afeb16ca31338a6b067da88e906ee
2020-12-21 13:18:35 +00:00
Michal Niewrzal
f3ef8088e7 satellite/metainfo/metabase: add Verify method for Pieces
This change adds Verify method for pieces to do some basic checks.

Change-Id: I0ff4313b594d2cb3aad7da545f940e10ee654b77
2020-12-21 12:49:03 +00:00
Michal Niewrzal
18825d1e0b satellite/{metainfo,gracefulexit}: fix failing tests
Change-Id: I3428ea601255c36a316732c9f75135d6e5fa4d79
2020-12-21 12:22:32 +00:00
Kaloyan Raev
4d37d14929 satellite/{metrics,repair}: adjust monitoring to new metainfo loop
Change-Id: I87a2145daa5ed49bb2c08d6967baa09c0b14b4c6
2020-12-21 09:05:17 +02:00
Stefan Benten
866ce478bf
build: update node to v14.15.3 (#4007) 2020-12-20 17:20:19 +01:00
Stefan Benten
7eab859030
cmd: ensure proper arch is used for docker container 2020-12-20 09:26:23 +02:00
Jessica Grebenschikov
d961437889 satellite/orders: remove the config IncludeEncryptedMetadata
Since the Satellite now requires the order encryption functionality (since serial_number table is deprecated) to properly function, we can remove the config flag to turn on/off the feature.

Change-Id: Ie973f72a9a05a81cef9e53dc9c99d22c940c2488
2020-12-18 10:39:29 -08:00
Jessica Grebenschikov
da0327c9b7 satellite/dbcleanup: remove expired serial chore
Change-Id: Ib71d41eb6679d6435e5bc10b6244dac66380a74e
2020-12-18 09:36:28 -08:00
Jessica Grebenschikov
97a5e6c814 satellite/orders: stop inserting/reading from serial_numbers table
This PR contains the minimum changes needed to stop inserting into the serial_numbers table. This is the first step in completely deprecating that table.
The next step is to create another PR to remove the expiredSerial chore, fix more tests, and remove any other methods on the serial_number table.

Change-Id: I5f12a56ebf3fa4d1a1976141d2911f25a98d2cc3
2020-12-18 08:35:13 -08:00
NickolaiYurchenko
aaa4a9f31b web/storagenode: overused disk space added to chart
What: diskspace utilization extended with overused value.

Why: to show overused space on chart.

Change-Id: I84adf0d0cc94b19026d97655cb3060aad4560860
2020-12-18 14:58:55 +00:00
Egon Elbre
4e19839669 satellite/accounting: fix TestProjectUsage_ tests
Change-Id: Id3972ca592c4459cc95668a53daca09f818e34b7
2020-12-18 16:41:02 +02:00
VitaliiShpital
f645654d2e web/satellite: access grant CLI step copy token fix
WHAT:
copy button now copies restricted key instead of regular

WHY:
bug fix

Change-Id: I6696dfa4b5d804a64a6d7b49aa443ba16043e466
2020-12-18 16:22:11 +02:00
Egon Elbre
65b22be417 satellite/accounting/tally: use metabase
Change-Id: I6d49dc103a18e8a110bfa7775d53a65d208b6c2c
2020-12-18 16:18:03 +02:00
Michal Niewrzal
f7a31308db satellite/repair: enable TestRemoveExpiredSegmentFromQueue test
Change adds ability to set `now` time during test for repair.

Change-Id: Idb8826b7b58b8789b0abc65817b888ecdc752a3f
2020-12-18 10:58:05 +00:00
Michal Niewrzal
b3aa28cc02 satellite/gracefulexit: migrate to metabase
Change-Id: I8be9cc68894124427e4a30d7631126b3afb1f281
2020-12-18 10:57:39 +00:00
Michal Niewrzal
311b082838 satellite/metainfo: fix metainfo loop
This fix issues with passing observers between iteration methods.

It's not best implementation but I think we will need to optimize it
soon one way or another.

Change-Id: I574599bfd10822d84e2d2f1800bcd88e176a76ea
2020-12-18 10:38:12 +00:00
Kaloyan Raev
ce20db9f68 scripts/testdata: update satellite-config.yaml.lock
Change-Id: I6545a75b1de9834ec35ee172cf5db3daa7243295
2020-12-18 12:00:48 +02:00
Michal Niewrzal
2111740236 Merge 'master' branch
Change-Id: Ib73af0ff3ce0e9a1547b0b9fc55bf88704f6f394
2020-12-18 09:13:24 +01:00
Stefan Benten
e1456bc53f
Makefile: build wasm files only once (#4005) 2020-12-18 01:59:27 +01:00
littleskunk
2437d5b171
satellite/access-grants: default auth service url (#4002)
* satellite/access-grants: default auth service url
2020-12-17 23:38:16 +01:00
paul cannon
d3604a5e90 satellite/repair: use survivability model for segment health
The chief segment health models we've come up with are the "immediate
danger" model and the "survivability" model. The former calculates the
chance of losing a segment becoming lost in the next time period (using
the CDF of the binomial distribution to estimate the chance of x nodes
failing in that period), while the latter estimates the number of
iterations for which a segment can be expected to survive (using the
mean of the negative binomial distribution). The immediate danger model
was a promising one for comparing segment health across segments with
different RS parameters, as it is more precisely what we want to
prevent, but it turns out that practically all segments in production
have infinite health, as the chance of losing segments with any
reasonable estimate of node failure rate is smaller than DBL_EPSILON,
the smallest possible difference from 1.0 representable in a float64
(about 1e-16).

Leaving aside the wisdom of worrying about the repair of segments that
have less than a 1e-16 chance of being lost, we want to be extremely
conservative and proactive in our repair efforts, and the health of the
segments we have been repairing thus far also evaluates to infinity
under the immediate danger model. Thus, we find ourselves reaching for
an alternative.

Dr. Ben saves the day: the survivability model is a reasonably close
approximation of the immediate danger model, and even better, it is
far simpler to calculate and yields manageable values for real-world
segments. The downside to it is that it requires as input an estimate
of the total number of active nodes.

This change replaces the segment health calculation to use the
survivability model, and reinstates the call to SegmentHealth() where it
was reverted. It gets estimates for the total number of active nodes by
leveraging the reliability cache.

Change-Id: Ia5d9b9031b9f6cf0fa7b9005a7011609415527dc
2020-12-17 21:30:17 +00:00