Commit Graph

5029 Commits

Author SHA1 Message Date
Ethan Adams
6070018021
satellite/overlay: use AS OF SYSTEM TIME with Cockroach
Query nodes table using AS OF SYSTEM TIME '-10s' (by default) when on CRDB to alleviate contention on the nodes table and minimize CRDB retries. Queries for standard uploads are already cached, and node lookups for graceful exit uploads has retry logic so it isn't necessary for the nodes returned to be current.
2020-12-22 21:07:07 +02:00
Jennifer Johnson
7cccbdb766 cmd/uplink/share: add -- to flags referenced in help statements
Change-Id: I86cd08d51c5306effec14f338f37c53c2743d6b2
2020-12-22 17:03:00 +00:00
Bill Thorp
1b0424cad6 uplink/cmd: Export a RegisterAccess method.
Gateway-MT requires integration tests, which would be aided by having an
exported RegisterAccess() method in uplink/cmd.

To support this change, a little of the Uplink cmd logic was shifted around
and a method was made public.  I also normalized finding the access
between accessInspect and accessRegister.

Change-Id: I29369296521c2cc179e27233f5451b95f46109d8
2020-12-22 14:52:40 +00:00
Michal Niewrzal
9a8959d429 Merge 'master' branch
Change-Id: Iba69ea73ca4d3f1cd4ae94243eaaae033c5324e8
2020-12-22 14:55:57 +01:00
Kaloyan Raev
5934969dd6 satellite/orders: remove obsolete CreateDeleteOrderLimits method
Not used anywhere.

Change-Id: I878635d2d533ad4b06ba0d07a94908105546cb82
2020-12-22 13:15:11 +02:00
Kaloyan Raev
777f6e583b satellite/metainfo: replace a call to PointerDB with call to Metabase
Change-Id: I10aa89dacf91cb6c1528698031e6b53c52915bd9
2020-12-22 12:44:28 +02:00
Egon Elbre
c4578eb3ec satellite/gc: add test for pending object
Change-Id: Ifb076ab38442f88f94a3e0c2ae1b19528a55f724
2020-12-22 09:42:32 +00:00
Michal Niewrzal
66d4d5eb48 satellite/metainfo/metabase: implement IterateObjectsAllVersions for
pending/committed objects

Change-Id: Ibf390821b6a23919078de4f18c2653e308320904
2020-12-22 10:27:09 +01:00
Jeff Wendling
876e1be3b5 Blueprint: Sparse Order Storage
Change-Id: I1cf6f3bda84c9b9d5ccfbbaf51813d6ea9a65679
2020-12-21 18:04:54 +00:00
Ethan Adams
563197c628
satellite/overlay: Add index on nodes table (#4012)
satellite/accounting: Add index for project_id on bucket_storage_tallies
2020-12-21 12:48:48 -05:00
Michal Niewrzal
dad8360b39 satellite/metainfo/metabase: rename IterateObjectsAllVersions to
IterateObjectsAllVersionsWithStatus

We need different implementation for IterateObjectsAllVersions because
we want to iterate over all object without specifying object status.
Existing method will have new name but implementation details are not
changed.

Change-Id: I01b987996772fa7f8fd73da9910d52db2d1aa0d7
2020-12-21 16:47:32 +00:00
crawter
c78fdf4727 web/multinode: base router implemented
Change-Id: I831d087a87e05f6055646419b14d15e34d5de6b1
2020-12-21 16:42:36 +00:00
Ethan Adams
9b52283570
satellite/accounting: Add index for project_id on bucket_storage_tallies (#4010)
Change-Id: I47ab2d1e24f94307c3383c497cffe2a150fa8ab7
2020-12-21 11:42:00 -05:00
Kaloyan Raev
bafc6af992 ci: remove workaround for failing tests
Change-Id: I3eb673fae6c81bee17d7437cb870d5f5ba6978d5
2020-12-21 18:07:40 +02:00
Ethan Adams
6e501898c3
satellite/accounting: Performance improvements to getNodeIds used by GetBandwidthSince (#4009) 2020-12-21 16:37:01 +01:00
Stefan Benten
d14f2e4164
Makefile,scripts: move from aarch64 to arm64v8 (#4008)
It turns out that alpine dropped support/updates for the aarch64 image.
Instead they have been using the arm64v8 notation for quite a while,
which resulted in breaking our recent aarch64 builds due to missing
dependencies/updates.

Both arches are exactly the same, aarch64 was created originally by Apple
and arm64 by GNU. The backends have been merged by now and the arm64 became
the de facto standard.
2020-12-21 14:50:57 +01:00
Egon Elbre
529bae3052 satellite/accounting: fix TestBilling_AuditRepairTraffic
Change-Id: I1743b2ab9e1afeb16ca31338a6b067da88e906ee
2020-12-21 13:18:35 +00:00
Michal Niewrzal
f3ef8088e7 satellite/metainfo/metabase: add Verify method for Pieces
This change adds Verify method for pieces to do some basic checks.

Change-Id: I0ff4313b594d2cb3aad7da545f940e10ee654b77
2020-12-21 12:49:03 +00:00
Michal Niewrzal
18825d1e0b satellite/{metainfo,gracefulexit}: fix failing tests
Change-Id: I3428ea601255c36a316732c9f75135d6e5fa4d79
2020-12-21 12:22:32 +00:00
Kaloyan Raev
4d37d14929 satellite/{metrics,repair}: adjust monitoring to new metainfo loop
Change-Id: I87a2145daa5ed49bb2c08d6967baa09c0b14b4c6
2020-12-21 09:05:17 +02:00
Stefan Benten
866ce478bf
build: update node to v14.15.3 (#4007) 2020-12-20 17:20:19 +01:00
Stefan Benten
7eab859030
cmd: ensure proper arch is used for docker container 2020-12-20 09:26:23 +02:00
Jessica Grebenschikov
d961437889 satellite/orders: remove the config IncludeEncryptedMetadata
Since the Satellite now requires the order encryption functionality (since serial_number table is deprecated) to properly function, we can remove the config flag to turn on/off the feature.

Change-Id: Ie973f72a9a05a81cef9e53dc9c99d22c940c2488
2020-12-18 10:39:29 -08:00
Jessica Grebenschikov
da0327c9b7 satellite/dbcleanup: remove expired serial chore
Change-Id: Ib71d41eb6679d6435e5bc10b6244dac66380a74e
2020-12-18 09:36:28 -08:00
Jessica Grebenschikov
97a5e6c814 satellite/orders: stop inserting/reading from serial_numbers table
This PR contains the minimum changes needed to stop inserting into the serial_numbers table. This is the first step in completely deprecating that table.
The next step is to create another PR to remove the expiredSerial chore, fix more tests, and remove any other methods on the serial_number table.

Change-Id: I5f12a56ebf3fa4d1a1976141d2911f25a98d2cc3
2020-12-18 08:35:13 -08:00
NickolaiYurchenko
aaa4a9f31b web/storagenode: overused disk space added to chart
What: diskspace utilization extended with overused value.

Why: to show overused space on chart.

Change-Id: I84adf0d0cc94b19026d97655cb3060aad4560860
2020-12-18 14:58:55 +00:00
Egon Elbre
4e19839669 satellite/accounting: fix TestProjectUsage_ tests
Change-Id: Id3972ca592c4459cc95668a53daca09f818e34b7
2020-12-18 16:41:02 +02:00
VitaliiShpital
f645654d2e web/satellite: access grant CLI step copy token fix
WHAT:
copy button now copies restricted key instead of regular

WHY:
bug fix

Change-Id: I6696dfa4b5d804a64a6d7b49aa443ba16043e466
2020-12-18 16:22:11 +02:00
Egon Elbre
65b22be417 satellite/accounting/tally: use metabase
Change-Id: I6d49dc103a18e8a110bfa7775d53a65d208b6c2c
2020-12-18 16:18:03 +02:00
Michal Niewrzal
f7a31308db satellite/repair: enable TestRemoveExpiredSegmentFromQueue test
Change adds ability to set `now` time during test for repair.

Change-Id: Idb8826b7b58b8789b0abc65817b888ecdc752a3f
2020-12-18 10:58:05 +00:00
Michal Niewrzal
b3aa28cc02 satellite/gracefulexit: migrate to metabase
Change-Id: I8be9cc68894124427e4a30d7631126b3afb1f281
2020-12-18 10:57:39 +00:00
Michal Niewrzal
311b082838 satellite/metainfo: fix metainfo loop
This fix issues with passing observers between iteration methods.

It's not best implementation but I think we will need to optimize it
soon one way or another.

Change-Id: I574599bfd10822d84e2d2f1800bcd88e176a76ea
2020-12-18 10:38:12 +00:00
Kaloyan Raev
ce20db9f68 scripts/testdata: update satellite-config.yaml.lock
Change-Id: I6545a75b1de9834ec35ee172cf5db3daa7243295
2020-12-18 12:00:48 +02:00
Michal Niewrzal
2111740236 Merge 'master' branch
Change-Id: Ib73af0ff3ce0e9a1547b0b9fc55bf88704f6f394
2020-12-18 09:13:24 +01:00
Stefan Benten
e1456bc53f
Makefile: build wasm files only once (#4005) 2020-12-18 01:59:27 +01:00
littleskunk
2437d5b171
satellite/access-grants: default auth service url (#4002)
* satellite/access-grants: default auth service url
2020-12-17 23:38:16 +01:00
paul cannon
d3604a5e90 satellite/repair: use survivability model for segment health
The chief segment health models we've come up with are the "immediate
danger" model and the "survivability" model. The former calculates the
chance of losing a segment becoming lost in the next time period (using
the CDF of the binomial distribution to estimate the chance of x nodes
failing in that period), while the latter estimates the number of
iterations for which a segment can be expected to survive (using the
mean of the negative binomial distribution). The immediate danger model
was a promising one for comparing segment health across segments with
different RS parameters, as it is more precisely what we want to
prevent, but it turns out that practically all segments in production
have infinite health, as the chance of losing segments with any
reasonable estimate of node failure rate is smaller than DBL_EPSILON,
the smallest possible difference from 1.0 representable in a float64
(about 1e-16).

Leaving aside the wisdom of worrying about the repair of segments that
have less than a 1e-16 chance of being lost, we want to be extremely
conservative and proactive in our repair efforts, and the health of the
segments we have been repairing thus far also evaluates to infinity
under the immediate danger model. Thus, we find ourselves reaching for
an alternative.

Dr. Ben saves the day: the survivability model is a reasonably close
approximation of the immediate danger model, and even better, it is
far simpler to calculate and yields manageable values for real-world
segments. The downside to it is that it requires as input an estimate
of the total number of active nodes.

This change replaces the segment health calculation to use the
survivability model, and reinstates the call to SegmentHealth() where it
was reverted. It gets estimates for the total number of active nodes by
leveraging the reliability cache.

Change-Id: Ia5d9b9031b9f6cf0fa7b9005a7011609415527dc
2020-12-17 21:30:17 +00:00
littleskunk
3feee9f4f8
satellite/accounting: default project limits (#4001) 2020-12-17 22:27:05 +01:00
Cameron Ayer
28eaae66af satellite/satellitedb: drop num_healthy_pieces column from injuredsegments
This column is no longer used as it has been replaced by the segment_health
column.

Change-Id: I6b4df89cd4f994d8418976f88e8c5f57615f8115
2020-12-17 20:17:08 +00:00
VitaliiShpital
79a3a47805 build: added brotli compression for wasm bits
WHAT:
added brotli compression for wasm files and added copying of those files to static/wasm folder in Dockerfile

WHY:
those files are a part of web worker webpack bundle and I didn't find a way to compress them separately using webpack.
I'm open to any other ideas if they come up

Change-Id: I105cc1582e9816fd9b63052ba48358525c85a164
2020-12-17 19:23:53 +00:00
VitaliiShpital
f4bbd0f5df web/satellite: use brotli instead of gzip
WHAT:
we'll use brotli instead of gzip from now on

WHY:
better compression

Change-Id: Ibeadd6bfc783e9c15cf3f62f719af692071a7721
2020-12-17 19:23:44 +00:00
VitaliiShpital
50dd9fb11a web/satellite: move access grant web worker initialization to onlogin loading state
WHAT:
web worker is initialized during onlogin loading screen now

WHY:
removed unnecessary initializations and increased UX experience

Change-Id: I734f194f862c15b3fb08e436a161da32d8d4a8ac
2020-12-17 19:23:36 +00:00
Michal Niewrzal
b1712cc93b cmd/storj-sim: update default storj-sim access with real node id
Currently node id in access grant is '1' and it cannot be parsed to
valid node id. This change update access grant satellite address with
randomly generated node id.

Change-Id: Id1684ac71509bc5a8177b069a914355be3c72d43
2020-12-17 18:31:43 +00:00
crawter
9ea147d234 web/multinode: initial app and configs
Change-Id: I96f8ebcedf982139ff0d263266f25dd63746091c
2020-12-17 19:46:56 +02:00
crawter
a3c2711b2f mnd/nodes: db interface methods updated
Change-Id: I78643f5bdefa7e2f2cbeea06a5203627dbfa92ee
2020-12-17 17:05:18 +02:00
Yaroslav Vorobiov
26e65eeefc multinode/console: node list and get api
Change-Id: Icc3cc9e6997b715c865fee9f96c8a848b694f41f
2020-12-17 16:19:57 +02:00
Michal Niewrzal
70ba4deea9 satellite/repair/checker: adjust irreparable part of repair checker
Change-Id: I0732104a97ba18a5359de3966cd692677a0ff790
2020-12-17 14:11:22 +00:00
VitaliiShpital
cd2cdb616a web/satellite: create access grant: restricted key for CLI step, CLI step added to onboarding tour
Change-Id: I2cd0308a61ca724144720eb7f90ba83a1052aee1

WHAT:
CLI step now has restricted key.
CLI step added to onboarding tour

WHY:
bug fixing, extending onboarding flow

Change-Id: I496a23437d602e5dc9d5fdc64bf0e8b97b656e50
2020-12-17 13:11:26 +00:00
Kaloyan Raev
9aa61245d0 satellite/audits: migrate to metabase
Change-Id: I480c941820c5b0bd3af0539d92b548189211acb2
2020-12-17 14:38:48 +02:00
Michal Niewrzal
2381ca2810 Merge 'master' branch
Change-Id: I4a3e45a2a2cdacfd87d16b148cfb4c6671c20b15
2020-12-17 13:17:17 +01:00