Commit Graph

5014 Commits

Author SHA1 Message Date
Michal Niewrzal
57f374af24 Merge 'master' branch
Change-Id: Idf6b10ea7ca94e4d232e6a3b6a38ef2e646ba197
2020-12-15 08:26:53 +01:00
Jennifer Johnson
adb2c83e09 cmd/uplink: adds register, url, and dns flags to uplink share
and replaces access grant with access

uplink share <path> --> creates access grant

uplink share --register <path> --> registers access grant

uplink share --url <path> --> creates URL, implies register and public

uplink share --dns <hostname> <path> --> creates dns info, implies register and public

Change-Id: I7930c4973a602d3d721ec6f77170f90957dad8c0
2020-12-14 20:51:44 -05:00
Stefan Benten
9fe477899b satellite/satellitedb: add lint ignore rule to support staticcheck 2020.2
staticcheck 2020.2 is not liking our dbx files, so we need to ignore them.

Change-Id: I6becc3619bb088473f9776d0878ce240d4935936
2020-12-14 21:16:31 +00:00
Qweder93
12144a600b storagenode/console: payout tests and heldhistory joined_at rounding added
Change-Id: I1d43620fbafbf7ed92588b84cb9c6b8ced8832ef
2020-12-14 19:35:04 +02:00
Qweder93
c630037f34 multinodepb: diskspace, reputation and status added
Change-Id: I470fa8b59ce7f00f2fbedbd0c0878fb5fff0590c
2020-12-14 17:01:16 +00:00
Jessica Grebenschikov
3cc98de3ee satellite/console/wasm: reduce size to <9MB
Make changes so that we only import the necessary files from the console package so that the generated wasm code is as small as possible.

This change gets the compiled wasm code down to 8.6MB uncompressed and 2MB when compressed with `gzip --best`.

https://review.dev.storj.io/c/storj/storj/+/3396

Change-Id: Ifdd4be285810757b46bbbe43327c0d0139e5f8f7
2020-12-14 16:41:39 +00:00
JT Olio
d955946f15 satellite/compensation: don't abort entirely if a node isn't found
Change-Id: I1066fb6a281eece892ad179a24b01b2ff6615fe7
2020-12-14 15:56:59 +00:00
Ivan Fraixedes
2dddcffe43 satellite/accounting/rollout: Remove unused variable
Remove a declared variable that's set by never read nor passed to any
function so it's unused code.

Change-Id: I8daf9d1f71d29ab39d7a80011d1b4813ada1c67d
2020-12-14 14:11:41 +00:00
crawter
4a11ec2826 multinode/nodes: package created, api tests added, small restructuring
Change-Id: I9f8146760a2676a204eb1bd3410079c5fa017d70
2020-12-14 14:16:45 +02:00
Kaloyan Raev
fc85179a19 satellite/metainfo: refactor SegmentLocation.Index to SegmentPosition
Change-Id: Ic9403c8126712693326dd83d6ba4f3b84be3e0c7
2020-12-14 13:35:53 +02:00
Kaloyan Raev
7d8f19e94d satellite/metainfo: metainfo loop should yield StreamID for segments
Change-Id: If6c86add75ce79ffcfe95353225719c7d4b5a459
2020-12-14 13:04:38 +02:00
Michal Niewrzal
e7e6985ae9 satellite/metainfo/metabase: add UpdateSegmentPieces method
We need to be able to update just remote_pieces column in DB. This is
needed at least for repair process.

Change-Id: I20dcc9b06babfefbbf102f32b1d14946379f26c2
2020-12-14 10:25:09 +00:00
Kaloyan Raev
2bb010e7c5 cmd: remove segment reaper
It was designed to detect and remove zombie segments in the PointerDB.
This tool should be not relevant with the MetabaseDB anymore.

Change-Id: I112552203b1329a5a659f69a0043eb1f8dadb551
2020-12-14 09:36:37 +00:00
Michal Niewrzal
7e6e0d3e2e satellite/metainfo: metainfo loop implementation with metabase
Change-Id: Iadac469519de605a88e624df23265289771b2006
2020-12-11 16:15:57 +01:00
Brandon Iglesias
ca1e6b9756
Adding Fastly (#3994) 2020-12-11 15:53:05 +02:00
Stefan Benten
d559405ae0
Jenkinsfile: update to go v1.15.6 (#3993) 2020-12-11 11:32:16 +01:00
Stefan Benten
8fe829d5fd
build: add wasm bits to Dockerfile and bump to go v1.15.6 (#3992) 2020-12-11 02:23:39 +01:00
Aaron Covrig
46e26fa47d
Changed ChecksArea.vue audits/checks precision (#3987)
Fixes #3965
2020-12-10 20:03:11 +01:00
Stefan Benten
982c729132
.clabot: adding Doom4535 to approved list (#3989) 2020-12-10 19:42:22 +01:00
Jessica Grebenschikov
0649d2b930 satellite/repair: improve contention for injuredsegments table on CRDB
We migrated satelliteDB off of Postgres and over to CockroachDB (crdb), but there was way too high contention for the injuredsegments table so we had to rollback to Postgres for the repair queue. A couple things contributed to this problem:
1) crdb doesn't support `FOR UPDATE SKIP LOCKED`
2) the original crdb Select query was doing 2 full table scans and not using any indexes
3) the SLC Satellite (where we were doing the migration) was running 48 repair worker processes, each of which run up to 5 goroutines which all are trying to select out of the repair queue and this was causing a ton of contention.

The changes in this PR should help to reduce that contention and improve performance on CRDB.
The changes include:
1) Use an update/set query instead of select/update to capitalize on the new `UPDATE` implicit row locking ability in CRDB.
- Details: As of CRDB v20.2.2, there is implicit row locking with update/set queries (contention reduction and performance gains are described in this blog post: https://www.cockroachlabs.com/blog/when-and-why-to-use-select-for-update-in-cockroachdb/).

2) Remove the `ORDER BY` clause since this was causing a full table scan and also prevented the use of the row locking capability.
- While long term it is very important to `ORDER BY segment_health`, the change here is only suppose to be a temporary bandaid to get us migrated over to CRDB quickly. Since segment_health has been set to infinity for some time now (re: https://review.dev.storj.io/c/storj/storj/+/3224), it seems like it might be ok to continue not making use of this for the short term. However, long term this needs to be fixed with a redesign of the repair workers, possible in the trusted delegated repair design (https://review.dev.storj.io/c/storj/storj/+/2602) or something similar to what is recommended here on how to implement a queue on CRDB https://dev.to/ajwerner/quick-and-easy-exactly-once-distributed-work-queues-using-serializable-transactions-jdp, or migrate to rabbit MQ priority queue or something similar..

This PRs improved query uses the index to avoid full scans and also locks the row its going to update and CRDB retries for us if there are any lock errors.

Change-Id: Id29faad2186627872fbeb0f31536c4f55f860f23
2020-12-10 09:51:26 -08:00
Michal Niewrzal
b3acc1101a Merge 'master' branch
Change-Id: Iee99400c7095770e61cde94b3b2c8eb0ddec463d
2020-12-10 15:42:52 +01:00
Michal Niewrzal
c2a97aeb14 satellite/satellitedb: add ListAllBuckets method
We need to be able to list all buckets in DB without knowing project ID.
This method will be used to list buckets for metainfo loop
implementation based on metabase.

Change-Id: Iac75af0eee4f31e80a15577575a8249cbca787b2
2020-12-10 14:19:27 +00:00
Egon Elbre
588dff6d78 Upgrade to uplink v1.4.2
Change-Id: Ib1f5e70c76710d9c49a39347e4ed95bd4671efab
2020-12-10 15:47:11 +02:00
Fadila Khadar
15add3c752 satellite/metainfo: enable deletion of a pending object.
Change-Id: I1ac24ad340d411c52b92ef18f51a684d9214ed47
2020-12-10 13:56:14 +01:00
Tessa Nordgren
e322eac054
added myself to the clabot list. (#3988) 2020-12-09 19:34:22 +01:00
Egon Elbre
365410d10b satellite/metainfo/metabase: add commit with specifying a list of segments
Change-Id: Ibb9999545691d150f36e20ce70ac00a3802ad7b2
2020-12-09 15:16:38 +02:00
Michal Niewrzal
f077564bb7 satellite/metainfo: setup access to metabase in metainfo loop
This is initial change to replace metainfo loop implementation with
metabase.

Change-Id: Ib23cb047bbbe6cff146305935a16ba833d40351a
2020-12-09 09:50:04 +01:00
Egon Elbre
5b6fcfb725 cmd/storj-sim: fix 32bit code
Change-Id: Ieec74f2a8ed1a207e7deff0f9072c1243cb1e3be
2020-12-09 09:49:33 +02:00
Stefan Benten
3c77825b19 cmd: move ca-certificates build step to be arch independent
It turns out, that running a docker image build for specific
arches is not possible from amd64 (eg. installing ca-certificates).

Change-Id: I8b8f002b7e532fb4a0c6542d5b573c294c501068
2020-12-08 21:49:41 +01:00
Malcolm Bouzi
1daadc11e7
web/satellite: dashboard lag message tooltip (#3982) 2020-12-08 19:01:51 +02:00
Qweder93
2f62cdf491 storagenode/console: diskSpaceInfo extended with overused diskspace, getDashboardData updated.
Change-Id: I44db26661a8dfb45b5d8e9fcb7511f63deb88cad
2020-12-08 14:55:55 +00:00
Kaloyan Raev
d033ec3dfa private/testplanet: fix TestDownloadWithSomeNodesOffline
Change-Id: Ia52f408dc4500c405d8b1e32d22c08b53d351247
2020-12-08 15:29:58 +02:00
Kaloyan Raev
3f5e07fb1f satellite/metainfo: more tests fixed
- TestBucketNameValidation
- TestBatch
- TestCommitObjectMetadataSize
- TestIDs

TestOverwriteZombieSegments is removed as not relevant to metabase.

Change-Id: I13cf5abe342089960628f185061303fd4f9d09a4
2020-12-08 14:00:24 +02:00
VitaliiShpital
c8e8fea052 web/satellite: replacing api keys with access grants
WHAT:
api keys appearence is replaced with access grants

WHY:
last step of access grants implementing

Change-Id: Ibef391849c7185fa56627b482218c76fb2d31b46
2020-12-08 13:34:59 +02:00
VitaliiShpital
70795c68ff web/satellite: update onboarding flow: create access grant
WHAT:
create access grant flow for onboarding tour

WHY:
new onboarding tour

Change-Id: I75e84b00edb71cac07f8e6623ce06655ae771894
2020-12-08 10:39:15 +00:00
VitaliiShpital
55b495faa2 web/satellite: update onboarding flow: payment step update
WHAT:
added untitled project creation after adding payment method.
UI was slightly changed.

WHY:
new onboarding flow

Change-Id: I6cca1c784bd0462f20f2b06cc6595b9920d4f992
2020-12-08 10:39:08 +00:00
VitaliiShpital
701474acd5 web/satellite: update onboarding tour: overview step
WHAT:
updated overview step of onboarding tour. It shows upload data methods instead of tour steps now

WHY:
new onboarding tour

Change-Id: I7ffe9b2b91c2e17dd0c27e5e80a15301f6de16aa
2020-12-08 10:39:01 +00:00
VitaliiShpital
d5f1245cc7 web/satellite: update onboarding flow: removed redundant steps, routing applied
WHAT:
redundant steps removed from onboarding tour, new routing applied

WHY:
flow simplification

Change-Id: I153c5fe30914317ddd9bf7008d4e5e6a24f77f9a
2020-12-08 10:38:54 +00:00
VitaliiShpital
aaec8bfc5c web/satellite: update onboarding tour: routing
WHAT:
first step of updating onboarding tour - adding routes

WHY:
onboarding your is being reworked so adding routes will make it easier to operate over it's states

Change-Id: Ide830989e39a6222e975bd2a6106b0efbb3839f9
2020-12-08 10:38:47 +00:00
Kaloyan Raev
d0d0a192cd satellite/metainfo: fix TestEndpoint_DeleteObjectPieces tests
This also removes the
TestEndpoint_DeleteObjectPieces_ObjectWithoutLastSegment test case as it
does not seem relevant to metabase.

Change-Id: I06a0ecaa8232c10c15e433517a7ba056933bf858
2020-12-08 10:22:05 +00:00
Michal Niewrzal
fb5244e8f1 satellite/metainfo: migrate GetObjectIP endpoint method to metabase
Change-Id: I0da346fc2ce84787a09d2bb7c4ffc02a5d8a3b2f
2020-12-08 08:50:15 +00:00
Stefan Benten
a71c908c1e
cmd: add ca-certificates to Docker images (#3986) 2020-12-08 01:38:33 +01:00
Michal Niewrzal
bc3be9c416 Jenkinsfile: enable build failure on integration tests failure
Change-Id: Ifabcd9e72cd51de1300b5f8a4453c87ca292c60e
2020-12-07 15:31:46 +00:00
Kaloyan Raev
3269401515 deps: update to latest uplink from multipart-upload branch
This resolves an issue in Uplink CLI with listing size 0 for files
uploaded with multipart upload.

Change-Id: I80e0b11a96f87ed6a87eb5301034c08dbc09e8aa
2020-12-07 14:54:51 +00:00
Michal Niewrzal
218bbeaffa Merge 'master' branch
Change-Id: Ica5c25607a951076dd9f77e35e308062f71ce3f0
2020-12-07 15:05:52 +01:00
Michal Niewrzal
cdeea1c999 private/testplanet: add helper OpenProject method to testplanet uplink
This will simplify opening pure uplink.Project in tests.

Change-Id: I076875e15e21608f49dc875bb445412f34609bdb
2020-12-07 13:45:47 +00:00
VitaliiShpital
7eccfccfda web/satellite: create access grant: bucket search feature
WHAT:
search for bucket names during creating access grant flow

WHY:
make user be able to search for needed buckets. Just in case if the number of buckets is too large.

Change-Id: I73bcaa160c7a1f433d8f0f7213999e7e40543bbc
2020-12-07 11:54:05 +00:00
Stefan Benten
494bd5db81
all: golangci-lint v1.33.0 fixes (#3985) 2020-12-05 17:01:42 +01:00
Yingrong Zhao
746315672f scripts/tests/testversions: fix indentation
Change-Id: Iaa5aec27f0ad78e1d8bf1a68aa5a62762c8ab537
2020-12-04 21:54:55 +00:00
Yingrong Zhao
0faf7d5293 scripts/tests/testversions: fix race in install_sim
Change-Id: I0792686d99a222d5977fd913425e2b94d100c40e
2020-12-04 18:18:14 +00:00