Commit Graph

4799 Commits

Author SHA1 Message Date
Kaloyan Raev
fc85179a19 satellite/metainfo: refactor SegmentLocation.Index to SegmentPosition
Change-Id: Ic9403c8126712693326dd83d6ba4f3b84be3e0c7
2020-12-14 13:35:53 +02:00
Kaloyan Raev
7d8f19e94d satellite/metainfo: metainfo loop should yield StreamID for segments
Change-Id: If6c86add75ce79ffcfe95353225719c7d4b5a459
2020-12-14 13:04:38 +02:00
Michal Niewrzal
e7e6985ae9 satellite/metainfo/metabase: add UpdateSegmentPieces method
We need to be able to update just remote_pieces column in DB. This is
needed at least for repair process.

Change-Id: I20dcc9b06babfefbbf102f32b1d14946379f26c2
2020-12-14 10:25:09 +00:00
Kaloyan Raev
2bb010e7c5 cmd: remove segment reaper
It was designed to detect and remove zombie segments in the PointerDB.
This tool should be not relevant with the MetabaseDB anymore.

Change-Id: I112552203b1329a5a659f69a0043eb1f8dadb551
2020-12-14 09:36:37 +00:00
Michal Niewrzal
7e6e0d3e2e satellite/metainfo: metainfo loop implementation with metabase
Change-Id: Iadac469519de605a88e624df23265289771b2006
2020-12-11 16:15:57 +01:00
Michal Niewrzal
b3acc1101a Merge 'master' branch
Change-Id: Iee99400c7095770e61cde94b3b2c8eb0ddec463d
2020-12-10 15:42:52 +01:00
Michal Niewrzal
c2a97aeb14 satellite/satellitedb: add ListAllBuckets method
We need to be able to list all buckets in DB without knowing project ID.
This method will be used to list buckets for metainfo loop
implementation based on metabase.

Change-Id: Iac75af0eee4f31e80a15577575a8249cbca787b2
2020-12-10 14:19:27 +00:00
Egon Elbre
588dff6d78 Upgrade to uplink v1.4.2
Change-Id: Ib1f5e70c76710d9c49a39347e4ed95bd4671efab
2020-12-10 15:47:11 +02:00
Fadila Khadar
15add3c752 satellite/metainfo: enable deletion of a pending object.
Change-Id: I1ac24ad340d411c52b92ef18f51a684d9214ed47
2020-12-10 13:56:14 +01:00
Tessa Nordgren
e322eac054
added myself to the clabot list. (#3988) 2020-12-09 19:34:22 +01:00
Egon Elbre
365410d10b satellite/metainfo/metabase: add commit with specifying a list of segments
Change-Id: Ibb9999545691d150f36e20ce70ac00a3802ad7b2
2020-12-09 15:16:38 +02:00
Michal Niewrzal
f077564bb7 satellite/metainfo: setup access to metabase in metainfo loop
This is initial change to replace metainfo loop implementation with
metabase.

Change-Id: Ib23cb047bbbe6cff146305935a16ba833d40351a
2020-12-09 09:50:04 +01:00
Egon Elbre
5b6fcfb725 cmd/storj-sim: fix 32bit code
Change-Id: Ieec74f2a8ed1a207e7deff0f9072c1243cb1e3be
2020-12-09 09:49:33 +02:00
Stefan Benten
3c77825b19 cmd: move ca-certificates build step to be arch independent
It turns out, that running a docker image build for specific
arches is not possible from amd64 (eg. installing ca-certificates).

Change-Id: I8b8f002b7e532fb4a0c6542d5b573c294c501068
2020-12-08 21:49:41 +01:00
Malcolm Bouzi
1daadc11e7
web/satellite: dashboard lag message tooltip (#3982) 2020-12-08 19:01:51 +02:00
Qweder93
2f62cdf491 storagenode/console: diskSpaceInfo extended with overused diskspace, getDashboardData updated.
Change-Id: I44db26661a8dfb45b5d8e9fcb7511f63deb88cad
2020-12-08 14:55:55 +00:00
Kaloyan Raev
d033ec3dfa private/testplanet: fix TestDownloadWithSomeNodesOffline
Change-Id: Ia52f408dc4500c405d8b1e32d22c08b53d351247
2020-12-08 15:29:58 +02:00
Kaloyan Raev
3f5e07fb1f satellite/metainfo: more tests fixed
- TestBucketNameValidation
- TestBatch
- TestCommitObjectMetadataSize
- TestIDs

TestOverwriteZombieSegments is removed as not relevant to metabase.

Change-Id: I13cf5abe342089960628f185061303fd4f9d09a4
2020-12-08 14:00:24 +02:00
VitaliiShpital
c8e8fea052 web/satellite: replacing api keys with access grants
WHAT:
api keys appearence is replaced with access grants

WHY:
last step of access grants implementing

Change-Id: Ibef391849c7185fa56627b482218c76fb2d31b46
2020-12-08 13:34:59 +02:00
VitaliiShpital
70795c68ff web/satellite: update onboarding flow: create access grant
WHAT:
create access grant flow for onboarding tour

WHY:
new onboarding tour

Change-Id: I75e84b00edb71cac07f8e6623ce06655ae771894
2020-12-08 10:39:15 +00:00
VitaliiShpital
55b495faa2 web/satellite: update onboarding flow: payment step update
WHAT:
added untitled project creation after adding payment method.
UI was slightly changed.

WHY:
new onboarding flow

Change-Id: I6cca1c784bd0462f20f2b06cc6595b9920d4f992
2020-12-08 10:39:08 +00:00
VitaliiShpital
701474acd5 web/satellite: update onboarding tour: overview step
WHAT:
updated overview step of onboarding tour. It shows upload data methods instead of tour steps now

WHY:
new onboarding tour

Change-Id: I7ffe9b2b91c2e17dd0c27e5e80a15301f6de16aa
2020-12-08 10:39:01 +00:00
VitaliiShpital
d5f1245cc7 web/satellite: update onboarding flow: removed redundant steps, routing applied
WHAT:
redundant steps removed from onboarding tour, new routing applied

WHY:
flow simplification

Change-Id: I153c5fe30914317ddd9bf7008d4e5e6a24f77f9a
2020-12-08 10:38:54 +00:00
VitaliiShpital
aaec8bfc5c web/satellite: update onboarding tour: routing
WHAT:
first step of updating onboarding tour - adding routes

WHY:
onboarding your is being reworked so adding routes will make it easier to operate over it's states

Change-Id: Ide830989e39a6222e975bd2a6106b0efbb3839f9
2020-12-08 10:38:47 +00:00
Kaloyan Raev
d0d0a192cd satellite/metainfo: fix TestEndpoint_DeleteObjectPieces tests
This also removes the
TestEndpoint_DeleteObjectPieces_ObjectWithoutLastSegment test case as it
does not seem relevant to metabase.

Change-Id: I06a0ecaa8232c10c15e433517a7ba056933bf858
2020-12-08 10:22:05 +00:00
Michal Niewrzal
fb5244e8f1 satellite/metainfo: migrate GetObjectIP endpoint method to metabase
Change-Id: I0da346fc2ce84787a09d2bb7c4ffc02a5d8a3b2f
2020-12-08 08:50:15 +00:00
Stefan Benten
a71c908c1e
cmd: add ca-certificates to Docker images (#3986) 2020-12-08 01:38:33 +01:00
Michal Niewrzal
bc3be9c416 Jenkinsfile: enable build failure on integration tests failure
Change-Id: Ifabcd9e72cd51de1300b5f8a4453c87ca292c60e
2020-12-07 15:31:46 +00:00
Kaloyan Raev
3269401515 deps: update to latest uplink from multipart-upload branch
This resolves an issue in Uplink CLI with listing size 0 for files
uploaded with multipart upload.

Change-Id: I80e0b11a96f87ed6a87eb5301034c08dbc09e8aa
2020-12-07 14:54:51 +00:00
Michal Niewrzal
218bbeaffa Merge 'master' branch
Change-Id: Ica5c25607a951076dd9f77e35e308062f71ce3f0
2020-12-07 15:05:52 +01:00
Michal Niewrzal
cdeea1c999 private/testplanet: add helper OpenProject method to testplanet uplink
This will simplify opening pure uplink.Project in tests.

Change-Id: I076875e15e21608f49dc875bb445412f34609bdb
2020-12-07 13:45:47 +00:00
VitaliiShpital
7eccfccfda web/satellite: create access grant: bucket search feature
WHAT:
search for bucket names during creating access grant flow

WHY:
make user be able to search for needed buckets. Just in case if the number of buckets is too large.

Change-Id: I73bcaa160c7a1f433d8f0f7213999e7e40543bbc
2020-12-07 11:54:05 +00:00
Stefan Benten
494bd5db81
all: golangci-lint v1.33.0 fixes (#3985) 2020-12-05 17:01:42 +01:00
Yingrong Zhao
746315672f scripts/tests/testversions: fix indentation
Change-Id: Iaa5aec27f0ad78e1d8bf1a68aa5a62762c8ab537
2020-12-04 21:54:55 +00:00
Yingrong Zhao
0faf7d5293 scripts/tests/testversions: fix race in install_sim
Change-Id: I0792686d99a222d5977fd913425e2b94d100c40e
2020-12-04 18:18:14 +00:00
Caleb Case
c1bdb88d3c
Upgrade to uplink v1.4.1
Change-Id: Ie823f9b72d41f3da8e929d96a870fffc2e7ec12b
2020-12-04 11:38:19 -05:00
Kaloyan Raev
cc9e9ee1f5 storj-sim: use gateway from multipart-upload branch
Change-Id: I0886d277b3b757c8b00975a3e95c2d0d1228488b
2020-12-04 15:15:47 +02:00
Ethan Adams
f90ea10a4a
Allow for DB application names per process. (#3983) 2020-12-04 11:24:39 +01:00
Moby von Briesen
d75e4be11f satellite/{accounting, contact}: Remove periods and spaces from metrics.
Change-Id: I84179c2931293e3a1eb0ff8050416d25e481ce07
2020-12-03 15:33:01 +00:00
Yingrong Zhao
13555f3983 scripts/tests/testversions: enable concurrent installation for each
version

The test-versions test currently takes 1h 40min to run each time. By
running each installation concurrently, hopefully, it will reduce the execution
time for the whole test.

Change-Id: I680c7d9945e982894b11825c9075c167f754e087
2020-12-03 15:01:37 +00:00
Kaloyan Raev
a7685f50c9 satellite/metainfo/metabase: set maxParts to MaxListLimit if greater
We should set the client requested maxParts to MaxListLimit if it is
greater than that value instead of returning an error.

MinIO default value for maxParts is 10,000 while the satellite's
MaxListLimit is 1,000. If we return an error, the ListParts with default
maxParts will throw an error.

Change-Id: I06739e1d8d8f96803eba491585395da0443aec04
2020-12-03 11:32:07 +00:00
Fadila Khadar
fd7cc20a7e satellite/metainfo/metabase: add DeletePendingObject
Change-Id: I7ba7d3d7f179712940e91b05501310279bbb97a4
2020-12-03 10:21:04 +00:00
Moby von Briesen
3fc76f4ffe satellite/downtime: Remove deprecated downtime tracking service.
We are no longer planning on implementing downtime penalization using
the method described in
docs/blueprints/archive/storage-node-downtime-tracking-deprecated.md.
Now, we are implementing the design described in
docs/blueprints/storage-node-downtime-tracking-with-audits.md.

This change removes the downtime estimation chores from the satellite
core as well as the package satellite/downtime. A future change will
remove the database table.

Change-Id: I1a1d3cf9dceeba36255d25243294865b89925518
2020-12-02 15:16:13 -05:00
Caleb Case
6ba8dd58e6 Update to uplink v1.4.0
Change-Id: Ia01fec012031ceb2867b80a37e6a6c2a7d179cca
2020-12-02 13:44:32 +00:00
Kaloyan Raev
bc25cc8ddd satellite/metainfo: add encryption info in metadata of list items
This is needed to resolve 'encryption type 0 is not supported' errors in
gateway.

Change-Id: I33c486968d5a1a1310b82e9778a8ce01429f7043
2020-12-02 13:34:41 +02:00
Michal Niewrzal
83e7cd2a40 satellite/metainfo/metabase: replace SQL substring with go code
We have some issues with SUBSTRING function on cockroachdb so for now we
are removing it from SQL query and replacing with go code.

Change-Id: I5be921211067d42e7d1a4997076bcfdbed9617a1
2020-12-02 10:47:18 +01:00
Bill Thorp
47e008b714 cmd/uplink: Write to AWS Credentials file more safely from uplink access register
Jeff provided feedback on https://review.dev.storj.io/c/storj/storj/+/3176 after
the changeset was already merged.  I attempt to address that feedback here.

Change-Id: Ibc7dba3e4e2c73736042fe4b4ee49ce679ba7f44
2020-12-01 17:47:14 +00:00
JT Olio
1728c3a992 satellite/dbx: standardize on assignment
Change-Id: I8f87bc8391e765e4480b0590d92d3601248e1f93
2020-12-01 16:10:18 +00:00
Jessica Grebenschikov
b261110352 satellite/orders: get bucketID from encrypted metadata in order instead of serial_numbers table
We want to stop using the serial_numbers table in satelliteDB. One of the last places using the serial_numbers table is when storagenodes settle orders, we look up the bucket name and project ID from the serial number from the serial_numbers table.

Now that we have support to add encrypted metadata into the OrderLimit, this PR makes use of that and now attempts to read the project ID and bucket name from the encrypted orderLimit metadata instead of from the serial_numbers table. For backwards compatibility and to ensure no errors, we will still fallback to the old way of getting that info from the serial_numbers table, but this will be removed in the next release as long as there are no errors.

All processes that create orderLimits must have an orders.encryption-keys set. The services that create orderLimits (and thus need to encrypt the order metadata) are the satellite apiProcess, the repair process, audit service (core process), and graceful exit (core process). Only the satellite api process decrypts the order metadata when storagenodes settle orders. This means that the same encryption key needs to be provided in the config for the satellite api process, repair process, and the core process like so:
orders.include-encrypted-metadata=true
orders.encryption-keys="<"encryptionKeyID>=<encryptionKey>"

Change-Id: Ie2c037971713d6fbf69d697bfad7f8b672eedd66
2020-12-01 15:29:32 +00:00
Kaloyan Raev
e8378a87bf satellite/metainfo: include redundancy in satStreamID of ListObjects
This fixes the issue with using UploadID from the ListMultipartUploads
response.

Change-Id: If077a5804d03cc139f75febf472d5fb73f0f7929
2020-12-01 15:11:34 +00:00