JakeHillion/storj
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,852 Commits 5 Branches 0 Tags 94 MiB
4e19839669
Commit Graph

422 Commits

Author SHA1 Message Date
Egon Elbre
65b22be417 satellite/accounting/tally: use metabase 
Change-Id: I6d49dc103a18e8a110bfa7775d53a65d208b6c2c
 2020-12-18 16:18:03 +02:00
Michal Niewrzal
311b082838 satellite/metainfo: fix metainfo loop 
This fix issues with passing observers between iteration methods.

It's not best implementation but I think we will need to optimize it
soon one way or another.

Change-Id: I574599bfd10822d84e2d2f1800bcd88e176a76ea
 2020-12-18 10:38:12 +00:00
Michal Niewrzal
2111740236 Merge 'master' branch 
Change-Id: Ib73af0ff3ce0e9a1547b0b9fc55bf88704f6f394
 2020-12-18 09:13:24 +01:00
littleskunk
3feee9f4f8
satellite/accounting: default project limits (#4001) 2020-12-17 22:27:05 +01:00
Michal Niewrzal
70ba4deea9 satellite/repair/checker: adjust irreparable part of repair checker 
Change-Id: I0732104a97ba18a5359de3966cd692677a0ff790
 2020-12-17 14:11:22 +00:00
Kaloyan Raev
9aa61245d0 satellite/audits: migrate to metabase 
Change-Id: I480c941820c5b0bd3af0539d92b548189211acb2
 2020-12-17 14:38:48 +02:00
Michal Niewrzal
8d3ea9c251 satellite/repair/repairer: implement SegmentRepairer with metabase 
Change-Id: I647c625e00a626c44e812602ad9bc3e85a7b602c
 2020-12-17 10:47:21 +00:00
Kaloyan Raev
6206aa88e4 {satellite/metainfo,private/testplanet} use TestingAllSegments in tests 
Change-Id: I8c641a24fabf3ea537312978a42501eab8d6a339
 2020-12-17 09:58:31 +00:00
Egon Elbre
1728e45e55 satellite/metainfo/metabase: optimize DeleteBucket 
Change-Id: If34cdeae0f688cb96717905fc8287c66ea3034be
 2020-12-16 14:39:33 +00:00
Egon Elbre
4706f01876 satellite/metainfo: add TestingAll{Segments,Objects} 
Change-Id: Ia758c119d5ebd7bbb21216a463c99c2e9afcdeb0
 2020-12-16 10:21:36 +00:00
Kaloyan Raev
4fba9921f6 satellite/metainfo/metabase: define ErrSegmentNotFound error class 
This makes it easier to callers of GetSegmentByPosition to determine if
the segment is missing.

Change-Id: I2d8546dddf07dcf790d2f7c08d308ed589b34f2f
 2020-12-15 15:48:41 +02:00
Michal Niewrzal
934ae32ca4 satellite/repair/checker: fix checker tests 
Change-Id: I63d3368a07b800fdb10bb93b847eb32927b8c0dc
 2020-12-15 10:47:42 +00:00
Kaloyan Raev
fc85179a19 satellite/metainfo: refactor SegmentLocation.Index to SegmentPosition 
Change-Id: Ic9403c8126712693326dd83d6ba4f3b84be3e0c7
 2020-12-14 13:35:53 +02:00
Kaloyan Raev
7d8f19e94d satellite/metainfo: metainfo loop should yield StreamID for segments 
Change-Id: If6c86add75ce79ffcfe95353225719c7d4b5a459
 2020-12-14 13:04:38 +02:00
Michal Niewrzal
e7e6985ae9 satellite/metainfo/metabase: add UpdateSegmentPieces method 
We need to be able to update just remote_pieces column in DB. This is
needed at least for repair process.

Change-Id: I20dcc9b06babfefbbf102f32b1d14946379f26c2
 2020-12-14 10:25:09 +00:00
Kaloyan Raev
2bb010e7c5 cmd: remove segment reaper 
It was designed to detect and remove zombie segments in the PointerDB.
This tool should be not relevant with the MetabaseDB anymore.

Change-Id: I112552203b1329a5a659f69a0043eb1f8dadb551
 2020-12-14 09:36:37 +00:00
Michal Niewrzal
7e6e0d3e2e satellite/metainfo: metainfo loop implementation with metabase 
Change-Id: Iadac469519de605a88e624df23265289771b2006
 2020-12-11 16:15:57 +01:00
Michal Niewrzal
b3acc1101a Merge 'master' branch 
Change-Id: Iee99400c7095770e61cde94b3b2c8eb0ddec463d
 2020-12-10 15:42:52 +01:00
Michal Niewrzal
c2a97aeb14 satellite/satellitedb: add ListAllBuckets method 
We need to be able to list all buckets in DB without knowing project ID.
This method will be used to list buckets for metainfo loop
implementation based on metabase.

Change-Id: Iac75af0eee4f31e80a15577575a8249cbca787b2
 2020-12-10 14:19:27 +00:00
Fadila Khadar
15add3c752 satellite/metainfo: enable deletion of a pending object. 
Change-Id: I1ac24ad340d411c52b92ef18f51a684d9214ed47
 2020-12-10 13:56:14 +01:00
Egon Elbre
365410d10b satellite/metainfo/metabase: add commit with specifying a list of segments 
Change-Id: Ibb9999545691d150f36e20ce70ac00a3802ad7b2
 2020-12-09 15:16:38 +02:00
Michal Niewrzal
f077564bb7 satellite/metainfo: setup access to metabase in metainfo loop 
This is initial change to replace metainfo loop implementation with
metabase.

Change-Id: Ib23cb047bbbe6cff146305935a16ba833d40351a
 2020-12-09 09:50:04 +01:00
Kaloyan Raev
3f5e07fb1f satellite/metainfo: more tests fixed 
- TestBucketNameValidation
- TestBatch
- TestCommitObjectMetadataSize
- TestIDs

TestOverwriteZombieSegments is removed as not relevant to metabase.

Change-Id: I13cf5abe342089960628f185061303fd4f9d09a4
 2020-12-08 14:00:24 +02:00
Kaloyan Raev
d0d0a192cd satellite/metainfo: fix TestEndpoint_DeleteObjectPieces tests 
This also removes the
TestEndpoint_DeleteObjectPieces_ObjectWithoutLastSegment test case as it
does not seem relevant to metabase.

Change-Id: I06a0ecaa8232c10c15e433517a7ba056933bf858
 2020-12-08 10:22:05 +00:00
Michal Niewrzal
fb5244e8f1 satellite/metainfo: migrate GetObjectIP endpoint method to metabase 
Change-Id: I0da346fc2ce84787a09d2bb7c4ffc02a5d8a3b2f
 2020-12-08 08:50:15 +00:00
Michal Niewrzal
218bbeaffa Merge 'master' branch 
Change-Id: Ica5c25607a951076dd9f77e35e308062f71ce3f0
 2020-12-07 15:05:52 +01:00
Stefan Benten
494bd5db81
all: golangci-lint v1.33.0 fixes (#3985) 2020-12-05 17:01:42 +01:00
Ethan Adams
f90ea10a4a
Allow for DB application names per process. (#3983) 2020-12-04 11:24:39 +01:00
Kaloyan Raev
a7685f50c9 satellite/metainfo/metabase: set maxParts to MaxListLimit if greater 
We should set the client requested maxParts to MaxListLimit if it is
greater than that value instead of returning an error.

MinIO default value for maxParts is 10,000 while the satellite's
MaxListLimit is 1,000. If we return an error, the ListParts with default
maxParts will throw an error.

Change-Id: I06739e1d8d8f96803eba491585395da0443aec04
 2020-12-03 11:32:07 +00:00
Fadila Khadar
fd7cc20a7e satellite/metainfo/metabase: add DeletePendingObject 
Change-Id: I7ba7d3d7f179712940e91b05501310279bbb97a4
 2020-12-03 10:21:04 +00:00
Kaloyan Raev
bc25cc8ddd satellite/metainfo: add encryption info in metadata of list items 
This is needed to resolve 'encryption type 0 is not supported' errors in
gateway.

Change-Id: I33c486968d5a1a1310b82e9778a8ce01429f7043
 2020-12-02 13:34:41 +02:00
Michal Niewrzal
83e7cd2a40 satellite/metainfo/metabase: replace SQL substring with go code 
We have some issues with SUBSTRING function on cockroachdb so for now we
are removing it from SQL query and replacing with go code.

Change-Id: I5be921211067d42e7d1a4997076bcfdbed9617a1
 2020-12-02 10:47:18 +01:00
Kaloyan Raev
e8378a87bf satellite/metainfo: include redundancy in satStreamID of ListObjects 
This fixes the issue with using UploadID from the ListMultipartUploads
response.

Change-Id: If077a5804d03cc139f75febf472d5fb73f0f7929
 2020-12-01 15:11:34 +00:00
Michal Niewrzal
d5c0264163 satellite/metainfo: implement deleteBucketObjects with metabase objects 
iterator

This method replaces `deleteByPrefix` as at the moment only function of
this method was to delete objects in a bucket.

Change-Id: I5266103672003fbd64f3847f53760b1ba0016fe2
 2020-12-01 14:54:23 +00:00
Michal Niewrzal
f08e34f15e satellite/metainfo: fix listing objects when cursor is specified 
We need to combine prefix + cursor to iterate objects correctly.

Change-Id: I2f4a29ba3ee65ce52eb1e8b88a3b09cf93e0b2bb
 2020-12-01 15:01:44 +01:00
Michal Niewrzal
e4c4ab06ba satellite/metainfo: add testing methods for getting objects and segments 
Change-Id: I4229d80090609df27d998a90bdb3137469dba042
 2020-12-01 12:44:08 +00:00
Kaloyan Raev
46839b992a satellite/metainfo: expired deletion service to use Metabase 
It also sets the Expires time correctly in the Metabase when uploading
objects.

Change-Id: Iec1b3ec8d11346a91dfc2ba23a1b08edec4a84d3
 2020-12-01 09:12:14 +00:00
Michal Niewrzal
5a7bc9657d Merge 'master' branch 
Change-Id: If583132a821274dc4b78cf5f72b853ba8460c619
 2020-11-30 12:57:22 +01:00
Michal Niewrzal
21602e0494 satellite/metainfo: enable commented test 
Test was commented to make uplink refactoring possible. Now we can bring
back this test.

Change-Id: I0511b76073efaafed8aac97f8e845dcec93dd059
 2020-11-30 10:49:23 +01:00
Michal Niewrzal
cfb45a785c satellite/metainfo/metabase: add TotalPlainSize to objects table 
We need to know how large is object in general and we shouldn't query
segments for that.

Change-Id: Ie316e1ec38bfc39065e6046b79b55d58f3ce9d53
 2020-11-25 14:35:28 +01:00
Michal Niewrzal
8ceef9f357 satellite/metainfo: temporary disable one assertion in test 
This is need to merge https://review.dev.storj.io/c/storj/uplink/+/3208
, after that this code will be back.

Change-Id: If9f2f1db95c7a1bba64a41c45a39bd3096a519e7
 2020-11-25 13:21:41 +00:00
Michal Niewrzal
983b1737c2 satellite/metainfo: implement IsBucketEmpty with metabase 
Use objects iterator to check if bucket is empty.

Change-Id: I590bb7258a47dd816f079039a1151df809dd2637
 2020-11-25 12:20:30 +00:00
Michal Niewrzal
46102c1942 satellite/metainfo: change ListSegments required permission to 
Read/Download

We should treat listing segments as a part of download process, not
listing.

Change-Id: Iae72c97c77976c427f668a96486af613d58da3b9
 2020-11-24 17:50:33 +01:00
Egon Elbre
dad36179c6 satellite/metainfo/metabase: fix segment listing with cursor.Part 
Change-Id: I910888d435b11d2d4687ea0042998a1078cd6443
 2020-11-24 15:31:57 +02:00
Kaloyan Raev
67c210a64c satellite/metainfo: set list cursor to version 1 
Otherwise, if left to default version 0, the iterator will include the
cursor item in the result, which fails some tests.

Change-Id: I85103a36852477f371ec46c673a82c2e129978b7
 2020-11-24 10:17:42 +00:00
Michal Niewrzal
5be4d413b0 satellite/metainfo: set PlainSize for inline segment 
Plain size is required by metabase implementation.

Change-Id: I6739744bff4ce485e0066f2a49acdf38560d31fb
 2020-11-20 13:54:14 +01:00
Kaloyan Raev
05fe497a6f satellite/metainfo: add stream id when listing pending objects 
Change-Id: Idcdffd70b82c0fe4f79635fc193604b66101ef76
 2020-11-19 14:22:09 +02:00
Michal Niewrzal
1465539220 satellite/metainfo: ignore empty inline segments for backward 
compatibility

Change-Id: I2567d77e172391945dee24cea8e110fa1963d663
 2020-11-18 14:12:03 +00:00
Michal Niewrzal
70b1c7aa17 satellite/metainfo: check if EncryptedMetadataNonce was set 
It possible that old uplinks won't set it.

Change-Id: Ifd3c4d4a45c5636cfeb70b0a37bcf1305a097727
 2020-11-18 14:04:31 +01:00
Kaloyan Raev
a30f5d7ec0 satellite/metainfo: use Object.Status from ObjectListRequest 
Change-Id: Ie2e48d2eedf69bd0f04b7c9d6e7e217fec6b0267
 2020-11-18 13:31:52 +02:00