Commit Graph

39 Commits

Author SHA1 Message Date
Kaloyan Raev
b8c6fb764c satellite/metainfo: add metabase to metainfo service
Change-Id: Ie3ff238b138d8a57d99e32b13f7a71aa624d53e3
2020-10-30 12:49:47 +02:00
Kaloyan Raev
1f386db566
cmd/satellite: remove metainfo commands (#3955) 2020-10-22 13:33:09 +03:00
Kaloyan Raev
b409b53f7f cmd/satellite: command for verifying piece hashes
Jira: https://storjlabs.atlassian.net/browse/PG-69

There are a number of segments with piece_hashes_verified = false in
their metadata) on US-Central-1, Europe-West-1, and Asia-East-1
satellites. Most probably, this happened due to a bug we had in the
past. We want to verify them before executing the main migration to
metabase. This would simplify the main migration to metabase with one
less issue to think about.

Change-Id: I8831af1a254c560d45bb87d7104e49abd8242236
2020-09-29 10:58:24 +00:00
Kaloyan Raev
34613e4588 cmd/satellite: command for fixing old-style objects
Jira: https://storjlabs.atlassian.net/browse/PG-67

There are a number of old-style objects (without the number of segments
in their metadata) on US-Central-1, Europe-West-1, and Asia-East-1
satellites. We want to migrate their metadata to contain the number of
segments before executing the main migration to metabase. This would
simplify the main migration to metabase with one less issue to think
about.

Change-Id: I42497ae0375b5eb972aab08c700048b9a93bb18f
2020-09-21 14:46:19 +00:00
Michal Niewrzal
aa47e70f03 satellite/metainfo: use metabase.SegmentKey with metainfo.Service
Instead of using string or []byte we will be using dedicated type
SegmentKey.

Change-Id: I6ca8039f0741f6f9837c69a6d070228ed10f2220
2020-09-03 15:11:32 +00:00
Cameron Ayer
46bdb1dbfe Revert "satellite/metainfo: log if replacing pieces in pointer goes wrong"
This reverts commit 9edb363558.

Change-Id: Ia6219f4cca4e766c7919a6d830dfa6900fd4f6e0
2020-07-23 13:42:50 +00:00
Rafael Gomes
375d76638d satellite/metainfo/objectdeletion: Object deletion implementation
Improve our delete logic to require fewer database requests.
This PR creates a new objectdeletion package

Change-Id: I0500173bb9b8c771accb350f076329ede6dbb42c
2020-07-20 16:09:48 +00:00
Egon Elbre
080ba47a06 all: fix dots
Change-Id: I6a419c62700c568254ff67ae5b73efed2fc98aa2
2020-07-16 14:58:28 +00:00
Jennifer Johnson
784a156eea satellite: prevents uplink from creating a bucket once it exceeds the max bucket allocation.
Change-Id: I4b3822ed723c03dbbc0df136b2201027e19ba0cd
2020-07-15 17:27:05 +00:00
Egon Elbre
d88394c3d2 satellite/metainfo: verify pointer in service
Change-Id: Ib489a177c93739f48d3a9669a56d9892640c74ef
2020-06-11 13:45:58 +00:00
Cameron Ayer
9edb363558 satellite/metainfo: log if replacing pieces in pointer goes wrong
Every now and then we see the repair error, "piece to add already exists".
With these new logs we should be able to verify if it is due to a change in
the pointer. These logs are only temporary

Change-Id: I029390cc4816668707546df14ed2cfe7ca192b0b
2020-05-26 21:48:40 +00:00
Egon Elbre
2c0d61b18e satellite/metainfo: avoid temporary list
Currently ListV2 loaded the whole data into memory, even when all the
data wasn't being used, using up more memory than needed.

Change-Id: I5846d979344729b447c108a6cc9f4227229ec981
2020-04-15 08:01:42 +00:00
Egon Elbre
11a44cdd88 all: don't depend on gogo/proto directly
Change-Id: I8822dea0d1b7b99e0b828e0373a0308a42dde2be
2020-04-08 17:32:15 +00:00
Egon Elbre
0a69da4ff1 all: switch to storj.io/common/uuid
Change-Id: I178a0a8dac691e57bce317b91411292fb3c40c9f
2020-03-31 19:16:41 +03:00
Egon Elbre
5342dd9fe6 go.mod: update uplink
Change-Id: I867a6a1eef8aa5d60bb676e5112b98c4192ce811
2020-02-21 16:08:12 +02:00
Egon Elbre
ef2f101495 satellite/metainfo: don't allow deleting non-empty bucket
Change-Id: I72a8b959e954c7f52e93fc8ea4006a957cc2941a
2020-02-14 14:36:22 +01:00
Kaloyan Raev
37cf42a9ae satellite/metainfo: overwrite zombie segments
Fixes https://storjlabs.atlassian.net/browse/USER-240

- Adds UnsynchronizedPut method to metainfo service that overwrites any
existing pointer under the same path
- Uses UnsynchronizedPut in the metainfo endpoint for committing the
segments

Change-Id: Icb43f31ea33f14066ca9dfdcf226eb3079b90948
2020-02-12 11:10:38 +00:00
Egon Elbre
082ec81714
uplink: move to storj.io/uplink (#3746) 2020-01-08 15:40:19 +02:00
Egon Elbre
6615ecc9b6 common: separate repository
Change-Id: Ibb89c42060450e3839481a7e495bbe3ad940610a
2019-12-27 14:11:15 +02:00
Ivan Fraixedes
770123de10 satellite/metainfo: Improve docs & use common funcs
* Use unexported existent method in logic that was duplicated in some
  exported methods.
* Log a forgotten internal error.
* Improve the documentation adding more and fixing some to fit to our
  code style conventions.

Change-Id: Ie6f8bc59f9089f92b8b0d1b4c09c2142c3f273f5
2019-12-13 16:00:35 +00:00
Egon Elbre
72d407559e satellite/metainfo: don't leak error implementation detail (#3722)
* satellite/metainfo: don't leak implementation detail

* add missing wrap
2019-12-10 15:21:30 -05:00
Ivan Fraixedes
c400471bbc
satellite/metainfo: Fix some docs comments
Fix a documentation comment for one method and apply our code
conventions to some that I stumbled.

Change-Id: I3baf5d004a128dcd561c3e27c080aab345c64461
2019-12-06 19:14:35 +01:00
paul cannon
0c025fa937 storage/: remove reverse-key-listing feature
We don't use reverse listing in any of our code, outside of tests, and
it is only exposed through libuplink in the
lib/uplink.(*Project).ListBuckets() API. We also don't know of any users
who might have a need for reverse listing through ListBuckets().

Since one of our prospective pointerdb backends can not support
backwards iteration, and because of the above considerations, we are
going to remove the reverse listing feature.

Change-Id: I8d2a1f33d01ee70b79918d584b8c671f57eef2a0
2019-11-12 18:47:51 +00:00
Egon Elbre
cc032d3151 satellite/metainfo: fix some uses of metainfo.Delete (#3513)
* satellite/metainfo: rename Delete to UnsynchronizedDelete

* fix deletes

* make db private

* fix typos

* also verify on commit object
2019-11-06 18:02:14 +01:00
Ethan Adams
2eb0cc56fe
satellite/gracefulexit: Check if node already has a piece in the pointer (#3434) 2019-11-05 14:13:45 -05:00
Egon Elbre
a744fdfef0 satellite/metainfo: remove Iterate from service (#3196)
* satellite/metainfo: remove Iterate from service

* fix test
2019-10-08 07:39:23 -07:00
Natalie Villasana
cc70cd2329
satellite/repair: add metric trackers for segment age before repair (#3056) 2019-09-17 15:18:48 -04:00
Maximillian von Briesen
7ada5d4152
satellite/metainfo: make piece hashes nil before storing pointer in metainfo.UpdatePieces() (#3050) 2019-09-16 12:11:12 -04:00
Egon Elbre
a801fab66a
all: add archview annotations (#2964) 2019-09-10 16:24:16 +03:00
Egon Elbre
5d0816430f
rename all the things (#2531)
* rename pkg/linksharing to linksharing
* rename pkg/httpserver to linksharing/httpserver
* rename pkg/eestream to uplink/eestream
* rename pkg/stream to uplink/stream
* rename pkg/metainfo/kvmetainfo to uplink/metainfo/kvmetainfo
* rename pkg/auth/signing to pkg/signing
* rename pkg/storage to uplink/storage
* rename pkg/accounting to satellite/accounting
* rename pkg/audit to satellite/audit
* rename pkg/certdb to satellite/certdb
* rename pkg/discovery to satellite/discovery
* rename pkg/overlay to satellite/overlay
* rename pkg/datarepair to satellite/repair
2019-07-28 08:55:36 +03:00
Kaloyan Raev
734c793deb
Add UpdatePieces method to Metainfo service (#2572) 2019-07-25 19:59:46 +03:00
Ivan Fraixedes
3c8f1370d2
[v3 2137] - Add more info to find out repair failures (#2623)
* pkg/datarepair/repairer: Track always time for repair
  Make a minor change in the worker function of the repairer, that when
  successful, always track the metric time for repair independently if the
  time since checker queue metric can be tracked.

* storage/postgreskv: Wrap error in Get func
  Wrap the returned error of the Get function as it is done when the
  query doesn't return any row.

* satellite/metainfo: Move debug msg to the right place
  NewStore function was writing a debug log message when the DB was
  connected, however it was always writing it out despite if an error
  happened when getting the connection.

* pkg/datarepair/repairer: Wrap error before logging it
  Wrap the error returned by process which is executed by the Run method
  of the repairer service to add context to the error log message.

* pkg/datarepair/repairer: Make errors more specific in worker
  Make the error messages of the "worker" method of the Service more
  specific and the logged message for such errors.

* pkg/storage/repair: Improve error reporting Repair
  In order of improving the error reporting by the
  pkg/storage/repair.Repair method, several errors of this method and
  functions/methods which this one relies one have been updated to be
  wrapper into their corresponding classes.

* pkg/storage/segments: Track path param of Repair method
  Track in monkit the path parameter passed to the Repair method.

* satellite/satellitedb: Wrap Error returned by Delete
  Wrap the error returned by repairQueue.Delete method to enhance the
  error with a class and stack and the
  pkg/storage/segments.Repairer.Repair method get a more contextualized
  error from it.
2019-07-23 16:28:06 +02:00
aligeti
29b576961f
value attribution merge fix and more test cases (#2588)
* value attribution merge fix and more test cases
2019-07-19 11:17:34 -04:00
Jess G
f11bf46a11
Jg/1967 mv bucket metadata uplink (#2505)
* add bucketstore, add init uplink bucket

* update uplink to use bucket rpc

* fix tests

* wrap metainfo client errors

* add allowedBucket struct, fix tests

* update comment

* add paging

* updates per CR

* add test for pagination

* fix lint

* fix uplink test so its easier tyo understand

* fix gateway pagination bug

* changes per cr

* fix bug w/allowedBuckets, add test to catch
2019-07-12 08:57:02 -04:00
Jess G
f9696d6c5e
satellite/metainfo: add buckets RPC and database (#2460)
* add db interface and methods, add sa metainfo endpoints and svc

* add bucket metainfo svc funcs

* add sadb bucekts

* bucket list gets all buckets

* filter buckets list on macaroon restrictions

* update pb cipher suite to be enum

* add conversion funcs

* updates per comments

* bucket settings should say default

* add direction to list buckets, add tests

* fix test bucket names

* lint err

* only support forward direction

* add comments

* minor refactoring

* make sure list up to limit

* update test

* update protolock file

* fix lint

* change per PR
2019-07-08 15:32:18 -07:00
Alexander Leitner
3587e1a579 Change pointerdb pointer to use time.Time for Creation date (#2483) 2019-07-09 00:16:50 +02:00
JT Olio
f1641af802 storage: add monkit task to missing places (#2122)
* storage: add monkit task to missing places

Change-Id: I9e17a6b14f7c25bbf698eeecf32785e9add3f26e

* fix tests

Change-Id: Id078276fa3de61a28eb3d01d4e751732ecbb173f

* import order

Change-Id: I814e33755b9f10b5219af37cd828cd75eb3da1a4

* remove part of other commit

Change-Id: Idaa4c95cd65e97567fb466de49718db8203cfbe1
2019-06-05 16:23:10 +02:00
JT Olio
29d16b4d68 satellite: add monkit task to missing places (#2108) 2019-06-04 13:55:37 +02:00
Michal Niewrzal
fe3dfc1587
Move pointerdb.Service to satellite (#1826) 2019-04-25 10:46:32 +02:00