Commit Graph

17 Commits

Author SHA1 Message Date
Michal Niewrzal
f0aeda3091 storj: remove from storj/pkg packages moved to storj/private repo
* debug
* traces
* cfgstruct
* process

Package `storj/private/version` will be removed as a separate change.

Change-Id: Iadc40faa782e6225513b28218952f02d9c240a9f
2020-03-24 09:56:29 +01:00
Egon Elbre
c6f94ce9e4 satellite/metainfo: remove support for boltdb based pointerDB
By previous changes we can now remove testplanet.New and
also remove metainfo boltdb support.

Change-Id: I5bdfbbbb45967492728e705b34b2fedb4f28c381
2020-01-23 13:54:00 +02:00
Simon Guindon
e1e7cebe49 satellite/metainfo: added rate limiting support to the metainfo loop.
As per discussed we decided to rate limit how fast we iterate through
the metainfo database in the metainfo loop. This puts in place a
mechanism for rate limiting and burst limiting if need be in the future.

The default for this rate limiting is still no limits so it stays the
same as our previous functionality.

Change-Id: I950f7192962b0e49f082d2c4284e2d52b0a925c7
2020-01-03 15:00:29 -05:00
Egon Elbre
6615ecc9b6 common: separate repository
Change-Id: Ibb89c42060450e3839481a7e495bbe3ad940610a
2019-12-27 14:11:15 +02:00
Ivan Fraixedes
9ac2c4d815 cmd/segment-reaper: Add test cases from/to processSegment
Add randomized test cases for testing the observer processSegment method
when the observer has a time range (from and to) set and there are
objects with segments whose creation date is outside of this range.

Change-Id: Ieac82a21f278f0850b95275bfdd2e8a812cc57a5
2019-12-13 17:23:32 +00:00
Michal Niewrzal
e1d99522b9 cmd/segment-reaper: test processSegment for single project
This is a test for `observer.processSegment` method from `segment-reaper
detect` command.

Change-Id: I14b7245848766b7aa01e5edffcf94c61a06a4c0a
2019-12-12 10:26:05 +00:00
Michal Niewrzal
b294569840 segment-reaper: fix for not analyzing last project in detect command
This change fixes issue where last project in DB was not analyzed to
find zombie segments

Change-Id: I9efd8a39a65f72d85c7aae325a0e4e3cc1cb9afb
2019-12-11 13:18:13 +00:00
Michal Niewrzal
011bb5d551 segment-reaper: test from/to detect parameters
Unit test for testing from/to parameters of `segment-reaper detect`
command.

Change-Id: I85c8839b676c61e7f046cb5ccc92d988ca51d6e3
2019-12-10 12:18:44 +01:00
Ivan Fraixedes
e961b1840c
cmd/segment-reaper: Implement unit test for observer (#3658)
Implement some unit test cases for the observer.processSegment method and fix a bug found by these tests.

A production snapshot is more certain but it's huge for having in the repository and run the test with it by the CI.
We want to have tests for the different cases to detect zombie segments and relaying on production data cannot guarantee to have all of them.

NOTE the test has been implemented with random values for not having always the same combination of segments list and the same values avoiding that the implementation gets stale due to the test. The issue about this is that it's harder to understand and we could get only sometimes failures in the CI in case that the implementation has some bug.

The random tests allow to eventually check cases that a static test may now cover because it is not expressed or because it is needed to implement a large number of cases.

Because we are worried that the test implementation is complex and we could have bugs on it despite that the same bugs should exist in both, the implementation and the test, moreover that we have to consider that the implementation and the tests have been written by different people. Because of that, we may replace entirely these random tests by a list of static ones.
2019-12-10 11:01:58 +01:00
Michal Niewrzal
27462f68e9
cmd/segment-reaper: add detecting and printing zombie segments (#3598) 2019-12-09 03:30:52 -08:00
Michal Niewrzal
b17d40ffd4
segment-reaper: delete command logic (#3660) 2019-12-09 00:25:39 -08:00
Ivan Fraixedes
0bf7ac5b20 cmd/segment-reaper: Use bitmask type in observer (#3661)
bitmask type was created for making easy to understand the observer bit
logic.
This commit makes the changes in observer for using the bitmask type.
2019-11-27 13:11:07 +01:00
Ivan Fraixedes
7288d01781
cmd/segment-reaper: several refactorings (#3637)
* Move the observer implementation and the type definitions related with
  it and helper functions to its own file.
* Cluster struct type is used as a key for a ObjectsMap type.
  Observer struct type has a field of ObjectsMap.
  Cluster has a field for project ID.
  Observer processSegment method uses its ObjectMap field for tracking
  objects.

  However Observer processSegment clears the map once the projectID
  diverges from the one kept in the Observer lastProjectID field, meaning
  that it isn't needed to keep the projectID as part of the ObjectMap key.

  For this reason, ObjectMap can use as a key just only the bucket name
  and Cluster struct isn't needed.

  Because of such change, the ObjectMap type has been renamed to a more
  descriptive name.
* Make the types defined for this specific package not being exported.
* Create a constructor function for observer to encapsulate the map
  allocation.
* Don't throw away the entirely buckets objects map when an empty one
  is used to reuse part of the allocations.

  Encapsulate the clearing up logic into a method.
* Make the analyzeProject function to be a method of observer.
2019-11-27 10:28:43 +01:00
Michal Niewrzal
1e33995a89
cmd/segment-reaper: skeleton command for zombie deletion (#3599) 2019-11-26 05:01:31 -08:00
Ivan Fraixedes
77d4add101
cmd/segment-reaper: Implement bitmask type (#3626)
Define a bitmask type for implementing its logic to detect if an index
is set or not.
2019-11-22 11:53:48 +01:00
Michal Niewrzal
8ea09d5556
cmd/segment-reaper: add iteration over DB (#3576) 2019-11-18 09:13:20 -08:00
Michal Niewrzal
4912922631
cmd: skeleton for segment-reaper detect command (#3562) 2019-11-14 07:57:27 -08:00