Commit Graph

3018 Commits

Author SHA1 Message Date
Cameron Ayer
6fae361c31 replace planet.Start in tests with planet.Run
planet.Start starts a testplanet system, whereas planet.Run starts a testplanet
and runs a test against it with each DB backend (cockroach compat).

Change-Id: I39c9da26d9619ee69a2b718d24ab00271f9e9bc2
2019-12-10 16:55:54 +00:00
Yaroslav Vorobiov
8cf1aa6e4f
satellite/accounting: fix project limits migration (#3717) 2019-12-10 18:12:49 +02:00
Michal Niewrzal
f56107fc61
uplinkc: add download_range function (#3704) 2019-12-10 03:51:56 -08:00
Michal Niewrzal
011bb5d551 segment-reaper: test from/to detect parameters
Unit test for testing from/to parameters of `segment-reaper detect`
command.

Change-Id: I85c8839b676c61e7f046cb5ccc92d988ca51d6e3
2019-12-10 12:18:44 +01:00
Ivan Fraixedes
e961b1840c
cmd/segment-reaper: Implement unit test for observer (#3658)
Implement some unit test cases for the observer.processSegment method and fix a bug found by these tests.

A production snapshot is more certain but it's huge for having in the repository and run the test with it by the CI.
We want to have tests for the different cases to detect zombie segments and relaying on production data cannot guarantee to have all of them.

NOTE the test has been implemented with random values for not having always the same combination of segments list and the same values avoiding that the implementation gets stale due to the test. The issue about this is that it's harder to understand and we could get only sometimes failures in the CI in case that the implementation has some bug.

The random tests allow to eventually check cases that a static test may now cover because it is not expressed or because it is needed to implement a large number of cases.

Because we are worried that the test implementation is complex and we could have bugs on it despite that the same bugs should exist in both, the implementation and the test, moreover that we have to consider that the implementation and the tests have been written by different people. Because of that, we may replace entirely these random tests by a list of static ones.
2019-12-10 11:01:58 +01:00
Michal Niewrzal
de30d232cc uplinkc: add more functions to manage Scope (#3694) 2019-12-10 10:13:25 +01:00
littleskunk
71b58edb2c satellite/repair: decrease repair interval
Change-Id: Id9efdbfaa82521c35dc41e7a52b700522c197e77
2019-12-10 00:36:00 +00:00
Jeff Wendling
48da8baab5 storj-sim: work with cockroach:// urls for satellite databases
for storj-sim to work, we need to avoid schemas in cockroach urls
so we have storj-sim create namespaced databases instead of schemas
and we have the migrate command create the database in the same way
that it would create a schema for postgres. then it works!

a follow up commit will move the creation of the database/schemas
into storj-sim's setup step so that we can avoid doing these icky
creations during normal migration calls. it will also make the
pointerdb have an explicit call to migrate instead of just doing
it every time it's opened.

Change-Id: If69ef5cb96b6866b0438c761bd445afb3597ae5f
2019-12-09 23:44:00 +00:00
Andrew Harding
eb52ac623b storagenode/trust: source implementations
Change-Id: Ie36e79cc15257db88051f63e5b9463fd9d7b4736
2019-12-09 20:00:02 +00:00
littleskunk
6ab72a6e79 satellite/gracefulexit: enable graceful exit in production
Change-Id: I526ce4a4de9c318f1333b793e3167f5f86d65adc
2019-12-09 17:32:34 +00:00
Andrew Harding
7d0aadfeca storagenode/trust: satellite URL implementation
Satellite URL is a stricter form of the STORJ Node URL. It requires both
the ID and port specifier.

Change-Id: I7fd302064f864c1de8240a7915bf5263b898dfd1
2019-12-09 17:05:57 +00:00
Natalie Villasana
c3c02bec3c
satellite/satellitedb: reset storage node reputations to re-enable disqualification (#3693) 2019-12-09 12:04:00 -05:00
littleskunk
9d1faeee58 storagenode/garbagecollection: increase MaxTimeSkew to be higher than satellite MaxCommitInterval
Change-Id: I86f8d0b44bea3aa005ff26d52588611c59df5e9a
2019-12-09 16:03:55 +00:00
Egon Elbre
56a3b62bef satellite/satellitedb: ensure migration tests run (#3706)
satellitedb migration tests ran against multiple base versions, however after the merging all the steps the base versions didn't exists anymore - which meant none of the migration tests were actually running.
2019-12-09 09:26:58 -06:00
Vitalii Shpital
6ef359aba2
web/storagenode: blurred checks hint added for all satellites state (#3709) 2019-12-09 16:10:33 +02:00
Vitalii Shpital
fe5cfeb61b
web/storagenode: disk space chart's tooltip size fixed (#3684) 2019-12-09 15:59:20 +02:00
Nikolai Siedov
c6776ae6bb
error messages fixed (#3712) 2019-12-09 15:20:44 +02:00
Vitalii Shpital
fa5288c254
satellitedb: bucket search fixed (#3594) 2019-12-09 14:46:30 +02:00
Nikolay Yurchenko
ea92c68600
web/satelllite: default cc delete disabling (#3695) 2019-12-09 14:25:38 +02:00
Michal Niewrzal
ffd570eb04 uplink/metainfo: remove additional GetObject from download
This change removes one additional metainfo.GetObject call during
download.

Change-Id: I86b4e5165f299069e3e18944fe79c697c6a514d3
2019-12-09 11:48:48 +00:00
Michal Niewrzal
27462f68e9
cmd/segment-reaper: add detecting and printing zombie segments (#3598) 2019-12-09 03:30:52 -08:00
Ivan Fraixedes
8d49a99ad8 uplink/metainfo: Fix doc comments
Fix some documentation comments in a few methods and apply our code
style conventions to all of them in this source file of the package.

Change-Id: I78186cb1e7d45020d6cf7bbea8a39d146ada1cb6
2019-12-09 08:47:52 +00:00
Michal Niewrzal
b17d40ffd4
segment-reaper: delete command logic (#3660) 2019-12-09 00:25:39 -08:00
Ivan Fraixedes
c400471bbc
satellite/metainfo: Fix some docs comments
Fix a documentation comment for one method and apply our code
conventions to some that I stumbled.

Change-Id: I3baf5d004a128dcd561c3e27c080aab345c64461
2019-12-06 19:14:35 +01:00
Jeff Wendling
1df7b360d7 satellite/metainfo: Use cockroachdb client for metainfo db
Change-Id: I3cf7a00de4f654eacaffbb494f4841c64a2d9ce6
2019-12-05 10:33:54 -07:00
Jeff Wendling
f15192ea40 storage/cockroachkv: initial client implementation
Change-Id: I72ae558739b2ca532f31cb64f480b43437b6b309
2019-12-05 16:11:23 +00:00
Jennifer Johnson
8e9532dbd9 satellitedb/repairqueue: use errs.New() instead of fmt.Errorf() to retain stack trace
Change-Id: I47f1985aaeace556e3e0f7a20d2718410936db17
2019-12-05 15:52:25 +00:00
paul cannon
378b863b2b private,satellite: unite all the "temp db schema" things
first, so that they all work the same way, because it's getting
complicated, and second, so that we can do the appropriate thing
instead of CREATE SCHEMA for cockroachdb.

Change-Id: I27fbaeeb6223a3e06d97bcf692a2d014b31465f7
2019-12-05 15:36:59 +00:00
Vitalii Shpital
97fa000ca9
web/satellite: usage data converting implemented (#3673) 2019-12-05 13:51:11 +02:00
Cameron
28a66747c9 satellite/satellitedb: cockroachDB compatible transaction in offersDB.Create (#3686)
* use crdb wrapper for create/update offer transaction, ensure times returned are UTC

* round to nearest microsecond to compare times
2019-12-04 21:54:43 -06:00
Jennifer Johnson
7c5f777a4f satellitedb/repairqueue: runs a different implementation of the query within Select() for postgres vs cockroach
Change-Id: Ie34dbdb9d870d7d9f8f269702b6b3bad0c55b98e
2019-12-04 21:32:02 +00:00
Yingrong Zhao
7af42e3c10
satellite/metainfo, satellite/repair, uplink/eestream: add metric for download failed due to not enough pieces available (#3665) 2019-12-04 16:24:36 -05:00
paul cannon
850c358087 private/dbutil/pgutil: make QuerySchema work on crdb
Adjust the pg_constraint query so that it works without a LATERAL JOIN,
since CockroachDB doesn't like that.

This isn't hooked up to a cockroach test yet, but that's coming.

Change-Id: I0df6b477d958996b673fc121eaa1f7c35e5cc504
2019-12-04 18:55:26 +00:00
Jennifer Johnson
e9eff54473 satellitedb/attribution: updates valueAttrQuery to work for both postgres and cockroach
Change-Id: I55d6c21950a66bf782596e9152622efd700c324a
2019-12-04 18:42:02 +00:00
Jennifer Johnson
ecb960f506 private/dbutil: distinguishes between db drivers and implementations to allow for different implementations of SQL queries.
Change-Id: I2dc8d1d371139aa8bc805e92a2b80b71f580fd64
2019-12-04 18:31:26 +00:00
Michal Niewrzal
b779826194 uplinkc: basic support for Scope (#3689) 2019-12-04 18:36:01 +01:00
Cameron
c00f688930 format db name in query to match actual db name (#3688) 2019-12-04 08:55:10 -05:00
Ethan Adams
9420fa9fc5 satellite/gracefulexit: Add graceful exit completed/failed receipt verification to satellite CLI (#3679) 2019-12-03 17:09:39 -05:00
Andrew Harding
2461ccd469 pkg/private/fpath: subsume AtomicWriteFile
AtomicWriteFile is useful primitive to use throughout the codebase

Change-Id: I338fc4505ba20d5aece09ddc257286f46298e083
2019-12-03 18:14:08 +00:00
paul cannon
4d2881b711 satellite/satellitedb: return latest coupon time as UTC
it doesn't necessarily _have_ to be UTC; the time is correct as returned
either way, but this will make it a little less prone to variance.

also, there is a test that depends on the time being returned in UTC.

Change-Id: Ia71e24ecd9973ba70a1cfb5621a3030a5c82d004
2019-12-03 17:36:41 +00:00
Jess G
e0b9b5b317
satellitedb: fixes for cockroachdb compatibility (#3682)
* crdb compat fixes

* update sn accounting query
2019-12-03 08:24:46 -08:00
Ivan Fraixedes
d69482e938
satellite/metainfo: Improve piece hash validation (#3671)
Improve the piece hash validation filtering out a piece when an order
limit is not found for it.

The commit also improves the documentation of an internal metainfo
method and rename the parameters of 2 methods for clarifying what they
are.
2019-12-03 14:36:32 +01:00
Cameron
52851026a7
satellite/console : remove code for creating user credit in CreateUser, skip TestUserCredits (#3680)
* remove code for creating user credit in CreateUser, skip TestUserCredits
2019-12-02 15:23:12 -05:00
paul cannon
b1fa7cdfbb
satellite/satellitedb/satellitedbtest: flag to run cockroach tests (#3674)
This will make it so we don't need to comment out those lines every time
we want to enable the cockroachdb tests during development.

Once it's ready this flag can go away.
2019-12-02 14:06:21 -06:00
Nikolay Yurchenko
90b631c8bf
web/satellite: referral links and registration (#3678) 2019-12-02 21:27:56 +02:00
Fadila
526a126a5a
Satellite: log the project information for upload, download, and list activities (#3651) 2019-12-02 15:39:19 +01:00
Fadila
24cacc8adf installer/windows: fix minor typo (#3676) 2019-12-02 14:48:50 +01:00
Ivan Fraixedes
42c61138e8
storage: Improve doc comments delete methods (#3591)
Improve the documentation of several methods involved in the delete
operation to make clear their behavior without having to inspect their
logic.
2019-12-02 12:18:20 +01:00
Vitalii Shpital
ae60b6533b
web/satellite: titles styling fixed (#3662) 2019-11-29 23:31:52 +02:00
Yehor Butko
ae05fa0874
satellitedb/accounting: fixed query (#3672) 2019-11-29 17:53:57 +02:00