Commit Graph

3095 Commits

Author SHA1 Message Date
Andrew Harding
62c58f4a9a satellite: consistent report range arguments
This change updates the three satellite report commands that accept date
ranges to parse and treat those dates uniformly.

- End dates are now uniformly exclusive. Exclusive end dates helps
operators avoid one-off errors on month boundaries, as in the operator
does not have to remember how many days are in that month and can just
run the report from the 1st (inclusive) through the 1st (exclusive).
- Fixed the date range validity check which only failed if the start
date came after the end date (it should have failed dates that were
equal since the check happened after adjusting for inclusivity).

Change-Id: Ib2ee1c71ddb916c6e1906834d5ff0dc47d1a5801
2019-12-20 17:17:09 +00:00
Egon Elbre
9e4d833170 private/testplanet: use default interval
The default interval tries to balance:
1. ensure that most things run at least once during tests
2. ensure that they won't run over 10 times

Change-Id: I911b57b595ffbef1963654bf4a42efad1534b058
2019-12-20 17:01:30 +00:00
Yingrong Zhao
c6854accdf scripts: add test-versions stage to private Jenkins
test-sim-versions.sh tests upgrading the satellite, storagenodes, and uplinks from the most recent release to master, and ensures that compatibility across all uplink versions since v0.15 is maintained.

Change-Id: I80a54236d0eb2d681716caf4b825a883bdc25ef1
2019-12-20 15:52:54 +00:00
Ivan Fraixedes
46c8d2e9c7 private/testplanet: Wait until peer ends when closing it
Close a peer didn't guarantee that the peer ended its services and we
want that when a StopPeer method returns the peer service is actually
finished.

Change-Id: If97f41b7e404990555640c71e097ebc719678ae7
2019-12-20 14:23:25 +00:00
Egon Elbre
858f349a1a lib/uplinkc: add user agent
Change-Id: I6627851112823a72e4001b45d9a1b80ba76b8346
2019-12-20 13:11:25 +00:00
NikolaiYurchenko
9f1de67f6f web/storagenode: notification markup
Change-Id: Ie1eacd9a0f8c27a2450705833bb248ffe78e9257
2019-12-19 17:38:26 +00:00
Egon Elbre
ef8bc88328 ci: use external repository
Change-Id: If26a005df45f6067240511d603fb4dd613f92b79
2019-12-19 12:05:49 +00:00
JT Olio
389d1821ea uplink/paths/encryption: support commandline argument to override path cipher to be urlsafe base64 for lists and deletes (#2855) 2019-12-19 12:29:00 +01:00
Jeff Wendling
ffbc43d170 satellite/satellitedb/dbx: monitor the database calls
Change-Id: I9fbdc4a35fedfc7a1b4e1b630d2a3664b3218e67
2019-12-18 21:57:36 +00:00
Isaac Hess
d5d0c442ac satellite/accounting/rollup: Use lastRollup as zero-value
In satellite/accounting/rollup Service.RollupStorage we have a few
potential error scenarios that return time.Now(). Especially in the case
where we exit early because we have received 0 tallies since the *last*
rollup, this creates a potential race condition.

Between the time we call GetTalliesSince and realize it is empty, it's
possible a tally was inserted in that interval. As currently written we
are returning a latestTally time that excludes that tally.

We are currently protected because in Service.Rollup we don't save the
rollup unless we have populated the rollupStats. However, this change is
more correct and future-proof, because Service.RollupStorage should
always return a correct latestTally time, which in case of errors and
empty tallies, is the last successful tally.

Change-Id: I2521a2cc9802c8f06e512dde4422803a272e2a0a
2019-12-18 21:33:33 +00:00
Matt Robinson
2986923244 build: update golang to 1.13.5
Change-Id: I7f8e29ac682a015d6a18f7989a5d0d726e2a0ce7
2019-12-18 19:47:41 +00:00
Egon Elbre
2daf24a1ea private/testcontext: remove version dependency
Change-Id: Ibabf5ec774dcdb1e4fc2f200368281c69b62e6c2
2019-12-18 15:24:44 +00:00
Egon Elbre
9ed9d3516f pkg/peertls: move tests requiring redis or bolt
Change-Id: Ib9de8d5ac1123d109b0209de4174fb4da7c67078
2019-12-18 17:11:36 +02:00
Egon Elbre
1eaf9e9ed7 pkg/storj: move non-common types
Change-Id: I2dd15c95cd334660f29a528dfb3dec9c30b29cab
2019-12-18 14:14:53 +00:00
Egon Elbre
71d6eef0d7 pkg/storj: remove unnecessary interfaces
Change-Id: Ifb308d2654d61a5b9bb3e28ff20282d6a690f0ca
2019-12-18 14:14:43 +00:00
NikolaiYurchenko
de0aefaa6c web/satellite: card stars align fixed
Change-Id: I88ca22d9eb905db603c97ce9e99dae2f2a581c47
2019-12-18 13:46:24 +00:00
littleskunk
d5c5b57fac satellite/db: enable DeleteTallies
Change-Id: I1e2a6873b3e6398260e053592d676993272b960d
2019-12-18 13:16:06 +00:00
Kaloyan Raev
f8d0864630 satellite/metainfo: use KnownReliable in DeleteObjectPieces
This reduces the number of queries to the overlay when deleting objects.

Change-Id: I28ed2c2d225e0c5eb1a8d952235fa7e5837a48d1
2019-12-18 12:38:22 +00:00
paul cannon
af24581ac0
satellite/audit: do not report offline to overlay (#3547) 2019-12-18 04:51:24 -06:00
Ivan Fraixedes
366f4b9493
satellite: Create method for deleting pieces of SNs
Create a method for deleting pieces of the storage nodes and add a test
for it.

Change-Id: I5fffc8c43d900d317961098b1d135ce3223b73ea
2019-12-18 10:38:16 +01:00
Kaloyan Raev
5ee1a00857 satellite/overlay: filter reliable nodes from a list
Adds the KnownReliable method to Overlay Service that filters all nodes
from the given list to be only reliable nodes (online and qualified).
The method return []*pb.Node of reliable nodes. The pb.Node values are
ready for dialing.

The first use case is when deleting an object to efficiently dial all
reliable nodes holding a piece of that object and send them a delete
request.

Change-Id: I13e0a8666f3807c5c31ef1a1087476018a5d3acb
2019-12-17 21:20:08 +00:00
Jeff Wendling
efa08d4081 storage/cockroachkv: use different batch size for recursive iteration
the relatively small batch size of 128 was chosen so that if we have
a set of keys like

    a/1
    a/2
    ...
    a/100000
    b
    c

list operations would not have to walk 100k keys inside of a/ before
skipping to b. unfortunately, iteration is also used by the metainfo
loop. in that case, it's doing a recursive listing, and so there's
no need to skip large prefixes. thus, we can use a bigger batch
size when recursive listing is requested.

Change-Id: I87cf1ba385b6eb2928c5b7cc5e0f7a8c7bd126d9
2019-12-17 18:24:26 +00:00
Yingrong Zhao
1a625887ed scripts: Add script to automate testing against all highest release
points from major releases starting from v0.15.4 for uplink

Change-Id: I7a3a300466691a47b0324ee5440d70cac42df641
2019-12-17 17:47:17 +00:00
Bryan White
67892b4add private/testidentity: clone identities for each version test
Change-Id: Ic5e5c8e0b19d3b4f86d91e1ae22a26035fd63224
2019-12-17 17:21:16 +01:00
Egon Elbre
69dc514fcb lib/uplink: fix test data race
cfg was being modified concurrently.

Change-Id: I431f152e7e3f72582f1a7d39967c2496e0e0dc1e
2019-12-17 15:41:46 +00:00
Egon Elbre
afe05edff2 {storagenode,satellite}/gracefulexit: ensure workers finish their work
Fixes a data race caused by not waiting for workers to finish
before shutting down. Currently this ended up failing logging
because it was closed when test tried to write to it.

Change-Id: I074045cd83bbf49e658f51353aa7901e9a5d074b
2019-12-17 17:21:52 +02:00
Egon Elbre
7a36507a0a private/testcontext: ensure we call cleanup everywhere
Change-Id: Icb921144b651611d78f3736629430d05c3b8a7d3
2019-12-17 14:16:09 +00:00
Egon Elbre
7455ab771b pkg/peertls/tlsopts: move test that requires testplanet
For splitting core repository we need it not to pull in testplanet
even in tests.

Change-Id: I04d46b418e6e908185a4da694cf47dc3c5cc65f0
2019-12-17 13:45:51 +00:00
Egon Elbre
b04f9996c5 pkg/rpc: move test that needs testplanet
Move rpc test that uses testplanet into private/testplanet.

This ensures that rpc doesn't have the whole system as a dependency
making it easier to separate.

This unfortunately leaves pkg/rpc without specific tests, but
we would need to write new tests that only use the core packages.

Change-Id: I402ab3c2d50282af159c2ef3371d23b0997fef0a
2019-12-17 13:31:12 +00:00
VitaliiShpital
04d459a4a8 web/satellite: buckets page adaptation fixed
Change-Id: I41527680453b1e38ec383ce9c1446725b130dfa3
2019-12-17 13:00:05 +00:00
Michal Niewrzal
0e12c23431
uplinkc: add restrict_scope function (#3724) 2019-12-17 04:35:17 -08:00
littleskunk
08947e177d storagenode/garbagecollection: enable in production
Change-Id: I627b7a37ca4a85eb19936ca2c7ca907d7cc63f5b
2019-12-16 22:44:04 +00:00
Cameron Ayer
a4f9865b47 satellite: adds and enables cockroachdb compatibility for tests
Change-Id: I85a3ad8c3b9d7e15ea8675b6c55af0002933db57
2019-12-16 22:29:25 +00:00
Simon Guindon
a47d7ac89b scripts: Add script that filters postgres plaintext backup to cockroachdb compat.
Change-Id: I457e8f0566186fc76b7ae61db77c01153c3e1079
2019-12-16 22:11:16 +00:00
Simon Guindon
8242eecea6 Adding benchmarking script that reports response times.
Change-Id: Ide6f439849ec51cd41f491eb3ff00a7ad0f8a560
2019-12-16 16:34:01 -05:00
Isaac Hess
0008aebf80 pkg/rpc: Change drpcheader to save a packet
This changes when we write the drpcheader. Rather than making it its own
write to the connection, it now prepends the drpc header to the first
write on the connection (typically the tls handshake). This results in
one less packet being sent at the beginning of each drpc connection.

For an operation like uploading a file from uplink, this results in many
packets being dropped: one when communicating with the satellite, and
one for each communication with the storage nodes.

Change-Id: I7644b46e90ffa7acea73ac56831396307352ed7a
2019-12-16 13:33:39 -07:00
paul cannon
2f7465c294 private/dbutil: register "cockroach" as sql.DB driver
this will allow us to inspect the type of `db.Driver()` on *sql.DB
connections to correctly differentiate between pg and crdb conns.

as a bonus, this moves all concerns about when to replace "cockroach://"
with "postgres://" out of view, letting the thin shim "driver" take care
of that.

Change-Id: Ib24103ab7c508231e681f89a7321b623e4e125e9
2019-12-16 19:10:00 +00:00
paul cannon
b5ddfc6fa5 satellite/satellitedb: unexport satellitedb.DB
Backstory: I needed a better way to pass around information about the
underlying driver and implementation to all the various db-using things
in satellitedb (at least until some new "cockroach driver" support makes
it to DBX). After hitting a few dead ends, I decided I wanted to have a
type that could act like a *dbx.DB but which would also carry
information about the implementation, etc. Then I could pass around that
type to all the things in satellitedb that previously wanted *dbx.DB.

But then I realized that *satellitedb.DB was, essentially, exactly that
already.

One thing that might have kept *satellitedb.DB from being directly
usable was that embedding a *dbx.DB inside it would make a lot of dbx
methods publicly available on a *satellitedb.DB instance that previously
were nicely encapsulated and hidden. But after a quick look, I realized
that _nothing_ outside of satellite/satellitedb even needs to use
satellitedb.DB at all. It didn't even need to be exported, except for
some trivially-replaceable code in migrate_postgres_test.go. And once
I made it unexported, any concerns about exposing new methods on it were
entirely moot.

So I have here changed the exported *satellitedb.DB type into the
unexported *satellitedb.satelliteDB type, and I have changed all the
places here that wanted raw dbx.DB handles to use this new type instead.
Now they can just take a gander at the implementation member on it and
know all they need to know about the underlying database.

This will make it possible for some other pending code here to
differentiate between postgres and cockroach backends.

Change-Id: I27af99f8ae23b50782333da5277b553b34634edc
2019-12-16 19:09:30 +00:00
Egon Elbre
6fb9b4b231 ci: remove Jenkinsfile.drpc
Change-Id: I3efa15df271bad42a946f8eb3b3a0d5156cb6cf4
2019-12-16 18:46:06 +00:00
NikolaiYurchenko
6dc1249c0e web/satellite: billing banner changed
Change-Id: Ia5356f600fcb42ec31613ae1ef012d75ba16ab59
2019-12-16 20:19:42 +02:00
Jessica Grebenschikov
c5116cb2a0 satellitedb: fix migration cockroach test
Change-Id: Ie3b4a4b0795d156238d50a58078282cc0918a334
2019-12-16 18:02:31 +00:00
Vitalii Shpital
53d9bc4530
storagenode/notifications: db created (#3707) 2019-12-16 19:59:01 +02:00
NikolaiYurchenko
11db709066 web/satellite: project limits
Change-Id: Ia9c3ee9b5bc3dc1bc03e613c8715d299fce569dc
2019-12-16 19:35:15 +02:00
Moby von Briesen
ab777e823e do not update pointer for failed audits
Change-Id: If88dce8928db28d6f53c3dc771e14ea97aae9661
2019-12-16 10:50:54 -05:00
Michal Niewrzal
45f8bb5084 uplink/storage: remove unused Meta methods
Change-Id: Ia18609646efd6238dc5b9142146839ee2abf388e
2019-12-16 14:46:41 +01:00
littleskunk
c2ea75208f
storagenode/orderdb: fix db lock
Change-Id: Id1add0ba7ae1b20bd98099bd4d3aff0fcfdd90c9
2019-12-15 23:41:22 +01:00
Andrew Harding
cb89496569 storagenode/trust: wire up list into pool
- also updated ping chore to pick up trust changes
- fixed small typo in blueprint
- fixed flags for storj-sim
- wired up changes to testplanet

Change-Id: I02982f3a63a1b4150b82a009ee126b25ed51917d
2019-12-13 20:32:50 +00:00
Ivan Fraixedes
9ac2c4d815 cmd/segment-reaper: Add test cases from/to processSegment
Add randomized test cases for testing the observer processSegment method
when the observer has a time range (from and to) set and there are
objects with segments whose creation date is outside of this range.

Change-Id: Ieac82a21f278f0850b95275bfdd2e8a812cc57a5
2019-12-13 17:23:32 +00:00
Egon Elbre
fdd86d17ac lib/uplinkc: use SliceHeader instead of [1<<30]
Conversions such as will be invalid with Go 1.14

    (*[1<<30]byte)(unsafe.Pointer(data))

The recommended way is to use:

    *(*[]byte)(unsafe.Pointer(
        &reflect.SliceHeader{
            Data: uintptr(unsafe.Pointer(data)),
            Len: int(length),
            Cap: int(length),
        },
    ))

Also fixed a memory leak.

Change-Id: I1768b7b85505e6b57b49deb62a510474f1bf84c1
2019-12-13 16:49:41 +00:00
Ivan Fraixedes
770123de10 satellite/metainfo: Improve docs & use common funcs
* Use unexported existent method in logic that was duplicated in some
  exported methods.
* Log a forgotten internal error.
* Improve the documentation adding more and fixing some to fit to our
  code style conventions.

Change-Id: Ie6f8bc59f9089f92b8b0d1b4c09c2142c3f273f5
2019-12-13 16:00:35 +00:00