Commit Graph

3250 Commits

Author SHA1 Message Date
Egon Elbre
f41d440944 all: reduce number of log messages
Remove starting up messages from peers. We expect all of them to start,
if they don't, then they should return an error why they don't start.
The only informative message is when a service is disabled.

When doing initial database setup then each migration step isn't
informative, hence print only a single line with the final version.

Also use shorter log scopes.

Change-Id: Ic8b61411df2eeae2a36d600a0c2fbc97a84a5b93
2020-01-06 19:03:46 +00:00
paul cannon
a33734bee7 satellite/satellitedb/dbx: add cockroach driver type
Change-Id: I7a0da6e066c67a521fc1b23b085ab8554eee0d4c
2020-01-06 18:01:03 +00:00
Moby von Briesen
ea84af578b scripts/tests/rollingupgrade: create new test files for final upload
stage

The test-versions script no longer uses the `testfiles` directory, which
the final upload for the rolling-upgrade script depended on. This change
creates and populates a `testfiles` diirectory during the final upload
stage of the rolling upgrade test.

Change-Id: Iabeccbadc55a8c85a1febbd5eb4e7d889a57a8dc
2020-01-06 12:31:12 -05:00
Egon Elbre
91947311f5 ci: always try to pull latest image
Change-Id: Ic1422a96705f8e66876f5c724060d7c389c5da3d
2020-01-06 15:28:04 +00:00
Yingrong Zhao
07a1702f41 scripts/tests/rollingupgrade: fix test-versions.sh path referrence
Change-Id: I5c696e5d38c087c50f025796e2f48876883d0f4a
2020-01-04 19:42:15 -05:00
Yingrong Zhao
71c5c2213f scripts/tests/testversions: make binary installation and upload/download running in parallel
Change-Id: I16d87f7e16e2daf30e4d7ee5490b76c175b06930
2020-01-04 16:39:45 +00:00
Simon Guindon
80b41af8f1 satellite/metainfo: Fixed bug that discarded context cancellation errors
When the context was being cancelled the error was being discarded within the rate limiting error handling which caused tests to fail.
Change-Id: I5c6458c16da09a11531233ea0ee80d914969cb3f
2020-01-03 22:48:12 -05:00
Ivan Fraixedes
fc4ea28695 satellite/metainfo: Return ErrObjectNotFound
deletePointer must return an ErrObjectNotFound rather than a rpc status
error NotFound because the callers must distinguish such error if it
comes from the getPointer or from the UnsynchronizedDelete.

Change-Id: I68b4e45a2765e63b73bf85c2c39a5fc0198373f6
2020-01-03 22:01:29 +00:00
Jeff Wendling
29fe206b9a satellite/gc: add timeout to retain requests
We don't want slowloris nodes to be able to indefinitely block
up the satellite, so add a timeout. Some monitoring inspection
showed the largest success times being on the order of 30s, so
a 1min timeout should be sufficient to kill the misbehaving nodes.

Change-Id: I5e2c3480a15f6304e37262d0a4d30d07eae99bb3
2020-01-03 21:46:46 +00:00
Jeff Wendling
828d0b9984 pkg/server: set TCP_USER_TIMEOUT and monitor leaked conns
Go will, by default, set tcp keep alives on sockets. But
the kernel does not send keep alives to sockets that have
a non-empty send queue. That can cause connections that
hang forever.

So we set TCP_USER_TIMEOUT on all of the sockets as well.
That option will close any connection that has not received
an ack for any sent data (keep alive or otherwise) in the
configured time period. This places an upper bound on the
amount of time a socket can be stuck due to a client not
acknowleding data.

See https://blog.cloudflare.com/when-tcp-sockets-refuse-to-die/
for more information on what these options do and how they
interact.

Additionally, make sure that we close every connection coming
from the listeners by wrapping them in a type with a finalizer
that closes the connection, much like the os package does for
file handles. It monitors if a connection was closed due to a
finalizer so that we can go and look for the bug if we ever
see a non-zero value.

Change-Id: Idc6c0564224b8dc2e4c9d769e80374ed1fe8cce0
2020-01-03 21:31:09 +00:00
Cameron Ayer
0038abb51b private/testplanet: use redis for live accounting
storing live accounting in memory will not work, as the core and api each create
their own instance. Using redis will allow each to access the same store

Change-Id: I4c8250b579d7b6b6d8991bc890894573626effe6
2020-01-03 21:04:50 +00:00
Simon Guindon
e1e7cebe49 satellite/metainfo: added rate limiting support to the metainfo loop.
As per discussed we decided to rate limit how fast we iterate through
the metainfo database in the metainfo loop. This puts in place a
mechanism for rate limiting and burst limiting if need be in the future.

The default for this rate limiting is still no limits so it stays the
same as our previous functionality.

Change-Id: I950f7192962b0e49f082d2c4284e2d52b0a925c7
2020-01-03 15:00:29 -05:00
Ethan
05b406e992 satellite:{downtime,overlay}: Implement offline node detection chore
https://storjlabs.atlassian.net/browse/V3-3398

Change-Id: I598c3bad819026377d1d113c099dc9bba8b02742
2020-01-03 17:10:03 +00:00
Matt Robinson
5aac77c2a1
Slack the build team instead of everyone (#3739)
Change-Id: If55105fa99ebb32fa84dd595258f83838d3150f1
2020-01-03 11:54:01 -05:00
Yaroslav
389567fc9e satellite/console: add credit card charges to billing history
Change-Id: I82a08c42c01086dc7fb9508da5c6c0baa2438124
2020-01-03 17:34:59 +02:00
Bryan White
325790703f installer/windows: batch file improvements (#3441) 2020-01-03 15:28:04 +02:00
Yaroslav
0cc7056a9a satellite/console: convert dates to UTC in advanced usage reports
Change-Id: I5c72c869533a7613bffdb8077fdedff2a4e203d0
2020-01-03 14:17:37 +02:00
Michal Niewrzal
38eff60698 satellite/metainfo: adjust old API test to new API
We are missing some tests for new Metainfo API that we have for old API.
This is first change to adjust old tests to new API.

Change-Id: Ie2b16bf85de8633662f952e863dbf3d409d801d9
2020-01-03 11:05:14 +01:00
Moby von Briesen
e34ac3ef3a ci,scripts/tests/rolling-upgrade: run rolling upgrade test on private jenkins
Change-Id: Ic1c9f7539ee0ac371bcb856bdbcac2ff6c0ccc65
2020-01-02 16:27:41 -05:00
Moby von Briesen
aecea820fc scripts: add rolling upgrade test script
Change-Id: Ibf79c8e40da54520ce17e2e1f66124c117b32b53
2020-01-02 13:38:56 -05:00
Ethan
8859c36234 satellite/{downtime,contact}: Add CheckNodeAvailability for use within the downtime tracking chores.
https://storjlabs.atlassian.net/browse/V3-2545

Change-Id: I1dd54a0c77cb4905bb1f350beeb82c6f7700ee70
2020-01-02 18:24:11 +00:00
Egon Elbre
6098f606a2 lib/uplink: move to uplink
Setup aliases in lib/uplink and set uplink as the default.

Change-Id: Ic06b5f3d59fe402faaed2143fdf4b2314e3d06b9
2020-01-02 17:56:16 +00:00
Moby von Briesen
ff74b44c5f satellite/overlay: Add ability for overlay to get offline nodes ordered by last checked time
This is required for the downtime tracking service: https://storjlabs.atlassian.net/browse/V3-2545

Change-Id: I286cdc07d802393948eb10c25c45ba78cc3ceafc
2020-01-02 16:39:38 +00:00
Ivan Fraixedes
c3b58f1656 satellte/metainfo: Make BeginDeleteObject to delete pieces
For improving the deletion performance we are shifting the
responsibility to delete the pieces of the object from Uplink to the
Satellite.

BeginDeleteObject was the first call to return the stream ID which was
used for after retrieving the list of segments and then get addressed
order limits for deleting the pieces (of each segment) from the storage
nodes.

Now we want the Satellite deletes the pieces of all the object segments
from the storage nodes hence we don't need anymore to have several
network round trips between the Uplink and the Satellite because the
Satellite can delete all of them in the initial BegingDeleteObject
request.

satellite/metainfo.ListSegments has been changed to return 0 items if
the pointer of the last segment of an object is not found because we
need to preserve the backward compatibility with Uplinks that won't be
updated to the last release and they rely on listing the segments after
calling BeginDeleteObject for retrieving the addressed order limits
to contact the storage nodes to delete the pieces.

Change-Id: I5f99ecf27d62d65b0a062936b9b17581ef692af0
2020-01-02 15:53:59 +00:00
Bryan White
71ea2b0dc0 uplinkc: object test fix
require object creation time to be less than 60 seconds ago instead of less than or equal to 2

Change-Id: I3ec348e12ead1144524509092265b2b4d15109e3
2020-01-02 14:34:28 +00:00
Ivan Fraixedes
105a9a4848 web/satellite/tests/unit/common: Fix hardcoded year check
Don't use a hardcoded year number for tests which depends on the current
year for avoiding that they don't pass when a new year starts.

Change-Id: I52b6248f6c3ddd2df89a4b04caf3a228b0d564e0
2020-01-02 12:38:10 +00:00
Egon Elbre
3e873158f4 ci: increase cockroach max-memory to reduce flakiness
Change-Id: I88343c8668ee57abfdc30f2d8d4772c61e636b78
2020-01-02 11:07:00 +00:00
Egon Elbre
3528c56c6f satellite/satellitedb/satellitedbtest: skip unconfigured db
Change-Id: Ib6ea58208ef19410146845e82eb08724888e85ae
2020-01-02 10:51:59 +00:00
Egon Elbre
e03d3fb577 uplink: move configs to cmd/uplink/cmd
Change-Id: Ifc1d3440dcef429c2a6142c16f3e991abf49f1d2
2020-01-02 09:40:57 +00:00
Egon Elbre
2680bae88c private/testplanet: remove dependency to uplink
Remove direct dependency on uplink.RSConfig, this simplifies
moving the config file without introducing weird dependencies.

Change-Id: I7fd2a145401e0205d7047631df9d2810241efeec
2020-01-02 09:40:46 +00:00
Jeff Wendling
a35eb8027e lib/uplinkc: do some checks around conversions and allocations
ensure that every integer conversion fits into the destination
type, and for any unchecked ones, annotate why they are safe.
additionally, any integers we pass into slice headers need to
check that they are not negative.

all of our allocations should check for allocation failure and
return an error if there was a problem. even though an allocation
just failed, we don't pre-allocate the failure because we don't
want the callers to free it like they have to with the other
errors we return. maybe we should just panic.

Change-Id: Id4dfad802d312d35e565041efc9872453c90d793
2020-01-01 17:36:34 +00:00
Natalie Ventura Villasana
aa3e183c2e
satellite/gracefulexit: add ge eligibility check
Adds check to see if storage nodes are eligible to initiate
graceful exit, by checking their CreatedAt date and seeing if
their "age" is greater than the new config value:
NodeMinAgeInMonths
The default for this value is 6 months for now.

https://storjlabs.atlassian.net/browse/V3-3357

Change-Id: Ib807ab8987ddb5a38a27a83886490f73fe8c5816
2019-12-31 09:31:58 -05:00
Ivan Fraixedes
7266155375 satellite/metainfo: List segments manually limit check
The endpoint listSegmentsManually method misses a check for the limit
parameter, otherwise it can return inconsistent results when it's 0 or
negative.

When 0 or negative, without the check, it returns no segments but also
that there isn't more segments and that isn't correct.

The function is only called from the Endpoint.ListSegments method and
the function cares to ensure that limit is always greater than 0, but if
the method doesn't check that a new future caller could misuse it and
provoke a bug.

Additionally:

* Documentation for the modified function has been written
* The part of the function that repeated the logic of the
  Endpoint.getPointer method has been removed for using that method.
* Added logging before returning an internal error in
  Endpoint.getPointer.

Change-Id: I5c4f0db2292da0162db6b7d63553895808d0925a
2019-12-31 07:44:46 +00:00
Moby von Briesen
bb3baf5a4e satellite/satellitedb: Add nodes_offline_times table for downtime tracking
Change-Id: If6b80fe0a20d88cedacaf4b76b75aa21d0af2465
2019-12-30 15:45:02 -05:00
Stefan Benten
758fe35aba
storagenode/orders: adding jitter to sending (#3725) 2019-12-30 21:35:26 +01:00
Stefan Benten
82ee13b00b
Update Coverage URL (#3737) 2019-12-30 21:21:24 +01:00
NikolaiYurchenko
e99bdac944 web/satellite: ux bugs fixes
Change-Id: I8d7ff98fd23f7a653857969e57b39c4aba464665
2019-12-28 14:06:38 +02:00
VitaliiShpital
090603b8e0 web/storagenode: wording on DQ info message updated
Change-Id: I141de181c726dffd61a08b64c19c3d9a2d3a17b1
2019-12-27 14:37:48 +00:00
Ivan Fraixedes
059537c16e satellite/metainfo: Add new TODOs & remove old ones
Do some cleanup for adding new identified TODOs (associated with ticket
https://storjlabs.atlassian.net/browse/V3-3406) and remove an old one.

Change-Id: I5d20dbe1c4dee0a8279e08b05b907f4cc9dba278
2019-12-27 13:16:09 +00:00
Egon Elbre
6615ecc9b6 common: separate repository
Change-Id: Ibb89c42060450e3839481a7e495bbe3ad940610a
2019-12-27 14:11:15 +02:00
Fadila
115b8b0fc8 storagenode/piecestore: delete several pieces in a single request
This is part of the deletion performance improvement.
See https://storjlabs.atlassian.net/browse/V3-3349

Change-Id: Idcd83a302f2bd5cc3299e1a4195a7e177f452599
2019-12-27 10:58:04 +00:00
Kaloyan Raev
7df3c9efc3 cmd/uplink: use arguments in share command as allowed path prefixes
Fixes Least Authority Issue F:
https://storjlabs.atlassian.net/browse/V3-3409

If the --allowed-path-prefix flag is not set to the `share` command, any
command arguments will be used as allowed path prefixes.

This patch also improves the output of the `share` command to print the
state of all restrictions, so users can confirm they match their
intention.

Change-Id: Id1b4df20b182d3fe04cb2196feea090975fce8b4
2019-12-27 10:02:33 +00:00
Isaac Hess
7d1e28ea30 storagenode: Include trash space when calculating space used
This commit adds functionality to include the space used in the trash
directory when calculating available space on the node.

It also includes this trash value in the space used cache, with methods
to keep the cache up-to-date as files are trashed, restored, and
emptied.

As part of the commit, the RestoreTrash and EmptyTrash methods have
slightly changed signatures. RestoreTrash now also returns the keys that
were restored, while EmptyTrash also returns the total disk space
recovered. Each of these changes makes it possible to keep the cache
up-to-date and know how much space is being used/recovered.

Also changed is the signature of PieceStoreAccess.ContentSize method.
Previously this method returns only the content size of the blob,
removing the size of any header data. This method has been renamed
`Size` and returns both the full disk size and content size of the blob.
This allows us to only stat the file once, and in some instances (i.e.
cache) knowing the full file size is useful.

Note: This commit simply adds the trash size data to the piece size data
we were already collecting. The piece size data is not accurate for all
use-cases (e.g. because it does not contain piece header data); however,
this commit does not fix that problem. Now that the ContentSize (Size)
method returns the full size of the file, it should be easier to fix
this problem in a future commit.

Change-Id: I4a6cae09e262c8452a618116d1dc66b687f59f85
2019-12-23 19:07:03 -07:00
Egon Elbre
3849b9a21a pkr/rpc: remove RawConn
Change-Id: I61bc10b82f178a16f27279b85b627553d122c174
2019-12-22 18:01:01 +02:00
Egon Elbre
d55288cf68 pkg/rpc: replace methods with direct calls to pb
Change-Id: I8bd015d8d316a2c12c1daceca1d9fd257f6f57bc
2019-12-22 17:12:43 +02:00
Egon Elbre
006baa9ca6 pkg/rpc: remove drpc aliases
We need to split up pb package, which means we cannot have a core package
that depends on them.

Change-Id: I7f4f6fd82f89a51a9b2ad08bf2b1207253b8a215
2019-12-22 16:58:08 +02:00
ccase
6f1eaef8d4 cmd/uplink: Pass -- in tests to avoid treating generated arg strings as flags.
Change-Id: I41c50b9f645b57ddc8832b0fc92f1c6bfaf2de8d
2019-12-21 15:15:15 +00:00
Egon Elbre
acb4435a67 satellite/satellitedb: improve Cockroach migrate test
Load schemas in parallel instead of one-by-one.

Optimizes from 2m30s to 1m15s.

Change-Id: I0bf6381a0ae99b44271fe55d4ee658683064c097
2019-12-21 10:58:43 +00:00
Egon Elbre
31fbdcc8f7 pkg/encryption: better EncodingDecodingStress
The test case wasn't testing all the combinations.

But, testing all 3 byte combinations would take too long,
instead test special values and +- 1 of them, with some
additional noise characters.

Change-Id: If53ff25863a1f27c534922bd399fbbbdfefda441
2019-12-21 12:45:24 +02:00
Egon Elbre
98243360d6 cmd/storagenode-updater: faster update test
flate compression with default settings plays very poorly
together with race, causing test to take a significant amount
of time.

Use pass-through compression to avoid the issue.

Improves test from 2m45s to 17s.

Change-Id: Iadf1381c538736d48e018164697bdfd3356e24b8
2019-12-21 02:59:31 +02:00