Commit Graph

143 Commits

Author SHA1 Message Date
JT Olio
9c5708da32 pkg/*: add monkit task to missing places (#2109) 2019-06-04 13:36:27 +02:00
ethanadams
16e3b77cf5
Enable Scopelint Linter (#2049)
* added scopelint and correcte issues found

* corrected scopelint issue

* made updates based on Ivan's suggestions

Most were around naming conventions
Some were false positives, but I kept them since the test.Run could eventually be changed to run in parallel, which could cause a bug
Others were false positives.  Added // nolint: scopelint
2019-05-29 09:30:16 -04:00
ethanadams
268dc6b7e4
Enable gocritic linter (#2051)
* first round cleanup based on go-critic

* more issues resolved for ifelsechain and unlambda checks

* updated from master and gocritic found a new ifElseChain issue

* disable appendAssign. i reports false positives

* re-enabled go-critic appendAssign and disabled lint check at code line level

* fixed go-critic lint error

* fixed // nolint add gocritic specifically
2019-05-29 09:14:25 -04:00
Maximillian von Briesen
c07162beef address potential divide by 0` (#2065) 2019-05-28 08:54:30 -06:00
Maximillian von Briesen
5a4ff2c855 add repair monkit stats (#2045)
* add repair monkit stats

* rename values, use meter instead of counter, use success threshold instead of repair threshold

* Counter -> Meter

* add repair segment size

* update names and use ratios for healthy before/after repair

* restart jenkins
2019-05-28 16:10:26 +02:00
Jeff Wendling
1bd52b9f90 server side macaroons (#1945)
What: Adds macaroon support to the server side

Why: So that api keys are now macaroons
2019-05-24 10:51:27 -06:00
littleskunk
878e79dd79 Bugfix: Repair increase success counter (#2015) 2019-05-21 14:23:00 +02:00
littleskunk
8e023b8bbf improve logging (#2004) 2019-05-20 16:18:16 +02:00
Bill Thorp
09065b8dec call GetRemotePieces once (#2003) 2019-05-20 15:22:03 +02:00
littleskunk
c974e0ce8a
Store repaired Segments and improve Repair Condition (#2000)
* repair no cutoff longtail

* commit repair pieces even if not hitting success threshold

* commit repair pieces even if not hitting success threshold

* remove useless condition

* better error message
2019-05-20 12:50:13 +02:00
littleskunk
d2c95c1d62 improve repair logs (#1999) 2019-05-20 10:37:46 +02:00
Egon Elbre
1103fa63c0
disable flaky TestSegmentStoreRepair (#1994) 2019-05-17 23:13:37 +03:00
aligeti
60cf1dafb0
repair segment reassess it missing pieces just before repair (#1939)
* repair segment reaccess it missing pieces just before repair to see if it actually needs repair
2019-05-16 09:49:10 -04:00
Egon Elbre
80400e7811
delete unused buckets/mocks (#1849) 2019-04-26 12:09:23 +03:00
Michal Niewrzal
fe3dfc1587
Move pointerdb.Service to satellite (#1826) 2019-04-25 10:46:32 +02:00
Kaloyan Raev
8fc5fe1d6f
Refactor pb.Node protobuf (#1785) 2019-04-22 12:07:50 +03:00
Philip Hutchins
a840948671
Moving failure for putting pieces to Debug as this is expected behavior (#1777) 2019-04-18 08:35:44 -04:00
paul cannon
0ae0de75bc use SerializableMeta to store bucket attributes (#1658) 2019-04-10 18:27:04 -04:00
Maximillian von Briesen
ed8fc126aa
add error in ec client Repair() if not enough successful nodes by end (#1722) 2019-04-09 18:46:43 -04:00
Cameron
62deae6a0a
fix bucketID bug in repair (#1719) 2019-04-09 13:20:00 -04:00
Maximillian von Briesen
bb3b4e4816 Data repair integration test (#1582) 2019-04-08 13:33:47 -04:00
Kaloyan Raev
bfdee78f05
Introduce NodeDossier type and cleanup overlay.DB interface (#1626)
Co-authored-by: Natalie Villasana <navillasa@gmail.com>
Co-authored-by: Bill Thorp <bill3000@hotmail.com>
2019-04-04 19:34:36 +03:00
Maximillian von Briesen
4d925f783c
Fix repairer unit test (#1557) 2019-04-03 19:00:25 -04:00
paul cannon
e4a70e3fac
plumb EncryptionScheme, RedundancyScheme through to buckets (#1638)
We want to use those fields in the bucket-level Pointer objects as
bucket defaults, but we need to be able to get at them first.

I don't see any strong reason not to make these available, except
that it was kind of a pain.
2019-04-02 15:15:31 -06:00
Michal Niewrzal
f80750693c Store bandwidth from orders on satellite (#1586) 2019-04-01 16:14:58 -04:00
Egon Elbre
be06fdfd6c Create orders.Service (#1593) 2019-03-28 22:09:23 +02:00
Egon Elbre
94e79eda6d
remove overlay endpoint (#1521) 2019-03-23 10:06:11 +02:00
Egon Elbre
694b6dc1da
make tests run faster (#1553) 2019-03-22 15:14:17 +02:00
Maximillian von Briesen
db64d6590b Add repairer tests (#1494) 2019-03-21 16:26:56 +02:00
Michal Niewrzal
d7feafe56b Move psserver tests (#1522) 2019-03-20 23:12:00 +02:00
Kaloyan Raev
d057efb05e
Add Repair method to ECClient (#1509) 2019-03-19 15:14:59 +02:00
Egon Elbre
117edec54c
Add serial number type (#1508) 2019-03-18 15:08:24 +02:00
Egon Elbre
05d148aeb5
Storage node and upload/download protocol refactor (#1422)
refactor storage node server
refactor upload and download protocol
2019-03-18 12:55:06 +02:00
Kaloyan Raev
5fa7a4a7c6 Ensure ECClient upload timer is stopped when no more status is expected (#1397)
This change ensures that the upload timer of ECClient is always stopped after no more status is expected from uploaded pieces. It also ensures that the "Timer expired" message will be logged only if the context is not already cancelled.

This is to avoid confusing logs where a "Timer expired" message is logged significantly later and mixes with similar messages logged from the upload of the next file segments.
2019-03-04 14:48:13 +01:00
Michal Niewrzal
4122c98cb7
Validate piece hash on satellite (#1359)
The satellite receives pieces signed hashes in Pointer. If signed hash cannot be validated then piece is removed from Pointer and not saved in DB.
2019-02-28 15:14:54 +01:00
Michal Niewrzal
81408a3c9e
Use SignedHash on client/uplink side (#1354)
* psclient receives storage node hash and compare it to own hash for verification
* uplink sends delete request when hashes don't match
* valid hashes are propagated up to segments.Store for future sending to satellite
2019-02-25 16:57:54 +01:00
Bill Thorp
373b301736
BWA aliases (#1333)
aliased RBAs and PBAs
2019-02-22 16:17:35 -05:00
Natalie Villasana
c3d3f41d30 removes some SignedMessage use (#1258)
Removes most instances of pb.SignedMessage (there's more to take out but they shouldn't hurt anyone as is).

There used to be places in psserver where a PieceID was hmac'd with the SatelliteID, which was gotten from a SignedMessage. This PR makes it so some functions access the SatelliteID from the Payer Bandwidth Allocation instead.

This requires passing a SatelliteID into psserver functions where they weren't before, so the following proto messages have been changed:

 * PieceId - satellite_id field added
   This is so the psserver.Piece function has access to the SatelliteID when it needs to get the namespaced pieceID.
   This proto message should probably be renamed to PieceRequest, or a new PieceRequest message should be created so this isn't misnamed.

 * PieceDelete - satellite_id field added
   This is so the psserver.Delete function has access to the SatelliteID when receiving a request to Delete.
2019-02-19 23:36:08 -06:00
JT Olio
2a59679766 pkg/transport: require tls configuration for dialing (#1286)
* separate TLS options from server options (because we need them for dialing too)
* stop creating transports in multiple places
* ensure that we actually check revocation, whitelists, certificate signing, etc, for all connections.
2019-02-11 13:17:32 +02:00
paul cannon
c35b93766d
Unite all cryptographic signing and verifying (#1244)
this change removes the cryptopasta dependency.

a couple possible sources of problem with this change:

 * the encoding used for ECDSA signatures on SignedMessage has changed.
   the encoding employed by cryptopasta was workable, but not the same
   as the encoding used for such signatures in the rest of the world
   (most particularly, on ECDSA signatures in X.509 certificates). I
   think we'll be best served by using one ECDSA signature encoding from
   here on, but if we need to use the old encoding for backwards
   compatibility with existing nodes, that can be arranged.

 * since there's already a breaking change in SignedMessage, I changed
   it to send and receive public keys in raw PKIX format, instead of
   PEM. PEM just adds unhelpful overhead for this case.
2019-02-07 14:39:20 -06:00
Kaloyan Raev
3c73d3a33c Fixes intermitent failures in storage/ec TestPut (#1239) 2019-02-05 18:49:52 +02:00
Kaloyan Raev
dd76829d10
Improve logic for cutting the long tail during upload (#909) 2019-02-05 12:54:25 +02:00
Kaloyan Raev
a90aa18582 Fix panic in PointerDB.Get (#1209)
* Fix panic in PointerDB.Get

* Allow nil nodes, check if v is nil instead

* More checks like this

* Remove unnecessary check

* More checks
2019-02-01 10:55:47 -07:00
Egon Elbre
d5346982c2
Delete provider package (#1177) 2019-01-30 22:47:21 +02:00
Egon Elbre
6132ce86b7
Remove utils.LogClose (#1169) 2019-01-29 22:42:27 +02:00
Bill Thorp
60946c2024
make bandwidth agreements sensible: without []byte's (#1152)
removed []byte's from bandwidth agreement protocol buffers
2019-01-28 14:45:25 -05:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Egon Elbre
99d3b7a3c8
Fix import grouping (#1111) 2019-01-22 17:48:23 +02:00
Bill Thorp
fb4a11ebb0
Differentiate GET and PUT operations in the tally service (#965)
* draft of new bandwidth agreement types
* updated storagenode report for new types
* use correct pba types
2019-01-10 06:41:57 -05:00
Matt Robinson
c0e6b62708
Test all-in-one (#900)
* Add test for aio

* Don't trust the user to have images built for a version

* Make travis run the aio test

* Add missing values to docker-compose, sort some things, consider the gateway image

* today's changes

* config changed, again

* more fixes

* Expose satellite port on localhost:7778

* Add retries and a timeout around the big-testfile test in AIO

* Another config value changed

* Make this error message a little more useful

* Fix nil condition
2019-01-03 14:54:27 -05:00