Commit Graph

62 Commits

Author SHA1 Message Date
littleskunk
8e023b8bbf improve logging (#2004) 2019-05-20 16:18:16 +02:00
littleskunk
c974e0ce8a
Store repaired Segments and improve Repair Condition (#2000)
* repair no cutoff longtail

* commit repair pieces even if not hitting success threshold

* commit repair pieces even if not hitting success threshold

* remove useless condition

* better error message
2019-05-20 12:50:13 +02:00
littleskunk
d2c95c1d62 improve repair logs (#1999) 2019-05-20 10:37:46 +02:00
Kaloyan Raev
8fc5fe1d6f
Refactor pb.Node protobuf (#1785) 2019-04-22 12:07:50 +03:00
Philip Hutchins
a840948671
Moving failure for putting pieces to Debug as this is expected behavior (#1777) 2019-04-18 08:35:44 -04:00
paul cannon
0ae0de75bc use SerializableMeta to store bucket attributes (#1658) 2019-04-10 18:27:04 -04:00
Maximillian von Briesen
ed8fc126aa
add error in ec client Repair() if not enough successful nodes by end (#1722) 2019-04-09 18:46:43 -04:00
Michal Niewrzal
d7feafe56b Move psserver tests (#1522) 2019-03-20 23:12:00 +02:00
Kaloyan Raev
d057efb05e
Add Repair method to ECClient (#1509) 2019-03-19 15:14:59 +02:00
Egon Elbre
117edec54c
Add serial number type (#1508) 2019-03-18 15:08:24 +02:00
Egon Elbre
05d148aeb5
Storage node and upload/download protocol refactor (#1422)
refactor storage node server
refactor upload and download protocol
2019-03-18 12:55:06 +02:00
Kaloyan Raev
5fa7a4a7c6 Ensure ECClient upload timer is stopped when no more status is expected (#1397)
This change ensures that the upload timer of ECClient is always stopped after no more status is expected from uploaded pieces. It also ensures that the "Timer expired" message will be logged only if the context is not already cancelled.

This is to avoid confusing logs where a "Timer expired" message is logged significantly later and mixes with similar messages logged from the upload of the next file segments.
2019-03-04 14:48:13 +01:00
Michal Niewrzal
81408a3c9e
Use SignedHash on client/uplink side (#1354)
* psclient receives storage node hash and compare it to own hash for verification
* uplink sends delete request when hashes don't match
* valid hashes are propagated up to segments.Store for future sending to satellite
2019-02-25 16:57:54 +01:00
Bill Thorp
373b301736
BWA aliases (#1333)
aliased RBAs and PBAs
2019-02-22 16:17:35 -05:00
Natalie Villasana
c3d3f41d30 removes some SignedMessage use (#1258)
Removes most instances of pb.SignedMessage (there's more to take out but they shouldn't hurt anyone as is).

There used to be places in psserver where a PieceID was hmac'd with the SatelliteID, which was gotten from a SignedMessage. This PR makes it so some functions access the SatelliteID from the Payer Bandwidth Allocation instead.

This requires passing a SatelliteID into psserver functions where they weren't before, so the following proto messages have been changed:

 * PieceId - satellite_id field added
   This is so the psserver.Piece function has access to the SatelliteID when it needs to get the namespaced pieceID.
   This proto message should probably be renamed to PieceRequest, or a new PieceRequest message should be created so this isn't misnamed.

 * PieceDelete - satellite_id field added
   This is so the psserver.Delete function has access to the SatelliteID when receiving a request to Delete.
2019-02-19 23:36:08 -06:00
JT Olio
2a59679766 pkg/transport: require tls configuration for dialing (#1286)
* separate TLS options from server options (because we need them for dialing too)
* stop creating transports in multiple places
* ensure that we actually check revocation, whitelists, certificate signing, etc, for all connections.
2019-02-11 13:17:32 +02:00
paul cannon
c35b93766d
Unite all cryptographic signing and verifying (#1244)
this change removes the cryptopasta dependency.

a couple possible sources of problem with this change:

 * the encoding used for ECDSA signatures on SignedMessage has changed.
   the encoding employed by cryptopasta was workable, but not the same
   as the encoding used for such signatures in the rest of the world
   (most particularly, on ECDSA signatures in X.509 certificates). I
   think we'll be best served by using one ECDSA signature encoding from
   here on, but if we need to use the old encoding for backwards
   compatibility with existing nodes, that can be arranged.

 * since there's already a breaking change in SignedMessage, I changed
   it to send and receive public keys in raw PKIX format, instead of
   PEM. PEM just adds unhelpful overhead for this case.
2019-02-07 14:39:20 -06:00
Kaloyan Raev
3c73d3a33c Fixes intermitent failures in storage/ec TestPut (#1239) 2019-02-05 18:49:52 +02:00
Kaloyan Raev
dd76829d10
Improve logic for cutting the long tail during upload (#909) 2019-02-05 12:54:25 +02:00
Egon Elbre
d5346982c2
Delete provider package (#1177) 2019-01-30 22:47:21 +02:00
Egon Elbre
6132ce86b7
Remove utils.LogClose (#1169) 2019-01-29 22:42:27 +02:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Egon Elbre
99d3b7a3c8
Fix import grouping (#1111) 2019-01-22 17:48:23 +02:00
Matt Robinson
c0e6b62708
Test all-in-one (#900)
* Add test for aio

* Don't trust the user to have images built for a version

* Make travis run the aio test

* Add missing values to docker-compose, sort some things, consider the gateway image

* today's changes

* config changed, again

* more fixes

* Expose satellite port on localhost:7778

* Add retries and a timeout around the big-testfile test in AIO

* Another config value changed

* Make this error message a little more useful

* Fix nil condition
2019-01-03 14:54:27 -05:00
Jennifer Li Johnson
a2fa5c4c5a Proper NodeType Handling (#873)
* adds enums to nodetype

* updating nodetype todos

* ran pb updates

* reorder nodetypes

* adding checks

* wip

* wip

* wip

* bug in test-captplanet

* wip

* add values to storagenode, satellite, captplanet binaries

* Cleanup

* more cleanup

* wip

* lint

* lint

* wip

* fixes bug

* regenerate protos

Change-Id: Id270212e8c7479e52641058042cf23b5317ab773

* limit node type changes to kademlia

Change-Id: I9c1a6cc4a79e05086627f0fdeb5028c62ce754f4

* dpanic

Change-Id: Id952a2ad13c807ebaea0ec0a875405e267d81c3e

* review comments

Change-Id: I7f9b77ef22779dd012fd490375b136014f51f834
2019-01-02 11:47:34 -07:00
Kaloyan Raev
252da15f0d
Randomize node selection during GETs (#827) 2018-12-11 18:05:14 +02:00
Egon Elbre
1e4556f88a
Fix import groupings (#739) 2018-11-30 15:40:13 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Bryan White
ee62e2a9d8
Use transport client and cleanup all the clients (#574)
* wip

* linter fixes

* linter fixes

* test fixes

* linter fixes

* fix merge + restructure piecestore packages

* review feedback

* linter fixes

* linter fixes

* remove unnecessary aliases to piecestore

* more merge fixing
2018-11-06 18:49:17 +01:00
aligeti
771c7e6ef9
download with specific number of nodes V3-353 (#564)
* download with specific number of nodes

* changes per code review comments
2018-11-02 11:22:01 -04:00
Alexander Leitner
7e4149138a
Piecestore proto change (#556)
* add action and storage node id to piecestore.proto. renamed payer and renter to satellite_id and uplink_id
2018-10-30 15:03:41 -04:00
Michal Niewrzal
0d03f2fbb5
Send bandwidth alloc from satellite to storage node (#538)
* Send bandwidth alloc from satellite to storage node

* Remove unecessary nil checks

* set Renter field

* fix tests

* goimports

* Update README.md

* Update README.md

* Update README.md

* Update README.md (#550)

* Skip flaky TestPing (#552)
2018-10-30 17:24:46 +01:00
Egon Elbre
21026b35f5
Fix error message (#502) 2018-10-18 17:48:57 +03:00
Michal Niewrzal
f9248c21d4
Satellite verification on storage node (#469)
* Satellite verification on storage node

* fix formatting

* fix formatting

* rename SignatureAuth to SignedMessage

* fixes after review

* fix linter errors

* improve errors handling

* remove SignedMessageProvider

* fix liter errors

* params changed to authorization, signed message in audit, minor fixes

* fix formatting
2018-10-17 13:40:11 +02:00
Cameron
118e9bec64
handle nil nodes in ec Put (#454)
* handle nil nodes in ec Put

* read and discard readers for nil nodes

* test 2 nil nodes, unique wont return false with nil nodes

* Discard reader data for nil nodes

* edit control flow
2018-10-16 11:53:25 -04:00
Egon Elbre
2dbada73ab
Separate storj.io packages in imports (#481)
Format with goimports -w -local storj.io
Add flag to golang lint
Remove commented out imports as they mess with goimports
2018-10-16 14:43:44 +03:00
Stefan Benten
586ed1b58d
Improve path handling + filepath handling framework (#423)
* Initial Layout

* Commit to test File Handling OS independed

* Hide struct properties to prevent manual interaction

* Fix Linting Errors

* 1st Working Windows Version

* Add missing Error Handling

* Fix Linting Errors

* Remove dependencies

* Further Improvements

* Remove commented code

* Improve comments and error messages

* No pointers to FPath

* Improve comment

* Do not filepath.ToSlash URL path

* Extract helper functions for parsing local path and Storj path

* Minor Improvements based on PR Comments

* Fix Linting Error and make Regex private

* Improve Layout

* Rework FPath and add tests

* Add more tests cases for windows

* Use for-loop instead of goto

* Use FPath in all uplink commands

* Add guard checks

* Add Test Cases and add comments
2018-10-15 16:10:20 +02:00
Egon Elbre
caecb2801d
Cleanups related to transport (#475)
* Remove DialUnauthenticated method
* Use more specific node transport enum
* Formatting
* Rename Provider.g -> Provider.grpc
* Fix naming in dialer
2018-10-15 15:04:21 +03:00
aligeti
cf329b1f37
Handle CTRL+C and clean up the partial segments and pieces (#381)
* merged the lasted master changes

* debug working of handling ctrl+c

* Handling of clean up of partially uploaded segments and pieces

* code cleanup per code comment

* updates based on code review comments
2018-10-04 09:52:12 -04:00
Kaloyan Raev
f94c584768
Rename: DecodedBlockSize>StripeSize, EncodedBlockSize>ErasureShareSize (#392) 2018-09-27 14:52:18 +03:00
Kaloyan Raev
bc0f697929
Pointer stores only nodes with successfully uploaded pieces (#390) 2018-09-27 13:45:19 +03:00
Egon Elbre
2019803c5e
Use RepairThreshold naming consistently (#385) 2018-09-26 17:23:33 +03:00
aligeti
97cbacb299
storage node quick check and startup validation (#368)
* storage node quick check and startup validation

*  rearranged the startup validation and quick check logic

* travis lint warning fixes

* travis lint warning fixes

* travis lint warning fixes

* code changes per review comments

* code clean dev debug info

* travis lint wranings

* code changes per code review comments

* code changes per code review comments

* code update per review

* sqlite SUM is having issue when getting the SUM of an empty column; filepath was checking a directory that doesn't exist when starting server; Example updated to print allocated and used space

* storage node quick check and startup validation

*  rearranged the startup validation and quick check logic

* travis lint warning fixes

* travis lint warning fixes

* travis lint warning fixes

* code changes per review comments

* code clean dev debug info

* travis lint wranings

* code changes per code review comments

* code changes per code review comments

* code update per review

* no file or directory error

* Updated mock PSClient
2018-09-25 12:49:55 -04:00
Egon Elbre
b6b6111173
Flatten proto definitions into a single package (#360)
* protos: move streams to pb
* protos: move overlay to pb
* protos: move pointerdb to pb
* protos: move piecestore to pb
* fix statdb import naming
2018-09-18 07:39:06 +03:00
Kaloyan Raev
ee5f32b0f7
ConcatRanger closes segment readers as soon as data is read (#349)
* ConcatRanger closes a segment readers as soon as data is read

* Fix linter errors

* Fix data race

* Nicer call to CombineErrors
2018-09-14 10:10:43 -04:00
Alexander Leitner
8f07c4d746
Keep track of used space (#338)
* Keep track of space at all times

* Create index on expires

* Updated mock psclient
2018-09-11 08:40:45 -04:00
Jennifer Li Johnson
d0f87f0de1
Overlay Bulk Lookup (#279)
* implements bulk node lookup from overlay cache
2018-09-11 00:52:14 -04:00
Michal Niewrzal
dfc52e4394 [WIP] Piece Rangers should connect lazily to storage nodes (second approach) (#321)
* Piece Rangers should connect lazily to storage nodes

* code reorganization

* fixed unit tests

* remove if
2018-09-08 16:52:19 +03:00
JT Olio
a4d1070d68
storage/ec: fix some Get socket leaks (#308)
* storage/ec: fix some Get socket leaks

* fix linters
2018-09-05 10:08:39 -07:00
Alexander Leitner
899e1e68f1
Add functions for signing and verifying during bandwidth exchange (#246)
* Added initial functions for signing and verifying

* whoops

* Get client up to speed

* Added initial functions for signing and verifying

* whoops

* Get client up to speed

* wip

* wip

* actual signatures in tests

(cherry picked from commit 1464853b737f1d712d64fbf90147f535525c8fd9)

* bugfixing

* Generate private key in example

* Generate signatures for pieceranger tests

* Update examples to use TLS

* Use private key from identity inside of example

* Use crypto.PrivateKey interface

* Change err name in defers

* Pass tests

* Pass identity Key to PSClient

* Get tests passing on travis

* Resolve linter complaints
2018-08-27 14:35:27 -04:00