Commit Graph

15 Commits

Author SHA1 Message Date
Michal Niewrzal
81408a3c9e
Use SignedHash on client/uplink side (#1354)
* psclient receives storage node hash and compare it to own hash for verification
* uplink sends delete request when hashes don't match
* valid hashes are propagated up to segments.Store for future sending to satellite
2019-02-25 16:57:54 +01:00
Bill Thorp
373b301736
BWA aliases (#1333)
aliased RBAs and PBAs
2019-02-22 16:17:35 -05:00
Natalie Villasana
c3d3f41d30 removes some SignedMessage use (#1258)
Removes most instances of pb.SignedMessage (there's more to take out but they shouldn't hurt anyone as is).

There used to be places in psserver where a PieceID was hmac'd with the SatelliteID, which was gotten from a SignedMessage. This PR makes it so some functions access the SatelliteID from the Payer Bandwidth Allocation instead.

This requires passing a SatelliteID into psserver functions where they weren't before, so the following proto messages have been changed:

 * PieceId - satellite_id field added
   This is so the psserver.Piece function has access to the SatelliteID when it needs to get the namespaced pieceID.
   This proto message should probably be renamed to PieceRequest, or a new PieceRequest message should be created so this isn't misnamed.

 * PieceDelete - satellite_id field added
   This is so the psserver.Delete function has access to the SatelliteID when receiving a request to Delete.
2019-02-19 23:36:08 -06:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Bryan White
ee62e2a9d8
Use transport client and cleanup all the clients (#574)
* wip

* linter fixes

* linter fixes

* test fixes

* linter fixes

* fix merge + restructure piecestore packages

* review feedback

* linter fixes

* linter fixes

* remove unnecessary aliases to piecestore

* more merge fixing
2018-11-06 18:49:17 +01:00
Michal Niewrzal
f9248c21d4
Satellite verification on storage node (#469)
* Satellite verification on storage node

* fix formatting

* fix formatting

* rename SignatureAuth to SignedMessage

* fixes after review

* fix linter errors

* improve errors handling

* remove SignedMessageProvider

* fix liter errors

* params changed to authorization, signed message in audit, minor fixes

* fix formatting
2018-10-17 13:40:11 +02:00
Egon Elbre
2dbada73ab
Separate storj.io packages in imports (#481)
Format with goimports -w -local storj.io
Add flag to golang lint
Remove commented out imports as they mess with goimports
2018-10-16 14:43:44 +03:00
aligeti
97cbacb299
storage node quick check and startup validation (#368)
* storage node quick check and startup validation

*  rearranged the startup validation and quick check logic

* travis lint warning fixes

* travis lint warning fixes

* travis lint warning fixes

* code changes per review comments

* code clean dev debug info

* travis lint wranings

* code changes per code review comments

* code changes per code review comments

* code update per review

* sqlite SUM is having issue when getting the SUM of an empty column; filepath was checking a directory that doesn't exist when starting server; Example updated to print allocated and used space

* storage node quick check and startup validation

*  rearranged the startup validation and quick check logic

* travis lint warning fixes

* travis lint warning fixes

* travis lint warning fixes

* code changes per review comments

* code clean dev debug info

* travis lint wranings

* code changes per code review comments

* code changes per code review comments

* code update per review

* no file or directory error

* Updated mock PSClient
2018-09-25 12:49:55 -04:00
Egon Elbre
b6b6111173
Flatten proto definitions into a single package (#360)
* protos: move streams to pb
* protos: move overlay to pb
* protos: move pointerdb to pb
* protos: move piecestore to pb
* fix statdb import naming
2018-09-18 07:39:06 +03:00
Kaloyan Raev
ee5f32b0f7
ConcatRanger closes segment readers as soon as data is read (#349)
* ConcatRanger closes a segment readers as soon as data is read

* Fix linter errors

* Fix data race

* Nicer call to CombineErrors
2018-09-14 10:10:43 -04:00
Alexander Leitner
8f07c4d746
Keep track of used space (#338)
* Keep track of space at all times

* Create index on expires

* Updated mock psclient
2018-09-11 08:40:45 -04:00
Kaloyan Raev
130acf57a5
Make PSClient a Closer, so it can be closed with utils.LogClose() (#247) 2018-08-20 18:11:54 +03:00
Alexander Leitner
b0db33f919
Bandwidth accounting (#134)
* captplanet standalone farmer setup

* Bandwidth Allocation

* utils.Close method changed to utils.LogClose

* Get build temporarily working

* Get/Put for PSClient should take payer bandwidth allocations rather than the NewPSClient function

* Update example client to reflect changes in client API

* Update ecclient to use latest PSClient, Make NewPSClient return error also

* Updated pieceranger tests to check for errors; sign method should take byte array

* Handle defers in store.go better

* Fix defer functions in psdb.go

* fun times

* Protobuf bandwidthallocation data is now a byte array

* Remove psservice package and merge it into pstore server

* Write wrapper for database calls

* Change all expiration names in protobuf to be more informative; add defer in retrieve; remove old comment

* Make PSDB tests implementation independent rather than method independent

* get rid of payer, renter in ecclient

* add context monitoring in store and retrieve
2018-08-17 13:40:15 -04:00
Bryan White
5b913c45b9
Transport security (#63)
* wip initial transport security

* wip: transport security (add tests / refactor)

* wip tests

* refactoring - still wip

* refactor, improve tests

* wip tls testing

* fix typo

* wip testing

* wip testing

* wip

* tls_test passing

* code-style improvemente / refactor; service and tls tests passing!

* code-style auto-format

* add TestNewServer_LoadTLS

* refactor; test improvements

* refactor

* add client cert

* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* files created

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* add separate `Process` tests for bolt and redis-backed overlay

* more testing

* fix gitignore

* fix linter error

* goimports goimports GOIMPORTS GoImPortS!!!!

* wip

* fix port madness

* forgot to add

* add `mux` as handler and shorten context timeouts

* gofreakingimports

* fix comments

* refactor test & add logger/monkit registry

* debugging travis

* add comment

* Set redisAddress to empty string for bolt-test

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* Merge branch 'tls' into tls-upstream

* tls:
  add client cert
  refactor
  refactor; test improvements
  add TestNewServer_LoadTLS
  code-style auto-format
  code-style improvemente / refactor; service and tls tests passing!
  tls_test passing
  wip
  wip testing
  wip testing
  fix typo
  wip tls testing
  refactor, improve tests
  refactoring - still wip
  wip tests
  wip: transport security (add tests / refactor)
  wip initial transport security

* fixing linter things

* wip

* remove bkad dependencie from tests

* wip

* wip

* wip

* wip

* wip

* updated coyle/kademlia

* wip

* cleanup

* ports

* overlay upgraded

* linter fixes

* piecestore kademlia newID

* Merge branch 'master' into tls-upstream

* master:
  Add error to the return values of Ranger.Range method (#90)
  udp-forwarding: demo week work! (#84)

* Merge branch 'kad-tests' into tls-upstream

* kad-tests:
  piecestore kademlia newID
  linter fixes
  overlay upgraded
  ports
  cleanup
  wip
  updated coyle/kademlia
  wip
  wip
  wip
  wip
  wip
  remove bkad dependencie from tests
  wip
  wip
  files created
  port changes

* wip

* finish merging service tests

* add test for different client/server certs

* wip

* Merge branch 'master' into tls-upstream

* master:
  Add context to Ranger.Range method (#99)
  Coyle/kad client (#91)

* wip

* wip; refactoring/cleanup

* wip

* Merge branch 'master' into tls

* master:
  Bolt backed overlay cache (#94)
  internal/test: switch errors to error classes (#96)

* wip - test passing

* cleanup

* remove port.go

* cleanup

* Merge branch 'master' into tls

* master:
  hardcode version (#111)
  Coyle/docker fix (#109)
  pkg/kademlia tests and restructuring (#97)
  Use continue instead of return in table tests (#106)
  prepend storjlabs to docker tag (#108)
  Automatically build, tag and push docker images on merge to master (#103)

* more belated merging

* more belated merging

* more belated merging

* add copyrights

* cleanup

* goimports

* refactoring

* wip

* wip

* implement `TLSFileOptions#loadTLS`, refactoring:

`peertls.TestNewClient_LoadTLS` is the failing holdout; Still trying to figure out why I'm getting ECDSA verification is failing.

* not sure if actually working:

Tests are now passing (no more "ECDSA verification failed"); however,
`len(*tls.Certificates.Certificate) == 1` which I don't think should be
the case if the root and leaf are being created correctly.

* Experimenting/bugfixing?:

I think leaf certs should be properly signed by the parent now but not
entirely certain. It's also unclear to me why in
`VerifyPeerCertificate`, `len(rawCerts) == 1` when the certs should
contain both the root and leaf afaik.

* Properly write/read certificate chain (root/leaf):

I think I'm now properly reading and writing the root and leaf
certificate chain such that they're both being received by
`VerifyPeerCertificate`.

The next step is to parse the certificates with `x509.ParseCertificate`
(or similar) and verify that the public keys and signatures match.

* Add tls certificate chain signature veification (spike):

+ `VerifyPeerCertificate` verifies signatures of certificates using the
key of it's parent if there is one; otherwise, it verifies the
certificate is self-signed
+ TODO: refactor
+ TODO: test

* refactoring `VerifyPeerCertificate`

* cleanup

* refactor

* Merge branch 'master' into tls

* master:
  Remove some structural folders we don't seem to be using. (#125)
  license code with agplv3 (#126)
  Update .clabot (#124)
  added team memebers (#123)
  clabot file added (#121)
  ECClient (#110)
  docker image issue fixed (#118)
  Piecestore Farmer CLI  (#92)
  Define Path type (#101)
  adds netstate pagination (#95)
  Transport Client (#89)
  Implement psclient interface (#107)
  pkg/process: start replacing pkg/process with cobra helpers (#98)
  protos/netstate: remove stuff we're not using (#100)
  adding coveralls / code coverage  (#112)

* responding to review feedback / cleanup / add copywrite headers

* suggestions

* realitive

* Merge pull request #1 from coyle/coyle/tls

suggestions

* remove unnecessary `_`s

* Merge branch 'tls' of github.com:bryanchriswhite/storj into tls

* 'tls' of github.com:bryanchriswhite/storj:
  realitive
  suggestions

* Responding to review feedback:

+ refactor `VerifyPeerCertificate`

* remove tls expiration

* remove "hosts" and "clien option" from tls options

* goimports

* linter fixes
2018-07-09 19:43:13 +02:00
Kaloyan Raev
d8f1ec1db6
ECClient (#110)
* WIP ECClient

* Get returns RangeCloser

* Introduce RedundancyStrategy

* Constructor takes max buffer memory

* Remove unnecessary NopCloser wrapper

* Added telemetry

* Tests

* Adapt to PSClient from master

* Decode should report error if empty rrs map is passed

* collectErrors helper

* Move to /pkg/storage

* Move to /pkg/storage/ec

* Rename ecclient.go to client.go

* Better logging

* Rename ec.ECClient to ec.Client

* Fix some test execution

* Adopt Transport Client from master
2018-07-03 11:35:01 +03:00