Commit Graph

18 Commits

Author SHA1 Message Date
Alexander Leitner
db80358f32
bandwidth agreements protobuf (#505)
* protobuf for sending bandwidth agreements to satellite from storage nodes

* Setup process for sending agreements

* Add payer_id to db with bandwidth agreements for better sorting

* Renamed payer to satellite in psdb
2018-10-31 14:47:25 -04:00
Michal Niewrzal
f9248c21d4
Satellite verification on storage node (#469)
* Satellite verification on storage node

* fix formatting

* fix formatting

* rename SignatureAuth to SignedMessage

* fixes after review

* fix linter errors

* improve errors handling

* remove SignedMessageProvider

* fix liter errors

* params changed to authorization, signed message in audit, minor fixes

* fix formatting
2018-10-17 13:40:11 +02:00
Egon Elbre
b6b6111173
Flatten proto definitions into a single package (#360)
* protos: move streams to pb
* protos: move overlay to pb
* protos: move pointerdb to pb
* protos: move piecestore to pb
* fix statdb import naming
2018-09-18 07:39:06 +03:00
Egon Elbre
4486e265bb
Cleanup after tests (#341) 2018-09-11 16:57:12 +03:00
Egon Elbre
e7e2d4d7c9
use golangci-lint and add missing error checks (#340) 2018-09-11 16:13:25 +03:00
Alexander Leitner
c68cd1232d Prevent sql injection for farmer (#322)
* Validate incoming ID

* Pass tests, cleanup code

* Smarter regex

* Update server.go
2018-09-08 11:06:44 -06:00
Egon Elbre
a575a8e293
Safer and faster psdb (#324) 2018-09-08 18:34:55 +03:00
Egon Elbre
706a6ac26e
Use better test ports (#311)
Other interface or ports may be blocked or require permissions when running tests.
2018-09-07 12:42:04 +03:00
Alexander Leitner
899e1e68f1
Add functions for signing and verifying during bandwidth exchange (#246)
* Added initial functions for signing and verifying

* whoops

* Get client up to speed

* Added initial functions for signing and verifying

* whoops

* Get client up to speed

* wip

* wip

* actual signatures in tests

(cherry picked from commit 1464853b737f1d712d64fbf90147f535525c8fd9)

* bugfixing

* Generate private key in example

* Generate signatures for pieceranger tests

* Update examples to use TLS

* Use private key from identity inside of example

* Use crypto.PrivateKey interface

* Change err name in defers

* Pass tests

* Pass identity Key to PSClient

* Get tests passing on travis

* Resolve linter complaints
2018-08-27 14:35:27 -04:00
Egon Elbre
0f5a2f4ef5 Enable more linters (#272)
* enable more linters

* Run gofmt -s

* run goimports

* run unconvert

* fix naked return

* fix misspellings

* fix ineffectual assigments

* fix missing declaration

* don't use deprecated grpc.Errof

* check errors in tests

* run gofmt -w -r "assert.Nil(err) -> assert.NoError(err)"

* fix directory permissions

* don't use nil Context

* simplify boolean expressions

* use bytes.Equal instead of bytes.Compare

* merge variable declarations, remove redundant returns

* fix some golint errors

* run goimports

* handle more errors

* delete empty TestMain

* delete empty TestMain

* ignore examples for now

* fix lint errors

* remove unused values

* more fixes

* run gofmt -w -s .

* add more comments

* fix naming

* more lint fixes

* try switching travis to go1.11

* fix unnecessary conversions

* fix deprecated methods

* use go1.10 and disable gofmt/goimports for now

* switch to 1.10

* don't re-enable gofmt and goimports

* switch covermode to atomic because of -race

* gofmt
2018-08-27 11:28:16 -06:00
aligeti
09da23737a
rename farmers to storagenode (#275)
* rename farmers to storagenode

* review changes

* merge conflicts
2018-08-24 21:52:58 -04:00
JT Olio
560e9fd9ec
uplink: make ul/uplink consistent (#286)
* uplink: make ul/uplink consistent

* pstore: make test more robust
2018-08-24 14:02:42 -06:00
Egon Elbre
f80185a807
Preliminary support for windows (#266)
* Don't use url.Parse for bolt paths: filepaths may not be valid URL-s.
* go.mod: update dependencies
* README.md: add Windows instructions
* pkg/overlay: check for the correct path and text in error
* pkg/overlay: fix tests for windows
* pkg/piecestore: make windows tests pass
* pkg/telemetry: skip test, as it doesn't shutdown nicely
* storage/redis: ensure that redis is clean before running tests
2018-08-23 10:05:56 +03:00
Alexander Leitner
b0db33f919
Bandwidth accounting (#134)
* captplanet standalone farmer setup

* Bandwidth Allocation

* utils.Close method changed to utils.LogClose

* Get build temporarily working

* Get/Put for PSClient should take payer bandwidth allocations rather than the NewPSClient function

* Update example client to reflect changes in client API

* Update ecclient to use latest PSClient, Make NewPSClient return error also

* Updated pieceranger tests to check for errors; sign method should take byte array

* Handle defers in store.go better

* Fix defer functions in psdb.go

* fun times

* Protobuf bandwidthallocation data is now a byte array

* Remove psservice package and merge it into pstore server

* Write wrapper for database calls

* Change all expiration names in protobuf to be more informative; add defer in retrieve; remove old comment

* Make PSDB tests implementation independent rather than method independent

* get rid of payer, renter in ecclient

* add context monitoring in store and retrieve
2018-08-17 13:40:15 -04:00
Alexander Leitner
cde0ffaa52 Add dockerfile and yaml for setting up piecestore servers (#102)
* Add dockerfile and yaml for setting up piecestore servers

* Fix dockerfile for @aleitner (#115)

* Fix dockerfile for @aleitner

* Move files for @coyle

* Update yaml

* My linter had some errors so I resolved them

* Make jenkins do the needful

* Make piecestore-farmer look like overlay's build process

* Fix service spec to work in staging

* Make Jenkins push images, but not deploy them, yet.

* Modify entrypoint to fit new verbs

* Update piecestore-farmer entrypoint script to handle new app output
2018-07-19 15:42:50 -04:00
Bryan White
1ebd66d880 Cleanup & bolt-backed cache (#87)
* wip post-demos cleanup

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* Reorganize:

storage
├── common
│   └── common.go `package storage`
├── boltdb
│   └── ...
└── redis
    └── ...

storage
├── common.go `package storage`
├── boltdb
│   └── ...
└── redis
    └── ...

storage
├── common
│   ├── common.go `package storage`
│   └── testing.go `package storage` <--
└── ...

internal
├── app
│   └── cli
├── pkg
│   └── readcloser
└── test
    └── util.go `package test` <--

* remove comment

* add and use goimports

* add test types & fix some lint errors

* better typing

* fixing linter issues/comments

* goimports

* goimports

* more linter fixes; replace panic with NoError assertions in tests

* fix typo/more linter errors

* moar better linter fixes

* even moar better linter fixes

* linter

* add localPort back

* fixing exports, imports, and add comments
2018-06-13 14:22:32 -04:00
Alexander Leitner
db17557d0d
piece store accept data when size isn't specified (#72)
* added spawn scripts

* wip

* Make server not require length

* Added test for 0 length

* Added test for accepting length of 0

* linter errors

* Make store not take anything more than an id and maybe a ttl if you want
2018-06-04 17:30:42 -04:00
Alexander Leitner
ff8e191a9a
Added rpc server (#40)
* Added piecestore

* gofmt

* Added requested changes

* Added cli

* Removed ranger because I wanted something that can stand alone

* Add example of http server using piece store

* Changed piecestore code to make it more optial for error handelling

* Merged with piecestore

* Added missing package

* Forgot io import

* gofmt

* gofmt

* Forgot io

* Make path by hash exported

* updated to simplify again whoops

* Updated server to work real good

* Forgot ampersand

* Updated to match FilePiece

* Merged in cam's delete code

* Remove unused io

* Added RPC code

* Give the download request a reader

* Removed http server stuff; changed receive stream to say io.reader

* Added expiration date to shardInfo

* Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions

* added ttl info request

* Move scripts to http server pr; added close method for Retrieve api

* added rpc server tests for getting piece meta data and retrieval routes

* Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf

* Imported cams test

* Bump gometalinter deadline

* WIP adding tests

* added tests for store and delete routes

* Add changes as requested by Kaloyan, also cleaned up some code

* Get the code actually working whoops

* More cleanup

* Separating database calls from api.go

* need to rename expiration

* Added some changes requested by JT

* Fix read size

* Fixed total amount to read

* added tests

* Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages

* Moved rpc client and server to piece store

* Moved piecestore protobuf to the /protos folder

* Cleaned up messages

* Clean up linter errors

* Added missing sqlite import

* Add ability to do iterative reads and writes to pstore

* Incrementally read data

* Fix linter and import errors

* Solve linter Error

* Change return types

* begin test refactor

* refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go

* Cleaning up tests

* Added retrieve tests

* refactored delete tests

* Deleted old tests

* Updated cmd/piecestore to reflect changes to piecestore

* Refactored server tests and server/client store code

* gofmt

* WIP implementing TTL struct

* Read 4k at a time when Retrieving

* implemented ttl struct

* Accidentally removed fpiece dependency?

* Double resolve merge conflict

* Moved client to the examples since it is an example

* Change hash to id in protobuf. TODO: Change client and server to reflect these changes

* Update client and protobuf

* changed hash to id

* Handle eof properly in retrieve

* Id -> ID

* forgot to change import for client after moving

* Moved client and server main to examples

* Made changes requested by JT

* checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close()

* Print not fatal

* Handle overflow when reading from server

* added const IDLength

* removed types from comments

* Add reader/writer for download data from and uploading data to server

* goimports and comments

* fixed nits, casts, added OK const, DBCleanup now exits program on error

* Add stream reader and writer for server

* Fix errors

* i beforee except after c lol

* customizable data dir

* Forgot to rename variable

* customizable data dir

* Handle closing of server stream properly

* linter

* pass on inserting the same data twice

* linter

* linter

* Do the easy things JT asked for

* Handle overflow from reads properly; handle custom db path

* Handle overflow for server stream read; TODO Combine server and client stream reads

* Allow for TTL of 0 to stay forever

* Make Client cleaner for user
2018-06-02 14:14:59 -04:00