Commit Graph

30 Commits

Author SHA1 Message Date
Alexander Leitner
3399376eb2
Limit to only 1 ttl database write when storing data on storage nodes (#348)
* Limit to only 1 database write

* Check file system rather than database

* Move check to storefile. We need to figure out how to fix this mess

* piecestore should not overwrite data, it should fail when trying to write to a file that already exists

* Format errors, delete unused function in psdb for checking if TTL exists

* Combine errors better
2018-09-13 10:30:45 -04:00
Alexander Leitner
4397b90349
Only create index if not exists (#347) 2018-09-12 12:23:21 -04:00
Egon Elbre
4486e265bb
Cleanup after tests (#341) 2018-09-11 16:57:12 +03:00
Egon Elbre
e7e2d4d7c9
use golangci-lint and add missing error checks (#340) 2018-09-11 16:13:25 +03:00
Alexander Leitner
8f07c4d746
Keep track of used space (#338)
* Keep track of space at all times

* Create index on expires

* Updated mock psclient
2018-09-11 08:40:45 -04:00
Alexander Leitner
2eb660d4b7 Bandwidth allocation pipeline data (#276)
* Moving retrieve into multiple goroutines

* Make sure we pass nil errors into err channel

* restore tests

* incorporate locks in retrieve.go

* deserialize data only if we have something to deserealize when receiving bandwidth allocation in server store

* Adding logic for retrieve to be more efficient

* Add channel?

* hmm

* implement Throttle concurrency primitive

* using throttle

* Remove unused variables

* Egon comments addressed

* Get ba total correct

* Consume without waiting

* incrementally increase signing size

* Get downloads working with throttle

* Removed logging

* Make sure we handle errors properly

* Fix tests
>
>
Co-authored-by: Kaloyan <kaloyan@storj.io>

* Can't Fatalf in goroutine

* Add missing returns to tests

* add capacity to channel, smarter allocations

* rename things and don't use size as limit

* replace things with sync2.Throttle

* fix compilation errors

* add note about security

* fix ordering

* Max length is actually 64 bytes for piece ID

* Max length is actually 64 bytes for piece ID

* fix limit

* error comes from pending allocs, so no need to relog

* Optimize throughput

* TODO

* Deleted allocation manager

* Return when someone sends a smaller bandwidth allocation than the previous message

* review comments
2018-09-10 03:18:41 -06:00
Alexander Leitner
c68cd1232d Prevent sql injection for farmer (#322)
* Validate incoming ID

* Pass tests, cleanup code

* Smarter regex

* Update server.go
2018-09-08 11:06:44 -06:00
Egon Elbre
a575a8e293
Safer and faster psdb (#324) 2018-09-08 18:34:55 +03:00
Egon Elbre
706a6ac26e
Use better test ports (#311)
Other interface or ports may be blocked or require permissions when running tests.
2018-09-07 12:42:04 +03:00
Alexander Leitner
899e1e68f1
Add functions for signing and verifying during bandwidth exchange (#246)
* Added initial functions for signing and verifying

* whoops

* Get client up to speed

* Added initial functions for signing and verifying

* whoops

* Get client up to speed

* wip

* wip

* actual signatures in tests

(cherry picked from commit 1464853b737f1d712d64fbf90147f535525c8fd9)

* bugfixing

* Generate private key in example

* Generate signatures for pieceranger tests

* Update examples to use TLS

* Use private key from identity inside of example

* Use crypto.PrivateKey interface

* Change err name in defers

* Pass tests

* Pass identity Key to PSClient

* Get tests passing on travis

* Resolve linter complaints
2018-08-27 14:35:27 -04:00
Egon Elbre
0f5a2f4ef5 Enable more linters (#272)
* enable more linters

* Run gofmt -s

* run goimports

* run unconvert

* fix naked return

* fix misspellings

* fix ineffectual assigments

* fix missing declaration

* don't use deprecated grpc.Errof

* check errors in tests

* run gofmt -w -r "assert.Nil(err) -> assert.NoError(err)"

* fix directory permissions

* don't use nil Context

* simplify boolean expressions

* use bytes.Equal instead of bytes.Compare

* merge variable declarations, remove redundant returns

* fix some golint errors

* run goimports

* handle more errors

* delete empty TestMain

* delete empty TestMain

* ignore examples for now

* fix lint errors

* remove unused values

* more fixes

* run gofmt -w -s .

* add more comments

* fix naming

* more lint fixes

* try switching travis to go1.11

* fix unnecessary conversions

* fix deprecated methods

* use go1.10 and disable gofmt/goimports for now

* switch to 1.10

* don't re-enable gofmt and goimports

* switch covermode to atomic because of -race

* gofmt
2018-08-27 11:28:16 -06:00
aligeti
09da23737a
rename farmers to storagenode (#275)
* rename farmers to storagenode

* review changes

* merge conflicts
2018-08-24 21:52:58 -04:00
JT Olio
560e9fd9ec
uplink: make ul/uplink consistent (#286)
* uplink: make ul/uplink consistent

* pstore: make test more robust
2018-08-24 14:02:42 -06:00
Egon Elbre
f80185a807
Preliminary support for windows (#266)
* Don't use url.Parse for bolt paths: filepaths may not be valid URL-s.
* go.mod: update dependencies
* README.md: add Windows instructions
* pkg/overlay: check for the correct path and text in error
* pkg/overlay: fix tests for windows
* pkg/piecestore: make windows tests pass
* pkg/telemetry: skip test, as it doesn't shutdown nicely
* storage/redis: ensure that redis is clean before running tests
2018-08-23 10:05:56 +03:00
JT Olio
40f4f7ec31
piecestore: only check ttl db every hour instead of every 300 nanoseconds (#250) 2018-08-20 15:47:24 -04:00
Kaloyan Raev
130acf57a5
Make PSClient a Closer, so it can be closed with utils.LogClose() (#247) 2018-08-20 18:11:54 +03:00
Alexander Leitner
b0db33f919
Bandwidth accounting (#134)
* captplanet standalone farmer setup

* Bandwidth Allocation

* utils.Close method changed to utils.LogClose

* Get build temporarily working

* Get/Put for PSClient should take payer bandwidth allocations rather than the NewPSClient function

* Update example client to reflect changes in client API

* Update ecclient to use latest PSClient, Make NewPSClient return error also

* Updated pieceranger tests to check for errors; sign method should take byte array

* Handle defers in store.go better

* Fix defer functions in psdb.go

* fun times

* Protobuf bandwidthallocation data is now a byte array

* Remove psservice package and merge it into pstore server

* Write wrapper for database calls

* Change all expiration names in protobuf to be more informative; add defer in retrieve; remove old comment

* Make PSDB tests implementation independent rather than method independent

* get rid of payer, renter in ecclient

* add context monitoring in store and retrieve
2018-08-17 13:40:15 -04:00
Kaloyan Raev
3644d77b45 PSClient handles io.ErrUnexpectedEOF (#211)
* PSClient handles io.ErrUnexpectedEOF

* Fix my bad English

* Even better log message
2018-08-09 11:42:35 -04:00
Kaloyan Raev
34629e5b80 Group erasure shares before sending them to storage nodes (#210) 2018-08-09 11:14:05 -04:00
Alexander Leitner
cde0ffaa52 Add dockerfile and yaml for setting up piecestore servers (#102)
* Add dockerfile and yaml for setting up piecestore servers

* Fix dockerfile for @aleitner (#115)

* Fix dockerfile for @aleitner

* Move files for @coyle

* Update yaml

* My linter had some errors so I resolved them

* Make jenkins do the needful

* Make piecestore-farmer look like overlay's build process

* Fix service spec to work in staging

* Make Jenkins push images, but not deploy them, yet.

* Modify entrypoint to fit new verbs

* Update piecestore-farmer entrypoint script to handle new app output
2018-07-19 15:42:50 -04:00
Cameron
6463b87ebe
Errcheck (#133)
* add errcheck

* fixed linter errors

* fixes

* errcheck fixes in pkg/paths

* Fix errchecks in PieceID.Derive

* Fix ecclient tests

* Move closeConn a little bit above in the exectution flow

* fix new lint errors

* Fatalf -> Printf

* address eclipsed errors

* rename err to closeErr

* rename err to closeErr for differentiation
2018-07-16 15:22:34 -04:00
Kaloyan Raev
a6bafa993a
Obscure piece id across piece store nodes (#120)
* Obscure piece id across piece store nodes

* Add line separator in imports

* Avoid potential panic if hash < 32 bytes
2018-07-06 11:51:13 +03:00
Alexander Leitner
900f67e3d0 Implement psclient interface (#107)
* Implement psclient interface

* Add string method to pieceID type

* try to fix linter errors

* Whoops missed an error

* More linter errors

* Typo

* Lol double typo

*  Get everything working, begin adding tests for psclient rpc

* goimports

* Forgot to change the piecestore cli when changed the piecestore code

* Fix CLI

* remove ID length, added validator to pieceID

* Move grpc ranger to client
Change client PUT api to take a reader rather than return a writer

* GRPCRanger -> PieceRanger; Make PieceRanger a RangeCloser

* Forgot to remove offset

* Added message upon successful store

* Do that thing dennis and kaloyan wanted

* goimports

* Make closeConn a part of the interface for psclient

* Use interface

* Removed uneccessary new lines

* goimport

* Whoops

* Actually we don't want to use the interface in Piece Ranger

* Renamed piecestore in examples to piecestore-client; moved piecestore-cli to examples

* Make comments look nicer
2018-06-27 21:42:54 +03:00
Kaloyan Raev
045dae5f0d
Add context to Ranger.Range method (#99) 2018-06-19 18:59:09 +03:00
Bryan White
1ebd66d880 Cleanup & bolt-backed cache (#87)
* wip post-demos cleanup

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* Reorganize:

storage
├── common
│   └── common.go `package storage`
├── boltdb
│   └── ...
└── redis
    └── ...

storage
├── common.go `package storage`
├── boltdb
│   └── ...
└── redis
    └── ...

storage
├── common
│   ├── common.go `package storage`
│   └── testing.go `package storage` <--
└── ...

internal
├── app
│   └── cli
├── pkg
│   └── readcloser
└── test
    └── util.go `package test` <--

* remove comment

* add and use goimports

* add test types & fix some lint errors

* better typing

* fixing linter issues/comments

* goimports

* goimports

* more linter fixes; replace panic with NoError assertions in tests

* fix typo/more linter errors

* moar better linter fixes

* even moar better linter fixes

* linter

* add localPort back

* fixing exports, imports, and add comments
2018-06-13 14:22:32 -04:00
JT Olio
258d5fec44 Piecestore: demo week work (#82)
* piecestore: connect to kad

* piecestore: Linter errors

* piecestore: added pstore config command line utility

* piecestore: removed main.go, implement methods and structs

* piecestore: Import Cam's config code into piecestore-farmer

* piecestore: moving farmer to urfavecli

* piecestore: added create command

* piecestore: Removed old config, added server start code to cli

* piecestore: Get server code working

* piecestore: Changed default dir for storing piece store data; added ttl to config

* piecestore: Generate id; add bootstrap ip for kad

* piecestore: Separate kad port and server port

* piecestore: goimports

* piecestore: Removed print

* piecestore: use pkg/process

* piecestore: Better config file

* piecestore: base58 encode for id

* piecestore: base58 encode and clean up cli

* piecestore: Typo

* piecestore: removed unnecessary variable

* piecestore: Fixed more typos

* piecestore: Place data in a directory based on nodeid

* piecestore: base58 encode instead

* piecestore: Add dependency to go.mod

* piecestore: Fix typo in rpc server start; clear data on failed piece upload
2018-06-12 11:03:45 -04:00
JT Olio
6be2baf9f9
pkg/piecestore: use readersource (#75)
* pkg/piecestore: use readersource

* pkg/piecestore: fix linting
2018-06-05 08:00:48 -06:00
Alexander Leitner
db17557d0d
piece store accept data when size isn't specified (#72)
* added spawn scripts

* wip

* Make server not require length

* Added test for 0 length

* Added test for accepting length of 0

* linter errors

* Make store not take anything more than an id and maybe a ttl if you want
2018-06-04 17:30:42 -04:00
Kaloyan Raev
916e0b0ee0 gRPC Ranger (#44)
* Added piecestore

* gofmt

* Added requested changes

* Added cli

* Removed ranger because I wanted something that can stand alone

* Add example of http server using piece store

* Changed piecestore code to make it more optial for error handelling

* Merged with piecestore

* Added missing package

* Forgot io import

* gofmt

* gofmt

* Forgot io

* Make path by hash exported

* updated to simplify again whoops

* Updated server to work real good

* Forgot ampersand

* Updated to match FilePiece

* Merged in cam's delete code

* Remove unused io

* Added RPC code

* Give the download request a reader

* Removed http server stuff; changed receive stream to say io.reader

* Added expiration date to shardInfo

* gRPC Ranger

* Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions

* Adapt to latest changes in piece store rpc api

* added ttl info request

* Initialize grpcRanger type with named fields

* Move scripts to http server pr; added close method for Retrieve api

* added rpc server tests for getting piece meta data and retrieval routes

* Adapt to PieceStreamReader now being a ReadCloser

* Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf

* Imported cams test

* Bump gometalinter deadline

* Adapt to package name changes

* Remove Garbage

* Adapt to latest changes in piece store rpc api

* NewCustomRoute constructor to allow mocking the gRPC client

* Name struct values in constructor.
2018-06-02 22:20:17 -06:00
Alexander Leitner
ff8e191a9a
Added rpc server (#40)
* Added piecestore

* gofmt

* Added requested changes

* Added cli

* Removed ranger because I wanted something that can stand alone

* Add example of http server using piece store

* Changed piecestore code to make it more optial for error handelling

* Merged with piecestore

* Added missing package

* Forgot io import

* gofmt

* gofmt

* Forgot io

* Make path by hash exported

* updated to simplify again whoops

* Updated server to work real good

* Forgot ampersand

* Updated to match FilePiece

* Merged in cam's delete code

* Remove unused io

* Added RPC code

* Give the download request a reader

* Removed http server stuff; changed receive stream to say io.reader

* Added expiration date to shardInfo

* Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions

* added ttl info request

* Move scripts to http server pr; added close method for Retrieve api

* added rpc server tests for getting piece meta data and retrieval routes

* Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf

* Imported cams test

* Bump gometalinter deadline

* WIP adding tests

* added tests for store and delete routes

* Add changes as requested by Kaloyan, also cleaned up some code

* Get the code actually working whoops

* More cleanup

* Separating database calls from api.go

* need to rename expiration

* Added some changes requested by JT

* Fix read size

* Fixed total amount to read

* added tests

* Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages

* Moved rpc client and server to piece store

* Moved piecestore protobuf to the /protos folder

* Cleaned up messages

* Clean up linter errors

* Added missing sqlite import

* Add ability to do iterative reads and writes to pstore

* Incrementally read data

* Fix linter and import errors

* Solve linter Error

* Change return types

* begin test refactor

* refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go

* Cleaning up tests

* Added retrieve tests

* refactored delete tests

* Deleted old tests

* Updated cmd/piecestore to reflect changes to piecestore

* Refactored server tests and server/client store code

* gofmt

* WIP implementing TTL struct

* Read 4k at a time when Retrieving

* implemented ttl struct

* Accidentally removed fpiece dependency?

* Double resolve merge conflict

* Moved client to the examples since it is an example

* Change hash to id in protobuf. TODO: Change client and server to reflect these changes

* Update client and protobuf

* changed hash to id

* Handle eof properly in retrieve

* Id -> ID

* forgot to change import for client after moving

* Moved client and server main to examples

* Made changes requested by JT

* checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close()

* Print not fatal

* Handle overflow when reading from server

* added const IDLength

* removed types from comments

* Add reader/writer for download data from and uploading data to server

* goimports and comments

* fixed nits, casts, added OK const, DBCleanup now exits program on error

* Add stream reader and writer for server

* Fix errors

* i beforee except after c lol

* customizable data dir

* Forgot to rename variable

* customizable data dir

* Handle closing of server stream properly

* linter

* pass on inserting the same data twice

* linter

* linter

* Do the easy things JT asked for

* Handle overflow from reads properly; handle custom db path

* Handle overflow for server stream read; TODO Combine server and client stream reads

* Allow for TTL of 0 to stay forever

* Make Client cleaner for user
2018-06-02 14:14:59 -04:00