Commit Graph

14 Commits

Author SHA1 Message Date
Bryan White
1ebd66d880 Cleanup & bolt-backed cache (#87)
* wip post-demos cleanup

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* Reorganize:

storage
├── common
│   └── common.go `package storage`
├── boltdb
│   └── ...
└── redis
    └── ...

storage
├── common.go `package storage`
├── boltdb
│   └── ...
└── redis
    └── ...

storage
├── common
│   ├── common.go `package storage`
│   └── testing.go `package storage` <--
└── ...

internal
├── app
│   └── cli
├── pkg
│   └── readcloser
└── test
    └── util.go `package test` <--

* remove comment

* add and use goimports

* add test types & fix some lint errors

* better typing

* fixing linter issues/comments

* goimports

* goimports

* more linter fixes; replace panic with NoError assertions in tests

* fix typo/more linter errors

* moar better linter fixes

* even moar better linter fixes

* linter

* add localPort back

* fixing exports, imports, and add comments
2018-06-13 14:22:32 -04:00
Dylan Lott
325a70d514 Cache (#67)
* add reference to dht to overlay client struct

* wip

* wip

* Implement FindNode

* get nodes

* WIP

* Merge in Dennis kademlia code, get it working with our code

* ping and moar

* WIP trying to get cache working with kademlia

* WIP more wiring up

* WIP

* Update service cli commands

* WIP

* added GetNodes

* added nodes to Kbucket

* default transport changed to TCP

* GetBuckets interface changed

* filling in more routing

* timestamp methods

* removed store

* Added initial network overlay explorer page

* Updating and building with dockerfile

* Working on adding bootstrap node code

* WIP merging in dennis' code

* WIP

* connects cache to pkg/kademlia implementation

* WIP redis cache

* testing

* Add bootstrap network function for CLI usage

* cleanup

* call bootstrap on init network

* Add BootstrapNetwork function to interface

* Merge in dennis kad code

* WIP updates to redis/overlay client interface

* WIP trying to get the DHT connected to the cache

* go mod & test

* deps

* Bootstrap node now setting up correctly

- Need to pass it through CLI commands better

* WIP adding refresh and walk functions, added cli flags

- added cli flags for custom bootstrap port and ip

* PR comments addressed

* adding FindStorageNodes to overlay cache

* fix GetBucket

* using SplitHostPort

* Use JoinHostPort

* updates to findstoragenodes response and request

* WIP merge in progress, having issues with a panic

* wip

* adjustments

* update port for dht bootstrap test

* Docker

* wip

* dockerfile

* fixes

* makefile changes

* Update port in NewKademlia call

* Update local kademlia DHT config

* kubernetes yaml

* cleanup

* making tests pass

* k8s yaml

* lint issues

* Edit cli flags to allow for configurable bootstrap IP and Port args

* cleanup

* cache walking the network now

* Rough prototype of Walk function laid out

* Move walk function into bootstrap function

* Update dht.go

* changes to yaml

* goimports
2018-06-05 17:06:37 -04:00
Alexander Leitner
ff8e191a9a
Added rpc server (#40)
* Added piecestore

* gofmt

* Added requested changes

* Added cli

* Removed ranger because I wanted something that can stand alone

* Add example of http server using piece store

* Changed piecestore code to make it more optial for error handelling

* Merged with piecestore

* Added missing package

* Forgot io import

* gofmt

* gofmt

* Forgot io

* Make path by hash exported

* updated to simplify again whoops

* Updated server to work real good

* Forgot ampersand

* Updated to match FilePiece

* Merged in cam's delete code

* Remove unused io

* Added RPC code

* Give the download request a reader

* Removed http server stuff; changed receive stream to say io.reader

* Added expiration date to shardInfo

* Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions

* added ttl info request

* Move scripts to http server pr; added close method for Retrieve api

* added rpc server tests for getting piece meta data and retrieval routes

* Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf

* Imported cams test

* Bump gometalinter deadline

* WIP adding tests

* added tests for store and delete routes

* Add changes as requested by Kaloyan, also cleaned up some code

* Get the code actually working whoops

* More cleanup

* Separating database calls from api.go

* need to rename expiration

* Added some changes requested by JT

* Fix read size

* Fixed total amount to read

* added tests

* Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages

* Moved rpc client and server to piece store

* Moved piecestore protobuf to the /protos folder

* Cleaned up messages

* Clean up linter errors

* Added missing sqlite import

* Add ability to do iterative reads and writes to pstore

* Incrementally read data

* Fix linter and import errors

* Solve linter Error

* Change return types

* begin test refactor

* refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go

* Cleaning up tests

* Added retrieve tests

* refactored delete tests

* Deleted old tests

* Updated cmd/piecestore to reflect changes to piecestore

* Refactored server tests and server/client store code

* gofmt

* WIP implementing TTL struct

* Read 4k at a time when Retrieving

* implemented ttl struct

* Accidentally removed fpiece dependency?

* Double resolve merge conflict

* Moved client to the examples since it is an example

* Change hash to id in protobuf. TODO: Change client and server to reflect these changes

* Update client and protobuf

* changed hash to id

* Handle eof properly in retrieve

* Id -> ID

* forgot to change import for client after moving

* Moved client and server main to examples

* Made changes requested by JT

* checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close()

* Print not fatal

* Handle overflow when reading from server

* added const IDLength

* removed types from comments

* Add reader/writer for download data from and uploading data to server

* goimports and comments

* fixed nits, casts, added OK const, DBCleanup now exits program on error

* Add stream reader and writer for server

* Fix errors

* i beforee except after c lol

* customizable data dir

* Forgot to rename variable

* customizable data dir

* Handle closing of server stream properly

* linter

* pass on inserting the same data twice

* linter

* linter

* Do the easy things JT asked for

* Handle overflow from reads properly; handle custom db path

* Handle overflow for server stream read; TODO Combine server and client stream reads

* Allow for TTL of 0 to stay forever

* Make Client cleaner for user
2018-06-02 14:14:59 -04:00
JT Olds
dd2fbb5069 don't require shallow clones of old commits (#38)
* don't require shallow clones of old commits

doesn't look like github allows it. hints here:
https://stackoverflow.com/questions/31278902/how-to-shallow-clone-a-specific-commit-with-depth-1

* don't try to update submodules
2018-05-09 16:21:53 +03:00
JT Olio
78989a2d5a fix hanging travis test 2018-05-08 17:16:16 -06:00
JT Olds
2ef8e581b2 fix travis, and get a clean lint (#32)
* fix travis

* lint fixes

* travis: enable tests

* fix some tests, disable a broken eestream test for now
2018-05-08 18:02:01 -04:00
Dennis Coyle
52741af9ce make travis work? 2018-04-30 15:27:30 -04:00
Dennis Coyle
bb8d4f8d22 cleanup 2018-04-25 10:55:26 -04:00
Dennis Coyle
fe60ba02a3 lint step for copyright headers (#16)
* overlay proto

* grpc server and client

* fix import to storj

* tests

* change imports

* imports cleanup/comments

* PR comments addressed from @jtolds

* lint step for copyright headers

* spelling

* PR comments addressed from @jtolds
2018-04-23 10:05:52 -06:00
Bryan White
2e1ca41d39 bugfix: https://github.com/alecthomas/gometalinter#installing 2018-04-18 14:13:18 -04:00
Dennis Coyle
bd0d2778f0 add linter to build-dev-deps 2018-04-17 14:18:39 -04:00
Dennis Coyle
7d9d0e940c updating how we get dev deps 2018-04-17 13:29:15 -04:00
Dennis Coyle
1975b98eba grpc server and client 2018-04-16 10:09:01 -06:00
Dennis Coyle
250b36f27b wip on structure 2018-04-06 12:32:34 -04:00