Commit Graph

12 Commits

Author SHA1 Message Date
JT Olio
7280ae64a6 netstate: rename piece_name to piece_id (#83) 2018-06-13 14:10:16 -04:00
Dylan Lott
325a70d514 Cache (#67)
* add reference to dht to overlay client struct

* wip

* wip

* Implement FindNode

* get nodes

* WIP

* Merge in Dennis kademlia code, get it working with our code

* ping and moar

* WIP trying to get cache working with kademlia

* WIP more wiring up

* WIP

* Update service cli commands

* WIP

* added GetNodes

* added nodes to Kbucket

* default transport changed to TCP

* GetBuckets interface changed

* filling in more routing

* timestamp methods

* removed store

* Added initial network overlay explorer page

* Updating and building with dockerfile

* Working on adding bootstrap node code

* WIP merging in dennis' code

* WIP

* connects cache to pkg/kademlia implementation

* WIP redis cache

* testing

* Add bootstrap network function for CLI usage

* cleanup

* call bootstrap on init network

* Add BootstrapNetwork function to interface

* Merge in dennis kad code

* WIP updates to redis/overlay client interface

* WIP trying to get the DHT connected to the cache

* go mod & test

* deps

* Bootstrap node now setting up correctly

- Need to pass it through CLI commands better

* WIP adding refresh and walk functions, added cli flags

- added cli flags for custom bootstrap port and ip

* PR comments addressed

* adding FindStorageNodes to overlay cache

* fix GetBucket

* using SplitHostPort

* Use JoinHostPort

* updates to findstoragenodes response and request

* WIP merge in progress, having issues with a panic

* wip

* adjustments

* update port for dht bootstrap test

* Docker

* wip

* dockerfile

* fixes

* makefile changes

* Update port in NewKademlia call

* Update local kademlia DHT config

* kubernetes yaml

* cleanup

* making tests pass

* k8s yaml

* lint issues

* Edit cli flags to allow for configurable bootstrap IP and Port args

* cleanup

* cache walking the network now

* Rough prototype of Walk function laid out

* Move walk function into bootstrap function

* Update dht.go

* changes to yaml

* goimports
2018-06-05 17:06:37 -04:00
Alexander Leitner
db17557d0d
piece store accept data when size isn't specified (#72)
* added spawn scripts

* wip

* Make server not require length

* Added test for 0 length

* Added test for accepting length of 0

* linter errors

* Make store not take anything more than an id and maybe a ttl if you want
2018-06-04 17:30:42 -04:00
nfarah86
f9af3b6ee4
Mutex/auth grpc (#50)
* initial commit for PUT request authorization

* inital auth for put request

* auth. request working for all req

* deleted library

* removed .db files

* work in progress for modifying test suite to accomodate credentials

* modified tests

* gofmt, fixed code based on suggestions; test passed

* gofmt again

* merged nat's update on pointers, passed tests, cleanup from git rebase

* fixed fmt

* fixed fmt on tests

* work in progress

* reduced code

* fixed naming conventions

* added line in code

* fixed server bug, merged new code to server, added env

* fixed linter; getting cright issues on piecestore

* added comments for what passes on the creds
2018-06-04 09:45:07 -07:00
Kaloyan Raev
916e0b0ee0 gRPC Ranger (#44)
* Added piecestore

* gofmt

* Added requested changes

* Added cli

* Removed ranger because I wanted something that can stand alone

* Add example of http server using piece store

* Changed piecestore code to make it more optial for error handelling

* Merged with piecestore

* Added missing package

* Forgot io import

* gofmt

* gofmt

* Forgot io

* Make path by hash exported

* updated to simplify again whoops

* Updated server to work real good

* Forgot ampersand

* Updated to match FilePiece

* Merged in cam's delete code

* Remove unused io

* Added RPC code

* Give the download request a reader

* Removed http server stuff; changed receive stream to say io.reader

* Added expiration date to shardInfo

* gRPC Ranger

* Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions

* Adapt to latest changes in piece store rpc api

* added ttl info request

* Initialize grpcRanger type with named fields

* Move scripts to http server pr; added close method for Retrieve api

* added rpc server tests for getting piece meta data and retrieval routes

* Adapt to PieceStreamReader now being a ReadCloser

* Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf

* Imported cams test

* Bump gometalinter deadline

* Adapt to package name changes

* Remove Garbage

* Adapt to latest changes in piece store rpc api

* NewCustomRoute constructor to allow mocking the gRPC client

* Name struct values in constructor.
2018-06-02 22:20:17 -06:00
Alexander Leitner
ff8e191a9a
Added rpc server (#40)
* Added piecestore

* gofmt

* Added requested changes

* Added cli

* Removed ranger because I wanted something that can stand alone

* Add example of http server using piece store

* Changed piecestore code to make it more optial for error handelling

* Merged with piecestore

* Added missing package

* Forgot io import

* gofmt

* gofmt

* Forgot io

* Make path by hash exported

* updated to simplify again whoops

* Updated server to work real good

* Forgot ampersand

* Updated to match FilePiece

* Merged in cam's delete code

* Remove unused io

* Added RPC code

* Give the download request a reader

* Removed http server stuff; changed receive stream to say io.reader

* Added expiration date to shardInfo

* Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions

* added ttl info request

* Move scripts to http server pr; added close method for Retrieve api

* added rpc server tests for getting piece meta data and retrieval routes

* Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf

* Imported cams test

* Bump gometalinter deadline

* WIP adding tests

* added tests for store and delete routes

* Add changes as requested by Kaloyan, also cleaned up some code

* Get the code actually working whoops

* More cleanup

* Separating database calls from api.go

* need to rename expiration

* Added some changes requested by JT

* Fix read size

* Fixed total amount to read

* added tests

* Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages

* Moved rpc client and server to piece store

* Moved piecestore protobuf to the /protos folder

* Cleaned up messages

* Clean up linter errors

* Added missing sqlite import

* Add ability to do iterative reads and writes to pstore

* Incrementally read data

* Fix linter and import errors

* Solve linter Error

* Change return types

* begin test refactor

* refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go

* Cleaning up tests

* Added retrieve tests

* refactored delete tests

* Deleted old tests

* Updated cmd/piecestore to reflect changes to piecestore

* Refactored server tests and server/client store code

* gofmt

* WIP implementing TTL struct

* Read 4k at a time when Retrieving

* implemented ttl struct

* Accidentally removed fpiece dependency?

* Double resolve merge conflict

* Moved client to the examples since it is an example

* Change hash to id in protobuf. TODO: Change client and server to reflect these changes

* Update client and protobuf

* changed hash to id

* Handle eof properly in retrieve

* Id -> ID

* forgot to change import for client after moving

* Moved client and server main to examples

* Made changes requested by JT

* checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close()

* Print not fatal

* Handle overflow when reading from server

* added const IDLength

* removed types from comments

* Add reader/writer for download data from and uploading data to server

* goimports and comments

* fixed nits, casts, added OK const, DBCleanup now exits program on error

* Add stream reader and writer for server

* Fix errors

* i beforee except after c lol

* customizable data dir

* Forgot to rename variable

* customizable data dir

* Handle closing of server stream properly

* linter

* pass on inserting the same data twice

* linter

* linter

* Do the easy things JT asked for

* Handle overflow from reads properly; handle custom db path

* Handle overflow for server stream read; TODO Combine server and client stream reads

* Allow for TTL of 0 to stay forever

* Make Client cleaner for user
2018-06-02 14:14:59 -04:00
Natalie Villasana
6723064bfb
Enables netstate service to save pointers (#49)
* adds pointer to netstate proto file

* generated updated netstate proto

* changes boltdb netstate to save pointers as values

* updates netstate Put to save Pointers, updates client example to put a pointer, adds grpc status errors, updates tests, changes boltdb 'File' struct to 'PointerEntry'

* updates netstate client example and client test to save pointers, updates netstate List and Delete

* begins adding netstate-http tests

* removes netstate http service

* re-adds netstate auth

* updates boltdb netstate test

* changes encrypted_unencrypted_size from int64 to bytes in netstate proto

* updates READMEs
2018-05-29 22:47:40 -04:00
Natalie Villasana
1a06c2b0c7 adding grpc interface to network state (#33)
* adds proto files for netstate crud

* moves netstate grpc client lib into pkg/netstate where grpc netstate service is defined

* starts adding grpc client and server tests

* moves creation of grpc server into cmd/netstate/main.go, removes pkg/netstate/service.go, adds more client testing

* changed all 'Path' and 'Value' fields from strings to bytes, updated tests

* changes Get and Delete in proto file to receive 'requests' instead of 'file paths', adds tests for Get, List, and Delete

* changes netstate-routes to get 'fileValue' bytes not 'fileInfo'

* adds example rpc client in 'examples' and adds more specific debug logs

* adds readmes for netstate rpc services and updates netstate-routes
2018-05-14 18:31:26 -06:00
JT Olds
2ef8e581b2 fix travis, and get a clean lint (#32)
* fix travis

* lint fixes

* travis: enable tests

* fix some tests, disable a broken eestream test for now
2018-05-08 18:02:01 -04:00
JT Olds
4f40280c71 some small import cleanups, go generate overlay protos (#25)
checking in go generated stuff is recommended by
https://blog.golang.org/generate
2018-05-01 13:24:32 -04:00
Dennis Coyle
9e31cfc7a0 overlay proto 2018-04-16 10:09:01 -06:00
Dennis Coyle
b236d7dffb keep 2018-04-06 12:36:55 -04:00