Commit Graph

158 Commits

Author SHA1 Message Date
Bryan White
249244536a
CSR Service (part 2): cert signing rpc (#950)
* CSR Service:

+ implement certificate sign rpc method
+ implement certificate signer client/server
+ refactor `AuthorizationDB#Create`
+ refactor `NewTestIdentity`
+ add `AuthorizationDB#Claim`
+ add `Token#Equal`
+ fix `Authorizations#Marshal` when marshaling identities and certificates
+ tweak `Authorization#String` format
+ cert debugging improvements (jsondiff)
+ receive context arg in `NewTestIdentity`
+ misc. fixes
2019-01-02 12:39:17 -05:00
Bryan White
2016ce9fd6
Certificate revocation (#836)
* wip certificate revocation

* refactor tests

* wip testing

* testing

* review fixes

* integration fix attempt #1

* review fixes

* integration fix attempt #2

* linter fixes

* add copywrite

* integration fix attemp #3

* more testing

* more tests

* go mod tidy

* review fixes

* linter fixes
2018-12-13 21:01:43 +01:00
Bryan White
228aa34ff6
TLS extension processing (#771) 2018-12-07 14:44:25 +01:00
Matt Robinson
4237afb1c1
Attempt at fixing the all-in-one environment (#736) 2018-12-04 16:10:23 -05:00
Egon Elbre
3e461df470
Linting protobuf files (#676) 2018-11-30 17:02:01 +02:00
Matt Robinson
471f9004aa
Fix arm builds (#595) 2018-11-26 09:11:09 -05:00
Egon Elbre
42ff835e6e
Fail linter when copyright is missing. (#664) 2018-11-15 14:47:52 +02:00
Matt Robinson
c5f5915588
build the gateway binary too (#592) 2018-11-07 13:19:09 -05:00
Matt Robinson
d1e7534f9e
use GOPROXY if it's set in the environment (#495) 2018-10-17 15:06:26 -04:00
Matt Robinson
48eb8e5c8e
sign windows builds (#494) 2018-10-17 14:21:18 -04:00
Matt Robinson
bcd18a44bc
give our appearence on windows some love (#429) 2018-10-16 14:48:17 -04:00
Matt Robinson
00358fc368
clean up Makefile (#487) 2018-10-16 14:47:43 -04:00
Matt Robinson
a1f93285e2 figure out which deployments need to be patched during a deployment instead of guessing (#438) 2018-10-08 12:52:27 -04:00
Egon Elbre
e7e2d4d7c9
use golangci-lint and add missing error checks (#340) 2018-09-11 16:13:25 +03:00
Egon Elbre
6ee6f0fdf5
Remove duplicated mock store implementations (#337) 2018-09-11 10:27:12 +03:00
Matt Robinson
75c64035a2 Normalize the Docker Bits (#325)
* Normalize the dockerfiles and entrypoints

* fix formattting and docker-compose file

* add missing env var to satellite dockerfile

* Remove build args no longer in use

* More changes to make it all work
2018-09-07 15:46:08 -04:00
Matt Robinson
2c315db84b Fix the binaries and Docker Images (#306)
* Finally fix the binary builds for a couple os/arch combinations

* Make the file executable before adding it to the zip

* This cache hack wasn't suppose to make it into the PR

* Guard against calling the binary target by itself

* Order matters on OS X

* Vendored my image

* No 386 builds until storage is fixed.
2018-09-05 16:40:47 -04:00
Matt Robinson
0a78e741ec
Build and upload binaries (#296)
* First pass at building binaries

* Finished building images

* Reworked some bits to save binaries in a better place.

* First pass at uploading binaries

* Builds for freebsd fail currently

* Add ignore file for docker builds to not invalidate the cache as often

* Docker image for 1.11-alpine exists now

* Name windows binaries correctly

* The makefile is used in tests.

* Take a swing at updating the tests for 1.11

* Switch to something with glibc so the race detector works.

* Remove unused .PHONY targets
2018-08-31 11:21:44 -04:00
aligeti
743ec65204
hc to satellite rename (#282)
* hc to satellite rename

* rebase with master and fix spelling

* missed out rename

* rebase merge fix
2018-08-29 14:32:41 -04:00
Egon Elbre
23231a29b1
Reduce linter concurrency (#297)
Limit concurrency to reduce memory usage.
2018-08-28 18:00:18 +03:00
JT Olio
58c810519f
cross-compilation: fix darwin (#293)
* cross-compilation: fix darwin

* minio without fsnotify

* update Gopkg.lock

* make release target
2018-08-27 15:06:55 -06:00
Egon Elbre
0f5a2f4ef5 Enable more linters (#272)
* enable more linters

* Run gofmt -s

* run goimports

* run unconvert

* fix naked return

* fix misspellings

* fix ineffectual assigments

* fix missing declaration

* don't use deprecated grpc.Errof

* check errors in tests

* run gofmt -w -r "assert.Nil(err) -> assert.NoError(err)"

* fix directory permissions

* don't use nil Context

* simplify boolean expressions

* use bytes.Equal instead of bytes.Compare

* merge variable declarations, remove redundant returns

* fix some golint errors

* run goimports

* handle more errors

* delete empty TestMain

* delete empty TestMain

* ignore examples for now

* fix lint errors

* remove unused values

* more fixes

* run gofmt -w -s .

* add more comments

* fix naming

* more lint fixes

* try switching travis to go1.11

* fix unnecessary conversions

* fix deprecated methods

* use go1.10 and disable gofmt/goimports for now

* switch to 1.10

* don't re-enable gofmt and goimports

* switch covermode to atomic because of -race

* gofmt
2018-08-27 11:28:16 -06:00
aligeti
09da23737a
rename farmers to storagenode (#275)
* rename farmers to storagenode

* review changes

* merge conflicts
2018-08-24 21:52:58 -04:00
JT Olio
560e9fd9ec
uplink: make ul/uplink consistent (#286)
* uplink: make ul/uplink consistent

* pstore: make test more robust
2018-08-24 14:02:42 -06:00
aligeti
4edba19986 [Ready for review]Rename gw uplink (#273)
Rename gw uplink
2018-08-24 10:22:24 -06:00
Matt Robinson
3456a127cf
More fixes to the staging deploy process (#278) 2018-08-23 17:17:47 -04:00
Matt Robinson
631331236e
Fix staging deploy of satellite and storage-node (#277) 2018-08-23 17:04:51 -04:00
Matt Robinson
026997e725
All-in-one docker-compose project (#267)
* First pass at an all-in-one compose project

* Sort out dependencies so uplink can build in docker

* Added the final bits to make this work

* Add readme for all-in-one

* Fix sed to work on OS X and Linux

* Make the readme for all-in-one better

* Only bring up the satellite and uplink after fixing the mock overlay

* Use a dummy value for mock overlay for now

* Finishing touches

* Fix url form for aws cli tool

* Move files out of all-in-one directory to help with directory confusion.

* Added a make target to make all-in-one even better
2018-08-23 11:48:03 -04:00
Matt Robinson
5841065420 Attempt to fix jenkins builds (#260)
* Attempt to clean up docker image tags

* Clean only master

* Add back go version and actually use it
2018-08-21 22:54:56 -04:00
Matt Robinson
1687ce25c7 Fix the sha3 error with the build (#230)
* Fix the build
2018-08-15 14:49:30 -04:00
Maximillian von Briesen
f80ddf9358
Test captplanet with travis (#226)
* test captplanet in travis

* add comparison between uploaded and downloaded file

* test inline and remote files

* move captplanet testing to bash script
2018-08-14 12:58:16 -04:00
Philip Hutchins
c77d1b7f0b
Updating service names for docker tags (#207) 2018-08-08 13:48:59 -04:00
Dennis Coyle
5de5428d3a
First pass at node restrictions (#146)
* WIP First pass at node restrictions

* adjustments to storage clients

* fix cover

* undo previous import change

* adding copyright

* fix the tests

* check for keys

* moar fixes to tests

* linter

* change how we handle restrictions

* add generated pb.go file

* PR comments addressed

* MockKeyValueStore

* pr comments addressed

* missing )

* past my bedtime

* moar merge issues

* cleanup
2018-08-01 10:15:38 -04:00
Maximillian von Briesen
36c394c295
StatDB (#144)
* add statdb proto and example client

* server logic

* update readme

* remove boltdb from service.go

* sqlite3

* add statdb server executable file

* create statdb node table if it does not exist already

* get UpdateBatch working

* update based on jt review

* remove some commented lines

* fix linting issues

* reformat

* apiKey -> APIKey

* update statdb client apiKey->APIKey
2018-07-27 17:11:44 -04:00
JT Olio
4cff003ec9 fix docker makefile (#170)
* fix makefile
2018-07-26 16:01:38 -04:00
Matt Robinson
0cbbc69777 Add files for testing builds in docker (#161)
* Add files for testing builds in docker

* Make tests check for redis running before trying to start redis-server, which may not exist.

* Clean redis server before any tests use it.

* Add more debugging for travis

* Explicitly requiring redis for travis
2018-07-25 17:47:02 -04:00
Alexander Leitner
cde0ffaa52 Add dockerfile and yaml for setting up piecestore servers (#102)
* Add dockerfile and yaml for setting up piecestore servers

* Fix dockerfile for @aleitner (#115)

* Fix dockerfile for @aleitner

* Move files for @coyle

* Update yaml

* My linter had some errors so I resolved them

* Make jenkins do the needful

* Make piecestore-farmer look like overlay's build process

* Fix service spec to work in staging

* Make Jenkins push images, but not deploy them, yet.

* Modify entrypoint to fit new verbs

* Update piecestore-farmer entrypoint script to handle new app output
2018-07-19 15:42:50 -04:00
Cameron
6463b87ebe
Errcheck (#133)
* add errcheck

* fixed linter errors

* fixes

* errcheck fixes in pkg/paths

* Fix errchecks in PieceID.Derive

* Fix ecclient tests

* Move closeConn a little bit above in the exectution flow

* fix new lint errors

* Fatalf -> Printf

* address eclipsed errors

* rename err to closeErr

* rename err to closeErr for differentiation
2018-07-16 15:22:34 -04:00
Matt Robinson
544cf46b10 Fix jenkins (#139)
* Attempt to fix Jenkinsfile

* Only deploy on master
2018-07-13 11:02:33 -04:00
Dennis Coyle
f2b6c18f2a adding coveralls / code coverage (#112)
* adding coveralls

* adding code coverage badge

* fixing badges

* verbose

* swap tests and coverage

* extra line

* maybe

* maybe

* moar

* gover maybe

* testing

* cleanup
2018-06-26 13:48:02 +02:00
Dennis Coyle
2b4a1ef17d
Coyle/docker fix (#109)
* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge branch 'master' of https://github.com/storj/storj

* fixing tag name

* no idea

* testing

* changes

* testing on travis

* testing

* changes to travis build

* new approach

* Merge branch 'master' into coyle/docker-fix
2018-06-22 16:17:37 -04:00
Dennis Coyle
a5ff5016c9
pkg/kademlia tests and restructuring (#97)
* port changes

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* Merge remote-tracking branch 'upstream/master'

* files created

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* Merge remote-tracking branch 'upstream/master' into coyle/kad-tests

* wip

* remove bkad dependencie from tests

* wip

* wip

* wip

* wip

* wip

* updated coyle/kademlia

* wip

* cleanup

* ports

* overlay upgraded

* linter fixes

* piecestore kademlia newID

* add changes from kad demo

* PR comments addresses

* go func

* force travis build

* fixed merge conflicts

* fixed merge conflicts

* Merge branch 'coyle/kad-tests' of https://github.com/coyle/storj into coyle/kad-tests

* linter issues

* linting issues

* fixed merge conflicts

* linter is stupid
2018-06-22 09:33:57 -04:00
Dennis Coyle
43cdba29e3 prepend storjlabs to docker tag (#108)
* port changes

* fixing tag name
2018-06-21 18:08:31 -06:00
Dennis Coyle
2181932e31 Automatically build, tag and push docker images on merge to master (#103)
* port changes

* build overlay on successful merge to master

* fixes to Makefile

* permissions

* dep ensure

* gopath

* let's try vgo

* remove dep

* maybe alpine is the issue

* tagging go version on build

* stupid vgo

* vgo

* adding tags to push

* quotes

* local linting fixes & stupid travis
2018-06-21 15:36:39 -04:00
Bryan White
1ebd66d880 Cleanup & bolt-backed cache (#87)
* wip post-demos cleanup

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* Reorganize:

storage
├── common
│   └── common.go `package storage`
├── boltdb
│   └── ...
└── redis
    └── ...

storage
├── common.go `package storage`
├── boltdb
│   └── ...
└── redis
    └── ...

storage
├── common
│   ├── common.go `package storage`
│   └── testing.go `package storage` <--
└── ...

internal
├── app
│   └── cli
├── pkg
│   └── readcloser
└── test
    └── util.go `package test` <--

* remove comment

* add and use goimports

* add test types & fix some lint errors

* better typing

* fixing linter issues/comments

* goimports

* goimports

* more linter fixes; replace panic with NoError assertions in tests

* fix typo/more linter errors

* moar better linter fixes

* even moar better linter fixes

* linter

* add localPort back

* fixing exports, imports, and add comments
2018-06-13 14:22:32 -04:00
Dylan Lott
325a70d514 Cache (#67)
* add reference to dht to overlay client struct

* wip

* wip

* Implement FindNode

* get nodes

* WIP

* Merge in Dennis kademlia code, get it working with our code

* ping and moar

* WIP trying to get cache working with kademlia

* WIP more wiring up

* WIP

* Update service cli commands

* WIP

* added GetNodes

* added nodes to Kbucket

* default transport changed to TCP

* GetBuckets interface changed

* filling in more routing

* timestamp methods

* removed store

* Added initial network overlay explorer page

* Updating and building with dockerfile

* Working on adding bootstrap node code

* WIP merging in dennis' code

* WIP

* connects cache to pkg/kademlia implementation

* WIP redis cache

* testing

* Add bootstrap network function for CLI usage

* cleanup

* call bootstrap on init network

* Add BootstrapNetwork function to interface

* Merge in dennis kad code

* WIP updates to redis/overlay client interface

* WIP trying to get the DHT connected to the cache

* go mod & test

* deps

* Bootstrap node now setting up correctly

- Need to pass it through CLI commands better

* WIP adding refresh and walk functions, added cli flags

- added cli flags for custom bootstrap port and ip

* PR comments addressed

* adding FindStorageNodes to overlay cache

* fix GetBucket

* using SplitHostPort

* Use JoinHostPort

* updates to findstoragenodes response and request

* WIP merge in progress, having issues with a panic

* wip

* adjustments

* update port for dht bootstrap test

* Docker

* wip

* dockerfile

* fixes

* makefile changes

* Update port in NewKademlia call

* Update local kademlia DHT config

* kubernetes yaml

* cleanup

* making tests pass

* k8s yaml

* lint issues

* Edit cli flags to allow for configurable bootstrap IP and Port args

* cleanup

* cache walking the network now

* Rough prototype of Walk function laid out

* Move walk function into bootstrap function

* Update dht.go

* changes to yaml

* goimports
2018-06-05 17:06:37 -04:00
Alexander Leitner
ff8e191a9a
Added rpc server (#40)
* Added piecestore

* gofmt

* Added requested changes

* Added cli

* Removed ranger because I wanted something that can stand alone

* Add example of http server using piece store

* Changed piecestore code to make it more optial for error handelling

* Merged with piecestore

* Added missing package

* Forgot io import

* gofmt

* gofmt

* Forgot io

* Make path by hash exported

* updated to simplify again whoops

* Updated server to work real good

* Forgot ampersand

* Updated to match FilePiece

* Merged in cam's delete code

* Remove unused io

* Added RPC code

* Give the download request a reader

* Removed http server stuff; changed receive stream to say io.reader

* Added expiration date to shardInfo

* Change all instances of Shard to Piece; change protobuf name; moved client insance to outside functions

* added ttl info request

* Move scripts to http server pr; added close method for Retrieve api

* added rpc server tests for getting piece meta data and retrieval routes

* Resolved linter errors, moved to prc server to pkg, updated go.mod to use latest protobuf

* Imported cams test

* Bump gometalinter deadline

* WIP adding tests

* added tests for store and delete routes

* Add changes as requested by Kaloyan, also cleaned up some code

* Get the code actually working whoops

* More cleanup

* Separating database calls from api.go

* need to rename expiration

* Added some changes requested by JT

* Fix read size

* Fixed total amount to read

* added tests

* Simplify protobuf, add store tests, edited api to handle invalid stores properly, return errors instead of messages

* Moved rpc client and server to piece store

* Moved piecestore protobuf to the /protos folder

* Cleaned up messages

* Clean up linter errors

* Added missing sqlite import

* Add ability to do iterative reads and writes to pstore

* Incrementally read data

* Fix linter and import errors

* Solve linter Error

* Change return types

* begin test refactor

* refactored to implement only 1 db connection, moved SQLite row checking into separate function and removed defer on rows.Close(), fixed os.tempDir in rpc_test.go

* Cleaning up tests

* Added retrieve tests

* refactored delete tests

* Deleted old tests

* Updated cmd/piecestore to reflect changes to piecestore

* Refactored server tests and server/client store code

* gofmt

* WIP implementing TTL struct

* Read 4k at a time when Retrieving

* implemented ttl struct

* Accidentally removed fpiece dependency?

* Double resolve merge conflict

* Moved client to the examples since it is an example

* Change hash to id in protobuf. TODO: Change client and server to reflect these changes

* Update client and protobuf

* changed hash to id

* Handle eof properly in retrieve

* Id -> ID

* forgot to change import for client after moving

* Moved client and server main to examples

* Made changes requested by JT

* checkEntries is now private, created currentTime variable for checkEntries, added defer rows.Close()

* Print not fatal

* Handle overflow when reading from server

* added const IDLength

* removed types from comments

* Add reader/writer for download data from and uploading data to server

* goimports and comments

* fixed nits, casts, added OK const, DBCleanup now exits program on error

* Add stream reader and writer for server

* Fix errors

* i beforee except after c lol

* customizable data dir

* Forgot to rename variable

* customizable data dir

* Handle closing of server stream properly

* linter

* pass on inserting the same data twice

* linter

* linter

* Do the easy things JT asked for

* Handle overflow from reads properly; handle custom db path

* Handle overflow for server stream read; TODO Combine server and client stream reads

* Allow for TTL of 0 to stay forever

* Make Client cleaner for user
2018-06-02 14:14:59 -04:00
JT Olds
dd2fbb5069 don't require shallow clones of old commits (#38)
* don't require shallow clones of old commits

doesn't look like github allows it. hints here:
https://stackoverflow.com/questions/31278902/how-to-shallow-clone-a-specific-commit-with-depth-1

* don't try to update submodules
2018-05-09 16:21:53 +03:00
JT Olio
78989a2d5a fix hanging travis test 2018-05-08 17:16:16 -06:00
JT Olds
2ef8e581b2 fix travis, and get a clean lint (#32)
* fix travis

* lint fixes

* travis: enable tests

* fix some tests, disable a broken eestream test for now
2018-05-08 18:02:01 -04:00
Dennis Coyle
52741af9ce make travis work? 2018-04-30 15:27:30 -04:00
Dennis Coyle
bb8d4f8d22 cleanup 2018-04-25 10:55:26 -04:00
Dennis Coyle
fe60ba02a3 lint step for copyright headers (#16)
* overlay proto

* grpc server and client

* fix import to storj

* tests

* change imports

* imports cleanup/comments

* PR comments addressed from @jtolds

* lint step for copyright headers

* spelling

* PR comments addressed from @jtolds
2018-04-23 10:05:52 -06:00
Bryan White
2e1ca41d39 bugfix: https://github.com/alecthomas/gometalinter#installing 2018-04-18 14:13:18 -04:00
Dennis Coyle
bd0d2778f0 add linter to build-dev-deps 2018-04-17 14:18:39 -04:00
Dennis Coyle
7d9d0e940c updating how we get dev deps 2018-04-17 13:29:15 -04:00
Dennis Coyle
1975b98eba grpc server and client 2018-04-16 10:09:01 -06:00
Dennis Coyle
250b36f27b wip on structure 2018-04-06 12:32:34 -04:00