Commit Graph

52 Commits

Author SHA1 Message Date
Michal Niewrzal
034a34069a
Merge bwagreement db into satellite master db (#770)
* Merge bwagreement db into satellite master db

* adjust to recent tally changes

* linter problems

* linter problems

* returning db structs in more optimal way

* added pointer for assignment

* error message changed

* better param message
2018-12-07 10:59:31 +01:00
JT Olio
b6c936340b
clean up uplink config section (#786) 2018-12-06 17:18:11 -07:00
Michal Niewrzal
44a6cfa1c1
Initial satellite master database setup (#735)
* Initial satellite master database setup

* fixed unsed ctx and /nolint added for temp change

* tests for two db drivers + connection param renamed
2018-12-05 10:35:50 +01:00
Maximillian von Briesen
2ab15196d9
Get rid of statdb grpc stuff (#761)
* remove sdbclient

* do not register statdb grpc server/remove apikey from statdb

* start audit service after statdb in captplanet
2018-12-04 13:47:58 -05:00
Michal Niewrzal
06c196d28c
Allow config generation into empty directory (#752)
* Allow config generation into empty directory

* unifed messages

* move main method to fpath

* fix linter problems
2018-12-04 12:42:16 +01:00
Michal Niewrzal
1206ef9126
Use OS related default setup directory (#741)
* Use OS related default setup directory

* added missing comment

* adjust test-captplanet.sh

* ApplicationDir moved to fpath
2018-12-03 16:51:56 +01:00
Dylan Lott
0d05cb26bf
Kademlia Inspector CLI (#657)
* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* Fix error return

* Trying to get tests passing

* Adds method on dht mock for tests

* Add dbx files back

* Fix package import error in dbx file

* Adds copyrights to pass linter

* tidy go mod

* Updates from code review

* Updates inspector to take flag arguments for address

* Format list-buckets output more prettier
2018-11-21 10:31:27 -07:00
Dennis Coyle
e7e0d1daaa
Coyle/disable mock (#627)
* disabled mock overlay from captplanet
2018-11-20 11:54:52 -05:00
Jennifer Li Johnson
93c5f385a8
Enable checker in captplanet and staging (#643)
* enable checker

* add option to use mock overlay in checker

* adds logs to checker

* appease linter
2018-11-20 10:54:22 -05:00
Alexander Leitner
5e9377524f
Refactor basystem (#641)
* Signature verification

* Clean up agreement sender to have less errors

* overlay address in captnplanet

* Refactor bandwidth.proto to not use streams

* Make sure the send worked

* Handle connection to satellite

* Save renter public key inside of renter bandwidth allocations

* Default diag to sqlite. Make configurable

* Separate bw server and dbm; regenerate dbx files

* Make sure test uses protobufs

* Demonstrate creating bandwidth allocations
2018-11-15 14:06:09 -05:00
Yaroslav Vorobiov
c442205b3a
account api: infrastructure, login, register, getUser (#611) 2018-11-14 12:50:15 +02:00
Alexander Leitner
50d657af11
Cleanup agreement sender (#630)
* Signature verification

* code review comments & lint warnings

* Clean up agreement sender to have less errors

* overlay address in captnplanet
2018-11-13 10:37:49 -05:00
Natalie Villasana
3b572264ca
adds audit job (#521) 2018-11-07 01:16:43 +00:00
Bryan White
ee62e2a9d8
Use transport client and cleanup all the clients (#574)
* wip

* linter fixes

* linter fixes

* test fixes

* linter fixes

* fix merge + restructure piecestore packages

* review feedback

* linter fixes

* linter fixes

* remove unnecessary aliases to piecestore

* more merge fixing
2018-11-06 18:49:17 +01:00
Cameron
de46a999bc
Integrate SegmentStore Repair method with repair service (#582)
* add storeConfig struct and getSegmentStore helper for creating a segment store

* implement segment store in repairer, remove unnecessary repairer Repair method

* change repair method parameter from int to int32 to match type being passed in

* implement repairer service in captplanet

* rework Config, set Config defaults in captplanet/setup
2018-11-06 09:52:11 -05:00
Dennis Coyle
1d3367bb09
bypass data checker and repairer in captplanet (#563) 2018-10-31 20:46:47 -04:00
Dennis Coyle
a3becb8a7b
Add repairer & checker to Satelite (#561)
* Added repairer & checker to Satellite

* fixed repairer and checker configs
2018-10-31 12:22:35 -04:00
JT Olio
a42973f089
captplanet: fix output line name (#549)
Change-Id: I37fbb76586ee3c0a40d24aad72aff2fcfc039a25
2018-10-31 10:00:58 -06:00
Michal Niewrzal
b2caac4323
Option for generating sample TLS certs for Minio GW (#525)
* Option for generating sample TLS certs for Minio GW

* os.Link errors handled

* rename option and change cert template
2018-10-24 22:17:28 +02:00
Maximillian von Briesen
94128d63a1
fix captplanet and satellite overwrite flags (#516)
* fix captplanet and satellite overwrite flags
2018-10-19 11:06:02 -04:00
Dylan Lott
6b88f0a36b
Moves CollectErrors into pkg/utils (#493)
Merging but creating a PR for tests shortly
2018-10-17 14:23:44 -06:00
Dylan Lott
19d8b7a2df
return multiple errors when captplanet fails (#478)
* implements CombineErrors on runCfg error chan

* implement error channel timeout and CombineErrors

* implements CombineErrors with time.After timeout

* changes d to duration in args

* changes comment to better reflect what collectErrors does

* appeases the linter gods

* Gets rid of timeout bug in collectErrors
2018-10-17 09:56:39 -06:00
Matt Robinson
bcd18a44bc
give our appearence on windows some love (#429) 2018-10-16 14:48:17 -04:00
Egon Elbre
2dbada73ab
Separate storj.io packages in imports (#481)
Format with goimports -w -local storj.io
Add flag to golang lint
Remove commented out imports as they mess with goimports
2018-10-16 14:43:44 +03:00
Alexander Leitner
3e1b16ea99
Start redis (#470)
* Start miniredis, repairer, and checker with captplanet
2018-10-12 14:04:16 -04:00
Egon Elbre
db42772b4d
Enable goimports and fix formatting. (#459) 2018-10-11 23:25:54 +03:00
Michal Niewrzal
ad327bedb1
Use context to propagate API Key (#383)
* Satellite signs proofs

* wip

* remove direct apikey usage from pdbclient

* adjusting unit tests

* fix linter errors

* unit tests

* linter errors

* remove usless interface

* remove unused code

* improve unit tests

* signature generation

* code review changes

* code review comments

* back to satellite-id signature generation

* remove go-grpc-middlewar dependency

* small step back

* linter fixes

* fix tests

* packages reorganization

* Move TestAPIKeyInjector to grpcauth package
2018-10-09 16:39:14 +02:00
littleskunk
ddbe039908
captplanet: api and encryption key setup (#441)
* captplanet: encryption api and key setup
2018-10-08 22:04:32 +02:00
Dennis Coyle
dee2c137c8
Remove BKAD dependency from pkg/kademlia (#294)
* slowly but surely

* hardcode ID for tests so we can get predictable results

* skipping bad test

* removing tests of bkad

* wip

* new algorithm for worker

* clean up

* remove skipped test

* changes

* uncomment

* fixed conflicts

* maybe done ?

* cleanup

* boot bkad

* wip

* cleanup

* undo change

* fixes

* wip

* wip

* moving nodeID around

* wip

* wip

* fixes

* fixes after merge

* added TODO

* fixed tests post identity

* linter fixes

* wip

* PR review comments

* wip

* fixing tests

* fix tests

* force db directory

* bad test

* fixes race condition

* small cleanups

* adding db folder

* testing

* wip

* cleanup

* cleanup

* linters

* export Restrict

* add timeout

* testing

* linters

* forgot one

* moar fixes from master merge

* PR comments

* moar PR comments

* removed stun flag

* remove duplicate declaration

* remove old tests

* remove timeout

* fix tests

* missed one

* changed StringToID >> IDFromString

* PR comments

* stupid linter

* moevd overlay mock

* fixed merge conflicts

* fixes

* linter
2018-10-08 11:09:37 -04:00
Alexander Leitner
dc8bea2cd1
Repairer points to redis server (#427)
* Let's do it right this time

* Oh travis...

* Handle redis URL

* Travis... why u gotta be like this?

* Handle when address does not use redis scheme

* Start repairer

* Match provider.Responsibility interface

* Simplify if statement

* Config doesn't need to be a pointer

* Initialize doesn't need to be exported

* Don't run checker or repairer on startup

* Fix travis complaints
2018-10-05 11:58:07 -04:00
Dylan Lott
f7afafe4e1
Repair cron (#420)
* Creates cron-job for checker, adds it to captplanet and satellite

* removes datarepair from satellite & captplanet run

* Delete config.go

* removes unused datarepair imports

* adds comments to fix linter
2018-10-04 15:40:34 -06:00
Egon Elbre
3b98b93f05
captplanet: add ctrl+\ for dumping goroutines (#416) 2018-10-03 15:27:51 +03:00
Maximillian von Briesen
821d0b6f1d
Stream encryption (#302)
* begin adding encryption for remote pieces

* begin adding decryption

* add encryption key as arg to Put and Get

* move encryption/decryption to object store

* Add encryption key to object store constructor

* Add the erasure scheme to object store constructor

* Ensure decrypter is initialized with the stripe size used by encrypter

* Revert "Ensure decrypter is initialized with the stripe size used by encrypter"

This reverts commit 07272333f461606edfb43ad106cc152f37a3bd46.

* Revert "Add the erasure scheme to object store constructor"

This reverts commit ea5e793b536159d993b96e3db69a37c1656a193c.

* move encryption to stream store

* move decryption stuff to stream store

* revert changes in object store

* add encryptedBlockSize and close rangers on error during Get

* calculate padding sizes correctly

* encryptedBlockSize -> encryptionBlockSize

* pass encryption key and block size into stream store

* remove encryption key and block size from object store constructor

* move encrypter/decrypter initialization

* remove unnecessary cast

* Fix padding issue

* Fix linter

* add todos

* use random encryption key for data encryption. Store an encrypted copy of this key in segment metadata

* use different encryption key for each segment

* encrypt data in one step if it is small enough

* refactor and move encryption stuff

* fix errors related to nil slices passed to copy

* fix encrypter vs. decrypter bug

* put encryption stuff in eestream

* get captplanet test to pass

* fix linting errors

* add types for encryption keys/nonces and clean up

* fix tests

* more review changes

* add Cipher type for encryption stuff

* fix rs_test

* Simplify type casting of key and nonce

* Init starting nonce to the segment index

* don't copy derived key

* remove default encryption key; force user to explicitly set it

* move getSegmentPath to streams package

* dont require user to specify encryption key for captplanet

* rename GenericKey and GenericNonce to Key and Nonce

* review changes

* fix linting error

* Download uses the encryption type from metadata

* Store enc block size in metadata and use it for download
2018-09-26 09:32:23 -04:00
Brandon Iglesias
f1a82e588c updating the reed solomon numbers (#350)
* Update config.go

* updating captplanet

* Fix captplanet setup command for >= 100 nodes
2018-09-25 18:38:37 +03:00
JT Olio
e0226790e6
miniogw: simplify output (#330) 2018-09-08 12:53:52 -06:00
aligeti
743ec65204
hc to satellite rename (#282)
* hc to satellite rename

* rebase with master and fix spelling

* missed out rename

* rebase merge fix
2018-08-29 14:32:41 -04:00
JT Olio
5f54f2ca8b
captplanet: customizable mock overlay host (#284) 2018-08-27 12:17:19 -06:00
Egon Elbre
0f5a2f4ef5 Enable more linters (#272)
* enable more linters

* Run gofmt -s

* run goimports

* run unconvert

* fix naked return

* fix misspellings

* fix ineffectual assigments

* fix missing declaration

* don't use deprecated grpc.Errof

* check errors in tests

* run gofmt -w -r "assert.Nil(err) -> assert.NoError(err)"

* fix directory permissions

* don't use nil Context

* simplify boolean expressions

* use bytes.Equal instead of bytes.Compare

* merge variable declarations, remove redundant returns

* fix some golint errors

* run goimports

* handle more errors

* delete empty TestMain

* delete empty TestMain

* ignore examples for now

* fix lint errors

* remove unused values

* more fixes

* run gofmt -w -s .

* add more comments

* fix naming

* more lint fixes

* try switching travis to go1.11

* fix unnecessary conversions

* fix deprecated methods

* use go1.10 and disable gofmt/goimports for now

* switch to 1.10

* don't re-enable gofmt and goimports

* switch covermode to atomic because of -race

* gofmt
2018-08-27 11:28:16 -06:00
aligeti
09da23737a
rename farmers to storagenode (#275)
* rename farmers to storagenode

* review changes

* merge conflicts
2018-08-24 21:52:58 -04:00
aligeti
4edba19986 [Ready for review]Rename gw uplink (#273)
Rename gw uplink
2018-08-24 10:22:24 -06:00
JT Olio
fa390c9ec9
Mock overlay move (#245)
* move mock overlay from client to server

this doesn't really change much, but it does allow you to
run a standalone gateway against captain planet. it still does
not allow you to run a standalone gateway against a standalone
heavy client

* pointerdb: small error fixes

* some cleanups

* fix tests
2018-08-20 14:24:11 -04:00
Alexander Leitner
b0db33f919
Bandwidth accounting (#134)
* captplanet standalone farmer setup

* Bandwidth Allocation

* utils.Close method changed to utils.LogClose

* Get build temporarily working

* Get/Put for PSClient should take payer bandwidth allocations rather than the NewPSClient function

* Update example client to reflect changes in client API

* Update ecclient to use latest PSClient, Make NewPSClient return error also

* Updated pieceranger tests to check for errors; sign method should take byte array

* Handle defers in store.go better

* Fix defer functions in psdb.go

* fun times

* Protobuf bandwidthallocation data is now a byte array

* Remove psservice package and merge it into pstore server

* Write wrapper for database calls

* Change all expiration names in protobuf to be more informative; add defer in retrieve; remove old comment

* Make PSDB tests implementation independent rather than method independent

* get rid of payer, renter in ecclient

* add context monitoring in store and retrieve
2018-08-17 13:40:15 -04:00
JT Olio
1412d5a110
fix gw and hc (#225)
* fix gw and hc

* resolve paths

* i keep forgetting gw is different
2018-08-13 12:29:13 -06:00
JT Olio
df726b3ee8
fix services after identity merge (#224)
unfortunately, we can't enable server-side tls yet because
client-side tls hasn't been implemented
2018-08-13 09:39:42 -06:00
JT Olio
ab029301bd
Consolidate command configuration and setup (#221) 2018-08-13 09:07:05 -06:00
Bryan White
5d20cf8829
Node Identity (#193)
* peertls: don't log errors for double close

understood that this part of the code is undergoing heavy change
right now, but just want to make sure this fix gets incorporated
somewhere

* git cleanup: node-id stuff

* cleanup

* rename identity_util.go

* wip `CertificateAuthority` refactor

* refactoring

* gitignore update

* wip

* Merge remote-tracking branch 'storj/doubleclose' into node-id3

* storj/doubleclose:
  peertls: don't log errors for double close

* add peertls tests & gomports

* wip:

+ refactor
+ style changes
+ cleanup
+ [wip] add version to CA and identity configs
+ [wip] heavy client setup

* refactor

* wip:

+ refactor
+ style changes
+ add `CAConfig.Load`
+ add `CAConfig.Save`

* wip:

+ add `LoadOrCreate` and `Create` to CA and Identity configs
+ add overwrite to CA and identity configs
+ heavy client setup
+ refactor
+ style changes
+ cleanup

* wip

* fixing things

* fixing things

* wip hc setup

* hc setup:

+ refactor
+ bugfixing

* improvements based on reveiw feedback

* goimports

* improvements:

+ responding to review feedback
+ refactor

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* cleanup

* refactoring CA and Identity structs

* Merge branch 'master' into node-id3

* move version field to setup config structs for CA and identity

* fix typo

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* Merge branch 'master' into node-id3

* fix gateway setup finally

* go imports

* fix `FullCertificateAuthority.GenerateIdentity`

* cleanup overlay tests

* bugfixing

* update ca/identity setup

* go imports

* fix peertls test copy/paste fail

* responding to review feedback

* setup tweaking

* update farmer setup
2018-08-13 10:39:45 +02:00
Maximillian von Briesen
889afd0ae5
add overwrite flag for captplanet, hc, and gw setup (#209)
* add overwrite flag for captplanet setup

* add overwrite flag for heavy client and gateway
2018-08-08 18:22:59 -04:00
JT Olio
e66a0d289a
miniogw: make a Config.NewGateway helper, bypass overlay for now (#197) 2018-08-02 21:37:20 -06:00
JT Olio
30e3b503a4
captplanet: configure api key (#195) 2018-08-02 13:30:57 -06:00
JT Olio
79d2639ba1
cmd/hc, cmd/gw, cmd/captplanet: simplify setup/run commands (#178)
also allows much more customization of services within captain planet,
such as reconfiguring the overlay service to use redis
2018-07-30 01:38:31 -06:00