Commit Graph

39 Commits

Author SHA1 Message Date
Egon Elbre
d8df4b5f6b
Simplify provider tests (#862) 2018-12-17 17:09:52 +02:00
Bryan White
d8db7c3049
RevocationDB fixes (#866) 2018-12-14 21:45:53 +01:00
Bill Thorp
c17a6ba584
fixed temp dir issues w/ rev db (#859) 2018-12-13 21:24:35 -05:00
Bryan White
2016ce9fd6
Certificate revocation (#836)
* wip certificate revocation

* refactor tests

* wip testing

* testing

* review fixes

* integration fix attempt #1

* review fixes

* integration fix attempt #2

* linter fixes

* add copywrite

* integration fix attemp #3

* more testing

* more tests

* go mod tidy

* review fixes

* linter fixes
2018-12-13 21:01:43 +01:00
Egon Elbre
969fcabf72
Remove utils.LogClose from identity saving (#837) 2018-12-13 15:50:05 +02:00
JT Olio
ceb590fa67
capt: reduce nodes to 10 (#793)
Change-Id: Ief380fe29e3043657705cd7505c266fd774181a4
2018-12-11 11:40:54 -07:00
Bryan White
228aa34ff6
TLS extension processing (#771) 2018-12-07 14:44:25 +01:00
Egon Elbre
1e4556f88a
Fix import groupings (#739) 2018-11-30 15:40:13 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Dennis Coyle
e7e0d1daaa
Coyle/disable mock (#627)
* disabled mock overlay from captplanet
2018-11-20 11:54:52 -05:00
Jennifer Li Johnson
93c5f385a8
Enable checker in captplanet and staging (#643)
* enable checker

* add option to use mock overlay in checker

* adds logs to checker

* appease linter
2018-11-20 10:54:22 -05:00
Dennis Coyle
e4baec8fce
Coyle/tls verify (#612)
* add nodeID to dial opts

* fix tests to use real identity
2018-11-09 17:08:33 -05:00
Bryan White
8b9711cb5e
better waitlist-gating (#557)
* better waitlist-gating

(cherry picked from commit 490fe02b7c3558da18678dfb651c92ec9c4a75b5)

* fix broken test

* linter fixes

* linter fixes

* make extension verification optional

* add certifcate gating script for captplanet

* fixing tests

* linter fixes

* linter fixes?

* moar linter fixes

* Revert "moar linter fixes"

This reverts commit 8139ccbd73cbbead987b7667567844f50f7df2c8.

* just kill me

* refactor

* refactor tests

* liniter...

* cleanup
2018-11-01 16:48:43 +01:00
Dennis Coyle
3b7b2afb1f
Added ping support to node client (#491)
* added ping support to node client

* Added tests to Ping

* Added connection creation responsibility to Connection Pool
2018-10-26 12:38:22 -04:00
Bryan White
df1f7a6214 Wait-list gating (#534) 2018-10-26 09:52:37 -04:00
Egon Elbre
2dbada73ab
Separate storj.io packages in imports (#481)
Format with goimports -w -local storj.io
Add flag to golang lint
Remove commented out imports as they mess with goimports
2018-10-16 14:43:44 +03:00
Egon Elbre
caecb2801d
Cleanups related to transport (#475)
* Remove DialUnauthenticated method
* Use more specific node transport enum
* Formatting
* Rename Provider.g -> Provider.grpc
* Fix naming in dialer
2018-10-15 15:04:21 +03:00
Michal Niewrzal
351dc523d6
Satellite signature generation (#453)
* Satellite signature generation

* Add unit test

* remove unused var

* remove base64 encoding from signature generation
2018-10-11 16:35:55 +02:00
Michal Niewrzal
ad327bedb1
Use context to propagate API Key (#383)
* Satellite signs proofs

* wip

* remove direct apikey usage from pdbclient

* adjusting unit tests

* fix linter errors

* unit tests

* linter errors

* remove usless interface

* remove unused code

* improve unit tests

* signature generation

* code review changes

* code review comments

* back to satellite-id signature generation

* remove go-grpc-middlewar dependency

* small step back

* linter fixes

* fix tests

* packages reorganization

* Move TestAPIKeyInjector to grpcauth package
2018-10-09 16:39:14 +02:00
Kaloyan Raev
70caa86d89
Reduce garbage errors in captplanet when streaming (#435) 2018-10-09 10:29:00 +03:00
Dennis Coyle
dee2c137c8
Remove BKAD dependency from pkg/kademlia (#294)
* slowly but surely

* hardcode ID for tests so we can get predictable results

* skipping bad test

* removing tests of bkad

* wip

* new algorithm for worker

* clean up

* remove skipped test

* changes

* uncomment

* fixed conflicts

* maybe done ?

* cleanup

* boot bkad

* wip

* cleanup

* undo change

* fixes

* wip

* wip

* moving nodeID around

* wip

* wip

* fixes

* fixes after merge

* added TODO

* fixed tests post identity

* linter fixes

* wip

* PR review comments

* wip

* fixing tests

* fix tests

* force db directory

* bad test

* fixes race condition

* small cleanups

* adding db folder

* testing

* wip

* cleanup

* cleanup

* linters

* export Restrict

* add timeout

* testing

* linters

* forgot one

* moar fixes from master merge

* PR comments

* moar PR comments

* removed stun flag

* remove duplicate declaration

* remove old tests

* remove timeout

* fix tests

* missed one

* changed StringToID >> IDFromString

* PR comments

* stupid linter

* moevd overlay mock

* fixed merge conflicts

* fixes

* linter
2018-10-08 11:09:37 -04:00
Egon Elbre
4486e265bb
Cleanup after tests (#341) 2018-09-11 16:57:12 +03:00
Egon Elbre
e7e2d4d7c9
use golangci-lint and add missing error checks (#340) 2018-09-11 16:13:25 +03:00
Egon Elbre
83df0ee1b0
Implement ListV2 with storage rework (#303)
1. Added KeyValueStore.Iterate for implementing the different List, ListV2 etc. implementations. This allows for more efficient use of memory depending on the situation.
2. Implemented an inmemory teststore for running tests. This should allow to replace MockKeyValueStore in most places.
3. Rewrote tests
4. Pulled out logger from bolt implementation so it can be used for all other storage implementations.
5. Fixed multiple things in bolt and redis implementations.
2018-09-05 19:10:35 +03:00
Bryan White
746b63f685
CA and identity commands (#235)
* wip ca/ident cmds

* minor improvements and commenting

* combine id and ca commands and add $CONFDIR

* add `NewIdenity` test

* refactor `NewCA` benchmarks

* linter fixes
2018-08-28 00:23:48 +02:00
Alexander Leitner
899e1e68f1
Add functions for signing and verifying during bandwidth exchange (#246)
* Added initial functions for signing and verifying

* whoops

* Get client up to speed

* Added initial functions for signing and verifying

* whoops

* Get client up to speed

* wip

* wip

* actual signatures in tests

(cherry picked from commit 1464853b737f1d712d64fbf90147f535525c8fd9)

* bugfixing

* Generate private key in example

* Generate signatures for pieceranger tests

* Update examples to use TLS

* Use private key from identity inside of example

* Use crypto.PrivateKey interface

* Change err name in defers

* Pass tests

* Pass identity Key to PSClient

* Get tests passing on travis

* Resolve linter complaints
2018-08-27 14:35:27 -04:00
Egon Elbre
0f5a2f4ef5 Enable more linters (#272)
* enable more linters

* Run gofmt -s

* run goimports

* run unconvert

* fix naked return

* fix misspellings

* fix ineffectual assigments

* fix missing declaration

* don't use deprecated grpc.Errof

* check errors in tests

* run gofmt -w -r "assert.Nil(err) -> assert.NoError(err)"

* fix directory permissions

* don't use nil Context

* simplify boolean expressions

* use bytes.Equal instead of bytes.Compare

* merge variable declarations, remove redundant returns

* fix some golint errors

* run goimports

* handle more errors

* delete empty TestMain

* delete empty TestMain

* ignore examples for now

* fix lint errors

* remove unused values

* more fixes

* run gofmt -w -s .

* add more comments

* fix naming

* more lint fixes

* try switching travis to go1.11

* fix unnecessary conversions

* fix deprecated methods

* use go1.10 and disable gofmt/goimports for now

* switch to 1.10

* don't re-enable gofmt and goimports

* switch covermode to atomic because of -race

* gofmt
2018-08-27 11:28:16 -06:00
aligeti
09da23737a
rename farmers to storagenode (#275)
* rename farmers to storagenode

* review changes

* merge conflicts
2018-08-24 21:52:58 -04:00
Maximillian von Briesen
553def620a TLS Support (#252)
* begin adding tls

* remove incomplete line in gw/main.go

* identity fixes+:

+ fix `peertls.NewCert` public key issue
+ fix `peertls.verfiyChain` issue
+ fix identity dial option
+ rename `GenerateCA` to `NewCA` and `generateCAWorker` to `newCAWorker` for better consistency/convention

* use pdbclient instead of pointerdb in miniogw

* fix tests

* go fmt

* make review changes

* modify how context.Background() is used

* more context stuff
2018-08-23 22:01:03 -06:00
Bryan White
0e2fd97029
Identity fixes+: (#270)
* identity fixes+:

+ fix `peertls.NewCert` public key issue
+ fix `peertls.verfiyChain` issue
+ fix identity dial option
+ rename `GenerateCA` to `NewCA` and `generateCAWorker` to `newCAWorker` for better consistency/convention
2018-08-23 16:08:26 +02:00
Egon Elbre
f80185a807
Preliminary support for windows (#266)
* Don't use url.Parse for bolt paths: filepaths may not be valid URL-s.
* go.mod: update dependencies
* README.md: add Windows instructions
* pkg/overlay: check for the correct path and text in error
* pkg/overlay: fix tests for windows
* pkg/piecestore: make windows tests pass
* pkg/telemetry: skip test, as it doesn't shutdown nicely
* storage/redis: ensure that redis is clean before running tests
2018-08-23 10:05:56 +03:00
Egon Elbre
fe3decc42f
all: fix govet warnings (#255)
Fixes go1.11 vet warnings.

Cancel on WithTimeout must always be called to avoid memory leak:

pkg/provider/provider.go:73: the cancel function returned by context.WithTimeout should be called, not discarded, to avoid a context leak

Range over non-copyable things:

pkg/pool/connection_pool_test.go:32: range var v copies lock: struct{pool pool.ConnectionPool; key string; expected pool.TestFoo; expectedError error} contains pool.ConnectionPool contains sync.RWMutex
pkg/pool/connection_pool_test.go:56: range var v copies lock: struct{pool pool.ConnectionPool; key string; value pool.TestFoo; expected pool.TestFoo; expectedError error} contains pool.ConnectionPool contains sync.RWMutex
pkg/pool/connection_pool_test.go:83: range var v copies lock: struct{pool pool.ConnectionPool; key string; value pool.TestFoo; expected interface{}; expectedError error} contains pool.ConnectionPool contains sync.RWMutex

zeebo/errs package always requires formatting directives:

pkg/peertls/peertls.go:50: Class.New call has arguments but no formatting directives
pkg/peertls/utils.go:47: Class.New call has arguments but no formatting directives
pkg/peertls/utils.go:87: Class.New call has arguments but no formatting directives
pkg/overlay/cache.go:94: Class.New call has arguments but no formatting directives
pkg/provider/certificate_authority.go:98: New call has arguments but no formatting directives
pkg/provider/identity.go:96: New call has arguments but no formatting directives
pkg/provider/utils.go:124: New call needs 1 arg but has 2 args
pkg/provider/utils.go:136: New call needs 1 arg but has 2 args
storage/redis/client.go:44: Class.New call has arguments but no formatting directives
storage/redis/client.go:64: Class.New call has arguments but no formatting directives
storage/redis/client.go:75: Class.New call has arguments but no formatting directives
storage/redis/client.go:80: Class.New call has arguments but no formatting directives
storage/redis/client.go:92: Class.New call has arguments but no formatting directives
storage/redis/client.go:96: Class.New call has arguments but no formatting directives
storage/redis/client.go:102: Class.New call has arguments but no formatting directives
storage/redis/client.go:126: Class.New call has arguments but no formatting directives
2018-08-22 09:39:57 +03:00
nfarah86
d396eb6be9
Bug/download (#216)
* fixed downloading empty file

* fixed formatting

* fixed error handling

* fixed cache_test.go to accomodate for error

* fixed error verbose messaging for wrong downlaods

* clean up
2018-08-14 08:22:29 -07:00
JT Olio
c7adc1dfb8
shorter ids (#227)
* shorter ids

* fix test
2018-08-13 17:05:31 -06:00
JT Olio
df726b3ee8
fix services after identity merge (#224)
unfortunately, we can't enable server-side tls yet because
client-side tls hasn't been implemented
2018-08-13 09:39:42 -06:00
Bryan White
5d20cf8829
Node Identity (#193)
* peertls: don't log errors for double close

understood that this part of the code is undergoing heavy change
right now, but just want to make sure this fix gets incorporated
somewhere

* git cleanup: node-id stuff

* cleanup

* rename identity_util.go

* wip `CertificateAuthority` refactor

* refactoring

* gitignore update

* wip

* Merge remote-tracking branch 'storj/doubleclose' into node-id3

* storj/doubleclose:
  peertls: don't log errors for double close

* add peertls tests & gomports

* wip:

+ refactor
+ style changes
+ cleanup
+ [wip] add version to CA and identity configs
+ [wip] heavy client setup

* refactor

* wip:

+ refactor
+ style changes
+ add `CAConfig.Load`
+ add `CAConfig.Save`

* wip:

+ add `LoadOrCreate` and `Create` to CA and Identity configs
+ add overwrite to CA and identity configs
+ heavy client setup
+ refactor
+ style changes
+ cleanup

* wip

* fixing things

* fixing things

* wip hc setup

* hc setup:

+ refactor
+ bugfixing

* improvements based on reveiw feedback

* goimports

* improvements:

+ responding to review feedback
+ refactor

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* cleanup

* refactoring CA and Identity structs

* Merge branch 'master' into node-id3

* move version field to setup config structs for CA and identity

* fix typo

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* Merge branch 'master' into node-id3

* fix gateway setup finally

* go imports

* fix `FullCertificateAuthority.GenerateIdentity`

* cleanup overlay tests

* bugfixing

* update ca/identity setup

* go imports

* fix peertls test copy/paste fail

* responding to review feedback

* setup tweaking

* update farmer setup
2018-08-13 10:39:45 +02:00
JT Olio
5170057c7f
log grpc server errors (#196) 2018-08-02 15:52:16 -06:00
JT Olio
45a3c2e974
pkg/provider: with pkg/provider merged, make a single heavy client binary, gateway binary, and deprecate old services (#165)
* pkg/provider: with pkg/provider merged, make a single heavy client binary and deprecate old services

* add setup to gw binary too

* captplanet: output what addresses everything is listening on

* revert peertls/io_util changes

* define config flag across all commands

* use trimsuffix
2018-07-26 08:21:35 -06:00
JT Olio
5f6607935b
captplanet (#159)
* captplanet

I kind of went overboard this weekend.

The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!

To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.

Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.

A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
 * Kademlia
 * OverlayCache
 * PointerDB
 * StatDB
 * PieceStore
 * etc.

A Provider is a collection of Responsibilities that
share an Identity, such as:
 * The heavy client
 * The farmer
 * The gateway

An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.

Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.

To make a Responsibility, one should create a "config"
struct, such as:

```
type Config struct {
  RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
  SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```

To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.

cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.

Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
 * cmd/captplanet/farmer/main.go (a new farmer binary)
 * cmd/captplanet/hc/main.go (a new heavy client binary)
 * cmd/captplanet/gw/main.go (a new minio gateway binary)

Usage:

```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```

Configuration is placed by default in `~/.storj/capt/`

Other changes:

 * introduces new config structs for currently existing
   Responsibilities that conform to the new Responsibility
   interface. Please see the `pkg/*/config.go` files for
   examples.

 * integrates the PointerDB API key with other global
   configuration via flags, instead of through environment
   variables through viper like it's been doing. (ultimately
   this should also change to use the PointerDB config
   struct but this is an okay shortterm solution).

 * changes the Overlay cache to use a URL for database
   configuration instead of separate redis and bolt config
   settings.

 * stubs out some peer identity skeleton code (but not the
   meat).

 * Fixes the SegmentStore to use the overlay client and
   pointerdb clients instead of gRPC client code directly

 * Leaves a very clear spot where we need to tie the object to
   stream to segment store together. There's sort of a "golden
   spike" opportunity to connect all the train tracks together
   at the bottom of pkg/miniogw/config.go, labeled with a
   bunch of TODOs.

Future stuff:

 * I now prefer this design over the original
   pkg/process.Service thing I had been pushing before (sorry!)

 * The experience of trying to have multiple farmers
   configurable concurrently led me to prefer config structs
   over global flags (I finally came around) or using viper
   directly. I think global flags are okay sometimes but in
   general going forward we should try and get all relevant
   config into config structs.

 * If you all like this direction, I think we can go delete my
   old Service interfaces and a bunch of flags and clean up a
   bunch of stuff.

 * If you don't like this direction, it's no sweat at all, and
   despite how much code there is here I'm not very tied to any
   of this! Considering a lot of this was written between midnight
   and 6 am, it might not be any good!

* bind tests
2018-07-24 10:08:28 -06:00