Commit Graph

200 Commits

Author SHA1 Message Date
paul cannon
17bdb5e9e5
move piece info into files (#2629)
Deprecate the pieceinfo database, and start storing piece info as a header to
piece files. Institute a "storage format version" concept allowing us to handle
pieces stored under multiple different types of storage. Add a piece_expirations
table which will still be used to track expiration times, so we can query it, but
which should be much smaller than the pieceinfo database would be for the
same number of pieces. (Only pieces with expiration times need to be stored in piece_expirations, and we don't need to store large byte blobs like the serialized
order limit, etc.) Use specialized names for accessing any functionality related
only to dealing with V0 pieces (e.g., `store.V0PieceInfo()`). Move SpaceUsed-
type functionality under the purview of the piece store. Add some generic
interfaces for traversing all blobs or all pieces. Add lots of tests.
2019-08-07 20:47:30 -05:00
Egon Elbre
c8edeb0257
satellite/overlay: rename overlay.Cache to overlay.Service (#2717) 2019-08-06 19:35:59 +03:00
paul cannon
3ebeda3334 storage/redis: make sure to return after put() (#2625) 2019-07-24 10:28:13 +03:00
Kaloyan Raev
0e1cb7bfb8
CompareAndSwap in KeyValueStore (#2602) 2019-07-23 22:46:33 +03:00
Ivan Fraixedes
3c8f1370d2
[v3 2137] - Add more info to find out repair failures (#2623)
* pkg/datarepair/repairer: Track always time for repair
  Make a minor change in the worker function of the repairer, that when
  successful, always track the metric time for repair independently if the
  time since checker queue metric can be tracked.

* storage/postgreskv: Wrap error in Get func
  Wrap the returned error of the Get function as it is done when the
  query doesn't return any row.

* satellite/metainfo: Move debug msg to the right place
  NewStore function was writing a debug log message when the DB was
  connected, however it was always writing it out despite if an error
  happened when getting the connection.

* pkg/datarepair/repairer: Wrap error before logging it
  Wrap the error returned by process which is executed by the Run method
  of the repairer service to add context to the error log message.

* pkg/datarepair/repairer: Make errors more specific in worker
  Make the error messages of the "worker" method of the Service more
  specific and the logged message for such errors.

* pkg/storage/repair: Improve error reporting Repair
  In order of improving the error reporting by the
  pkg/storage/repair.Repair method, several errors of this method and
  functions/methods which this one relies one have been updated to be
  wrapper into their corresponding classes.

* pkg/storage/segments: Track path param of Repair method
  Track in monkit the path parameter passed to the Repair method.

* satellite/satellitedb: Wrap Error returned by Delete
  Wrap the error returned by repairQueue.Delete method to enhance the
  error with a class and stack and the
  pkg/storage/segments.Repairer.Repair method get a more contextualized
  error from it.
2019-07-23 16:28:06 +02:00
Jennifer Li Johnson
53d96be44a
Stylistic Go Cleanup (#2524) 2019-07-22 15:10:04 -04:00
Stefan Benten
de300e9235 Network Wipe (Pre Beta) (#2566) 2019-07-16 18:31:29 +02:00
Jennifer Li Johnson
699ccea19f
Creates Routing Table Antechamber (#2318) 2019-07-01 17:20:19 -04:00
Egon Elbre
b6ad3e9c9f
internal/testrand: new package for random data (#2282) 2019-06-26 13:38:51 +03:00
Kaloyan Raev
8e29ef8a6b Use zap.Stringer instead of zap.String (#2223) 2019-06-18 01:37:43 +02:00
Egon Elbre
1a1a084477
testcontext: sanitize folder name (#2195) 2019-06-13 15:46:08 +03:00
JT Olio
cc47bf2df7
postgreskv: actually monitor the iterate call (#2129)
Change-Id: I3a5221d6641b6c95b7b746ec3f7ac4522f464b85
2019-06-05 11:22:46 -06:00
JT Olio
f1641af802 storage: add monkit task to missing places (#2122)
* storage: add monkit task to missing places

Change-Id: I9e17a6b14f7c25bbf698eeecf32785e9add3f26e

* fix tests

Change-Id: Id078276fa3de61a28eb3d01d4e751732ecbb173f

* import order

Change-Id: I814e33755b9f10b5219af37cd828cd75eb3da1a4

* remove part of other commit

Change-Id: Idaa4c95cd65e97567fb466de49718db8203cfbe1
2019-06-05 16:23:10 +02:00
JT Olio
679cdfda9e storage/filestore: add monkit task to missing places (#2124)
Change-Id: I61f8056617a8d0596a00c232a6d0b330909e08f9
2019-06-05 15:06:06 +02:00
JT Olio
d02427e41a db: set max open conns, conn max lifetime, add db stat monitoring (#2117) 2019-06-04 23:30:21 +02:00
Kaloyan Raev
2ab95b533e
Check errors for possible outcomes from audit's DownloadShares (#2072) 2019-06-03 12:17:09 +03:00
ethanadams
16e3b77cf5
Enable Scopelint Linter (#2049)
* added scopelint and correcte issues found

* corrected scopelint issue

* made updates based on Ivan's suggestions

Most were around naming conventions
Some were false positives, but I kept them since the test.Run could eventually be changed to run in parallel, which could cause a bug
Others were false positives.  Added // nolint: scopelint
2019-05-29 09:30:16 -04:00
ethanadams
268dc6b7e4
Enable gocritic linter (#2051)
* first round cleanup based on go-critic

* more issues resolved for ifelsechain and unlambda checks

* updated from master and gocritic found a new ifElseChain issue

* disable appendAssign. i reports false positives

* re-enabled go-critic appendAssign and disabled lint check at code line level

* fixed go-critic lint error

* fixed // nolint add gocritic specifically
2019-05-29 09:14:25 -04:00
Egon Elbre
9c23c2d427 db: set max idle connections higher to avoid redialing all the time (#1991) 2019-05-21 17:30:06 +03:00
paul cannon
5d24af9b41 Revert "storage/postgreskv: use batch size 1000 (#1988)" (#1992)
This reverts commit f988543764.
2019-05-17 14:41:15 -04:00
Egon Elbre
f988543764 storage/postgreskv: use batch size 1000 (#1988) 2019-05-17 19:09:04 +02:00
Jess G
8518618b7a
add postgres support to storj-sim (#1908)
* add flags to sotrj-sim for SA dbs

* add schema to postgres

* add createschema with parse to sa

* add metainfo db postgres support

* add kv default as bolt

* add debug log to see db source

* add env var for postgres to test-sim.sh

* fix lint errs

* dynamically add postgres to args

* add postgres to integration tests

* add sqlite and postgres integration jenkins

* fix db name

* merge integration tests into one step

* test integration tests w/psql

* try using different schema

* debug failure

* use correct host for running storj-sim

* rm sqlite integration

* add back integration
2019-05-14 08:13:18 -07:00
Jess G
a9b8b50839
add boltDB batching for Put operation, add benchmark test (#1865)
* add boltDB batching for Put operation, add benchmark test

* add batchPut method to kademlia routingTable

* add BatchPut method for other KeyValueStore to satisfy interface

* return err not implemented

* add noSync to boltdb client

* rm boltDB noSync

* make batch block and fix tests

* changes per CR

* rm test setting so it matches prod code behavior

* fix lint errs
2019-05-06 13:47:12 -07:00
Egon Elbre
db939d37ec
cover all the things (#1818) 2019-04-26 16:39:11 +03:00
Egon Elbre
ac18432dc5 Public Jenkins (#1779)
* initial test

* add parenthesis

* remove pipeline

* add few todos

* use docker image for environment

* use pipeline

* fix

* add missing steps

* invoke with bash

* disable protoc

* try using golang image

* try as root

* Disable install-awscli.sh temporarily

* Debugging

* debugging part 2

* Set absolute path for debugging

* Remove absolute path

* Dont run as root

* Install unzip

* Dont forget to apt-get update

* Put into folder that is in PATH

* disable IPv6 Test

* add verbose info and check protobuf

* make integration non-parallel

* remove -v and make checkout part of build

* make a single block for linting

* fix echo

* update

* try using things directly

* try add xunit output

* fix name

* don't print empty lines

* skip testsuites without any tests

* remove coverage, because it's not showing the right thing

* try using dockerfile

* fix deb source

* fix typos

* setup postgres

* use the right flag

* try using postgresdb

* expose different port

* remove port mapping

* start postgres

* export

* use env block

* try using different host for integration tests

* eat standard ports

* try building images and binaries

* remove if statement

* add steps

* do before verification

* add go get goversioninfo

* make separate jenkinsfile

* add check

* don't add empty packages

* disable logging to reduce output size

* add timeout

* add comment about mfridman

* Revert Absolute Path

* Add aws to PATH

* PATH Changes

* Docker Env Fixes

* PATH Simplification

* Debugging the PATH

* Debug Logs

* Debugging

* Update PATH Handling

* Rename

* revert changes to Jenkinsfile
2019-04-22 15:45:53 +02:00
Natalie Villasana
8d1f614662 removes unused queue code, moves queue_test.go to repairqueue_test.go in satellitedb dir (#1783) 2019-04-22 13:35:52 +03:00
Bill Thorp
17a227e6e9
refactor injuredsegments db so that we can't have duplicates (#1717)
made repairqueue not use a true queue, forbid duplicates
2019-04-16 14:14:09 -04:00
Egon Elbre
0eee46524d update linter to v1.16 (#1741)
* update to v1.16

* use full version number

* fix linter issues

* fix comment

* nicer comment

* restart travis

* restart travis
2019-04-11 13:32:40 -04:00
Egon Elbre
80916ffb53
storagenode/pieces: ensure we can call Commit or Cancel only once (#1511) 2019-03-18 16:29:54 +02:00
Egon Elbre
05d148aeb5
Storage node and upload/download protocol refactor (#1422)
refactor storage node server
refactor upload and download protocol
2019-03-18 12:55:06 +02:00
Egon Elbre
773d6a7ad1
filestore changes for piece store (#1437) 2019-03-11 10:06:56 +02:00
Bill Thorp
9b580c5fb6 Repair checker is checking the same 1000 elements all the time (#1297)
* removed limit on repair, now using cycle

* added BatchIteratorOptions

* consolidated boltdb common.go

* PR feedback cleanup
2019-02-14 13:33:41 +01:00
Bill Thorp
b53f9896d3
Removed ReverseList from KeyValueStore interfaces (#1306)
Removed ReverseList from KeyValueStore interfaces
2019-02-13 12:27:03 -05:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Egon Elbre
99d3b7a3c8
Fix import grouping (#1111) 2019-01-22 17:48:23 +02:00
Bryan White
2b60adbc26 storelogger: truncate value output (#995) 2019-01-09 13:18:23 +02:00
Egon Elbre
0ca03b41e2
Use generated locking database implementation (#947) 2019-01-02 19:53:27 +02:00
Michal Niewrzal
b712fbcbb0
Fix 'empty queue' error when satellite starts (#939) 2019-01-02 17:00:32 +01:00
Egon Elbre
c4c9e75109
Use errs.Combine in storage (#923) 2018-12-21 12:54:20 +02:00
Bryan White
398379b149
CSR service (part 1): authorizations (#906) 2018-12-20 19:29:05 +01:00
Dennis Coyle
98429e39fc set TTL to 0 (#892)
* set TTL to 0

* added another comment
2018-12-17 15:16:28 -05:00
Bill Thorp
ba5f71810e
added tally to capt run.go (#751)
* added tally to capt run.go, ParseURL -> SplitDBURL
2018-12-12 08:15:34 -05:00
JT Olio
1c96db01ba
better database error handling (#784)
* better database error handling

Change-Id: I28dbd69cf6c2fa268e02405521ff6e6c1a68a702

* missing comments added

* missing comment added
2018-12-07 07:46:42 -07:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Michal Niewrzal
376bd74bed
Stops logging repairer errors for empty queue (#717) 2018-11-27 16:57:51 +01:00
Egon Elbre
eb07715d53
Make teststore thread-safe (#689) 2018-11-20 17:26:20 +02:00
aligeti
7958994ae2
Diagnostic tool to inspect repair queue (#656)
* initial repair queue diag tool development

* fixes linter warnings

* code review updates
2018-11-16 08:31:33 -05:00
Egon Elbre
c4b90f84dd
Use better defaults and naming for postgres database (#659) 2018-11-15 20:36:57 +02:00
Kaloyan Raev
0357e61bbd
metainfo objects tests (#662) 2018-11-15 17:31:33 +02:00
Bill Thorp
d3c347a0ac
Verify overlay cache is accurately and fully tested (#540)
* added intro node id, moved port range to 1024, listen in go routines
2018-11-14 16:30:07 -05:00
Egon Elbre
6d401a4351
Check for go.mod validity (#605) 2018-11-09 15:32:35 +02:00
Bill Thorp
07ed38c930
Added distqueue interface and redis and test queue (#555) 2018-11-08 08:53:27 -05:00
Bryan White
ee62e2a9d8
Use transport client and cleanup all the clients (#574)
* wip

* linter fixes

* linter fixes

* test fixes

* linter fixes

* fix merge + restructure piecestore packages

* review feedback

* linter fixes

* linter fixes

* remove unnecessary aliases to piecestore

* more merge fixing
2018-11-06 18:49:17 +01:00
paul cannon
e2c0dd437a
offer PostgreSQL storage for pointerdb (#440)
..although it ought to work for other storage.KeyValueStore needs as
well. it's just optimized to work pretty well for a largish hierarchy of
paths.

This includes the addition of "long benchmarks" for KeyValueStore
testing. These will only be run when -test-bench-long is added to the
test flags. In these benchmarks, a large corpus of paths matching a
natural ("real-life") hierarchy is read from paths.data.gz (which you
can get from https://github.com/storj/path-test-corpus) and imported
into a particular KeyValueStore. Recursive and non-recursive queries are
run on it to detect performance problems that arise only at scale.

This also includes alternate implementation of the postgreskv client,
which works in a less-bizarre way for non-recursive queries, but suffers
from poor performance in tests such as the long benchmarks. Once this
alternate impl is committed to the tree, we can remove it again; I just
want it to be available for future reference.
2018-10-25 12:11:28 -05:00
Egon Elbre
03bd93bba7
Make kademlia use less file-descriptors (#498) 2018-10-18 19:20:23 +03:00
Egon Elbre
2dbada73ab
Separate storj.io packages in imports (#481)
Format with goimports -w -local storj.io
Add flag to golang lint
Remove commented out imports as they mess with goimports
2018-10-16 14:43:44 +03:00
Alexander Leitner
dc8bea2cd1
Repairer points to redis server (#427)
* Let's do it right this time

* Oh travis...

* Handle redis URL

* Travis... why u gotta be like this?

* Handle when address does not use redis scheme

* Start repairer

* Match provider.Responsibility interface

* Simplify if statement

* Config doesn't need to be a pointer

* Initialize doesn't need to be exported

* Don't run checker or repairer on startup

* Fix travis complaints
2018-10-05 11:58:07 -04:00
Egon Elbre
ab64aa0b97
add missing copyright headers (#403) 2018-10-01 14:29:27 +03:00
Egon Elbre
c9cabaf73a
storage/filestore (#384) 2018-09-28 09:59:27 +03:00
Egon Elbre
4486e265bb
Cleanup after tests (#341) 2018-09-11 16:57:12 +03:00
Egon Elbre
e7e2d4d7c9
use golangci-lint and add missing error checks (#340) 2018-09-11 16:13:25 +03:00
Egon Elbre
6ee6f0fdf5
Remove duplicated mock store implementations (#337) 2018-09-11 10:27:12 +03:00
Jennifer Li Johnson
d0f87f0de1
Overlay Bulk Lookup (#279)
* implements bulk node lookup from overlay cache
2018-09-11 00:52:14 -04:00
Egon Elbre
00ac266bda Remove zap dependency from boltdb (#334)
* Remove zap dependency from boltdb

* Avoid pulling in testing in main binary
2018-09-10 05:52:53 -04:00
JT Olio
8f3398b349
merge uplink and storj (#323)
also remove testing package from import dependency chain
(testing defines some flags)
2018-09-07 09:01:04 -06:00
Egon Elbre
0a2839b2d7
Add IsPrefix support to piecestore (#313) 2018-09-07 17:20:15 +03:00
Egon Elbre
79354bf0ab
Enforce lookup limit for storage (#312) 2018-09-07 12:00:00 +03:00
Egon Elbre
8a4151f397
Fix Redis Iterators (#318)
Redis SCAN may return same item multiple times, add a basic check to avoid duplicate entries.
2018-09-06 23:50:22 +03:00
Egon Elbre
83df0ee1b0
Implement ListV2 with storage rework (#303)
1. Added KeyValueStore.Iterate for implementing the different List, ListV2 etc. implementations. This allows for more efficient use of memory depending on the situation.
2. Implemented an inmemory teststore for running tests. This should allow to replace MockKeyValueStore in most places.
3. Rewrote tests
4. Pulled out logger from bolt implementation so it can be used for all other storage implementations.
5. Fixed multiple things in bolt and redis implementations.
2018-09-05 19:10:35 +03:00
Egon Elbre
0f5a2f4ef5 Enable more linters (#272)
* enable more linters

* Run gofmt -s

* run goimports

* run unconvert

* fix naked return

* fix misspellings

* fix ineffectual assigments

* fix missing declaration

* don't use deprecated grpc.Errof

* check errors in tests

* run gofmt -w -r "assert.Nil(err) -> assert.NoError(err)"

* fix directory permissions

* don't use nil Context

* simplify boolean expressions

* use bytes.Equal instead of bytes.Compare

* merge variable declarations, remove redundant returns

* fix some golint errors

* run goimports

* handle more errors

* delete empty TestMain

* delete empty TestMain

* ignore examples for now

* fix lint errors

* remove unused values

* more fixes

* run gofmt -w -s .

* add more comments

* fix naming

* more lint fixes

* try switching travis to go1.11

* fix unnecessary conversions

* fix deprecated methods

* use go1.10 and disable gofmt/goimports for now

* switch to 1.10

* don't re-enable gofmt and goimports

* switch covermode to atomic because of -race

* gofmt
2018-08-27 11:28:16 -06:00
nfarah86
5f46a0696f
writing definition for LISTv2 (#289)
* writing definition for LISTv2

* fixed method to call storage.xx

* regenerate kvstore file

* fixed minor error in storage

* added panic

* added panic
2018-08-25 20:00:49 -07:00
Egon Elbre
f80185a807
Preliminary support for windows (#266)
* Don't use url.Parse for bolt paths: filepaths may not be valid URL-s.
* go.mod: update dependencies
* README.md: add Windows instructions
* pkg/overlay: check for the correct path and text in error
* pkg/overlay: fix tests for windows
* pkg/piecestore: make windows tests pass
* pkg/telemetry: skip test, as it doesn't shutdown nicely
* storage/redis: ensure that redis is clean before running tests
2018-08-23 10:05:56 +03:00
Egon Elbre
fe3decc42f
all: fix govet warnings (#255)
Fixes go1.11 vet warnings.

Cancel on WithTimeout must always be called to avoid memory leak:

pkg/provider/provider.go:73: the cancel function returned by context.WithTimeout should be called, not discarded, to avoid a context leak

Range over non-copyable things:

pkg/pool/connection_pool_test.go:32: range var v copies lock: struct{pool pool.ConnectionPool; key string; expected pool.TestFoo; expectedError error} contains pool.ConnectionPool contains sync.RWMutex
pkg/pool/connection_pool_test.go:56: range var v copies lock: struct{pool pool.ConnectionPool; key string; value pool.TestFoo; expected pool.TestFoo; expectedError error} contains pool.ConnectionPool contains sync.RWMutex
pkg/pool/connection_pool_test.go:83: range var v copies lock: struct{pool pool.ConnectionPool; key string; value pool.TestFoo; expected interface{}; expectedError error} contains pool.ConnectionPool contains sync.RWMutex

zeebo/errs package always requires formatting directives:

pkg/peertls/peertls.go:50: Class.New call has arguments but no formatting directives
pkg/peertls/utils.go:47: Class.New call has arguments but no formatting directives
pkg/peertls/utils.go:87: Class.New call has arguments but no formatting directives
pkg/overlay/cache.go:94: Class.New call has arguments but no formatting directives
pkg/provider/certificate_authority.go:98: New call has arguments but no formatting directives
pkg/provider/identity.go:96: New call has arguments but no formatting directives
pkg/provider/utils.go:124: New call needs 1 arg but has 2 args
pkg/provider/utils.go:136: New call needs 1 arg but has 2 args
storage/redis/client.go:44: Class.New call has arguments but no formatting directives
storage/redis/client.go:64: Class.New call has arguments but no formatting directives
storage/redis/client.go:75: Class.New call has arguments but no formatting directives
storage/redis/client.go:80: Class.New call has arguments but no formatting directives
storage/redis/client.go:92: Class.New call has arguments but no formatting directives
storage/redis/client.go:96: Class.New call has arguments but no formatting directives
storage/redis/client.go:102: Class.New call has arguments but no formatting directives
storage/redis/client.go:126: Class.New call has arguments but no formatting directives
2018-08-22 09:39:57 +03:00
Bryan White
f932c3ced8 Overlay cache commands (#236)
* wip overlay cache cmd

* overlay cache add and list commands

* use zap

* fixes

* simplify

* add restrictions

* cleanup

* remove defaults for overlay management commands
2018-08-20 14:21:41 -04:00
Natalie Villasana
ff65663867
adds Bucket Store (#213)
* adds foundation for bucketStore

* adds prefixedObjStore to buckets package, adjusts gateway-storj accordingly

* fixes multi value assignment problems in gateway-storj

* fixes more multi value assignment errors in gateway-storj

* starts changing miniogw tests to accommodate buckets

* creates bucket store mock

* wip - fixing test cases in object tests

* adds get, put, and list object tests, comments out two test cases

* adds happy scenario tests for bucket methods

* fixes bug in list, removes redundant parts from gateway tests

* fixes nit

* Clean up tests from #188

* Fix bug with timestamp conversion in segment store

* fixes segments.Meta test

* Fix regression in listing objects in a bucket

* adds check to see if bucket is empty before deleting

* updates DeleteBucket test to account for empty/full bucket

* adds TODOs for DeleteBucket and MakeBucket for some cases, adjusts tests, filters out minio errors in logging.go

* adds checks for if buckets already exist or not in DeleteBucket and MakeBucket functions; adjusts tests

* adds BucketNotFound error check in bucket store, removes todo

* adds make_bucket to Travis test, updates boltdb client constructor to always create a bucket (table)
2018-08-16 10:32:28 -04:00
nfarah86
d396eb6be9
Bug/download (#216)
* fixed downloading empty file

* fixed formatting

* fixed error handling

* fixed cache_test.go to accomodate for error

* fixed error verbose messaging for wrong downlaods

* clean up
2018-08-14 08:22:29 -07:00
Dennis Coyle
7348bf1e09
Adjustments to how find storage nodes handles limit/amout requested (#192)
* adjusted based on feedback from @jtolds and @kaloyan-raev

* generated protobuf

* fix crash

* WIP

* cleanup

* v >>found

* pointer

* test

* copyrights + cleanup

* fix tests
2018-08-03 09:15:52 -04:00
Yaroslav Vorobiov
f6fb41c5cc Testcoverage kademlia (#154)
* Unit test covarege increased for kademlia pkg

go style formatting added

Removed DHT param from newTestKademlia method, added comments for Bucket methods that informs that these tests will need to be updated

unnecessary comment deleted from newTestKademlia

Adjust Segment Store to the updated interface (#160)

* Adjust Segment Store to the updated interface

* Move /pkg/storage/segment to /pkg/storage/segments

* Fix overlay client tests

* Revert changes in NewOverlayClient return value

* Rename `rem` to `seg`

* Implement Meta()

captplanet (#159)

* captplanet

I kind of went overboard this weekend.

The major goal of this changeset is to provide an environment
for local development where all of the various services can
be easily run together. Developing on Storj v3 should be as
easy as running a setup command and a run command!

To do this, this changeset introduces a new tool called
captplanet, which combines the powers of the Overlay Cache,
the PointerDB, the PieceStore, Kademlia, the Minio Gateway,
etc.

Running 40 farmers and a heavy client inside the same process
forced a rethinking of the "services" that we had. To
avoid confusion by reusing prior terms, this changeset
introduces two new types: Providers and Responsibilities.
I wanted to avoid as many merge conflicts as possible, so
I left the existing Services and code for now, but if people
like this route we can clean up the duplication.

A Responsibility is a collection of gRPC methods and
corresponding state. The following systems are examples of
Responsibilities:
 * Kademlia
 * OverlayCache
 * PointerDB
 * StatDB
 * PieceStore
 * etc.

A Provider is a collection of Responsibilities that
share an Identity, such as:
 * The heavy client
 * The farmer
 * The gateway

An Identity is a public/private key pair, a node id, etc.
Farmers all need different Identities, so captplanet
needs to support running multiple concurrent Providers
with different Identities.

Each Responsibility and Provider should allow for configuration
of multiple copies on its own so creating Responsibilities and
Providers use a new workflow.

To make a Responsibility, one should create a "config"
struct, such as:

```
type Config struct {
  RepairThreshold int `help:"If redundancy falls below this number of
pieces, repair is triggered" default:"30"`
  SuccessThreshold int `help:"If redundancy is above this number then
no additional uploads are needed" default:"40"`
}
```

To use "config" structs, this changeset introduces another
new library called 'cfgstruct', which allows for the configuration
of arbitrary structs through flagsets, and thus through cobra and
viper.

cfgstruct relies on Go's "struct tags" feature to document
help information and default values. Config structs can be
configured via cfgstruct.Bind for binding the struct to a flagset.

Because this configuration system makes setup and configuration
easier *in general*, additional commands are provided that allow
for easy standup of separate Providers. Please make sure to
check out:
 * cmd/captplanet/farmer/main.go (a new farmer binary)
 * cmd/captplanet/hc/main.go (a new heavy client binary)
 * cmd/captplanet/gw/main.go (a new minio gateway binary)

Usage:

```
$ go install -v storj.io/storj/cmd/captplanet
$ captplanet setup
$ captplanet run
```

Configuration is placed by default in `~/.storj/capt/`

Other changes:

 * introduces new config structs for currently existing
   Responsibilities that conform to the new Responsibility
   interface. Please see the `pkg/*/config.go` files for
   examples.

 * integrates the PointerDB API key with other global
   configuration via flags, instead of through environment
   variables through viper like it's been doing. (ultimately
   this should also change to use the PointerDB config
   struct but this is an okay shortterm solution).

 * changes the Overlay cache to use a URL for database
   configuration instead of separate redis and bolt config
   settings.

 * stubs out some peer identity skeleton code (but not the
   meat).

 * Fixes the SegmentStore to use the overlay client and
   pointerdb clients instead of gRPC client code directly

 * Leaves a very clear spot where we need to tie the object to
   stream to segment store together. There's sort of a "golden
   spike" opportunity to connect all the train tracks together
   at the bottom of pkg/miniogw/config.go, labeled with a
   bunch of TODOs.

Future stuff:

 * I now prefer this design over the original
   pkg/process.Service thing I had been pushing before (sorry!)

 * The experience of trying to have multiple farmers
   configurable concurrently led me to prefer config structs
   over global flags (I finally came around) or using viper
   directly. I think global flags are okay sometimes but in
   general going forward we should try and get all relevant
   config into config structs.

 * If you all like this direction, I think we can go delete my
   old Service interfaces and a bunch of flags and clean up a
   bunch of stuff.

 * If you don't like this direction, it's no sweat at all, and
   despite how much code there is here I'm not very tied to any
   of this! Considering a lot of this was written between midnight
   and 6 am, it might not be any good!

* bind tests

Add files for testing builds in docker (#161)

* Add files for testing builds in docker

* Make tests check for redis running before trying to start redis-server, which may not exist.

* Clean redis server before any tests use it.

* Add more debugging for travis

* Explicitly requiring redis for travis

pkg/provider: with pkg/provider merged, make a single heavy client binary, gateway binary, and deprecate old services (#165)

* pkg/provider: with pkg/provider merged, make a single heavy client binary and deprecate old services

* add setup to gw binary too

* captplanet: output what addresses everything is listening on

* revert peertls/io_util changes

* define config flag across all commands

* use trimsuffix

fix docker makefile (#170)

* fix makefile

protos: update protobufs with go generate (#169)

the import for timestamp and duration should use
the path provided by a standard protocol buffer library
installation

Refactor List in PointerDB (#163)

* Refactor List in Pointer DB

* Fix pointerdb-client example

* Fix issue in Path type related to empty paths

* Test for the PointerDB service with some fixes

* Fixed debug message in example: trancated --> more

* GoDoc comments for unexported methods

* TODO comment to check if Put is overwriting

* Log warning if protobuf timestamp cannot be converted

* TODO comment to make ListPageLimit configurable

* Rename 'segment' package to 'segments' to reflect folder name

Minio integration with Object store (#156)

* initial WIP integration with Object store

* List WIP

* minio listobject function changes complete

* Code review changes and work in progress for the mock objectstore unit testing cases

* Warning fix redeclaration of err

* Warning fix redeclaration of err

*  code review comments & unit testing inprogress

* fix compilation bug

* Fixed code review comments & added GetObject Mock test case

* rearraged the mock test file and gateway storj test file in to the proper directory

* added the missing file

* code clean up

* fix lint error on the mock generated code

* modified per code review comments

* added the PutObject mock test case

* added the GetObjectInfo  mock test case

* added listobject mock test case

* fixed package from storj to miniogw

* resolved the gateway-storj.go initialization merge conflict

update readme (#174)

added assertion for unused errors (#152)

merging this PR to avoid future issues

updating github user to personal account (#171)

Test coverage ranger (#168)

* Fixed go panic for corner case

* Initial test coverage for ranger pkg

streamstore: add passthrough implementation (#176)

this doesn't implement streamstore, this just allows us to try and
get the june demo working again in the meantime

StatDB (#144)

* add statdb proto and example client

* server logic

* update readme

* remove boltdb from service.go

* sqlite3

* add statdb server executable file

* create statdb node table if it does not exist already

* get UpdateBatch working

* update based on jt review

* remove some commented lines

* fix linting issues

* reformat

* apiKey -> APIKey

* update statdb client apiKey->APIKey

Update README.md

Update README.md

overlay: correct dockerfile db (#179)

cmd/hc, cmd/gw, cmd/captplanet: simplify setup/run commands (#178)

also allows much more customization of services within captain planet,
such as reconfiguring the overlay service to use redis

pkg/process: don't require json formatting (#177)

Cleanup metadata across layers (#180)

* Cleanup metadata across layers

* Fix pointer db tests

Kademlia Routing Table (#164)

* adds comment

* runs deps

* creates boltdb kademlia routing table

* protobuf updates

* adds reverselist to mockkeyvaluestore interface

* xor wip

* xor wip

* fixes xor sort

* runs go fmt

* fixes

* goimports again

* trying to fix travis tests

* fixes mock tests

Ranger refactoring (#158)

* Fixed go panic for corner case

* Cosmetic changes, and small error fixes

miniogw: log all errors (#182)

* miniogw: log all errors

* tests added

* doc comment to satisfy linter

* fix test failure

Jennifer added to CLA list

* Temporary fix for storage/redis list method test
2018-08-02 14:36:57 -04:00
Dennis Coyle
5de5428d3a
First pass at node restrictions (#146)
* WIP First pass at node restrictions

* adjustments to storage clients

* fix cover

* undo previous import change

* adding copyright

* fix the tests

* check for keys

* moar fixes to tests

* linter

* change how we handle restrictions

* add generated pb.go file

* PR comments addressed

* MockKeyValueStore

* pr comments addressed

* missing )

* past my bedtime

* moar merge issues

* cleanup
2018-08-01 10:15:38 -04:00
Yehor Butko
7298c16f35 storage/redis covered with unit tests (#157)
* storage/redis covered with unit tests

* updated tests for redis
2018-07-31 11:23:55 -04:00
Jennifer Li Johnson
3230762041
Kademlia Routing Table (#164)
* adds comment

* runs deps

* creates boltdb kademlia routing table

* protobuf updates

* adds reverselist to mockkeyvaluestore interface

* xor wip

* xor wip

* fixes xor sort

* runs go fmt

* fixes

* goimports again

* trying to fix travis tests

* fixes mock tests
2018-07-30 15:25:18 -04:00
Natalie Villasana
80727ae90b adds netstate pagination (#95)
* adds netstate rpc server pagination, mocks pagination in test/util.go

* updates ns client example, combines ns client and server test to netstate_test, adds pagination to bolt client

* better organizes netstate test calls

* wip breaking netstate test into smaller tests

* wip modularizing netstate tests

* adds some test panics

* wip netstate test attempts

* testing bug in netstate TestDeleteAuth

* wip fixes global variable problem, still issues with list

* wip fixes get request params and args

* fixes bug in path when using MakePointers helper fn

* updates mockdb list func, adds test, changes Limit to int

* fixes merge conflicts

* fixes broken tests from merge

* remove unnecessary PointerEntry struct

* removes error when Get returns nil value from boltdb

* breaks boltdb client tests into smaller tests

* renames AssertNoErr test helper to HandleErr

* adds StartingKey and Limit parameters to redis list func, adds beginning of redis tests

* adds helper func for mockdb List function

* if no starting key provided for netstate List, the first value in storage will be used

* adds basic pagination for redis List function, adds tests

* adds list limit to call in overlay/server.go

* streamlines/fixes some nits from review

* removes use of obsolete EncryptedUnencryptedSize

* uses MockKeyValueStore instead of redis instance in redis client test

* changes test to expect nil returned for getting missing key

* remove error from `KeyValueStore#Get`

* fix bolt test

* Merge pull request #1 from bryanchriswhite/nat-pagination

remove error from `KeyValueStore#Get`

* adds Get returning error back to KeyValueStore interface and affected clients

* trying to appease travis: returns errors in Get calls in overlay/cache and cache_test

* handles redis get error when no key found
2018-06-29 16:06:25 -04:00
JT Olio
5ee6b76d41 internal/test: switch errors to error classes (#96)
* internal/test: switch errors to error classes

if you construct an error directly at package init time, you
won't get useful stack traces or anything. zeebo/errs expects that
you always construct an error (.New) when the error actually
happens. instead, you call Class at init time, then use (Class).Has
to test for error type membership.

* fix linter

* fix test
2018-06-19 20:03:46 +02:00
Bryan White
1ebd66d880 Cleanup & bolt-backed cache (#87)
* wip post-demos cleanup

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* wip

* Reorganize:

storage
├── common
│   └── common.go `package storage`
├── boltdb
│   └── ...
└── redis
    └── ...

storage
├── common.go `package storage`
├── boltdb
│   └── ...
└── redis
    └── ...

storage
├── common
│   ├── common.go `package storage`
│   └── testing.go `package storage` <--
└── ...

internal
├── app
│   └── cli
├── pkg
│   └── readcloser
└── test
    └── util.go `package test` <--

* remove comment

* add and use goimports

* add test types & fix some lint errors

* better typing

* fixing linter issues/comments

* goimports

* goimports

* more linter fixes; replace panic with NoError assertions in tests

* fix typo/more linter errors

* moar better linter fixes

* even moar better linter fixes

* linter

* add localPort back

* fixing exports, imports, and add comments
2018-06-13 14:22:32 -04:00
Dylan Lott
325a70d514 Cache (#67)
* add reference to dht to overlay client struct

* wip

* wip

* Implement FindNode

* get nodes

* WIP

* Merge in Dennis kademlia code, get it working with our code

* ping and moar

* WIP trying to get cache working with kademlia

* WIP more wiring up

* WIP

* Update service cli commands

* WIP

* added GetNodes

* added nodes to Kbucket

* default transport changed to TCP

* GetBuckets interface changed

* filling in more routing

* timestamp methods

* removed store

* Added initial network overlay explorer page

* Updating and building with dockerfile

* Working on adding bootstrap node code

* WIP merging in dennis' code

* WIP

* connects cache to pkg/kademlia implementation

* WIP redis cache

* testing

* Add bootstrap network function for CLI usage

* cleanup

* call bootstrap on init network

* Add BootstrapNetwork function to interface

* Merge in dennis kad code

* WIP updates to redis/overlay client interface

* WIP trying to get the DHT connected to the cache

* go mod & test

* deps

* Bootstrap node now setting up correctly

- Need to pass it through CLI commands better

* WIP adding refresh and walk functions, added cli flags

- added cli flags for custom bootstrap port and ip

* PR comments addressed

* adding FindStorageNodes to overlay cache

* fix GetBucket

* using SplitHostPort

* Use JoinHostPort

* updates to findstoragenodes response and request

* WIP merge in progress, having issues with a panic

* wip

* adjustments

* update port for dht bootstrap test

* Docker

* wip

* dockerfile

* fixes

* makefile changes

* Update port in NewKademlia call

* Update local kademlia DHT config

* kubernetes yaml

* cleanup

* making tests pass

* k8s yaml

* lint issues

* Edit cli flags to allow for configurable bootstrap IP and Port args

* cleanup

* cache walking the network now

* Rough prototype of Walk function laid out

* Move walk function into bootstrap function

* Update dht.go

* changes to yaml

* goimports
2018-06-05 17:06:37 -04:00
Natalie Villasana
6723064bfb
Enables netstate service to save pointers (#49)
* adds pointer to netstate proto file

* generated updated netstate proto

* changes boltdb netstate to save pointers as values

* updates netstate Put to save Pointers, updates client example to put a pointer, adds grpc status errors, updates tests, changes boltdb 'File' struct to 'PointerEntry'

* updates netstate client example and client test to save pointers, updates netstate List and Delete

* begins adding netstate-http tests

* removes netstate http service

* re-adds netstate auth

* updates boltdb netstate test

* changes encrypted_unencrypted_size from int64 to bytes in netstate proto

* updates READMEs
2018-05-29 22:47:40 -04:00
Dennis Coyle
d6ddb65bf8 basic structure (#39)
* wip - basic structure

* cleanup

* zap logger

* handling errors | using urfav/cli

* remove deadcode

* move NewLogger to utils

* single process method

* copyright

* more generic Process

* simplified

* added NewServer Back
2018-05-16 12:47:59 -06:00
Natalie Villasana
1a06c2b0c7 adding grpc interface to network state (#33)
* adds proto files for netstate crud

* moves netstate grpc client lib into pkg/netstate where grpc netstate service is defined

* starts adding grpc client and server tests

* moves creation of grpc server into cmd/netstate/main.go, removes pkg/netstate/service.go, adds more client testing

* changed all 'Path' and 'Value' fields from strings to bytes, updated tests

* changes Get and Delete in proto file to receive 'requests' instead of 'file paths', adds tests for Get, List, and Delete

* changes netstate-routes to get 'fileValue' bytes not 'fileInfo'

* adds example rpc client in 'examples' and adds more specific debug logs

* adds readmes for netstate rpc services and updates netstate-routes
2018-05-14 18:31:26 -06:00
JT Olds
2ef8e581b2 fix travis, and get a clean lint (#32)
* fix travis

* lint fixes

* travis: enable tests

* fix some tests, disable a broken eestream test for now
2018-05-08 18:02:01 -04:00
Dennis Coyle
5e9b3f7d6a merge conflicts 2018-05-07 14:04:53 -04:00
Natalie Ventura Villasana
7907733629 adds panics to netstate test 2018-05-07 11:03:59 -06:00
Natalie Ventura Villasana
4e8336f2ab adds zap logger, some flags, some nit fixes 2018-05-07 11:03:59 -06:00
Natalie Ventura Villasana
d93dacb53e fixes some error handling, updates storj import links 2018-05-07 11:03:59 -06:00
Natalie Ventura Villasana
8d8fd76748 adds suggested organization, zeebo, err handling 2018-05-07 11:03:59 -06:00
Natalie Ventura Villasana
5dec8fafaf adds bolt crud interface w put, get, list, delete 2018-05-07 11:03:59 -06:00
Dennis Coyle
bb8d4f8d22 cleanup 2018-04-25 10:55:26 -04:00
Dennis Coyle
af30d8ffa1 fixing import 2018-04-23 09:54:00 -06:00
Dennis Coyle
86e2657bdd PR comments from @jtolds 2018-04-23 09:54:00 -06:00
Dennis Coyle
e6e1dae7fc redis client 2018-04-23 09:54:00 -06:00
Dennis Coyle
250b36f27b wip on structure 2018-04-06 12:32:34 -04:00