Commit Graph

261 Commits

Author SHA1 Message Date
JT Olio
9aee5efd99
pointerdb: separate client and server packages (#261)
* pointerdb: separate client and server packages

the reason for this is so that things the server needs (bolt, auth)
don't get imported if all you need is the client. will result in
smaller binaries and less flag definitions

* review comments
2018-08-22 09:07:00 -06:00
Philip Hutchins
5279283b39 Adding some logging to Jenkinsfile for debug (#265) 2018-08-22 09:18:50 -04:00
JT Olio
ffa6f816d5 gw: support standard configuration via cmdline (#262) 2018-08-22 09:05:16 -04:00
Kaloyan Raev
62521bb3f8 Fixes downloads in Minio Browser (#264) 2018-08-22 13:15:13 +02:00
Egon Elbre
fe3decc42f
all: fix govet warnings (#255)
Fixes go1.11 vet warnings.

Cancel on WithTimeout must always be called to avoid memory leak:

pkg/provider/provider.go:73: the cancel function returned by context.WithTimeout should be called, not discarded, to avoid a context leak

Range over non-copyable things:

pkg/pool/connection_pool_test.go:32: range var v copies lock: struct{pool pool.ConnectionPool; key string; expected pool.TestFoo; expectedError error} contains pool.ConnectionPool contains sync.RWMutex
pkg/pool/connection_pool_test.go:56: range var v copies lock: struct{pool pool.ConnectionPool; key string; value pool.TestFoo; expected pool.TestFoo; expectedError error} contains pool.ConnectionPool contains sync.RWMutex
pkg/pool/connection_pool_test.go:83: range var v copies lock: struct{pool pool.ConnectionPool; key string; value pool.TestFoo; expected interface{}; expectedError error} contains pool.ConnectionPool contains sync.RWMutex

zeebo/errs package always requires formatting directives:

pkg/peertls/peertls.go:50: Class.New call has arguments but no formatting directives
pkg/peertls/utils.go:47: Class.New call has arguments but no formatting directives
pkg/peertls/utils.go:87: Class.New call has arguments but no formatting directives
pkg/overlay/cache.go:94: Class.New call has arguments but no formatting directives
pkg/provider/certificate_authority.go:98: New call has arguments but no formatting directives
pkg/provider/identity.go:96: New call has arguments but no formatting directives
pkg/provider/utils.go:124: New call needs 1 arg but has 2 args
pkg/provider/utils.go:136: New call needs 1 arg but has 2 args
storage/redis/client.go:44: Class.New call has arguments but no formatting directives
storage/redis/client.go:64: Class.New call has arguments but no formatting directives
storage/redis/client.go:75: Class.New call has arguments but no formatting directives
storage/redis/client.go:80: Class.New call has arguments but no formatting directives
storage/redis/client.go:92: Class.New call has arguments but no formatting directives
storage/redis/client.go:96: Class.New call has arguments but no formatting directives
storage/redis/client.go:102: Class.New call has arguments but no formatting directives
storage/redis/client.go:126: Class.New call has arguments but no formatting directives
2018-08-22 09:39:57 +03:00
Matt Robinson
5841065420 Attempt to fix jenkins builds (#260)
* Attempt to clean up docker image tags

* Clean only master

* Add back go version and actually use it
2018-08-21 22:54:56 -04:00
Jennifer Li Johnson
15657071d7
implements connection success and fail on kad routing table (#249)
* implements connection success and fail on kad routing table

* modifications from code review

* todo

* test fixes

* passes in node rather than id

* removes rpath

* test fix
2018-08-21 14:44:42 -04:00
JT Olio
9c252393f2
clabot: add egon (#259) 2018-08-21 12:23:28 -06:00
JT Olio
40f4f7ec31
piecestore: only check ttl db every hour instead of every 300 nanoseconds (#250) 2018-08-20 15:47:24 -04:00
JT Olio
520abc5c73
Update README.md 2018-08-20 13:46:45 -06:00
JT Olio
fa390c9ec9
Mock overlay move (#245)
* move mock overlay from client to server

this doesn't really change much, but it does allow you to
run a standalone gateway against captain planet. it still does
not allow you to run a standalone gateway against a standalone
heavy client

* pointerdb: small error fixes

* some cleanups

* fix tests
2018-08-20 14:24:11 -04:00
Bryan White
f932c3ced8 Overlay cache commands (#236)
* wip overlay cache cmd

* overlay cache add and list commands

* use zap

* fixes

* simplify

* add restrictions

* cleanup

* remove defaults for overlay management commands
2018-08-20 14:21:41 -04:00
Kaloyan Raev
d1c508e24a Avoid logging any minio error in miniogw logging wrapper (#240) 2018-08-20 13:55:39 -04:00
Kaloyan Raev
49577a47f8 Configure erasure-share-size to 1024 bytes (#241) 2018-08-20 13:54:30 -04:00
Kaloyan Raev
130acf57a5
Make PSClient a Closer, so it can be closed with utils.LogClose() (#247) 2018-08-20 18:11:54 +03:00
Jennifer Li Johnson
91bf6e19e8
Routing Table Replacement Cache (#229)
* creating replacement cache

* wip

* wip

* rewrites replacement cache

* replacement cache tests

* update and remove node tests

* check if dropped node got added to replacement cache from add node method

* wip

* wip

* making changes based on pr
2018-08-17 15:11:46 -04:00
Brandon Iglesias
a522cabfbd readme updates (Storj CLI) (#243)
* updates to the readme

Added section for the Storj CLI
Styled AWS S3 CLI section

* Update README.md

* removed file name in upload object commands

* updated title

* added table of contents
2018-08-17 15:05:03 -04:00
Alexander Leitner
b0db33f919
Bandwidth accounting (#134)
* captplanet standalone farmer setup

* Bandwidth Allocation

* utils.Close method changed to utils.LogClose

* Get build temporarily working

* Get/Put for PSClient should take payer bandwidth allocations rather than the NewPSClient function

* Update example client to reflect changes in client API

* Update ecclient to use latest PSClient, Make NewPSClient return error also

* Updated pieceranger tests to check for errors; sign method should take byte array

* Handle defers in store.go better

* Fix defer functions in psdb.go

* fun times

* Protobuf bandwidthallocation data is now a byte array

* Remove psservice package and merge it into pstore server

* Write wrapper for database calls

* Change all expiration names in protobuf to be more informative; add defer in retrieve; remove old comment

* Make PSDB tests implementation independent rather than method independent

* get rid of payer, renter in ecclient

* add context monitoring in store and retrieve
2018-08-17 13:40:15 -04:00
Kaloyan Raev
ae430b8c10 Use utils.LogClose for deferred closing in miniogw (#239) 2018-08-17 13:17:58 -04:00
Cameron
906643fce2
Storj CLT (#201)
* WIP first commit

* change args check

* rename to cli, rename folder to storj, const defaultConfDir, use process.Exec()

* typos

* WIP create helper for getting minio objectlayer, changed setup command to try to match changes in setting up identity

* add rb command

* changes for setting up identity

* remove concurrency field from setupCfg

* rename variable returned by url.Parse to u across all commands for consistency

* add marker functionality to ls command

* add download functionality to cp command

* fix typo in rb init, use short variable names
2018-08-16 14:34:55 -04:00
Philip Hutchins
bfac1297f2
Updating entrypoint and dockerfile (#237) 2018-08-16 14:03:15 -04:00
Natalie Villasana
ff65663867
adds Bucket Store (#213)
* adds foundation for bucketStore

* adds prefixedObjStore to buckets package, adjusts gateway-storj accordingly

* fixes multi value assignment problems in gateway-storj

* fixes more multi value assignment errors in gateway-storj

* starts changing miniogw tests to accommodate buckets

* creates bucket store mock

* wip - fixing test cases in object tests

* adds get, put, and list object tests, comments out two test cases

* adds happy scenario tests for bucket methods

* fixes bug in list, removes redundant parts from gateway tests

* fixes nit

* Clean up tests from #188

* Fix bug with timestamp conversion in segment store

* fixes segments.Meta test

* Fix regression in listing objects in a bucket

* adds check to see if bucket is empty before deleting

* updates DeleteBucket test to account for empty/full bucket

* adds TODOs for DeleteBucket and MakeBucket for some cases, adjusts tests, filters out minio errors in logging.go

* adds checks for if buckets already exist or not in DeleteBucket and MakeBucket functions; adjusts tests

* adds BucketNotFound error check in bucket store, removes todo

* adds make_bucket to Travis test, updates boltdb client constructor to always create a bucket (table)
2018-08-16 10:32:28 -04:00
Brandon Iglesias
0cdc331585
styling changes to the code of conduct (#234)
adding bullet points in "Our Standards" section
2018-08-16 10:07:30 -04:00
Brandon Iglesias
c17fc062ff Adding the Storj Code of Conduct (#233)
* Adding the Storj Code of Conduct to Github

* fixing styling

* Update CODE_OF_CONDUCT.md

* Update CODE_OF_CONDUCT.md
2018-08-15 14:22:58 -07:00
Matt Robinson
1687ce25c7 Fix the sha3 error with the build (#230)
* Fix the build
2018-08-15 14:49:30 -04:00
Maximillian von Briesen
f80ddf9358
Test captplanet with travis (#226)
* test captplanet in travis

* add comparison between uploaded and downloaded file

* test inline and remote files

* move captplanet testing to bash script
2018-08-14 12:58:16 -04:00
nfarah86
d396eb6be9
Bug/download (#216)
* fixed downloading empty file

* fixed formatting

* fixed error handling

* fixed cache_test.go to accomodate for error

* fixed error verbose messaging for wrong downlaods

* clean up
2018-08-14 08:22:29 -07:00
Natalie Villasana
6ab323bfec
adds stripe_size field to pointerdb proto file (#218)
* adds erasure_share_size to RS in pointerdb proto

* adds makeErasureScheme helper method to segments store

* cleans up makeErasureScheme function
2018-08-14 11:15:22 -04:00
Philip Hutchins
0440d5fbe6
Adding auto deploy to staging in kube (#219) 2018-08-14 10:00:39 -05:00
JT Olio
c7adc1dfb8
shorter ids (#227)
* shorter ids

* fix test
2018-08-13 17:05:31 -06:00
JT Olio
1412d5a110
fix gw and hc (#225)
* fix gw and hc

* resolve paths

* i keep forgetting gw is different
2018-08-13 12:29:13 -06:00
JT Olio
df726b3ee8
fix services after identity merge (#224)
unfortunately, we can't enable server-side tls yet because
client-side tls hasn't been implemented
2018-08-13 09:39:42 -06:00
JT Olio
1f8db2a4c7
captplanet standalone farmer setup (#202) 2018-08-13 09:33:21 -06:00
JT Olio
ab029301bd
Consolidate command configuration and setup (#221) 2018-08-13 09:07:05 -06:00
Bryan White
5d20cf8829
Node Identity (#193)
* peertls: don't log errors for double close

understood that this part of the code is undergoing heavy change
right now, but just want to make sure this fix gets incorporated
somewhere

* git cleanup: node-id stuff

* cleanup

* rename identity_util.go

* wip `CertificateAuthority` refactor

* refactoring

* gitignore update

* wip

* Merge remote-tracking branch 'storj/doubleclose' into node-id3

* storj/doubleclose:
  peertls: don't log errors for double close

* add peertls tests & gomports

* wip:

+ refactor
+ style changes
+ cleanup
+ [wip] add version to CA and identity configs
+ [wip] heavy client setup

* refactor

* wip:

+ refactor
+ style changes
+ add `CAConfig.Load`
+ add `CAConfig.Save`

* wip:

+ add `LoadOrCreate` and `Create` to CA and Identity configs
+ add overwrite to CA and identity configs
+ heavy client setup
+ refactor
+ style changes
+ cleanup

* wip

* fixing things

* fixing things

* wip hc setup

* hc setup:

+ refactor
+ bugfixing

* improvements based on reveiw feedback

* goimports

* improvements:

+ responding to review feedback
+ refactor

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* cleanup

* refactoring CA and Identity structs

* Merge branch 'master' into node-id3

* move version field to setup config structs for CA and identity

* fix typo

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* Merge branch 'master' into node-id3

* fix gateway setup finally

* go imports

* fix `FullCertificateAuthority.GenerateIdentity`

* cleanup overlay tests

* bugfixing

* update ca/identity setup

* go imports

* fix peertls test copy/paste fail

* responding to review feedback

* setup tweaking

* update farmer setup
2018-08-13 10:39:45 +02:00
Jennifer Li Johnson
198f7fd506
uncomments oc=globalmockoverlay (#222) 2018-08-10 19:19:19 -04:00
JT Olio
11e219d14f add phutchins to clabot (#220) 2018-08-10 18:11:44 -04:00
Brandon Iglesias
2269682280 spelling fix for read me (#217) 2018-08-10 09:21:43 -07:00
Brandon Iglesias
73a0bf3c7b Read me updates (#215)
* Read me updates

Readme updates based off the first two rounds of critic reviews

* Update README.md

* Update README.md
2018-08-10 08:28:38 -07:00
Kaloyan Raev
ffaebb57df
Improved error message in ECClient (#214)
* Improved error message in ECClient

* Fixed broken test
2018-08-10 12:14:23 +03:00
Jennifer Li Johnson
5f8142cdcc
Integrating on disk routing table with kademlia (#166)
* adds comment

* runs deps

* adds print statements for debugging add node bkad

* more print statements

* removes bkad from routing and integrates on disk routing table

tests failing :(

wip

* removes testbootstrap

* kademlia_test not working

* adds kad tests back in

* Adds skips for tests broken due to wip kademlia
2018-08-09 15:20:39 -04:00
Philip Hutchins
1f7d77ae69
Fixing Docker entrypoint for storage node (#212) 2018-08-09 12:22:39 -04:00
Kaloyan Raev
3644d77b45 PSClient handles io.ErrUnexpectedEOF (#211)
* PSClient handles io.ErrUnexpectedEOF

* Fix my bad English

* Even better log message
2018-08-09 11:42:35 -04:00
Kaloyan Raev
34629e5b80 Group erasure shares before sending them to storage nodes (#210) 2018-08-09 11:14:05 -04:00
Philip Hutchins
80c0a0ed2b
Fixing docker entrypoint for satellite (#208) 2018-08-08 19:34:25 -04:00
Maximillian von Briesen
889afd0ae5
add overwrite flag for captplanet, hc, and gw setup (#209)
* add overwrite flag for captplanet setup

* add overwrite flag for heavy client and gateway
2018-08-08 18:22:59 -04:00
JT Olio
fb77287325 fix env vars (#206) 2018-08-08 13:49:23 -04:00
Philip Hutchins
c77d1b7f0b
Updating service names for docker tags (#207) 2018-08-08 13:48:59 -04:00
Natalie Villasana
9ad4949825
adds segment store tests (#198)
* starts adding segmentStore tests

* adds mocked interfaces for segmentStore tests

* adds tests for put, get, delete, and list

* regenerates pointerdb mock and updates calls to accommodate new changes
2018-08-07 11:52:11 -04:00
Dylan Lott
4c6d359473
Overlay worker start (#190)
This is a naive implementation of the overlay worker. 

Future improvements / to dos: 
- Walk through the cache and remove nodes that don't respond 
- Better look ups for new nodes
- Better random ID generation
- Kademlia hooks for automatically adding new nodes to the cache

* adding refresh cache functionality, added schedule function

* update put in db

* Tests passing

* wip overlay tests for cache refresh

* update scheduler code

* update refresh function

* WIP adding random lookups to refresh worker

* remove quit channel

* updates fire on schedule and the refresh function finds near nodes

* updates to refresh function, getting more buckets and nodes

* updates to refresh function and cache operations

* add cancellation to context, fix k number of nodes in lookups
2018-08-06 17:32:47 -06:00