Commit Graph

6059 Commits

Author SHA1 Message Date
JT Olio
ffa6f816d5 gw: support standard configuration via cmdline (#262) 2018-08-22 09:05:16 -04:00
Kaloyan Raev
62521bb3f8 Fixes downloads in Minio Browser (#264) 2018-08-22 13:15:13 +02:00
Egon Elbre
fe3decc42f
all: fix govet warnings (#255)
Fixes go1.11 vet warnings.

Cancel on WithTimeout must always be called to avoid memory leak:

pkg/provider/provider.go:73: the cancel function returned by context.WithTimeout should be called, not discarded, to avoid a context leak

Range over non-copyable things:

pkg/pool/connection_pool_test.go:32: range var v copies lock: struct{pool pool.ConnectionPool; key string; expected pool.TestFoo; expectedError error} contains pool.ConnectionPool contains sync.RWMutex
pkg/pool/connection_pool_test.go:56: range var v copies lock: struct{pool pool.ConnectionPool; key string; value pool.TestFoo; expected pool.TestFoo; expectedError error} contains pool.ConnectionPool contains sync.RWMutex
pkg/pool/connection_pool_test.go:83: range var v copies lock: struct{pool pool.ConnectionPool; key string; value pool.TestFoo; expected interface{}; expectedError error} contains pool.ConnectionPool contains sync.RWMutex

zeebo/errs package always requires formatting directives:

pkg/peertls/peertls.go:50: Class.New call has arguments but no formatting directives
pkg/peertls/utils.go:47: Class.New call has arguments but no formatting directives
pkg/peertls/utils.go:87: Class.New call has arguments but no formatting directives
pkg/overlay/cache.go:94: Class.New call has arguments but no formatting directives
pkg/provider/certificate_authority.go:98: New call has arguments but no formatting directives
pkg/provider/identity.go:96: New call has arguments but no formatting directives
pkg/provider/utils.go:124: New call needs 1 arg but has 2 args
pkg/provider/utils.go:136: New call needs 1 arg but has 2 args
storage/redis/client.go:44: Class.New call has arguments but no formatting directives
storage/redis/client.go:64: Class.New call has arguments but no formatting directives
storage/redis/client.go:75: Class.New call has arguments but no formatting directives
storage/redis/client.go:80: Class.New call has arguments but no formatting directives
storage/redis/client.go:92: Class.New call has arguments but no formatting directives
storage/redis/client.go:96: Class.New call has arguments but no formatting directives
storage/redis/client.go:102: Class.New call has arguments but no formatting directives
storage/redis/client.go:126: Class.New call has arguments but no formatting directives
2018-08-22 09:39:57 +03:00
Matt Robinson
5841065420 Attempt to fix jenkins builds (#260)
* Attempt to clean up docker image tags

* Clean only master

* Add back go version and actually use it
2018-08-21 22:54:56 -04:00
Jennifer Li Johnson
15657071d7
implements connection success and fail on kad routing table (#249)
* implements connection success and fail on kad routing table

* modifications from code review

* todo

* test fixes

* passes in node rather than id

* removes rpath

* test fix
2018-08-21 14:44:42 -04:00
JT Olio
9c252393f2
clabot: add egon (#259) 2018-08-21 12:23:28 -06:00
JT Olio
40f4f7ec31
piecestore: only check ttl db every hour instead of every 300 nanoseconds (#250) 2018-08-20 15:47:24 -04:00
JT Olio
520abc5c73
Update README.md 2018-08-20 13:46:45 -06:00
JT Olio
fa390c9ec9
Mock overlay move (#245)
* move mock overlay from client to server

this doesn't really change much, but it does allow you to
run a standalone gateway against captain planet. it still does
not allow you to run a standalone gateway against a standalone
heavy client

* pointerdb: small error fixes

* some cleanups

* fix tests
2018-08-20 14:24:11 -04:00
Bryan White
f932c3ced8 Overlay cache commands (#236)
* wip overlay cache cmd

* overlay cache add and list commands

* use zap

* fixes

* simplify

* add restrictions

* cleanup

* remove defaults for overlay management commands
2018-08-20 14:21:41 -04:00
Kaloyan Raev
d1c508e24a Avoid logging any minio error in miniogw logging wrapper (#240) 2018-08-20 13:55:39 -04:00
Kaloyan Raev
49577a47f8 Configure erasure-share-size to 1024 bytes (#241) 2018-08-20 13:54:30 -04:00
Kaloyan Raev
130acf57a5
Make PSClient a Closer, so it can be closed with utils.LogClose() (#247) 2018-08-20 18:11:54 +03:00
Jennifer Li Johnson
91bf6e19e8
Routing Table Replacement Cache (#229)
* creating replacement cache

* wip

* wip

* rewrites replacement cache

* replacement cache tests

* update and remove node tests

* check if dropped node got added to replacement cache from add node method

* wip

* wip

* making changes based on pr
2018-08-17 15:11:46 -04:00
Brandon Iglesias
a522cabfbd readme updates (Storj CLI) (#243)
* updates to the readme

Added section for the Storj CLI
Styled AWS S3 CLI section

* Update README.md

* removed file name in upload object commands

* updated title

* added table of contents
2018-08-17 15:05:03 -04:00
Alexander Leitner
b0db33f919
Bandwidth accounting (#134)
* captplanet standalone farmer setup

* Bandwidth Allocation

* utils.Close method changed to utils.LogClose

* Get build temporarily working

* Get/Put for PSClient should take payer bandwidth allocations rather than the NewPSClient function

* Update example client to reflect changes in client API

* Update ecclient to use latest PSClient, Make NewPSClient return error also

* Updated pieceranger tests to check for errors; sign method should take byte array

* Handle defers in store.go better

* Fix defer functions in psdb.go

* fun times

* Protobuf bandwidthallocation data is now a byte array

* Remove psservice package and merge it into pstore server

* Write wrapper for database calls

* Change all expiration names in protobuf to be more informative; add defer in retrieve; remove old comment

* Make PSDB tests implementation independent rather than method independent

* get rid of payer, renter in ecclient

* add context monitoring in store and retrieve
2018-08-17 13:40:15 -04:00
Kaloyan Raev
ae430b8c10 Use utils.LogClose for deferred closing in miniogw (#239) 2018-08-17 13:17:58 -04:00
Cameron
906643fce2
Storj CLT (#201)
* WIP first commit

* change args check

* rename to cli, rename folder to storj, const defaultConfDir, use process.Exec()

* typos

* WIP create helper for getting minio objectlayer, changed setup command to try to match changes in setting up identity

* add rb command

* changes for setting up identity

* remove concurrency field from setupCfg

* rename variable returned by url.Parse to u across all commands for consistency

* add marker functionality to ls command

* add download functionality to cp command

* fix typo in rb init, use short variable names
2018-08-16 14:34:55 -04:00
Philip Hutchins
bfac1297f2
Updating entrypoint and dockerfile (#237) 2018-08-16 14:03:15 -04:00
Natalie Villasana
ff65663867
adds Bucket Store (#213)
* adds foundation for bucketStore

* adds prefixedObjStore to buckets package, adjusts gateway-storj accordingly

* fixes multi value assignment problems in gateway-storj

* fixes more multi value assignment errors in gateway-storj

* starts changing miniogw tests to accommodate buckets

* creates bucket store mock

* wip - fixing test cases in object tests

* adds get, put, and list object tests, comments out two test cases

* adds happy scenario tests for bucket methods

* fixes bug in list, removes redundant parts from gateway tests

* fixes nit

* Clean up tests from #188

* Fix bug with timestamp conversion in segment store

* fixes segments.Meta test

* Fix regression in listing objects in a bucket

* adds check to see if bucket is empty before deleting

* updates DeleteBucket test to account for empty/full bucket

* adds TODOs for DeleteBucket and MakeBucket for some cases, adjusts tests, filters out minio errors in logging.go

* adds checks for if buckets already exist or not in DeleteBucket and MakeBucket functions; adjusts tests

* adds BucketNotFound error check in bucket store, removes todo

* adds make_bucket to Travis test, updates boltdb client constructor to always create a bucket (table)
2018-08-16 10:32:28 -04:00
Brandon Iglesias
0cdc331585
styling changes to the code of conduct (#234)
adding bullet points in "Our Standards" section
2018-08-16 10:07:30 -04:00
Brandon Iglesias
c17fc062ff Adding the Storj Code of Conduct (#233)
* Adding the Storj Code of Conduct to Github

* fixing styling

* Update CODE_OF_CONDUCT.md

* Update CODE_OF_CONDUCT.md
2018-08-15 14:22:58 -07:00
Matt Robinson
1687ce25c7 Fix the sha3 error with the build (#230)
* Fix the build
2018-08-15 14:49:30 -04:00
Maximillian von Briesen
f80ddf9358
Test captplanet with travis (#226)
* test captplanet in travis

* add comparison between uploaded and downloaded file

* test inline and remote files

* move captplanet testing to bash script
2018-08-14 12:58:16 -04:00
nfarah86
d396eb6be9
Bug/download (#216)
* fixed downloading empty file

* fixed formatting

* fixed error handling

* fixed cache_test.go to accomodate for error

* fixed error verbose messaging for wrong downlaods

* clean up
2018-08-14 08:22:29 -07:00
Natalie Villasana
6ab323bfec
adds stripe_size field to pointerdb proto file (#218)
* adds erasure_share_size to RS in pointerdb proto

* adds makeErasureScheme helper method to segments store

* cleans up makeErasureScheme function
2018-08-14 11:15:22 -04:00
Philip Hutchins
0440d5fbe6
Adding auto deploy to staging in kube (#219) 2018-08-14 10:00:39 -05:00
JT Olio
c7adc1dfb8
shorter ids (#227)
* shorter ids

* fix test
2018-08-13 17:05:31 -06:00
JT Olio
1412d5a110
fix gw and hc (#225)
* fix gw and hc

* resolve paths

* i keep forgetting gw is different
2018-08-13 12:29:13 -06:00
JT Olio
df726b3ee8
fix services after identity merge (#224)
unfortunately, we can't enable server-side tls yet because
client-side tls hasn't been implemented
2018-08-13 09:39:42 -06:00
JT Olio
1f8db2a4c7
captplanet standalone farmer setup (#202) 2018-08-13 09:33:21 -06:00
JT Olio
ab029301bd
Consolidate command configuration and setup (#221) 2018-08-13 09:07:05 -06:00
Bryan White
5d20cf8829
Node Identity (#193)
* peertls: don't log errors for double close

understood that this part of the code is undergoing heavy change
right now, but just want to make sure this fix gets incorporated
somewhere

* git cleanup: node-id stuff

* cleanup

* rename identity_util.go

* wip `CertificateAuthority` refactor

* refactoring

* gitignore update

* wip

* Merge remote-tracking branch 'storj/doubleclose' into node-id3

* storj/doubleclose:
  peertls: don't log errors for double close

* add peertls tests & gomports

* wip:

+ refactor
+ style changes
+ cleanup
+ [wip] add version to CA and identity configs
+ [wip] heavy client setup

* refactor

* wip:

+ refactor
+ style changes
+ add `CAConfig.Load`
+ add `CAConfig.Save`

* wip:

+ add `LoadOrCreate` and `Create` to CA and Identity configs
+ add overwrite to CA and identity configs
+ heavy client setup
+ refactor
+ style changes
+ cleanup

* wip

* fixing things

* fixing things

* wip hc setup

* hc setup:

+ refactor
+ bugfixing

* improvements based on reveiw feedback

* goimports

* improvements:

+ responding to review feedback
+ refactor

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* cleanup

* refactoring CA and Identity structs

* Merge branch 'master' into node-id3

* move version field to setup config structs for CA and identity

* fix typo

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* Merge branch 'master' into node-id3

* fix gateway setup finally

* go imports

* fix `FullCertificateAuthority.GenerateIdentity`

* cleanup overlay tests

* bugfixing

* update ca/identity setup

* go imports

* fix peertls test copy/paste fail

* responding to review feedback

* setup tweaking

* update farmer setup
2018-08-13 10:39:45 +02:00
Jennifer Li Johnson
198f7fd506
uncomments oc=globalmockoverlay (#222) 2018-08-10 19:19:19 -04:00
JT Olio
11e219d14f add phutchins to clabot (#220) 2018-08-10 18:11:44 -04:00
Brandon Iglesias
2269682280 spelling fix for read me (#217) 2018-08-10 09:21:43 -07:00
Brandon Iglesias
73a0bf3c7b Read me updates (#215)
* Read me updates

Readme updates based off the first two rounds of critic reviews

* Update README.md

* Update README.md
2018-08-10 08:28:38 -07:00
Kaloyan Raev
ffaebb57df
Improved error message in ECClient (#214)
* Improved error message in ECClient

* Fixed broken test
2018-08-10 12:14:23 +03:00
Jennifer Li Johnson
5f8142cdcc
Integrating on disk routing table with kademlia (#166)
* adds comment

* runs deps

* adds print statements for debugging add node bkad

* more print statements

* removes bkad from routing and integrates on disk routing table

tests failing :(

wip

* removes testbootstrap

* kademlia_test not working

* adds kad tests back in

* Adds skips for tests broken due to wip kademlia
2018-08-09 15:20:39 -04:00
Philip Hutchins
1f7d77ae69
Fixing Docker entrypoint for storage node (#212) 2018-08-09 12:22:39 -04:00
Kaloyan Raev
3644d77b45 PSClient handles io.ErrUnexpectedEOF (#211)
* PSClient handles io.ErrUnexpectedEOF

* Fix my bad English

* Even better log message
2018-08-09 11:42:35 -04:00
Kaloyan Raev
34629e5b80 Group erasure shares before sending them to storage nodes (#210) 2018-08-09 11:14:05 -04:00
Philip Hutchins
80c0a0ed2b
Fixing docker entrypoint for satellite (#208) 2018-08-08 19:34:25 -04:00
Maximillian von Briesen
889afd0ae5
add overwrite flag for captplanet, hc, and gw setup (#209)
* add overwrite flag for captplanet setup

* add overwrite flag for heavy client and gateway
2018-08-08 18:22:59 -04:00
JT Olio
fb77287325 fix env vars (#206) 2018-08-08 13:49:23 -04:00
Philip Hutchins
c77d1b7f0b
Updating service names for docker tags (#207) 2018-08-08 13:48:59 -04:00
Natalie Villasana
9ad4949825
adds segment store tests (#198)
* starts adding segmentStore tests

* adds mocked interfaces for segmentStore tests

* adds tests for put, get, delete, and list

* regenerates pointerdb mock and updates calls to accommodate new changes
2018-08-07 11:52:11 -04:00
Dylan Lott
4c6d359473
Overlay worker start (#190)
This is a naive implementation of the overlay worker. 

Future improvements / to dos: 
- Walk through the cache and remove nodes that don't respond 
- Better look ups for new nodes
- Better random ID generation
- Kademlia hooks for automatically adding new nodes to the cache

* adding refresh cache functionality, added schedule function

* update put in db

* Tests passing

* wip overlay tests for cache refresh

* update scheduler code

* update refresh function

* WIP adding random lookups to refresh worker

* remove quit channel

* updates fire on schedule and the refresh function finds near nodes

* updates to refresh function, getting more buckets and nodes

* updates to refresh function and cache operations

* add cancellation to context, fix k number of nodes in lookups
2018-08-06 17:32:47 -06:00
JT Olio
2c500e7ba2
fix logo 2018-08-06 11:39:20 -06:00
Kaloyan Raev
aac7c6fbd1
ECClient pads data to fit RS block size (#199) 2018-08-06 17:24:30 +03:00