Commit Graph

62 Commits

Author SHA1 Message Date
Egon Elbre
9ac3517432
Reduce noise in logging (#733) 2018-11-29 22:59:26 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Michal Niewrzal
8dd669b37f
Bandwidth allocations need method designation (GET/PUT) (#708)
* Bandwidth allocations need method designation (GET/PUT)

* add method comment

* goimports
2018-11-26 19:21:44 +01:00
Bryan White
dabd87cb36
swap golang/protobuf out for gogo/protobuf (#692)
* swap golang/protobuf out for gogo/protobuf

* revert go.mod and go.sum changes

* linter fixes
2018-11-20 19:29:07 +01:00
Bryan White
54ccb460a9
rename size fields in protobuf messages for gogo compatibility (#690)
* rename `size` fields in protobuf messages for gogo compatibility

* linter fixes
2018-11-20 18:09:35 +01:00
Alexander Leitner
320c93368a
Add creation time to Payer Bandwidth Allocation (#581)
* Add creation time to Payer Bandwidth Allocation

* goimports
2018-11-08 11:04:52 -05:00
Bryan White
ee62e2a9d8
Use transport client and cleanup all the clients (#574)
* wip

* linter fixes

* linter fixes

* test fixes

* linter fixes

* fix merge + restructure piecestore packages

* review feedback

* linter fixes

* linter fixes

* remove unnecessary aliases to piecestore

* more merge fixing
2018-11-06 18:49:17 +01:00
Michal Niewrzal
deb015970d
Use Nodes array from pdb.Get (#578)
* Use Nodes array from pdb.Get

* fix problems with captplanet tests

* better comments
2018-11-06 18:03:11 +01:00
Michal Niewrzal
1129ead6ef
Fix 'message not signed by the satellite' problem (#560)
* Fix 'message not signed by the satellite' problem

* fixed build

* fix errors
2018-11-05 16:12:19 +01:00
Egon Elbre
c765dd98de
internal/testplanet (#528) 2018-11-03 14:17:14 +02:00
Bryan White
8b9711cb5e
better waitlist-gating (#557)
* better waitlist-gating

(cherry picked from commit 490fe02b7c3558da18678dfb651c92ec9c4a75b5)

* fix broken test

* linter fixes

* linter fixes

* make extension verification optional

* add certifcate gating script for captplanet

* fixing tests

* linter fixes

* linter fixes?

* moar linter fixes

* Revert "moar linter fixes"

This reverts commit 8139ccbd73cbbead987b7667567844f50f7df2c8.

* just kill me

* refactor

* refactor tests

* liniter...

* cleanup
2018-11-01 16:48:43 +01:00
Dennis Coyle
a3becb8a7b
Add repairer & checker to Satelite (#561)
* Added repairer & checker to Satellite

* fixed repairer and checker configs
2018-10-31 12:22:35 -04:00
Jennifer Li Johnson
1fb96689b8
creates run loop for data repair checker (#490)
* creates run loop for data repair checker

* moves actual checking and repairing under ticker case

* fixes mismatched queueaddrs
2018-10-30 15:16:40 -04:00
Alexander Leitner
7e4149138a
Piecestore proto change (#556)
* add action and storage node id to piecestore.proto. renamed payer and renter to satellite_id and uplink_id
2018-10-30 15:03:41 -04:00
Michal Niewrzal
0d03f2fbb5
Send bandwidth alloc from satellite to storage node (#538)
* Send bandwidth alloc from satellite to storage node

* Remove unecessary nil checks

* set Renter field

* fix tests

* goimports

* Update README.md

* Update README.md

* Update README.md

* Update README.md (#550)

* Skip flaky TestPing (#552)
2018-10-30 17:24:46 +01:00
Kaloyan Raev
99640225fd
Refactor Path type (#522)
The old paths.Path type is now replaced with the new storj.Path.

storj.Path is simply an alias to the built-in string type. As such it can be used just as any string, which simplifies a lot working with paths. No more conversions paths.New and path.String().

As an alias storj.Path does not define any methods. However, any functions applying to strings (like those from the strings package) gracefully apply to storj.Path too. In addition we have a few more functions defined:

    storj.SplitPath
    storj.JoinPaths
    encryption.EncryptPath
    encryption.DecryptPath
    encryption.DerivePathKey
    encryption.DeriveContentKey

All code in master is migrated to the new storj.Path type.

The Path example is also updated and is good for reference: /pkg/encryption/examples_test.go

This PR also resolve a nonce misuse issue in path encryption: https://storjlabs.atlassian.net/browse/V3-545
2018-10-25 23:28:16 +03:00
paul cannon
e2c0dd437a
offer PostgreSQL storage for pointerdb (#440)
..although it ought to work for other storage.KeyValueStore needs as
well. it's just optimized to work pretty well for a largish hierarchy of
paths.

This includes the addition of "long benchmarks" for KeyValueStore
testing. These will only be run when -test-bench-long is added to the
test flags. In these benchmarks, a large corpus of paths matching a
natural ("real-life") hierarchy is read from paths.data.gz (which you
can get from https://github.com/storj/path-test-corpus) and imported
into a particular KeyValueStore. Recursive and non-recursive queries are
run on it to detect performance problems that arise only at scale.

This also includes alternate implementation of the postgreskv client,
which works in a less-bizarre way for non-recursive queries, but suffers
from poor performance in tests such as the long benchmarks. Once this
alternate impl is committed to the tree, we can remove it again; I just
want it to be available for future reference.
2018-10-25 12:11:28 -05:00
Michal Niewrzal
f9248c21d4
Satellite verification on storage node (#469)
* Satellite verification on storage node

* fix formatting

* fix formatting

* rename SignatureAuth to SignedMessage

* fixes after review

* fix linter errors

* improve errors handling

* remove SignedMessageProvider

* fix liter errors

* params changed to authorization, signed message in audit, minor fixes

* fix formatting
2018-10-17 13:40:11 +02:00
Egon Elbre
2dbada73ab
Separate storj.io packages in imports (#481)
Format with goimports -w -local storj.io
Add flag to golang lint
Remove commented out imports as they mess with goimports
2018-10-16 14:43:44 +03:00
Egon Elbre
e39f9e42b4
Kademlia Sequential Lookup (#464) 2018-10-12 11:52:32 +03:00
Egon Elbre
db42772b4d
Enable goimports and fix formatting. (#459) 2018-10-11 23:25:54 +03:00
Michal Niewrzal
351dc523d6
Satellite signature generation (#453)
* Satellite signature generation

* Add unit test

* remove unused var

* remove base64 encoding from signature generation
2018-10-11 16:35:55 +02:00
Natalie Villasana
b04ea4639f moves all audit files to pkg/audit (#449) 2018-10-09 18:05:42 -04:00
Jennifer Li Johnson
6fb13896fb
Method to identify injured segments to repair (#398)
* creates checker

* tests offline nodes

* test id injured segs:

* Adds healthy pieces to injured segment struct

* changes inequality

* creates common files

* adds checker benchmarking

* creates more common files

* Replaces pointedb direct db with api call to a new iterate method on pointerdb

* move monkit

* removes identifyrequest proto

* remove healthypieces

* adds benchmarking

creates common file for datarepair

* recreates proto file

* api key on ctx
2018-10-09 12:09:33 -04:00
Michal Niewrzal
ad327bedb1
Use context to propagate API Key (#383)
* Satellite signs proofs

* wip

* remove direct apikey usage from pdbclient

* adjusting unit tests

* fix linter errors

* unit tests

* linter errors

* remove usless interface

* remove unused code

* improve unit tests

* signature generation

* code review changes

* code review comments

* back to satellite-id signature generation

* remove go-grpc-middlewar dependency

* small step back

* linter fixes

* fix tests

* packages reorganization

* Move TestAPIKeyInjector to grpcauth package
2018-10-09 16:39:14 +02:00
Kaloyan Raev
3de34ab6cb
Rename MinInlineSegmentSize to MinRemoteSegmentSize (#436) 2018-10-08 23:09:28 +03:00
Dennis Coyle
dee2c137c8
Remove BKAD dependency from pkg/kademlia (#294)
* slowly but surely

* hardcode ID for tests so we can get predictable results

* skipping bad test

* removing tests of bkad

* wip

* new algorithm for worker

* clean up

* remove skipped test

* changes

* uncomment

* fixed conflicts

* maybe done ?

* cleanup

* boot bkad

* wip

* cleanup

* undo change

* fixes

* wip

* wip

* moving nodeID around

* wip

* wip

* fixes

* fixes after merge

* added TODO

* fixed tests post identity

* linter fixes

* wip

* PR review comments

* wip

* fixing tests

* fix tests

* force db directory

* bad test

* fixes race condition

* small cleanups

* adding db folder

* testing

* wip

* cleanup

* cleanup

* linters

* export Restrict

* add timeout

* testing

* linters

* forgot one

* moar fixes from master merge

* PR comments

* moar PR comments

* removed stun flag

* remove duplicate declaration

* remove old tests

* remove timeout

* fix tests

* missed one

* changed StringToID >> IDFromString

* PR comments

* stupid linter

* moevd overlay mock

* fixed merge conflicts

* fixes

* linter
2018-10-08 11:09:37 -04:00
nfarah86
7812f1bbc0
Stripe selection (#375)
* initial commit- wip, working on testing and library

* wip working on testing library functioon to get poointer

* working on nil  reference for testing

* tests wip

* wip-working on getting tests  to  work

* working on tests

* put test passes

* working on test- need to export

* created pdclient, and now working on testing function

* tests working for list- getting object back

* wip - got derived piece id

* fixed making grpc public

* fixed linter errors and minor added method for size

* need to work on  testing, added random  integer function

* got psc server working for  testing

* working on ranger test and ranger  method

* testing creds for new computer

* working on getting segment metadata

* get random stripe

* added caveat to random fn

* fixed data types

* modified library to have one public function that returns a random stripe

* removed extra comments

* added commons.go file for audit

* added last path to be remembered

* changed random function to cryto/rand/ & worked on tests passing

* working on testing to get analysis of randomness

* changed to track last item in pagination

* finished testing randomness, cleaned up code

* fixed rebase errors

* removed error, kept common file

* fixed travis errors

* attempt to fix overlay issue

* fixed travis error

* updated pointer parameters

* made smaller functions, renamed audit

* made changes per suggestions

* removed gosum

* fixed pr per suggestions

* removed comment
2018-10-05 06:34:15 -07:00
Dylan Lott
ae38817fa3
Pointer lookup (#407)
* Loads cache from context for PointerDB access

* WIP adds overlay lookups to pointerdb requests

* Pointer lookup code is added for Get

* adds feature flag for pointerdb return

* refactors pointerdb code

* removes some unnecessary debug logs

* Fixes indent in config

* adds early return for non-remote pointers

* formats code, removes some comments

* Fixes tests broken by pointer proto changes

* adds error check and merges variable declaration

* removes commented out proto import

* adds error check to pdbclient
2018-10-04 15:00:19 -06:00
Egon Elbre
b6b6111173
Flatten proto definitions into a single package (#360)
* protos: move streams to pb
* protos: move overlay to pb
* protos: move pointerdb to pb
* protos: move piecestore to pb
* fix statdb import naming
2018-09-18 07:39:06 +03:00
Egon Elbre
6ee6f0fdf5
Remove duplicated mock store implementations (#337) 2018-09-11 10:27:12 +03:00
Egon Elbre
7a906285e3 Small cleanup at top-level (#336)
* cleanup some top-level packages

* remove unused index.html

* update indirect tags
2018-09-10 07:23:08 -07:00
Egon Elbre
00ac266bda Remove zap dependency from boltdb (#334)
* Remove zap dependency from boltdb

* Avoid pulling in testing in main binary
2018-09-10 05:52:53 -04:00
Kaloyan Raev
f62a3c383c
Filter prefix from list results (#328) 2018-09-09 09:34:23 +03:00
Egon Elbre
0a2839b2d7
Add IsPrefix support to piecestore (#313) 2018-09-07 17:20:15 +03:00
Egon Elbre
79354bf0ab
Enforce lookup limit for storage (#312) 2018-09-07 12:00:00 +03:00
Egon Elbre
83df0ee1b0
Implement ListV2 with storage rework (#303)
1. Added KeyValueStore.Iterate for implementing the different List, ListV2 etc. implementations. This allows for more efficient use of memory depending on the situation.
2. Implemented an inmemory teststore for running tests. This should allow to replace MockKeyValueStore in most places.
3. Rewrote tests
4. Pulled out logger from bolt implementation so it can be used for all other storage implementations.
5. Fixed multiple things in bolt and redis implementations.
2018-09-05 19:10:35 +03:00
Egon Elbre
0f5a2f4ef5 Enable more linters (#272)
* enable more linters

* Run gofmt -s

* run goimports

* run unconvert

* fix naked return

* fix misspellings

* fix ineffectual assigments

* fix missing declaration

* don't use deprecated grpc.Errof

* check errors in tests

* run gofmt -w -r "assert.Nil(err) -> assert.NoError(err)"

* fix directory permissions

* don't use nil Context

* simplify boolean expressions

* use bytes.Equal instead of bytes.Compare

* merge variable declarations, remove redundant returns

* fix some golint errors

* run goimports

* handle more errors

* delete empty TestMain

* delete empty TestMain

* ignore examples for now

* fix lint errors

* remove unused values

* more fixes

* run gofmt -w -s .

* add more comments

* fix naming

* more lint fixes

* try switching travis to go1.11

* fix unnecessary conversions

* fix deprecated methods

* use go1.10 and disable gofmt/goimports for now

* switch to 1.10

* don't re-enable gofmt and goimports

* switch covermode to atomic because of -race

* gofmt
2018-08-27 11:28:16 -06:00
Dylan Lott
7e52c81ebc Segment validation (#280)
* WIP adding inline segment size validation

* add better error handling

* updates error handling and removes unnecessary logs

* update minimum inline size check

* update check if condition and inline min config type
2018-08-27 10:04:46 -04:00
nfarah86
5f46a0696f
writing definition for LISTv2 (#289)
* writing definition for LISTv2

* fixed method to call storage.xx

* regenerate kvstore file

* fixed minor error in storage

* added panic

* added panic
2018-08-25 20:00:49 -07:00
Maximillian von Briesen
553def620a TLS Support (#252)
* begin adding tls

* remove incomplete line in gw/main.go

* identity fixes+:

+ fix `peertls.NewCert` public key issue
+ fix `peertls.verfiyChain` issue
+ fix identity dial option
+ rename `GenerateCA` to `NewCA` and `generateCAWorker` to `newCAWorker` for better consistency/convention

* use pdbclient instead of pointerdb in miniogw

* fix tests

* go fmt

* make review changes

* modify how context.Background() is used

* more context stuff
2018-08-23 22:01:03 -06:00
Egon Elbre
f80185a807
Preliminary support for windows (#266)
* Don't use url.Parse for bolt paths: filepaths may not be valid URL-s.
* go.mod: update dependencies
* README.md: add Windows instructions
* pkg/overlay: check for the correct path and text in error
* pkg/overlay: fix tests for windows
* pkg/piecestore: make windows tests pass
* pkg/telemetry: skip test, as it doesn't shutdown nicely
* storage/redis: ensure that redis is clean before running tests
2018-08-23 10:05:56 +03:00
JT Olio
9aee5efd99
pointerdb: separate client and server packages (#261)
* pointerdb: separate client and server packages

the reason for this is so that things the server needs (bolt, auth)
don't get imported if all you need is the client. will result in
smaller binaries and less flag definitions

* review comments
2018-08-22 09:07:00 -06:00
JT Olio
fa390c9ec9
Mock overlay move (#245)
* move mock overlay from client to server

this doesn't really change much, but it does allow you to
run a standalone gateway against captain planet. it still does
not allow you to run a standalone gateway against a standalone
heavy client

* pointerdb: small error fixes

* some cleanups

* fix tests
2018-08-20 14:24:11 -04:00
Natalie Villasana
ff65663867
adds Bucket Store (#213)
* adds foundation for bucketStore

* adds prefixedObjStore to buckets package, adjusts gateway-storj accordingly

* fixes multi value assignment problems in gateway-storj

* fixes more multi value assignment errors in gateway-storj

* starts changing miniogw tests to accommodate buckets

* creates bucket store mock

* wip - fixing test cases in object tests

* adds get, put, and list object tests, comments out two test cases

* adds happy scenario tests for bucket methods

* fixes bug in list, removes redundant parts from gateway tests

* fixes nit

* Clean up tests from #188

* Fix bug with timestamp conversion in segment store

* fixes segments.Meta test

* Fix regression in listing objects in a bucket

* adds check to see if bucket is empty before deleting

* updates DeleteBucket test to account for empty/full bucket

* adds TODOs for DeleteBucket and MakeBucket for some cases, adjusts tests, filters out minio errors in logging.go

* adds checks for if buckets already exist or not in DeleteBucket and MakeBucket functions; adjusts tests

* adds BucketNotFound error check in bucket store, removes todo

* adds make_bucket to Travis test, updates boltdb client constructor to always create a bucket (table)
2018-08-16 10:32:28 -04:00
nfarah86
d396eb6be9
Bug/download (#216)
* fixed downloading empty file

* fixed formatting

* fixed error handling

* fixed cache_test.go to accomodate for error

* fixed error verbose messaging for wrong downlaods

* clean up
2018-08-14 08:22:29 -07:00
Bryan White
5d20cf8829
Node Identity (#193)
* peertls: don't log errors for double close

understood that this part of the code is undergoing heavy change
right now, but just want to make sure this fix gets incorporated
somewhere

* git cleanup: node-id stuff

* cleanup

* rename identity_util.go

* wip `CertificateAuthority` refactor

* refactoring

* gitignore update

* wip

* Merge remote-tracking branch 'storj/doubleclose' into node-id3

* storj/doubleclose:
  peertls: don't log errors for double close

* add peertls tests & gomports

* wip:

+ refactor
+ style changes
+ cleanup
+ [wip] add version to CA and identity configs
+ [wip] heavy client setup

* refactor

* wip:

+ refactor
+ style changes
+ add `CAConfig.Load`
+ add `CAConfig.Save`

* wip:

+ add `LoadOrCreate` and `Create` to CA and Identity configs
+ add overwrite to CA and identity configs
+ heavy client setup
+ refactor
+ style changes
+ cleanup

* wip

* fixing things

* fixing things

* wip hc setup

* hc setup:

+ refactor
+ bugfixing

* improvements based on reveiw feedback

* goimports

* improvements:

+ responding to review feedback
+ refactor

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* feedback-based improvements

* cleanup

* refactoring CA and Identity structs

* Merge branch 'master' into node-id3

* move version field to setup config structs for CA and identity

* fix typo

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* responding to revieiw feedback

* Merge branch 'master' into node-id3

* fix gateway setup finally

* go imports

* fix `FullCertificateAuthority.GenerateIdentity`

* cleanup overlay tests

* bugfixing

* update ca/identity setup

* go imports

* fix peertls test copy/paste fail

* responding to review feedback

* setup tweaking

* update farmer setup
2018-08-13 10:39:45 +02:00
Jennifer Li Johnson
5f8142cdcc
Integrating on disk routing table with kademlia (#166)
* adds comment

* runs deps

* adds print statements for debugging add node bkad

* more print statements

* removes bkad from routing and integrates on disk routing table

tests failing :(

wip

* removes testbootstrap

* kademlia_test not working

* adds kad tests back in

* Adds skips for tests broken due to wip kademlia
2018-08-09 15:20:39 -04:00
Natalie Villasana
9ad4949825
adds segment store tests (#198)
* starts adding segmentStore tests

* adds mocked interfaces for segmentStore tests

* adds tests for put, get, delete, and list

* regenerates pointerdb mock and updates calls to accommodate new changes
2018-08-07 11:52:11 -04:00
Dennis Coyle
7348bf1e09
Adjustments to how find storage nodes handles limit/amout requested (#192)
* adjusted based on feedback from @jtolds and @kaloyan-raev

* generated protobuf

* fix crash

* WIP

* cleanup

* v >>found

* pointer

* test

* copyrights + cleanup

* fix tests
2018-08-03 09:15:52 -04:00