Commit Graph

431 Commits

Author SHA1 Message Date
Cameron
f5c115596e filter out excluded nodes when repairing (#426)
* add filter field into OverlayOptions message

* chooseFiltered method, add excluded parameter in populate method

* change excluded type to []dht.NodeID in ChooseFiltered, change comment

* change name filter to excluded_nodes in proto

* implement helper function contains

* delete ChooseFiltered and add its functionality into Choose method to keep original author's history, add excluded argument into Choose calls

* regenerate mock_client.go

* regenerate protobuf

* update test case to use new IDFromString function

* remove old kademlia test code
2018-10-15 13:42:36 -04:00
Egon Elbre
2042bab67e Delete dead code in overlay (#476)
* Delete dead code

* Delete some more dead code.

* Format
2018-10-15 13:14:29 -04:00
Maximillian von Briesen
3551b34ea7
Path Encryption (#448)
* begin adding path encryption

* do not encrypt/decrypt first element of path (bucket)

* add path encryption for delete and list

* use encrypted paths in streamstore.Meta

* fix listing with encrypted paths

* move encrypt/decryptAfterBucket to streamstore

* fix listing with no prefix

* remove duplicate logic for listing with no prefix
2018-10-15 11:39:09 -04:00
Matt Robinson
52951963f4 add wthorp to CLA (#477) 2018-10-15 09:22:00 -06:00
Stefan Benten
586ed1b58d
Improve path handling + filepath handling framework (#423)
* Initial Layout

* Commit to test File Handling OS independed

* Hide struct properties to prevent manual interaction

* Fix Linting Errors

* 1st Working Windows Version

* Add missing Error Handling

* Fix Linting Errors

* Remove dependencies

* Further Improvements

* Remove commented code

* Improve comments and error messages

* No pointers to FPath

* Improve comment

* Do not filepath.ToSlash URL path

* Extract helper functions for parsing local path and Storj path

* Minor Improvements based on PR Comments

* Fix Linting Error and make Regex private

* Improve Layout

* Rework FPath and add tests

* Add more tests cases for windows

* Use for-loop instead of goto

* Use FPath in all uplink commands

* Add guard checks

* Add Test Cases and add comments
2018-10-15 16:10:20 +02:00
Egon Elbre
9ba5b1156a
Run go mod tidy (#473) 2018-10-15 15:26:34 +03:00
Egon Elbre
caecb2801d
Cleanups related to transport (#475)
* Remove DialUnauthenticated method
* Use more specific node transport enum
* Formatting
* Rename Provider.g -> Provider.grpc
* Fix naming in dialer
2018-10-15 15:04:21 +03:00
Dylan Lott
7849d7fa63
Remove newID function (#468)
* Removes node.NewID usage from kademlia tests

* Updates more tests

* remove comments

* removes NewID method from overlay tests

* adds error checking to NewFullIdentity

* removes /x/tools from go.mod

* Removes commented out code
2018-10-12 15:10:50 -06:00
Alexander Leitner
3e1b16ea99
Start redis (#470)
* Start miniredis, repairer, and checker with captplanet
2018-10-12 14:04:16 -04:00
Jennifer Li Johnson
0e7f6358fb
creates configs for data repair package (#463)
* creates configs
2018-10-12 13:49:49 -04:00
littleskunk
f900167974 reproduce captplanet kademlia problems (#465) 2018-10-12 13:58:32 +03:00
Egon Elbre
e39f9e42b4
Kademlia Sequential Lookup (#464) 2018-10-12 11:52:32 +03:00
Dylan Lott
f87d295e5a
Kad identity (#458)
* Removes node.NewID usage from kademlia tests

* Updates more tests

* remove comments

* adds error checking to NewFullIdentity
2018-10-11 15:41:58 -06:00
Egon Elbre
db42772b4d
Enable goimports and fix formatting. (#459) 2018-10-11 23:25:54 +03:00
Jennifer Li Johnson
ea6fc3c532
use utils.combinederrors (#461) 2018-10-11 14:42:32 -04:00
Egon Elbre
ee5264f5fd Fix TestBootstrap and TestLookup (#460)
* Fix TestBootstrap and TestLookup

* Fix typos
2018-10-11 13:01:16 -04:00
Michal Niewrzal
351dc523d6
Satellite signature generation (#453)
* Satellite signature generation

* Add unit test

* remove unused var

* remove base64 encoding from signature generation
2018-10-11 16:35:55 +02:00
Natalie Villasana
404fc0329f
remove todo from audit pkg (#457) 2018-10-10 15:07:14 -04:00
Natalie Villasana
14b67af327
reorganizes audit package into Service, Cursor, and Verifier (#450) 2018-10-10 14:25:46 -04:00
Dennis Coyle
0d28101085
addnode fail to add should log not error (#446) 2018-10-10 11:05:05 -04:00
aligeti
4a51db2344
Tracking bandwidth v3-446 (#430)
* Added a new table 'mib'  with 'data', 'size' and 'method' columns

* added AddMIB() function and test case TestMIBHappyPath()

* added function and a test case  to add entries into bandwidth usage table

* added functionality to create an entry, update the entry and readback the entry based on a given date into/from bandwidth tbl

* added initial SumBandwidthSizes()

* added the functionality to retrieve the total bw usage based on start and end date

* Added the unit test case for AddBwUsageTbl

* changed the arguments to take time format as arg than Unix format

* changed the arguments to take time format as arg than Unix format

* changes per code review comments

* adding back go.sum

* changes per code review comments

* changes per code review comments

* changes per code review comments
2018-10-10 10:04:42 -04:00
Egon Elbre
b546ed9510
Remove unneeded constants (#452) 2018-10-10 11:33:17 +03:00
Egon Elbre
047ebdc327
metainfo database: add interfaces and types (#434) 2018-10-10 11:33:06 +03:00
Natalie Villasana
b04ea4639f moves all audit files to pkg/audit (#449) 2018-10-09 18:05:42 -04:00
Natalie Villasana
d36eb9c742
audits stripes (#380) 2018-10-09 17:10:37 -04:00
Dylan Lott
a69bc65a30
Kad flags (#443)
* Allows kademlia params to be configured with flags

* Removes commented out code

* updates kad tests to use kadconfig struct
2018-10-09 10:47:05 -06:00
Jennifer Li Johnson
6fb13896fb
Method to identify injured segments to repair (#398)
* creates checker

* tests offline nodes

* test id injured segs:

* Adds healthy pieces to injured segment struct

* changes inequality

* creates common files

* adds checker benchmarking

* creates more common files

* Replaces pointedb direct db with api call to a new iterate method on pointerdb

* move monkit

* removes identifyrequest proto

* remove healthypieces

* adds benchmarking

creates common file for datarepair

* recreates proto file

* api key on ctx
2018-10-09 12:09:33 -04:00
Michal Niewrzal
ad327bedb1
Use context to propagate API Key (#383)
* Satellite signs proofs

* wip

* remove direct apikey usage from pdbclient

* adjusting unit tests

* fix linter errors

* unit tests

* linter errors

* remove usless interface

* remove unused code

* improve unit tests

* signature generation

* code review changes

* code review comments

* back to satellite-id signature generation

* remove go-grpc-middlewar dependency

* small step back

* linter fixes

* fix tests

* packages reorganization

* Move TestAPIKeyInjector to grpcauth package
2018-10-09 16:39:14 +02:00
Michal Niewrzal
a4b5978259
Don't override file if remote file doesn't exist (#445) 2018-10-09 14:36:36 +02:00
Stefan Benten
7ec93aec03
Include GCC Hint for Windows (#442)
GCC is required for building our repo.
On Windows GCC is not available by default and needs to be added in order to build.
2018-10-09 12:32:10 +02:00
Kaloyan Raev
70caa86d89
Reduce garbage errors in captplanet when streaming (#435) 2018-10-09 10:29:00 +03:00
Maximillian von Briesen
81b106fb72 Add StatDB to Satellite (#396) 2018-10-08 18:15:54 -04:00
Dennis Coyle
783f055417 create db directory if it does not exist (#439)
* create db directory if it does not exist

* linter fix

* pass db path in from config

* change mkdir to mkdirAll

* windows love

* PR comments

* changing the path

* change the config default to $CONFDIR/kademlia
2018-10-08 16:37:52 -04:00
Kaloyan Raev
3de34ab6cb
Rename MinInlineSegmentSize to MinRemoteSegmentSize (#436) 2018-10-08 23:09:28 +03:00
littleskunk
ddbe039908
captplanet: api and encryption key setup (#441)
* captplanet: encryption api and key setup
2018-10-08 22:04:32 +02:00
Jennifer Li Johnson
cebdf57ad8
creates queue benchmark tests (#433)
* creates queue benchmark tests

* adds timer reset to benchmark

* update benchmark

* adds benchmarks for both redis and teststore
2018-10-08 15:37:27 -04:00
Matt Robinson
a1f93285e2 figure out which deployments need to be patched during a deployment instead of guessing (#438) 2018-10-08 12:52:27 -04:00
Dennis Coyle
dee2c137c8
Remove BKAD dependency from pkg/kademlia (#294)
* slowly but surely

* hardcode ID for tests so we can get predictable results

* skipping bad test

* removing tests of bkad

* wip

* new algorithm for worker

* clean up

* remove skipped test

* changes

* uncomment

* fixed conflicts

* maybe done ?

* cleanup

* boot bkad

* wip

* cleanup

* undo change

* fixes

* wip

* wip

* moving nodeID around

* wip

* wip

* fixes

* fixes after merge

* added TODO

* fixed tests post identity

* linter fixes

* wip

* PR review comments

* wip

* fixing tests

* fix tests

* force db directory

* bad test

* fixes race condition

* small cleanups

* adding db folder

* testing

* wip

* cleanup

* cleanup

* linters

* export Restrict

* add timeout

* testing

* linters

* forgot one

* moar fixes from master merge

* PR comments

* moar PR comments

* removed stun flag

* remove duplicate declaration

* remove old tests

* remove timeout

* fix tests

* missed one

* changed StringToID >> IDFromString

* PR comments

* stupid linter

* moevd overlay mock

* fixed merge conflicts

* fixes

* linter
2018-10-08 11:09:37 -04:00
aligeti
0b4a6188e3
multiple uploads of same file. handled the logic in stream store by deleting first before starting uploading (#412) 2018-10-08 10:19:54 -04:00
Egon Elbre
5b655d4a89
Restore go.sum (#432) 2018-10-06 20:57:53 +03:00
littleskunk
cce3ef2c37
benchmark list files and folders (#431) 2018-10-06 14:26:35 +02:00
Alexander Leitner
dc8bea2cd1
Repairer points to redis server (#427)
* Let's do it right this time

* Oh travis...

* Handle redis URL

* Travis... why u gotta be like this?

* Handle when address does not use redis scheme

* Start repairer

* Match provider.Responsibility interface

* Simplify if statement

* Config doesn't need to be a pointer

* Initialize doesn't need to be exported

* Don't run checker or repairer on startup

* Fix travis complaints
2018-10-05 11:58:07 -04:00
nfarah86
7812f1bbc0
Stripe selection (#375)
* initial commit- wip, working on testing and library

* wip working on testing library functioon to get poointer

* working on nil  reference for testing

* tests wip

* wip-working on getting tests  to  work

* working on tests

* put test passes

* working on test- need to export

* created pdclient, and now working on testing function

* tests working for list- getting object back

* wip - got derived piece id

* fixed making grpc public

* fixed linter errors and minor added method for size

* need to work on  testing, added random  integer function

* got psc server working for  testing

* working on ranger test and ranger  method

* testing creds for new computer

* working on getting segment metadata

* get random stripe

* added caveat to random fn

* fixed data types

* modified library to have one public function that returns a random stripe

* removed extra comments

* added commons.go file for audit

* added last path to be remembered

* changed random function to cryto/rand/ & worked on tests passing

* working on testing to get analysis of randomness

* changed to track last item in pagination

* finished testing randomness, cleaned up code

* fixed rebase errors

* removed error, kept common file

* fixed travis errors

* attempt to fix overlay issue

* fixed travis error

* updated pointer parameters

* made smaller functions, renamed audit

* made changes per suggestions

* removed gosum

* fixed pr per suggestions

* removed comment
2018-10-05 06:34:15 -07:00
Dylan Lott
f7afafe4e1
Repair cron (#420)
* Creates cron-job for checker, adds it to captplanet and satellite

* removes datarepair from satellite & captplanet run

* Delete config.go

* removes unused datarepair imports

* adds comments to fix linter
2018-10-04 15:40:34 -06:00
Dylan Lott
ae38817fa3
Pointer lookup (#407)
* Loads cache from context for PointerDB access

* WIP adds overlay lookups to pointerdb requests

* Pointer lookup code is added for Get

* adds feature flag for pointerdb return

* refactors pointerdb code

* removes some unnecessary debug logs

* Fixes indent in config

* adds early return for non-remote pointers

* formats code, removes some comments

* Fixes tests broken by pointer proto changes

* adds error check and merges variable declaration

* removes commented out proto import

* adds error check to pdbclient
2018-10-04 15:00:19 -06:00
aligeti
cf329b1f37
Handle CTRL+C and clean up the partial segments and pieces (#381)
* merged the lasted master changes

* debug working of handling ctrl+c

* Handling of clean up of partially uploaded segments and pieces

* code cleanup per code comment

* updates based on code review comments
2018-10-04 09:52:12 -04:00
JT Olio
d8c0f18059 uplink: encryption key setup (#414) 2018-10-04 15:36:26 +03:00
Egon Elbre
d10554eb8c
remove udp forwarder (#404) 2018-10-04 15:08:24 +03:00
Alexander Leitner
f80ec62e9d
Reorganize repair (#419)
* Reorganize repair

* Don't run the repair code yet

* Pass max repair from config to repairer initialize

* Add repairer Interface

* fix comment
2018-10-03 14:35:56 -04:00
Kaloyan Raev
3c985a553e
Remove multipart threshold config for AWS CLI (#418) 2018-10-03 17:47:35 +03:00