Commit Graph

155 Commits

Author SHA1 Message Date
Dylan Lott
0d05cb26bf
Kademlia Inspector CLI (#657)
* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* initial commit of inspector gadget wireup

* change name of comman dline tool, setup grpc server

* Get inspector cli working with grpc client

* Wired up CountNodes command

* WIP getting buckets response working

* Added GetBucket command

* WIP working on get buckets command

* WIP working on bucket list

* Still WIP

* WIP getting bucket counts to work

* Some clean up of unnecessary changes

* List Buckets and Get Bucket are working

* Removing logs, getting ready for review

* Fix error return

* Trying to get tests passing

* Adds method on dht mock for tests

* Add dbx files back

* Fix package import error in dbx file

* Adds copyrights to pass linter

* tidy go mod

* Updates from code review

* Updates inspector to take flag arguments for address

* Format list-buckets output more prettier
2018-11-21 10:31:27 -07:00
Kaloyan Raev
8c56198090
Use designated error classes for common bucket and object errors (#691) 2018-11-21 13:17:28 +02:00
Michal Niewrzal
7e40caafcd
Write access for uplink mount command (#587)
* Write access for uplink mount command

* fix linter problems

* Fix closing wrong variable and debug logs

* goimports

* fix problem with missing getattr for large files

* better map initialization

* fixes after comments

* add support for making directory

* add support for removing directories

* optimize OpenDir and Rmdir methods

* use storj.JoinPaths

* fix linter issues

* remove useless error check
2018-11-21 11:19:54 +01:00
Bryan White
dabd87cb36
swap golang/protobuf out for gogo/protobuf (#692)
* swap golang/protobuf out for gogo/protobuf

* revert go.mod and go.sum changes

* linter fixes
2018-11-20 19:29:07 +01:00
Dennis Coyle
e7e0d1daaa
Coyle/disable mock (#627)
* disabled mock overlay from captplanet
2018-11-20 11:54:52 -05:00
Jennifer Li Johnson
93c5f385a8
Enable checker in captplanet and staging (#643)
* enable checker

* add option to use mock overlay in checker

* adds logs to checker

* appease linter
2018-11-20 10:54:22 -05:00
Egon Elbre
6fe16e48ba
Simplify and fix tests (#660) 2018-11-16 18:31:14 +02:00
aligeti
7958994ae2
Diagnostic tool to inspect repair queue (#656)
* initial repair queue diag tool development

* fixes linter warnings

* code review updates
2018-11-16 08:31:33 -05:00
Alexander Leitner
5e9377524f
Refactor basystem (#641)
* Signature verification

* Clean up agreement sender to have less errors

* overlay address in captnplanet

* Refactor bandwidth.proto to not use streams

* Make sure the send worked

* Handle connection to satellite

* Save renter public key inside of renter bandwidth allocations

* Default diag to sqlite. Make configurable

* Separate bw server and dbm; regenerate dbx files

* Make sure test uses protobufs

* Demonstrate creating bandwidth allocations
2018-11-15 14:06:09 -05:00
Egon Elbre
c4b90f84dd
Use better defaults and naming for postgres database (#659) 2018-11-15 20:36:57 +02:00
Michal Niewrzal
b1cab0fc5a
Fix for uplink cat writing additional bytes (#655)
* Fix for uplink cat writing additional bytes

* rename progress parameter
2018-11-15 16:21:58 +01:00
Egon Elbre
42ff835e6e
Fail linter when copyright is missing. (#664) 2018-11-15 14:47:52 +02:00
Yaroslav Vorobiov
c442205b3a
account api: infrastructure, login, register, getUser (#611) 2018-11-14 12:50:15 +02:00
Kaloyan Raev
5cdb03515f
Configure path encryption in metainfo API (#644) 2018-11-14 11:26:18 +02:00
Alexander Leitner
50d657af11
Cleanup agreement sender (#630)
* Signature verification

* code review comments & lint warnings

* Clean up agreement sender to have less errors

* overlay address in captnplanet
2018-11-13 10:37:49 -05:00
aligeti
6576d31b29 Signature verification (#604)
* Signature verification

* code review comments & lint warnings
2018-11-12 16:59:30 -05:00
Dylan Lott
c77ca2b161
[WIP] MVP of Kad inspection tools (#596)
* WIP starting to wire up the kademlia CLI tool

* WIP wiring up kad cli tools

* WIP starting to wire up the kademlia CLI tool

* WIP wiring up kad cli tools

* Got everything wired up

* WIP starting to wire up the kademlia CLI tool

* WIP wiring up kad cli tools

* merge in upstream

* WIP wiring up kad cli tools

* Got everything wired up

* WIP trying to get CLI to connect

* Inspector connects to overlay now

* Some refactoring

* Linter fixes

* Linter fixes

* Switch to pkg/process instead of using rootCmd.Execute
2018-11-09 11:40:15 -07:00
aligeti
bc0b411e5c
Diagnostic tool to inspect storage node bandwidth allocation databases (#576)
* Diagnostic tool initial framework

* Changes per review comments

* changes per code review
2018-11-08 08:37:55 -05:00
aligeti
285b70048f
Diagnostic tool for Satellite (#586)
* Satellite diagnostic tool initial development

* changes for code review and lint warning fixes
2018-11-08 08:20:23 -05:00
Natalie Villasana
3b572264ca
adds audit job (#521) 2018-11-07 01:16:43 +00:00
Bryan White
ee62e2a9d8
Use transport client and cleanup all the clients (#574)
* wip

* linter fixes

* linter fixes

* test fixes

* linter fixes

* fix merge + restructure piecestore packages

* review feedback

* linter fixes

* linter fixes

* remove unnecessary aliases to piecestore

* more merge fixing
2018-11-06 18:49:17 +01:00
Cameron
de46a999bc
Integrate SegmentStore Repair method with repair service (#582)
* add storeConfig struct and getSegmentStore helper for creating a segment store

* implement segment store in repairer, remove unnecessary repairer Repair method

* change repair method parameter from int to int32 to match type being passed in

* implement repairer service in captplanet

* rework Config, set Config defaults in captplanet/setup
2018-11-06 09:52:11 -05:00
littleskunk
de7358fb30
s3-benchmark test with uplink (#577)
* s3-benchmark tests with uplink

* rename endpoint to s3-gateway
2018-11-06 14:19:43 +01:00
Dennis Coyle
1d3367bb09
bypass data checker and repairer in captplanet (#563) 2018-10-31 20:46:47 -04:00
Michal Niewrzal
a62b413de2
Uplink mount better prefix handling (#541)
* Uplink mount better prefix handling

* fix formatting
2018-10-31 21:38:32 +01:00
Dennis Coyle
a3becb8a7b
Add repairer & checker to Satelite (#561)
* Added repairer & checker to Satellite

* fixed repairer and checker configs
2018-10-31 12:22:35 -04:00
JT Olio
a42973f089
captplanet: fix output line name (#549)
Change-Id: I37fbb76586ee3c0a40d24aad72aff2fcfc039a25
2018-10-31 10:00:58 -06:00
Michal Niewrzal
79e1ed5d1f
Fix uplink mount for latest api (#539)
Fix uplink mount for changes in store.List
2018-10-26 16:22:20 +02:00
JT Olio
fdfb5f1da9
uplink: fix mount and cli/gateway logical merge conflict (#537)
Change-Id: Ib1661e1c9d3f06783854b2293579ab715d0f1c20
2018-10-25 16:00:15 -06:00
JT Olio
cd51787827 split gateway from uplink
Change-Id: I7504a630cb85539750e32780c1c2347742a9c9c4
Reviewed-on: https://review.gerrithub.io/430644
Reviewed-by: Kaloyan Raev <kaloyan@storj.io>
Tested-by: Kaloyan Raev <kaloyan@storj.io>
2018-10-25 15:01:44 -06:00
Kaloyan Raev
99640225fd
Refactor Path type (#522)
The old paths.Path type is now replaced with the new storj.Path.

storj.Path is simply an alias to the built-in string type. As such it can be used just as any string, which simplifies a lot working with paths. No more conversions paths.New and path.String().

As an alias storj.Path does not define any methods. However, any functions applying to strings (like those from the strings package) gracefully apply to storj.Path too. In addition we have a few more functions defined:

    storj.SplitPath
    storj.JoinPaths
    encryption.EncryptPath
    encryption.DecryptPath
    encryption.DerivePathKey
    encryption.DeriveContentKey

All code in master is migrated to the new storj.Path type.

The Path example is also updated and is good for reference: /pkg/encryption/examples_test.go

This PR also resolve a nonce misuse issue in path encryption: https://storjlabs.atlassian.net/browse/V3-545
2018-10-25 23:28:16 +03:00
Michal Niewrzal
b3c1ea1852
Add 'mount' command for uplink cli (read-only access) (#345)
* Add 'mount' command for uplink cli

* limit build to unix

* small changes

* Removed in-memory files map, fix in read function

* unmount on interrupt

* add delete support

* some Read improvements

* update to latest api version

* removed Println and better error handling

* fix linter errors

* fix linter errors

* disable async read + review comments

* minor fixes

* remove mutex

* code cleanup

* add prefix support

* simplifying if statement
2018-10-25 12:44:34 +02:00
Michal Niewrzal
b2caac4323
Option for generating sample TLS certs for Minio GW (#525)
* Option for generating sample TLS certs for Minio GW

* os.Link errors handled

* rename option and change cert template
2018-10-24 22:17:28 +02:00
Maximillian von Briesen
94128d63a1
fix captplanet and satellite overwrite flags (#516)
* fix captplanet and satellite overwrite flags
2018-10-19 11:06:02 -04:00
Dylan Lott
6b88f0a36b
Moves CollectErrors into pkg/utils (#493)
Merging but creating a PR for tests shortly
2018-10-17 14:23:44 -06:00
Dylan Lott
19d8b7a2df
return multiple errors when captplanet fails (#478)
* implements CombineErrors on runCfg error chan

* implement error channel timeout and CombineErrors

* implements CombineErrors with time.After timeout

* changes d to duration in args

* changes comment to better reflect what collectErrors does

* appeases the linter gods

* Gets rid of timeout bug in collectErrors
2018-10-17 09:56:39 -06:00
Matt Robinson
bcd18a44bc
give our appearence on windows some love (#429) 2018-10-16 14:48:17 -04:00
Michal Niewrzal
6ffded3186
Uplink cp upload improvements (#480)
* Uplick cp upload improvements

* fix formatting
2018-10-16 14:10:41 +02:00
Egon Elbre
2dbada73ab
Separate storj.io packages in imports (#481)
Format with goimports -w -local storj.io
Add flag to golang lint
Remove commented out imports as they mess with goimports
2018-10-16 14:43:44 +03:00
Stefan Benten
586ed1b58d
Improve path handling + filepath handling framework (#423)
* Initial Layout

* Commit to test File Handling OS independed

* Hide struct properties to prevent manual interaction

* Fix Linting Errors

* 1st Working Windows Version

* Add missing Error Handling

* Fix Linting Errors

* Remove dependencies

* Further Improvements

* Remove commented code

* Improve comments and error messages

* No pointers to FPath

* Improve comment

* Do not filepath.ToSlash URL path

* Extract helper functions for parsing local path and Storj path

* Minor Improvements based on PR Comments

* Fix Linting Error and make Regex private

* Improve Layout

* Rework FPath and add tests

* Add more tests cases for windows

* Use for-loop instead of goto

* Use FPath in all uplink commands

* Add guard checks

* Add Test Cases and add comments
2018-10-15 16:10:20 +02:00
Alexander Leitner
3e1b16ea99
Start redis (#470)
* Start miniredis, repairer, and checker with captplanet
2018-10-12 14:04:16 -04:00
Egon Elbre
db42772b4d
Enable goimports and fix formatting. (#459) 2018-10-11 23:25:54 +03:00
Michal Niewrzal
ad327bedb1
Use context to propagate API Key (#383)
* Satellite signs proofs

* wip

* remove direct apikey usage from pdbclient

* adjusting unit tests

* fix linter errors

* unit tests

* linter errors

* remove usless interface

* remove unused code

* improve unit tests

* signature generation

* code review changes

* code review comments

* back to satellite-id signature generation

* remove go-grpc-middlewar dependency

* small step back

* linter fixes

* fix tests

* packages reorganization

* Move TestAPIKeyInjector to grpcauth package
2018-10-09 16:39:14 +02:00
Michal Niewrzal
a4b5978259
Don't override file if remote file doesn't exist (#445) 2018-10-09 14:36:36 +02:00
Maximillian von Briesen
81b106fb72 Add StatDB to Satellite (#396) 2018-10-08 18:15:54 -04:00
littleskunk
ddbe039908
captplanet: api and encryption key setup (#441)
* captplanet: encryption api and key setup
2018-10-08 22:04:32 +02:00
Dennis Coyle
dee2c137c8
Remove BKAD dependency from pkg/kademlia (#294)
* slowly but surely

* hardcode ID for tests so we can get predictable results

* skipping bad test

* removing tests of bkad

* wip

* new algorithm for worker

* clean up

* remove skipped test

* changes

* uncomment

* fixed conflicts

* maybe done ?

* cleanup

* boot bkad

* wip

* cleanup

* undo change

* fixes

* wip

* wip

* moving nodeID around

* wip

* wip

* fixes

* fixes after merge

* added TODO

* fixed tests post identity

* linter fixes

* wip

* PR review comments

* wip

* fixing tests

* fix tests

* force db directory

* bad test

* fixes race condition

* small cleanups

* adding db folder

* testing

* wip

* cleanup

* cleanup

* linters

* export Restrict

* add timeout

* testing

* linters

* forgot one

* moar fixes from master merge

* PR comments

* moar PR comments

* removed stun flag

* remove duplicate declaration

* remove old tests

* remove timeout

* fix tests

* missed one

* changed StringToID >> IDFromString

* PR comments

* stupid linter

* moevd overlay mock

* fixed merge conflicts

* fixes

* linter
2018-10-08 11:09:37 -04:00
littleskunk
cce3ef2c37
benchmark list files and folders (#431) 2018-10-06 14:26:35 +02:00
Alexander Leitner
dc8bea2cd1
Repairer points to redis server (#427)
* Let's do it right this time

* Oh travis...

* Handle redis URL

* Travis... why u gotta be like this?

* Handle when address does not use redis scheme

* Start repairer

* Match provider.Responsibility interface

* Simplify if statement

* Config doesn't need to be a pointer

* Initialize doesn't need to be exported

* Don't run checker or repairer on startup

* Fix travis complaints
2018-10-05 11:58:07 -04:00
Dylan Lott
f7afafe4e1
Repair cron (#420)
* Creates cron-job for checker, adds it to captplanet and satellite

* removes datarepair from satellite & captplanet run

* Delete config.go

* removes unused datarepair imports

* adds comments to fix linter
2018-10-04 15:40:34 -06:00