paul cannon
331f0cc845
don't put nil nodes in GetResponse ( #1218 )
2019-02-05 23:12:27 +02:00
Yaroslav Vorobiov
2ff0d9d435
Add bucket to project relationship on satellite ( #1143 )
...
* integrate console api keys with buckets in pointerdb
* fix test
* fix tests kvmetainfo
* linter fix
* disable account activation
* fix test
* review fixes
* fix comments
* little refactoring
* remove debug println
* fix typo
* disable activation in a propper way
* fix test
* fix imports
* fix uplink count in testplanet
* move key creation to planet.newUplink
2019-02-05 19:22:17 +02:00
Kaloyan Raev
a90aa18582
Fix panic in PointerDB.Get ( #1209 )
...
* Fix panic in PointerDB.Get
* Allow nil nodes, check if v is nil instead
* More checks like this
* Remove unnecessary check
* More checks
2019-02-01 10:55:47 -07:00
Egon Elbre
d5346982c2
Delete provider package ( #1177 )
2019-01-30 22:47:21 +02:00
Bill Thorp
60946c2024
make bandwidth agreements sensible: without []byte's ( #1152 )
...
removed []byte's from bandwidth agreement protocol buffers
2019-01-28 14:45:25 -05:00
Bill Thorp
8ba4b61e59
Make Bandwidth Agreements Secure / Trustable ( #1117 )
...
* Added cert chains and nodeid verification to bandwidth agreement
2019-01-25 13:05:21 -05:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 ( #1133 )
2019-01-24 15:15:10 -05:00
Egon Elbre
5de7f8af7f
Satellite Peer ( #1119 )
2019-01-23 21:58:44 +02:00
Michal Niewrzal
eb2d858ffc
Use pointerdb.Service instead of pointerdb.Server ( #1099 )
2019-01-19 13:58:53 -05:00
Michal Niewrzal
cc1bdef8b7
Move pointerdb logic to a service ( #1037 )
...
* test
* Move pointerdb logic to service
* tset
* reorder constructor params
* restore field
2019-01-18 10:10:21 -05:00
Egon Elbre
78dc02b758
Satellite Peer ( #1034 )
...
* add satellite peer
* Add overlay
* reorganize kademlia
* add RunRefresh
* add refresh to storagenode.Peer
* add discovery
* add agreements and metainfo
* rename
* add datarepair checker
* add repair
* add todo notes for audit
* add testing interface
* add into testplanet
* fixes
* fix compilation errors
* fix compilation errors
* make testplanet run
* remove audit refrences
* ensure that audit tests run
* dev
* checker tests compilable
* fix discovery
* fix compilation
* fix
* fix
* dev
* fix
* disable auth
* fixes
* revert go.mod/sum
* fix linter errors
* fix
* fix copyright
* Add address param for SN dashboard (#1076 )
* Rename storj-sdk to storj-sim (#1078 )
* Storagenode logs and config improvements (#1075 )
* Add more info to SN logs
* remove config-dir from user config
* add output where config was stored
* add message for successful connection
* fix linter
* remove storage.path from user config
* resolve config path
* move success message to info
* log improvements
* Remove captplanet (#1070 )
* pkg/server: include production cert (#1082 )
Change-Id: Ie8e6fe78550be83c3bd797db7a1e58d37c684792
* Generate Payments Report (#1079 )
* memory.Size: autoformat sizes based on value entropy (#1081 )
* Jj/bytes (#1085 )
* run tally and rollup
* sets dev default tally and rollup intervals
* nonessential storj-sim edits (#1086 )
* Closing context doesn't stop storage node (#1084 )
* Print when cancelled
* Close properly
* Don't log nil
* Don't print error when closing dashboard
* Fix panic in inspector if ping fails (#1088 )
* Consolidate identity management to identity cli commands (#1083 )
* Consolidate identity management:
Move identity cretaion/signing out of storagenode setup command.
* fixes
* linters
* Consolidate identity management:
Move identity cretaion/signing out of storagenode setup command.
* fixes
* sava backups before saving signed certs
* add "-prebuilt-test-cmds" test flag
* linters
* prepare cli tests for travis
* linter fixes
* more fixes
* linter gods
* sp/sdk/sim
* remove ca.difficulty
* remove unused difficulty
* return setup to its rightful place
* wip travis
* Revert "wip travis"
This reverts commit 56834849dcf066d3cc0a4f139033fc3f6d7188ca.
* typo in travis.yaml
* remove tests
* remove more
* make it only create one identity at a time for consistency
* add config-dir for consitency
* add identity creation to storj-sim
* add flags
* simplify
* fix nolint and compile
* prevent overwrite and pass difficulty, concurrency, and parent creds
* goimports
2019-01-18 08:54:08 -05:00
Alexander Leitner
bfde515391
Clean up Storage node setup ( #1013 )
...
* Edit config on Setup
* Default to 1TiB storage space and 500GiB bandwidth
* Use human readable formats
* Use memory
* units of 1024 are measured with KiB/MiB etc
* pkg/cfgstruct: allow values to be configured with human readable sizes
Change-Id: Ic4e9ae461516d1d26fb81f6e44c5ac5cfccf777f
* Modify tests
* Removed comments
* More merge conflict stuff resolved
* Fix lint
* test fixin
Change-Id: I3a008206bf03a4446da19f642a2f9c1f9acaae36
* Remove commented code but secretly leave it in the histroy forever
* Move flag definition to struct
2019-01-14 16:19:15 -05:00
Cameron
ef50bbf8b6
satellites reject expired BWAs ( #1015 )
...
* add 45 day expiration to PBAs
* add expiration field to relevant areas, DeleteExpired placeholder
* reject expired BWAs
* test for expired BWAs
* add BwExpiration config value
2019-01-10 13:30:55 -05:00
Michal Niewrzal
cfbaeaf8f2
Some pointerdb methods doesn't validate api key ( #1019 )
2019-01-10 13:07:08 +01:00
aligeti
457f909a74
Satellite accepts manipulated paychecks V3-1023 ( #972 )
...
* fix - Satellite crashing on receiving a manipulated bandwidthagreement
* provider.PeerIdentityFromContext called twice. Remove one
* add storage node ID to serial number
* remove serialNum query and transaction
* add uuid to GeneratePayerBandwidthAllocation for testing
* enable expected failure on duplicate serialnum cases
* Revert "enable expected failure on duplicate serialnum cases"
This reverts commit 5948f43ed1741c280f0bb34a86c1c490365417bc.
* enable expected failure on duplicate serialnum cases
2019-01-09 10:02:03 -05:00
Egon Elbre
320a0708f4
Allow empty key and better error handling ( #986 )
2019-01-08 10:22:54 +02:00
aligeti
ce1db97ce6
fix manipulated paychecks V3-1023 ( #961 )
...
Satellite crashing on receiving a manipulated bandwidthagreement V3-1022
2019-01-04 11:26:26 -05:00
Jennifer Li Johnson
a2fa5c4c5a
Proper NodeType Handling ( #873 )
...
* adds enums to nodetype
* updating nodetype todos
* ran pb updates
* reorder nodetypes
* adding checks
* wip
* wip
* wip
* bug in test-captplanet
* wip
* add values to storagenode, satellite, captplanet binaries
* Cleanup
* more cleanup
* wip
* lint
* lint
* wip
* fixes bug
* regenerate protos
Change-Id: Id270212e8c7479e52641058042cf23b5317ab773
* limit node type changes to kademlia
Change-Id: I9c1a6cc4a79e05086627f0fdeb5028c62ce754f4
* dpanic
Change-Id: Id952a2ad13c807ebaea0ec0a875405e267d81c3e
* review comments
Change-Id: I7f9b77ef22779dd012fd490375b136014f51f834
2019-01-02 11:47:34 -07:00
Bryan White
249244536a
CSR Service (part 2): cert signing rpc ( #950 )
...
* CSR Service:
+ implement certificate sign rpc method
+ implement certificate signer client/server
+ refactor `AuthorizationDB#Create`
+ refactor `NewTestIdentity`
+ add `AuthorizationDB#Claim`
+ add `Token#Equal`
+ fix `Authorizations#Marshal` when marshaling identities and certificates
+ tweak `Authorization#String` format
+ cert debugging improvements (jsondiff)
+ receive context arg in `NewTestIdentity`
+ misc. fixes
2019-01-02 12:39:17 -05:00
Egon Elbre
c56307f5b9
datarepair: remove unneeded debug message ( #839 )
...
* datarepair: remove unneeded debug message
* Make overlay-cache and routing table logs more specific
2018-12-12 17:39:16 +02:00
Bill Thorp
ba5f71810e
added tally to capt run.go ( #751 )
...
* added tally to capt run.go, ParseURL -> SplitDBURL
2018-12-12 08:15:34 -05:00
Kaloyan Raev
9449010c15
Enable overlay by default ( #835 )
2018-12-11 18:52:45 +02:00
Kaloyan Raev
66476da1ae
metainfo: basic upload ( #726 )
2018-11-30 15:50:52 +02:00
Egon Elbre
9ac3517432
Reduce noise in logging ( #733 )
2018-11-29 22:59:26 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type ( #693 )
...
* preparing for use of `customtype` gogo extension with `NodeID` type
* review changes
* preparing for use of `customtype` gogo extension with `NodeID` type
* review changes
* wip
* tests passing
* wip fixing tests
* more wip test fixing
* remove NodeIDList from proto files
* linter fixes
* linter fixes
* linter/review fixes
* more freaking linter fixes
* omg just kill me - linterrrrrrrr
* travis linter, i will muder you and your family in your sleep
* goimports everything - burn in hell travis
* goimports update
* go mod tidy
2018-11-29 19:39:27 +01:00
Michal Niewrzal
8dd669b37f
Bandwidth allocations need method designation (GET/PUT) ( #708 )
...
* Bandwidth allocations need method designation (GET/PUT)
* add method comment
* goimports
2018-11-26 19:21:44 +01:00
Bryan White
dabd87cb36
swap golang/protobuf out for gogo/protobuf ( #692 )
...
* swap golang/protobuf out for gogo/protobuf
* revert go.mod and go.sum changes
* linter fixes
2018-11-20 19:29:07 +01:00
Bryan White
54ccb460a9
rename size fields in protobuf messages for gogo compatibility ( #690 )
...
* rename `size` fields in protobuf messages for gogo compatibility
* linter fixes
2018-11-20 18:09:35 +01:00
Alexander Leitner
320c93368a
Add creation time to Payer Bandwidth Allocation ( #581 )
...
* Add creation time to Payer Bandwidth Allocation
* goimports
2018-11-08 11:04:52 -05:00
Bryan White
ee62e2a9d8
Use transport client and cleanup all the clients ( #574 )
...
* wip
* linter fixes
* linter fixes
* test fixes
* linter fixes
* fix merge + restructure piecestore packages
* review feedback
* linter fixes
* linter fixes
* remove unnecessary aliases to piecestore
* more merge fixing
2018-11-06 18:49:17 +01:00
Michal Niewrzal
deb015970d
Use Nodes array from pdb.Get ( #578 )
...
* Use Nodes array from pdb.Get
* fix problems with captplanet tests
* better comments
2018-11-06 18:03:11 +01:00
Michal Niewrzal
1129ead6ef
Fix 'message not signed by the satellite' problem ( #560 )
...
* Fix 'message not signed by the satellite' problem
* fixed build
* fix errors
2018-11-05 16:12:19 +01:00
Egon Elbre
c765dd98de
internal/testplanet ( #528 )
2018-11-03 14:17:14 +02:00
Bryan White
8b9711cb5e
better waitlist-gating ( #557 )
...
* better waitlist-gating
(cherry picked from commit 490fe02b7c3558da18678dfb651c92ec9c4a75b5)
* fix broken test
* linter fixes
* linter fixes
* make extension verification optional
* add certifcate gating script for captplanet
* fixing tests
* linter fixes
* linter fixes?
* moar linter fixes
* Revert "moar linter fixes"
This reverts commit 8139ccbd73cbbead987b7667567844f50f7df2c8.
* just kill me
* refactor
* refactor tests
* liniter...
* cleanup
2018-11-01 16:48:43 +01:00
Dennis Coyle
a3becb8a7b
Add repairer & checker to Satelite ( #561 )
...
* Added repairer & checker to Satellite
* fixed repairer and checker configs
2018-10-31 12:22:35 -04:00
Jennifer Li Johnson
1fb96689b8
creates run loop for data repair checker ( #490 )
...
* creates run loop for data repair checker
* moves actual checking and repairing under ticker case
* fixes mismatched queueaddrs
2018-10-30 15:16:40 -04:00
Alexander Leitner
7e4149138a
Piecestore proto change ( #556 )
...
* add action and storage node id to piecestore.proto. renamed payer and renter to satellite_id and uplink_id
2018-10-30 15:03:41 -04:00
Michal Niewrzal
0d03f2fbb5
Send bandwidth alloc from satellite to storage node ( #538 )
...
* Send bandwidth alloc from satellite to storage node
* Remove unecessary nil checks
* set Renter field
* fix tests
* goimports
* Update README.md
* Update README.md
* Update README.md
* Update README.md (#550 )
* Skip flaky TestPing (#552 )
2018-10-30 17:24:46 +01:00
Kaloyan Raev
99640225fd
Refactor Path type ( #522 )
...
The old paths.Path type is now replaced with the new storj.Path.
storj.Path is simply an alias to the built-in string type. As such it can be used just as any string, which simplifies a lot working with paths. No more conversions paths.New and path.String().
As an alias storj.Path does not define any methods. However, any functions applying to strings (like those from the strings package) gracefully apply to storj.Path too. In addition we have a few more functions defined:
storj.SplitPath
storj.JoinPaths
encryption.EncryptPath
encryption.DecryptPath
encryption.DerivePathKey
encryption.DeriveContentKey
All code in master is migrated to the new storj.Path type.
The Path example is also updated and is good for reference: /pkg/encryption/examples_test.go
This PR also resolve a nonce misuse issue in path encryption: https://storjlabs.atlassian.net/browse/V3-545
2018-10-25 23:28:16 +03:00
paul cannon
e2c0dd437a
offer PostgreSQL storage for pointerdb ( #440 )
...
..although it ought to work for other storage.KeyValueStore needs as
well. it's just optimized to work pretty well for a largish hierarchy of
paths.
This includes the addition of "long benchmarks" for KeyValueStore
testing. These will only be run when -test-bench-long is added to the
test flags. In these benchmarks, a large corpus of paths matching a
natural ("real-life") hierarchy is read from paths.data.gz (which you
can get from https://github.com/storj/path-test-corpus ) and imported
into a particular KeyValueStore. Recursive and non-recursive queries are
run on it to detect performance problems that arise only at scale.
This also includes alternate implementation of the postgreskv client,
which works in a less-bizarre way for non-recursive queries, but suffers
from poor performance in tests such as the long benchmarks. Once this
alternate impl is committed to the tree, we can remove it again; I just
want it to be available for future reference.
2018-10-25 12:11:28 -05:00
Michal Niewrzal
f9248c21d4
Satellite verification on storage node ( #469 )
...
* Satellite verification on storage node
* fix formatting
* fix formatting
* rename SignatureAuth to SignedMessage
* fixes after review
* fix linter errors
* improve errors handling
* remove SignedMessageProvider
* fix liter errors
* params changed to authorization, signed message in audit, minor fixes
* fix formatting
2018-10-17 13:40:11 +02:00
Egon Elbre
2dbada73ab
Separate storj.io packages in imports ( #481 )
...
Format with goimports -w -local storj.io
Add flag to golang lint
Remove commented out imports as they mess with goimports
2018-10-16 14:43:44 +03:00
Egon Elbre
e39f9e42b4
Kademlia Sequential Lookup ( #464 )
2018-10-12 11:52:32 +03:00
Egon Elbre
db42772b4d
Enable goimports and fix formatting. ( #459 )
2018-10-11 23:25:54 +03:00
Michal Niewrzal
351dc523d6
Satellite signature generation ( #453 )
...
* Satellite signature generation
* Add unit test
* remove unused var
* remove base64 encoding from signature generation
2018-10-11 16:35:55 +02:00
Natalie Villasana
b04ea4639f
moves all audit files to pkg/audit ( #449 )
2018-10-09 18:05:42 -04:00
Jennifer Li Johnson
6fb13896fb
Method to identify injured segments to repair ( #398 )
...
* creates checker
* tests offline nodes
* test id injured segs:
* Adds healthy pieces to injured segment struct
* changes inequality
* creates common files
* adds checker benchmarking
* creates more common files
* Replaces pointedb direct db with api call to a new iterate method on pointerdb
* move monkit
* removes identifyrequest proto
* remove healthypieces
* adds benchmarking
creates common file for datarepair
* recreates proto file
* api key on ctx
2018-10-09 12:09:33 -04:00
Michal Niewrzal
ad327bedb1
Use context to propagate API Key ( #383 )
...
* Satellite signs proofs
* wip
* remove direct apikey usage from pdbclient
* adjusting unit tests
* fix linter errors
* unit tests
* linter errors
* remove usless interface
* remove unused code
* improve unit tests
* signature generation
* code review changes
* code review comments
* back to satellite-id signature generation
* remove go-grpc-middlewar dependency
* small step back
* linter fixes
* fix tests
* packages reorganization
* Move TestAPIKeyInjector to grpcauth package
2018-10-09 16:39:14 +02:00
Kaloyan Raev
3de34ab6cb
Rename MinInlineSegmentSize to MinRemoteSegmentSize ( #436 )
2018-10-08 23:09:28 +03:00
Dennis Coyle
dee2c137c8
Remove BKAD dependency from pkg/kademlia ( #294 )
...
* slowly but surely
* hardcode ID for tests so we can get predictable results
* skipping bad test
* removing tests of bkad
* wip
* new algorithm for worker
* clean up
* remove skipped test
* changes
* uncomment
* fixed conflicts
* maybe done ?
* cleanup
* boot bkad
* wip
* cleanup
* undo change
* fixes
* wip
* wip
* moving nodeID around
* wip
* wip
* fixes
* fixes after merge
* added TODO
* fixed tests post identity
* linter fixes
* wip
* PR review comments
* wip
* fixing tests
* fix tests
* force db directory
* bad test
* fixes race condition
* small cleanups
* adding db folder
* testing
* wip
* cleanup
* cleanup
* linters
* export Restrict
* add timeout
* testing
* linters
* forgot one
* moar fixes from master merge
* PR comments
* moar PR comments
* removed stun flag
* remove duplicate declaration
* remove old tests
* remove timeout
* fix tests
* missed one
* changed StringToID >> IDFromString
* PR comments
* stupid linter
* moevd overlay mock
* fixed merge conflicts
* fixes
* linter
2018-10-08 11:09:37 -04:00
