* add last_ip field to dbx model node, generate dbx
* add last_ip to node proto, generate pb
* migrate
* resolve address in transport.DialNode, update lastIp in cache.UpdateAddress
* use net.SplitHostPort to isolate host address from port
* define DistinctIPs flag
* add test for GetIP
* select last_ip when querying for nodes
* if distinctIPs flag == true, query for nodes with distinct IPs
* some basic tests
* change last_ip to field 14 in proto
* remove comments
* check err
* change distinctIPs to distinctIP
* exclude IPs from newNodes in query for reputable nodes
* add index on last_ip
* only add to excludedIPs if flag is true
* test half new nodes returns distinct IPs
* fix alignment
* add test
* rework ip filter query, add retry logic, add switch for database driver
* add retry to SelectNewNodes
* change discovery intervals so IPs don't get overwritten
* remove TestGetIP
* edit updating node stats in test
* split exclude into nodeIDs and IPs
* separate non-distinct IP query into other function
* trigger checks
* remove else block
* cmd/uplink: add share command to restrict an api key
This commit is an early bit of work to just implement restricting
macaroon api keys from the command line. It does not convert
api keys to be macaroons in general.
It also does not apply the path restriction caveats appropriately
yet because it does not encrypt them.
* cmd/uplink: fix path encryption for shares
It should now properly encrypt the path prefixes when adding
caveats to a macaroon.
* fix up linting problems
* print summary of caveat and require iso8601
* make clone part more clear
* Initial Webserver Draft for Version Controlling
* Rename type to avoid confusion
* Move Function Calls into Version Package
* Fix Linting and Language Typos
* Fix Linting and Spelling Mistakes
* Include Copyright
* Include Copyright
* Adjust Version-Control Server to return list of Versions
* Linting
* Improve Request Handling and Readability
* Add Configuration File Option
Add Systemd Service file
* Add Logging to File
* Smaller Changes
* Add Semantic Versioning and refuses outdated Software from Startup (#1612)
* implements internal Semantic Version library
* adds version logging + reporting to process
* Advance SemVer struct for easier handling
* Add Accepted Version Store
* Fix Function
* Restructure
* Type Conversion
* Handle Version String properly
* Add Note about array index
* Set temporary Default Version
* Add Copyright
* Adding Version to Dashboard
* Adding Version Info Log
* Renaming and adding CheckerProcess
* Iteration Sync
* Iteration V2
* linting
* made LogAndReportVersion a go routine
* Refactor to Go Routine
* Add Context to Go Routine and allow Operation if Lookup to Control Server fails
* Handle Unmarshal properly
* Linting
* Relocate Version Checks
* Relocating Version Check and specified default Version for now
* Linting Error Prevention
* Refuse Startup on outdated Version
* Add Startup Check Function
* Straighten Logging
* Dont force Shutdown if --dev flag is set
* Create full Service/Peer Structure for ControlServer
* Linting
* Straighting Naming
* Finish VersionControl Service Layout
* Improve Error Handling
* Change Listening Address
* Move Checker Function
* Remove VersionControl Peer
* Linting
* Linting
* Create VersionClient Service
* Renaming
* Add Version Client to Peer Definitions
* Linting and Renaming
* Linting
* Remove Transport Checks for now
* Move to Client Side Flag
* Remove check
* Linting
* Transport Client Version Intro
* Adding Version Client to Transport Client
* Add missing parameter
* Adding Version Check, to set Allowed = true
* Set Default to true, testing
* Restructuring Code
* Uplink Changes
* Add more proper Defaults
* Renaming of Version struct
* Dont pass Service use Pointer
* Set Defaults for Versioning Checks
* Put HTTP Server in go routine
* Add Versioncontrol to Storj-Sim
* Testplanet Fixes
* Linting
* Add Error Handling and new Server Struct
* Move Lock slightly
* Reduce Race Potentials
* Remove unnecessary files
* Linting
* Add Proper Transport Handling
* small fixes
* add fence for allowed check
* Add Startup Version Check and Service Naming
* make errormessage private
* Add Comments about VersionedClient
* Linting
* Remove Checks that refuse outgoing connections
* Remove release cmd
* Add Release Script
* Linting
* Update to use correct Values
* Change Timestamp handling
* Adding Protobuf changes back in
* Adding SatelliteDB Changes and adding Storj Node Version to PB
* Add Migration Table
* Add Default Stats for Creation
* Move to BigInt
* Proper SQL Migration
* Ensure minimum Version is passed to the node selection
* Linting...
* Remove VersionedClient and adjust smaller changes from prior merge
* Linting
* Fix PB Message Handling and Query for Node Selection
* some future-proofing type changes
Change-Id: I3cb5018dcccdbc9739fe004d859065992720caaf
* fix a compiler error
Change-Id: If66bb92d8b98e31cd618ecec9c6448ab9b037fa5
* Comment on Constant for Overlay
* Remove NOT NULL and add epoch call as function
* add versions to bootstrap and satellites
Change-Id: I436944589ea5f21600cdd997742a84fe0b16e47b
* Change Update Migration
* Fix DB Migration
* Increase Timeout temporarily, to see whats going on
* Remove unnecessary const and vars
Cleanup Function calls from deprecated NodeVersion struct
* Updated Protopuf, removed depcreated Code from Inspector
* Implement NodeVersion into InfoResponse
* Regenerated locked.go
* Linting
* Fix Tests
* Remove unnecessary constant
* Update Function and Flag Description
* Remove Empty Stat Creation
* return properly with error
* Remove unnecessary struct
* simplify migration step
* Update Inspector to return Version Info
* Update local Endpoint Version Handling
* Reset Travis Timeout
* Add Default for CommitHash
* single quotes
* define irreparable inspector protobuf
* add IrreparableDB method GetLimited
* fill out irreparable inspector API
* add IrreparableInspector server to satellite, fix small error
* refactor IrreparableDB to use pb.IrreparableSegment instead of irreparable.RemoteSegmentInfo
* Wiring up DumpNodes response for Inspector
* Finalize everything and test that it works
* remove changes to overlay inspector
* Replace list with iterate method in DumpNodes
* generate proto
* Merge in nodeinfo changes
* Update iterate function in DumpNodes
* Fix import order
Adds a new `Info` method to the Kademlia endpoint that returns the following local node info:
* ID
* Type
* Metadata (email and wallet)
* Restrictions (free storage and bandwidth)
The new endpoint is exposed as `inspector kad node-info` command too.
This PR includes a new package called testrouting, which implements a very algorithmically slow but hopefully easy-to-keep-operationally-correct in-memory routing table. The routing table also supports writing out its current structure as a DOT graph for visualization. testrouting is primarily meant to help in coming up with generic routing table integration tests.
This PR also includes a new routing table integration test suite that runs against all current routing table implementations. Our existing routing table passes a lot of the tests, but not all of them, still debugging why. I have confirmed the tests should pass with the visualization graphs though.
Removes most instances of pb.SignedMessage (there's more to take out but they shouldn't hurt anyone as is).
There used to be places in psserver where a PieceID was hmac'd with the SatelliteID, which was gotten from a SignedMessage. This PR makes it so some functions access the SatelliteID from the Payer Bandwidth Allocation instead.
This requires passing a SatelliteID into psserver functions where they weren't before, so the following proto messages have been changed:
* PieceId - satellite_id field added
This is so the psserver.Piece function has access to the SatelliteID when it needs to get the namespaced pieceID.
This proto message should probably be renamed to PieceRequest, or a new PieceRequest message should be created so this isn't misnamed.
* PieceDelete - satellite_id field added
This is so the psserver.Delete function has access to the SatelliteID when receiving a request to Delete.
now kad inspector features exist on every server that has
kademlia running. likewise, overlay and statdb.
this means kad inspection features are now available on
storage nodes
Change-Id: I343c873552341de13302bfb7a5d79cccc84fc6b8
* intial changes to migrate statdb to masterdb framework
* statdb refactor compiles
* added TestCreateDoesNotExist testcase
* Initial port of statdb to masterdb framework working
* refactored statdb proto def to pkg/statdb
* removed statdb/proto folder
* moved pb.Node to storj.NodeID
* CreateEntryIfNotExistsRequest moved pd.Node to storj.NodeID
* moved the fields from pb.Node to statdb.UpdateRequest
ported TestUpdateExists, TestUpdateUptimeExists, TestUpdateAuditSuccessExists TestUpdateBatchExists
* WIP possible discovery service impl
* Adds discovery service to CaptPlanet
* Updates the config and server for discovery service
* updates testplanet to use discovery package
* update satellite imports
* Removes unnecessary cache test
* linter fixes
* adds discovery startup to captplanet
* invoke refresh
* updates to discovery refresh cycle
* Make implementation more consistent with previous implementation
* add wait before trying to upload
* sleep a bit more
* remove kademlia bootstrap
* updates
* remove comments
* tallies up data stored on each node in pointerdb
* adds comments for data type enums
* changes Open to BeginTx because Go convention
* removes online status check from identify active nodes
* changes identifyactivenodes to calculatestaticdata
* updates accounting dbx names
* adds channel for getting node out of lookup
* WIP adding the channels to lookups
* WIP adding channel to node lookups
* Wires up FindNodes method with channels
* WIP adds a test suite for lookup - tests are still failing
* WIP wires up use of testplanet for kademlia lookup tests
* WIP merging in node id changes
* Merges in pkg/storj node type changes
* Tests passing
* Lookup node working via Inspector now
* updates
* WIP working on getting tests passing
* WIP getting tests passing
* FindNode works
* Linter fix
* Adds copyrights to lookup_test
* removes a fmt.Printf I missed
* Removes commented out lines
* Pulls statdb stats into overlay cache whenever cache.Put() is called
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to filter based on node stats
* Updates overlay.FindStorageNodes()/overlayClient.Choose() to exclude duplicate IP addresses
* preparing for use of `customtype` gogo extension with `NodeID` type
* review changes
* preparing for use of `customtype` gogo extension with `NodeID` type
* review changes
* wip
* tests passing
* wip fixing tests
* more wip test fixing
* remove NodeIDList from proto files
* linter fixes
* linter fixes
* linter/review fixes
* more freaking linter fixes
* omg just kill me - linterrrrrrrr
* travis linter, i will muder you and your family in your sleep
* goimports everything - burn in hell travis
* goimports update
* go mod tidy
* initial commit of inspector gadget wireup
* change name of comman dline tool, setup grpc server
* Get inspector cli working with grpc client
* Wired up CountNodes command
* WIP getting buckets response working
* Added GetBucket command
* WIP working on get buckets command
* WIP working on bucket list
* Still WIP
* WIP getting bucket counts to work
* Some clean up of unnecessary changes
* List Buckets and Get Bucket are working
* Removing logs, getting ready for review
* initial commit of inspector gadget wireup
* change name of comman dline tool, setup grpc server
* Get inspector cli working with grpc client
* Wired up CountNodes command
* WIP getting buckets response working
* Added GetBucket command
* WIP working on get buckets command
* WIP working on bucket list
* Still WIP
* WIP getting bucket counts to work
* Some clean up of unnecessary changes
* List Buckets and Get Bucket are working
* Removing logs, getting ready for review
* Fix error return
* Trying to get tests passing
* Adds method on dht mock for tests
* Add dbx files back
* Fix package import error in dbx file
* Adds copyrights to pass linter
* tidy go mod
* Updates from code review
* Updates inspector to take flag arguments for address
* Format list-buckets output more prettier
* Wiring up PING in kad inspector tools
* initial commit of inspector gadget wireup
* change name of comman dline tool, setup grpc server
* Get inspector cli working with grpc client
* Wired up CountNodes command
* WIP getting buckets response working
* Added GetBucket command
* WIP working on get buckets command
* WIP working on bucket list
* Still WIP
* WIP getting bucket counts to work
* Some clean up of unnecessary changes
* List Buckets and Get Bucket are working
* Removing logs, getting ready for review
* initial commit of inspector gadget wireup
* change name of comman dline tool, setup grpc server
* Get inspector cli working with grpc client
* Wired up CountNodes command
* WIP getting buckets response working
* Added GetBucket command
* WIP working on get buckets command
* WIP working on bucket list
* Still WIP
* WIP getting bucket counts to work
* Some clean up of unnecessary changes
* List Buckets and Get Bucket are working
* Removing logs, getting ready for review
* Fix error return
* Trying to get tests passing
* Adds method on dht mock for tests
* Add dbx files back
* Fix package import error in dbx file
* Adds copyrights to pass linter
* tidy go mod
* Updates from code review
* Updates inspector to take flag arguments for address
* Format list-buckets output more prettier
* Signature verification
* Clean up agreement sender to have less errors
* overlay address in captnplanet
* Refactor bandwidth.proto to not use streams
* Make sure the send worked
* Handle connection to satellite
* Save renter public key inside of renter bandwidth allocations
* Default diag to sqlite. Make configurable
* Separate bw server and dbm; regenerate dbx files
* Make sure test uses protobufs
* Demonstrate creating bandwidth allocations
* WIP starting to wire up the kademlia CLI tool
* WIP wiring up kad cli tools
* WIP starting to wire up the kademlia CLI tool
* WIP wiring up kad cli tools
* Got everything wired up
* WIP starting to wire up the kademlia CLI tool
* WIP wiring up kad cli tools
* merge in upstream
* WIP wiring up kad cli tools
* Got everything wired up
* WIP trying to get CLI to connect
* Inspector connects to overlay now
* Some refactoring
* Linter fixes
* Linter fixes
* Switch to pkg/process instead of using rootCmd.Execute
* protobuf for sending bandwidth agreements to satellite from storage nodes
* Setup process for sending agreements
* Add payer_id to db with bandwidth agreements for better sorting
* Renamed payer to satellite in psdb
This is an old definition from the very early stage of development. It
is not used anymore.
Change-Id: I6a033e4006e6edfa7c18acc6ae91c9e4e1df0e6a
Signed-off-by: Kaloyan Raev <kaloyan@storj.io>
Reviewed-on: https://review.gerrithub.io/429582
Reviewed-by: JT Olio <hello@jtolio.com>
Tested-by: JT Olio <hello@jtolio.com>
* add filter field into OverlayOptions message
* chooseFiltered method, add excluded parameter in populate method
* change excluded type to []dht.NodeID in ChooseFiltered, change comment
* change name filter to excluded_nodes in proto
* implement helper function contains
* delete ChooseFiltered and add its functionality into Choose method to keep original author's history, add excluded argument into Choose calls
* regenerate mock_client.go
* regenerate protobuf
* update test case to use new IDFromString function
* remove old kademlia test code
* creates checker
* tests offline nodes
* test id injured segs:
* Adds healthy pieces to injured segment struct
* changes inequality
* creates common files
* adds checker benchmarking
* creates more common files
* Replaces pointedb direct db with api call to a new iterate method on pointerdb
* move monkit
* removes identifyrequest proto
* remove healthypieces
* adds benchmarking
creates common file for datarepair
* recreates proto file
* api key on ctx
