* added satalite partner value attribution report. WIP
* WIP
* basic attribution report test completed. still a WIP
* cleanup
* fixed projectID conversion
* report display cleanup
* cleanup .added more test data
* added partnerID to query results
* fixed lint issues
* fix import order
* suggestions from PR review
* updated doc to reflect implementation
* clarification comments in the report SQL
* Changed based on PR suggestion
* More changes based on PR suggestions
* Changes based on PR suggestions
* reordered tests to make consistant with previous 2
* small comments cleanup
* More PR suggestions
* fixed lint issue and removed printf
* fixed var name
* Updates based on PR suggestions
* fixed message
* fixed test
* changes required after merge from master
* set to only listen on 127.0.0.1, move static files to same location, better template handling
* handle error
* fix path in storj-sim
* revert template handling changes
* code shouldn't panic on invalid tempalte
* do not rewrite once writing has started
* write correct error code
* use filepath for path handling
* revert change
* fix
* fix mod tidy
* use correct error code for not found, avoid infinite loop on failure
* Set up new port 8090 for in offers
Clean up commented code
Rename offers to offersweb
Remove unused code
Add todos for adding front-end templates
Add middleware for only allow local access
Add comment
Fix linting error
Remove commented code
Update storj-sim
Check request IP against Host IP
Use net pakcage to retrieve IP address
Rename service to marketing
* Add wrapper for all errors
* fix conflicts
* update the config file
* fix linting error
* remove unused packages
* remove global runtime var and add flag to storj-sim for mar static dir
* remove debugging lines
* add new config for test data and check if static dir flag is set before passing to mux
* change 'console' to 'marketing' for test data config
* fix linting errors
* update config flag
* Trigger Jenkins
* Trigger CLA
* uplink: Mark encryption key config field for setup
Set the "setup" property to the `EncryptionConfig.EncrptionKey` for
avoiding to save it in the configuration file.
This field is only meant for using in the command line parameters which
need to use a different encryption key than the one present in the key
file or use it when there is not set any encryption key file path.
* cmd/uplink: Setup non-interactive accept enc key
Change the uplink CLI setup command non-interactive to save the
encryption key into a file when it's passed through the flag
--enc.encryption-key
Previous to this change it wasn't possible to create an key file despite
of that the flag was provided, so it was useless on the setup command.
* cmd/uplink: Reuse logic to read pwd from terminal
Reuse the logic which is already implemented in the pkg/cfgstruct for
reading a password from the terminal on interactive mode, rather than
duplicating it in the setup command.
* cmd/gateway: Use encryption key file flags
The cmd/gateway was still using the `enc.key` configuration field which
doesn't exist anymore and its setup command wasn't using the
`enc.key-filepath` with combination of the `enc.encryption-key` for
generating a file with the encryption key.
This commit update the cmd/gateway appropriately and move to the uplink
package the function used by cmd/uplink to save the encryption key for
allowing to also be used by the cmd/gateway without duplicating the
logic.
* cmd/storj-sim: Adapt gateway config cmd changes
Adapt the cmd/storj-sim to correctly pass the parameters to the
cmd/gateway setup and run command.
* scripts: Don't pass the --enc.encryption-key flag
uplink configuration has changed to only support the
`--enc.encryption-key` flag for setup commands and consequently the
cmd/uplink and cmd/gateway don't accept this flag over other commands,
hence the test for the uplink had to be updated for no passing the
flag on the multiples calls that the test do to cmd/uplink.
* uplink: Remove func which aren't useful anymore
Remove the function which allows to user or load an encryption key
because it isn't needed anymore since the `--enc.encryption-key` flag is
only available for the setup command.
Consequently remove its usage from cmd/uplink and cmd/gateway, because
such flag will always be empty because in case that's passed Cobra will
return an error due to a "unknown flag".
* change BindSetup to be an option to Bind
* add process.Bind to allow composite structures
* hack fix for noprefix flags
* used tagged version of structs
Before this PR, some flags were created by calling `cfgstruct.Bind` and having their fields create a flag. Once the flags were parsed, `viper` was used to acquire all the values from them and config files, and the fields in the struct were set through the flag interface.
This doesn't work for slices of things on config structs very well, since it can only set strings, and for a string slice, it turns out that the implementation in `pflag` appends an entry rather than setting it.
This changes three things:
1. Only have a `Bind` call instead of `Bind` and `BindSetup`, and make `BindSetup` an option instead.
2. Add a `process.Bind` call that takes in a `*cobra.Cmd`, binds the struct to the command's flags, and keeps track of that struct in a global map keyed by the command.
3. Use `viper` to get the values and load them into the bound configuration structs instead of using the flags to propagate the changes.
In this way, we can support whatever rich configuration we want in the config yaml files, while still getting command like flags when important.
* added scopelint and correcte issues found
* corrected scopelint issue
* made updates based on Ivan's suggestions
Most were around naming conventions
Some were false positives, but I kept them since the test.Run could eventually be changed to run in parallel, which could cause a bug
Others were false positives. Added // nolint: scopelint
* first round cleanup based on go-critic
* more issues resolved for ifelsechain and unlambda checks
* updated from master and gocritic found a new ifElseChain issue
* disable appendAssign. i reports false positives
* re-enabled go-critic appendAssign and disabled lint check at code line level
* fixed go-critic lint error
* fixed // nolint add gocritic specifically
* add aws s3 benchmark script
* add s3 benchmark tests
* rearrange so smaller diff, fix spelling
* add configurable uplink config for s3-benchmark
* make new bucket w/unique name for each s3 test
* changes per CR
* update uplink cmd args
* add uplod/download benchmark, add script to run benchmarks
* fix lint err
* changes per CR comments
* export env var for linux
* fix lint
* add byte count to bench test
Uplink CLI was returning an error when the `--config-dir` flag was
used with interactive `setup` command because the directory used for
storing the encryption key file was the default configuration directory
rather than the value set in the flag.
This changes make the setup process to use the same directory, than the
one informed by the `--config-dir` flag value, for storing encryption
key file.
On the other hand, the default value for the encryption key file path
which was set with the purpose of showing it in the help message has
been removed because the cfgstruct doesn't show it unless that the flag
is set to the Cobra root CMD, so it was useless.
Also show the absolute path where the encryption key file has been saved when
the interactive setup command ends satisfactorily as it's equally done
with the configuration file.
What: allow packetfilter to work on packet destinations instead of only on metric destinations. this will allow us to filter what applications get sent to rothko.
Why: currently rothko is drowning in storj-sim data and it'd be nice to filter out.
* uplink: Add a new flag to set the filepath of the file which is used for
saving the encryption key and rename the one that hold the encryption key and
establish that it has priority over the key stored in the file to make the
configuration usable without having a huge refactoring in test-sim.
* cmd/uplink: Adapt the setup subcommand for storing the user input key to a file
and adapt the rest of the subcommands for reading the key from the key-file when
the key isn't explicitly set with a command line flag.
* cmd/gateway: Adapt it to read the encryption key from the key-file or use the
one passed by a command line flag.
* pkg/process: Export the default configuration filename so other packages which
use the same value can reference to it rather than having it hardcoded.
* Adapt several integrations (scripts, etc.) to consider the changes applied in uplink and cmd packages.
* cmd/uplink: add share command to restrict an api key
This commit is an early bit of work to just implement restricting
macaroon api keys from the command line. It does not convert
api keys to be macaroons in general.
It also does not apply the path restriction caveats appropriately
yet because it does not encrypt them.
* cmd/uplink: fix path encryption for shares
It should now properly encrypt the path prefixes when adding
caveats to a macaroon.
* fix up linting problems
* print summary of caveat and require iso8601
* make clone part more clear
* add flags to sotrj-sim for SA dbs
* add schema to postgres
* add createschema with parse to sa
* add metainfo db postgres support
* add kv default as bolt
* add debug log to see db source
* add env var for postgres to test-sim.sh
* fix lint errs
* dynamically add postgres to args
* add postgres to integration tests
* add sqlite and postgres integration jenkins
* fix db name
* merge integration tests into one step
* test integration tests w/psql
* try using different schema
* debug failure
* use correct host for running storj-sim
* rm sqlite integration
* add back integration
* tie defaults to releases
this change makes it so that by default, the flag defaults are
chosen based on whether the build was built as a release build or
an ordinary build. release builds by default get release defaults,
whereas ordinary builds by default get dev defaults.
any binary can have its defaults changed by specifying
--defaults=dev
or
--defaults=release
Change-Id: I6d216aa345d211c69ad913159d492fac77b12c64
* make release defaults more clear
this change extends cfgstruct structs to support either
a 'default' tag, or a pair of 'devDefault' and 'releaseDefault'
tags, but not both, for added clarity
Change-Id: Ia098be1fa84b932fdfe90a4a4d027ffb95e249c6
* clarify cfgstruct.DefaultsFlag
Change-Id: I55f2ff9080ebbc0ce83abf956e085242a92f883e
* hardcode debug ports for storj-sim
* add function to create port
* add 200 limit to sn
* remove iota and mv instance count
* remove singleIndex var
* remove err checking from port fx
* mv unneeded conversion
What: This change moves project-level bucket metadata encryption information to the volatile section, because it is unlikely to remain in future releases
Why: Ultimately, the web user interface will allow bucket management (creation, removal, etc), but not object management as that requires an encryption key for sure and we don't want to have users give the satellite their encryption keys.
At a high level, a (*Project) type should map to all of the things you can do inside the web user interface within a project, which by necessity cannot have an encryption key. So, we really don't want an encryption key in the non-volatile section of this library.
* releases: update alpha dockerfiles to embed release tags
Change-Id: I2978d3ed77d1b768be24ec9194bc404cd92e4700
* release script use plain shell
Change-Id: Ie4ca6633014e3bf00ac0c82e28905174a6ade95b
* removed /scripts from .dockerignore
Change-Id: Ib555236d62fb3dc354689284ac4a032d5b4b3aa8
* switch back to bash
Change-Id: Ie0958cde71cd396f44ac28c177f48df0bea7ba22
* fix go1.12 modules
Change-Id: If4c8364e7271f09febd1a6c8bfc731d723d344d9
* make sure the work tree is clean
Change-Id: Ie6e52af28e2bc52880d92a3b574d1f859fc8e88e
* internal/version: do version checks much earlier in the process initialization, take 2
Change-Id: Ida8c7e3757e0deea0ec7aea867d3d27ce97dc134
* linter and test failures
Change-Id: I45b02a16ec1c0f0981227dc842e68dbdf67fdbf4
Make separate "CreateCertificate" and "CreateSelfSignedCertificate"
functions to take the two roles of NewCert. These names should help
clarify that they actually make certificates and not just allocate new
"Cert" or "Certificate" objects.
Secondly, in the case of non-self-signed certs, require a public and a
private key to be passed in instead of two private keys, because it's
pretty hard to tell when reading code which one is meant to be the
signer and which one is the signee. With a public and private key, you
know.
(These are some changes I made in the course of the openssl port,
because the NewCert function kept being confusing to me. It's possible
I'm just being ridiculous, and this doesn't help improve readability for
anyone else, but if I'm not being ridiculous let's get this in)
* Initial Webserver Draft for Version Controlling
* Rename type to avoid confusion
* Move Function Calls into Version Package
* Fix Linting and Language Typos
* Fix Linting and Spelling Mistakes
* Include Copyright
* Include Copyright
* Adjust Version-Control Server to return list of Versions
* Linting
* Improve Request Handling and Readability
* Add Configuration File Option
Add Systemd Service file
* Add Logging to File
* Smaller Changes
* Add Semantic Versioning and refuses outdated Software from Startup (#1612)
* implements internal Semantic Version library
* adds version logging + reporting to process
* Advance SemVer struct for easier handling
* Add Accepted Version Store
* Fix Function
* Restructure
* Type Conversion
* Handle Version String properly
* Add Note about array index
* Set temporary Default Version
* Add Copyright
* Adding Version to Dashboard
* Adding Version Info Log
* Renaming and adding CheckerProcess
* Iteration Sync
* Iteration V2
* linting
* made LogAndReportVersion a go routine
* Refactor to Go Routine
* Add Context to Go Routine and allow Operation if Lookup to Control Server fails
* Handle Unmarshal properly
* Linting
* Relocate Version Checks
* Relocating Version Check and specified default Version for now
* Linting Error Prevention
* Refuse Startup on outdated Version
* Add Startup Check Function
* Straighten Logging
* Dont force Shutdown if --dev flag is set
* Create full Service/Peer Structure for ControlServer
* Linting
* Straighting Naming
* Finish VersionControl Service Layout
* Improve Error Handling
* Change Listening Address
* Move Checker Function
* Remove VersionControl Peer
* Linting
* Linting
* Create VersionClient Service
* Renaming
* Add Version Client to Peer Definitions
* Linting and Renaming
* Linting
* Remove Transport Checks for now
* Move to Client Side Flag
* Remove check
* Linting
* Transport Client Version Intro
* Adding Version Client to Transport Client
* Add missing parameter
* Adding Version Check, to set Allowed = true
* Set Default to true, testing
* Restructuring Code
* Uplink Changes
* Add more proper Defaults
* Renaming of Version struct
* Dont pass Service use Pointer
* Set Defaults for Versioning Checks
* Put HTTP Server in go routine
* Add Versioncontrol to Storj-Sim
* Testplanet Fixes
* Linting
* Add Error Handling and new Server Struct
* Move Lock slightly
* Reduce Race Potentials
* Remove unnecessary files
* Linting
* Add Proper Transport Handling
* small fixes
* add fence for allowed check
* Add Startup Version Check and Service Naming
* make errormessage private
* Add Comments about VersionedClient
* Linting
* Remove Checks that refuse outgoing connections
* Remove release cmd
* Add Release Script
* Linting
* Update to use correct Values
* Move vars private and set minimum default versions for testing builds
* Remove VersionedClient
* Better Error Handling and naked return removal
* Straighten the Regex and string conversion
* Change Check to allows testplanet and storj-sim to run without the
need to pass an LDFlag
* Cosmetic Change to Dashboard
* Cleanup Returns and remove commented code
* Remove Version Check if no build options are passed in
* Pass in Config Values instead of Pointers
* Handle missed Error
* Update Endpoint URL
* Change Type of Release Flag
* Add additional Logging
* Remove Versions Logging of other Services
* minor fixes
Change-Id: I5cc04a410ea6b2008d14dffd63eb5f36dd348a8b
* test-network-stalls tests... network stalls!
in particular, right now, it just tests whether an uplink correctly
times out after some amount of time when one of the nodes it's talking
to suddenly goes offline.
This tool is meant to be run under `storj-sim network test`.
Also included here:
* fix storj-sim-related test scripts on Mac
the default storj config dir on Mac has a space in it
('~/Library/Application Support/Storj'), which breaks everywhere it
shows up in an unquoted variable in a sh/bash script. easy enough to fix
as a one-off, but quoting bash vars avoids a dozen other potential
problems too.
change a few things using `head -c` to use `dd`. `head -c` works,
but is not as widely understood (as evidenced by each of these scripts
getting through code review, one at a time, with the comments not
matching the numbers actually used).
* storj-sim reports PIDs of worker processes to test
so that the tests can cause unfortunate "accidents" to befall the worker
processes in the course of the test, and find out whether everything
reacts correctly.
* define irreparable inspector protobuf
* add IrreparableDB method GetLimited
* fill out irreparable inspector API
* add IrreparableInspector server to satellite, fix small error
* refactor IrreparableDB to use pb.IrreparableSegment instead of irreparable.RemoteSegmentInfo
* Warn about permissions when creating identity
* Function to determine if directory is writeable
* Check if writable before authorizing
* Remove redeclatarion
* remove windows specific utils
* Nat nits
* Actually test if directory is writeable with file creation
* add private listener to grpc server
* add changes per init CR
* fix server.close
* add insecure grpc connection, update logs msg
* fix tests, move insecure client
* add private ports to storj-sim, add insecure client to other inspectors
* add ports to test so there arent conflicts
* fix lint err
* fix node started log msg, close public listener
* remove commented out line
* small identity refactor:
+ Optimize? iterative cert chain methods to use array instead of slice
+ Add `ToChain` helper for converting 1d to 2d cert chain
TODO: replace literal declarations with this
+ rename `ChainRaw/RestChainRaw` to `RawChain/RawRestChain`
(adjective noun, instead of nound adjective)
* add regression tests for V3-1320
* fix V3-1320
* separate `DialUnverifiedIDOption` from `DialOption`
* separate `PingNode` and `DialNode` from `PingAddress` and `DialAddress`
* update node ID while bootstrapping
* goimports & fix comment
* add test case
* Wiring up DumpNodes response for Inspector
* Finalize everything and test that it works
* Get Count and DumpNodes working for Overlay Cache
* WIP updating payment rollup to check statDB instead of overlay
* FIrst pass at updating statDB to take wallet and email
* Passing tests
* use pb.NodeOperator instead of Meta struct
* remove TODO
* revert go.mod
* Get SQL migration working correctly
* Changes Meta to Operator in NodeStats struct
* Adds update operator logic for statDB
* Fix db migrate tests - added v5 snapshot
* User friendly msg for missing snapshot version
* Passing tests
* Change node update to happen in discovery instead of in overlay
* Fix logic and update function calls
* Update comment on UpdateOperator interface method
* Update name of parameter
* Change type of argument to UpdateOperator
* Updates statDB tests
* Adding dockerfile for running the web UI for Satellite
* Updating to work with Makefile and from root directory of repo
* Updating satellite ui build process to run in a more production like mode by generating the assets the pulling those into the satellite container
* Updates to allow external traffic to UI, updates to storagenode for identity creation, and logging for bug tracking
* Adding auto cert generation for storagenode
* removing satellite-ui-image from main images flow in Makefile and adding latest tag to docker build for it
* Adding solid defaults, tuning dockerfiles, and moving to standard logging methods
* Updating logging to be more standard
* Updating to logger.Debug
* Removing unused library and unused identity creation code
Change-Id: I956453037e303693ea37f94318180af0ab7984d5
Adds a new `Info` method to the Kademlia endpoint that returns the following local node info:
* ID
* Type
* Metadata (email and wallet)
* Restrictions (free storage and bandwidth)
The new endpoint is exposed as `inspector kad node-info` command too.
this bandaid-fixes the identity tool issues (revocation
db defaults to /revocations.db due to the missing CONFDIR)
Change-Id: Ibdc8d1e9b64ec9a545bea592b0bd167ff5138117
* separate TLS options from server options (because we need them for dialing too)
* stop creating transports in multiple places
* ensure that we actually check revocation, whitelists, certificate signing, etc, for all connections.
this change removes the cryptopasta dependency.
a couple possible sources of problem with this change:
* the encoding used for ECDSA signatures on SignedMessage has changed.
the encoding employed by cryptopasta was workable, but not the same
as the encoding used for such signatures in the rest of the world
(most particularly, on ECDSA signatures in X.509 certificates). I
think we'll be best served by using one ECDSA signature encoding from
here on, but if we need to use the old encoding for backwards
compatibility with existing nodes, that can be arranged.
* since there's already a breaking change in SignedMessage, I changed
it to send and receive public keys in raw PKIX format, instead of
PEM. PEM just adds unhelpful overhead for this case.
* Divide uplink and gateway params set
* attempt to fix docker
* attempt to fix all in one
* test
* more reorganization
* fix compilation error
* fix imports order
* fix dependency
* rename structs
* keep minio params for now
* review comments
* remove manual flag check
* move payments command into satellite/main.go
* flag for db connection string in paymentsCmd
* refactor payments to satellite subcommand
* reports command, add payments arg descriptions
* report data prints to stdout unless --out is set
* fix small error in csv columns
* Consolidate identity management:
Move identity cretaion/signing out of storagenode setup command.
* fixes
* linters
* Consolidate identity management:
Move identity cretaion/signing out of storagenode setup command.
* fixes
* sava backups before saving signed certs
* add "-prebuilt-test-cmds" test flag
* linters
* prepare cli tests for travis
* linter fixes
* more fixes
* linter gods
* sp/sdk/sim
* remove ca.difficulty
* remove unused difficulty
* return setup to its rightful place
* wip travis
* Revert "wip travis"
This reverts commit 56834849dcf066d3cc0a4f139033fc3f6d7188ca.
* typo in travis.yaml
* remove tests
* remove more
* make it only create one identity at a time for consistency
* add config-dir for consitency
* add identity creation to storj-sim
* add flags
* simplify
* fix nolint and compile
* prevent overwrite and pass difficulty, concurrency, and parent creds
* goimports
* Add more info to SN logs
* remove config-dir from user config
* add output where config was stored
* add message for successful connection
* fix linter
* remove storage.path from user config
* resolve config path
* move success message to info
* log improvements
apparently, the presence of an "Identity" attribute in a config struct
does not imply that anything will fill it in or use it.
i tested all of these executables manually and individually this time.
This change removes automatic metrics reporting for everything going
through process.Exec(), and re-adds metrics reporting for those commands
which are expected to be long-lived. Other commands (which may have been
intermittently sending metrics before this, if they ran unusually long)
will no longer send any metrics.
For commands where it makes sense, a node ID is used as the metrics ID.
* Edit config on Setup
* Default to 1TiB storage space and 500GiB bandwidth
* Use human readable formats
* Use memory
* units of 1024 are measured with KiB/MiB etc
* pkg/cfgstruct: allow values to be configured with human readable sizes
Change-Id: Ic4e9ae461516d1d26fb81f6e44c5ac5cfccf777f
* Modify tests
* Removed comments
* More merge conflict stuff resolved
* Fix lint
* test fixin
Change-Id: I3a008206bf03a4446da19f642a2f9c1f9acaae36
* Remove commented code but secretly leave it in the histroy forever
* Move flag definition to struct
now kad inspector features exist on every server that has
kademlia running. likewise, overlay and statdb.
this means kad inspection features are now available on
storage nodes
Change-Id: I343c873552341de13302bfb7a5d79cccc84fc6b8
* Adding other top level vars to be parsed into config
* Adding additional vars for parsing
* Fixing shell script conditionals in remaining entrypoints
* Adding null default for remaining env variable if statements
* pkg/identity: use sha256 instead of sha3 for pow
Change-Id: I9b7a4f2c3e624a6e248a233e3653eaccaf23c6f3
* pkg/identity: restructure key generation a bit
Change-Id: I0061a5cc62f04b0c86ffbf046519d5c0a154e896
* cmd/identity: indefinite key generation command
you can start this command and leave it running and it will fill up your
hard drive with node certificate authority private keys ordered by
difficulty.
Change-Id: I61c7a3438b9ff6656e74b8d74fef61e557e4d95a
* pkg/storj: more node id difficulty testing
Change-Id: Ie56b1859aa14ec6ef5973caf42aacb4c494b87c7
* review comments
Change-Id: Iff019aa8121a7804f10c248bf2e578189e5b829d
* Add test for aio
* Don't trust the user to have images built for a version
* Make travis run the aio test
* Add missing values to docker-compose, sort some things, consider the gateway image
* today's changes
* config changed, again
* more fixes
* Expose satellite port on localhost:7778
* Add retries and a timeout around the big-testfile test in AIO
* Another config value changed
* Make this error message a little more useful
* Fix nil condition
