Commit Graph

74 Commits

Author SHA1 Message Date
Bill Thorp
0f662b8e38 sql based tally (#1166)
* WIP

* wacky changes

* more

* it builds.... IT BUILDS!!!

* fixed SQL, broke out saving tallies for testing

* shorter lines

* fixed SQL, moved tally_test to testplanet

* lint

* WIP logic error preventing PUT and GETs to same serialnum

* fixed BWA test

* fixed temporary brain failure

* eliminated magic numbers

* cleaned up satellite uplink stats

* use errs.Combine instead

* thrashing

* fixed tally erroneous error msg

* fixed tally test

* lint

* SQL syntax attempt to fix

* spelling error

* made bwa db test resist old postgres data

* postgres pk error msg hunting

* postgres pk error msg hunting

* postgres

* err might be nil?

* fixed error logging bug

* hopefully solved postgres issue

* using rebind

* moved tests to _test package

* fixing test dirs

* finally made sense of Egons package name feedback

* UTC, array fixes
2019-02-01 13:50:12 -05:00
Cameron
1403b15cc7
payments accesses database directly (#1135)
* move payments command into satellite/main.go

* flag for db connection string in paymentsCmd

* refactor payments to satellite subcommand

* reports command, add payments arg descriptions

* report data prints to stdout unless --out is set

* fix small error in csv columns
2019-01-30 16:44:50 -05:00
Jennifer Li Johnson
6355ae7a35
closes db in satellite, storage node and bootstrap cmdRun (#1159) 2019-01-30 00:22:58 -05:00
Bill Thorp
60946c2024
make bandwidth agreements sensible: without []byte's (#1152)
removed []byte's from bandwidth agreement protocol buffers
2019-01-28 14:45:25 -05:00
Egon Elbre
04d3699107
storj-sim configs (#1140) 2019-01-28 16:48:49 +02:00
Timothy Adams
2dbc06f7e1
fixing entrypoint script looking for invalid flag (#1147)
* fixing entrypoint script looking for invalid flag

* fixed other entrypoints

* removed ca_difficulty entries from compose.yaml
2019-01-28 07:53:16 -05:00
Egon Elbre
85b43926b4
Separate identity from server config (#1138) 2019-01-25 16:54:54 +02:00
Egon Elbre
187e9b2138
Code consistency between peers (#1126) 2019-01-24 22:28:06 +02:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Egon Elbre
5de7f8af7f
Satellite Peer (#1119) 2019-01-23 21:58:44 +02:00
Bryan White
322a2813f7
oops (#1110) 2019-01-22 15:34:40 +01:00
Bryan White
8edfd43abb
certificates config and --certs-dir fixes (#1093) 2019-01-22 13:35:48 +01:00
Egon Elbre
bbf81f2479 Consolidate identity management to identity cli commands (#1083)
* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* linters

* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* sava backups before saving signed certs

* add "-prebuilt-test-cmds" test flag

* linters

* prepare cli tests for travis

* linter fixes

* more fixes

* linter gods

* sp/sdk/sim

* remove ca.difficulty

* remove unused difficulty

* return setup to its rightful place

* wip travis

* Revert "wip travis"

This reverts commit 56834849dcf066d3cc0a4f139033fc3f6d7188ca.

* typo in travis.yaml

* remove tests

* remove more

* make it only create one identity at a time for consistency

* add config-dir for consitency

* add identity creation to storj-sim

* add flags

* simplify

* fix nolint and compile

* prevent overwrite and pass difficulty, concurrency, and parent creds

* goimports
2019-01-18 11:36:58 +01:00
Jennifer Li Johnson
c4b876ddc5
Jj/bytes (#1085)
* run tally and rollup

* sets dev default tally and rollup intervals
2019-01-17 14:39:32 -05:00
Jennifer Li Johnson
e6fbf63620
Generate Payments Report (#1079) 2019-01-17 13:34:13 -05:00
paul cannon
681d77c41a
use _real_ cert-path to get node ID for metrics (#1058)
apparently, the presence of an "Identity" attribute in a config struct
does not imply that anything will fill it in or use it.

i tested all of these executables manually and individually this time.
2019-01-16 15:43:29 -05:00
paul cannon
248ee6438f
use Node ID for metrics ID (#1052)
This change removes automatic metrics reporting for everything going
through process.Exec(), and re-adds metrics reporting for those commands
which are expected to be long-lived. Other commands (which may have been
intermittently sending metrics before this, if they ran unusually long)
will no longer send any metrics.

For commands where it makes sense, a node ID is used as the metrics ID.
2019-01-15 10:02:54 -05:00
Kaloyan Raev
a63abf8fab
Save only user-specific flags to storage node's config.yaml (#1051) 2019-01-15 15:55:33 +02:00
JT Olio
87925789de pkg/inspector: split functionality to be per-service (#1029)
now kad inspector features exist on every server that has
kademlia running. likewise, overlay and statdb.

this means kad inspection features are now available on
storage nodes

Change-Id: I343c873552341de13302bfb7a5d79cccc84fc6b8
2019-01-14 11:47:22 -07:00
Maximillian von Briesen
43ed9dea86
Remove overwrite flags (#1040)
remove overwrite flags from uplink, satellite, storagenode, captplanet, bootstrap

https://storjlabs.atlassian.net/browse/V3-1013
2019-01-14 10:57:58 -05:00
Philip Hutchins
b5f51014d3
Adding additional required values for satellite configuration (#1036)
* Adding other top level vars to be parsed into config

* Adding additional vars for parsing

* Fixing shell script conditionals in remaining entrypoints

* Adding null default for remaining env variable if statements
2019-01-11 22:47:18 -05:00
Egon Elbre
34125fe614 Reduce test CA difficulty (#1010)
* reduce test CA difficulty

* reduce docker ca difficulty
2019-01-10 15:23:33 +01:00
Bill Thorp
fb4a11ebb0
Differentiate GET and PUT operations in the tally service (#965)
* draft of new bandwidth agreement types
* updated storagenode report for new types
* use correct pba types
2019-01-10 06:41:57 -05:00
Philip Hutchins
c5549befd2 Reordering when we set the API_KEY env var in the docker entrypoint (#1017) 2019-01-10 12:35:06 +02:00
Michal Niewrzal
4e857ea133
Add setup tag for config structs (#968)
* Fix ignored setup arguments

* fix linter errors

* small params cleanup

* fix integration tests

* cleanup in configs

* Add `setup` tag for config structs

* fix broken if statement

* cleanup captplanet config

* remove reduntant return

* add missing Signer config

* review comments

* local variable renamed

* remove unused var
2019-01-07 12:06:10 +01:00
Michal Niewrzal
4be58e2a76
Fix ignored setup arguments (#944)
* Fix ignored setup arguments

* fix linter errors

* small params cleanup

* fix integration tests

* cleanup in configs

* Restart Travis
2019-01-07 10:48:16 +01:00
Jennifer Li Johnson
a2fa5c4c5a Proper NodeType Handling (#873)
* adds enums to nodetype

* updating nodetype todos

* ran pb updates

* reorder nodetypes

* adding checks

* wip

* wip

* wip

* bug in test-captplanet

* wip

* add values to storagenode, satellite, captplanet binaries

* Cleanup

* more cleanup

* wip

* lint

* lint

* wip

* fixes bug

* regenerate protos

Change-Id: Id270212e8c7479e52641058042cf23b5317ab773

* limit node type changes to kademlia

Change-Id: I9c1a6cc4a79e05086627f0fdeb5028c62ce754f4

* dpanic

Change-Id: Id952a2ad13c807ebaea0ec0a875405e267d81c3e

* review comments

Change-Id: I7f9b77ef22779dd012fd490375b136014f51f834
2019-01-02 11:47:34 -07:00
JT Olio
2c916a04c3 pkg/provider: split into pkg/server, pkg/identity (#953) 2019-01-02 12:23:25 +02:00
Egon Elbre
4346cd060f
Implement mutex around satellitedb (#932) 2018-12-27 11:56:25 +02:00
Matt Robinson
5224e92646
Fix detection of existing config in docker images (#929)
* If the directory is somehow persistent, but the config file itself isn't created yet, create it

* Other improvements to thd storagenode Docker image

* Other improvements to thd satellite Docker image

* Other improvements to thd gateway Docker image

* Other improvements to thd uplink Docker image
2018-12-21 11:56:57 -05:00
Cameron
f70b826fd4
repair queue masterDB support (#865)
* add injuredsegment model to satellitedb.dbx

* add context to queue.RepairQueue interface

* use queue.RepairQueue interface, use masterdb
2018-12-21 10:11:19 -05:00
littleskunk
e1c3f11cfa
add overwrite flag (#913)
* add storagenode setup --overwrite flag
* exit setup with error code
2018-12-20 14:27:51 +01:00
Matt Robinson
23fa3ac24a
Fix all-in-one, again (#896) 2018-12-18 09:27:46 -05:00
Matt Robinson
7c9ff4fc32
Fix segfault too (#883) 2018-12-17 14:27:19 -05:00
Matt Robinson
a5af25c9c2
Limit Jenkins to only one build at a time (#885)
* Fix docker images, again

* Limit jenkins to only one build of a branch at a time
2018-12-17 11:54:32 -05:00
Michal Niewrzal
f0662675bc
Add '--dir' param for all CLI parts (replace --base-path) (#833)
* Add '--dir' param for all CLI parts (replace --base-path)

* FindDirParam method moved

* fix compilation error

* make param global

* remove unused fields

* rename param

* remove config flag

* goimports
2018-12-14 22:14:59 +01:00
aligeti
5e1b02ca8b
Statdb master db v3 848 (#830)
* intial changes to migrate statdb to masterdb framework

* statdb refactor compiles

* added TestCreateDoesNotExist testcase

* Initial port of statdb to masterdb framework working

* refactored statdb proto def to pkg/statdb

* removed statdb/proto folder

* moved pb.Node to storj.NodeID

* CreateEntryIfNotExistsRequest moved pd.Node to storj.NodeID

* moved the fields from pb.Node to statdb.UpdateRequest

ported TestUpdateExists, TestUpdateUptimeExists, TestUpdateAuditSuccessExists TestUpdateBatchExists
2018-12-14 15:17:30 -05:00
Dylan Lott
9bdee7b106
Discovery service (#805)
* WIP possible discovery service impl

* Adds discovery service to CaptPlanet

* Updates the config and server for discovery service

* updates testplanet to use discovery package

* update satellite imports

* Removes unnecessary cache test

* linter fixes

* adds discovery startup to captplanet

* invoke refresh

* updates to discovery refresh cycle

* Make implementation more consistent with previous implementation

* add wait before trying to upload

* sleep a bit more

* remove kademlia bootstrap

* updates

* remove comments
2018-12-14 12:23:54 -07:00
Egon Elbre
7363e6cfd8
Better transport tests (#849) 2018-12-14 12:44:48 +02:00
Matt Robinson
32fdce7735
Fixes for latest code changes (#817) 2018-12-11 08:05:33 -05:00
Michal Niewrzal
7e2f66ee24
Small fixes around master db (#822) 2018-12-11 10:30:09 +01:00
Maximillian von Briesen
72a7726513
enable audit service on satellite (#803) 2018-12-07 13:52:58 -05:00
Michal Niewrzal
034a34069a
Merge bwagreement db into satellite master db (#770)
* Merge bwagreement db into satellite master db

* adjust to recent tally changes

* linter problems

* linter problems

* returning db structs in more optimal way

* added pointer for assignment

* error message changed

* better param message
2018-12-07 10:59:31 +01:00
Michal Niewrzal
44a6cfa1c1
Initial satellite master database setup (#735)
* Initial satellite master database setup

* fixed unsed ctx and /nolint added for temp change

* tests for two db drivers + connection param renamed
2018-12-05 10:35:50 +01:00
Matt Robinson
4237afb1c1
Attempt at fixing the all-in-one environment (#736) 2018-12-04 16:10:23 -05:00
Michal Niewrzal
06c196d28c
Allow config generation into empty directory (#752)
* Allow config generation into empty directory

* unifed messages

* move main method to fpath

* fix linter problems
2018-12-04 12:42:16 +01:00
Michal Niewrzal
1206ef9126
Use OS related default setup directory (#741)
* Use OS related default setup directory

* added missing comment

* adjust test-captplanet.sh

* ApplicationDir moved to fpath
2018-12-03 16:51:56 +01:00
Egon Elbre
1e4556f88a
Fix import groupings (#739) 2018-11-30 15:40:13 +02:00
Bryan White
2a0c4e60d2
preparing for use of customtype gogo extension with NodeID type (#693)
* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* preparing for use of `customtype` gogo extension with `NodeID` type

* review changes

* wip

* tests passing

* wip fixing tests

* more wip test fixing

* remove NodeIDList from proto files

* linter fixes

* linter fixes

* linter/review fixes

* more freaking linter fixes

* omg just kill me - linterrrrrrrr

* travis linter, i will muder you and your family in your sleep

* goimports everything - burn in hell travis

* goimports update

* go mod tidy
2018-11-29 19:39:27 +01:00
Bryan White
dabd87cb36
swap golang/protobuf out for gogo/protobuf (#692)
* swap golang/protobuf out for gogo/protobuf

* revert go.mod and go.sum changes

* linter fixes
2018-11-20 19:29:07 +01:00