Egon Elbre
b16f27c54b
Ensure discovery tests run against postgres ( #1230 )
2019-02-05 19:03:16 +02:00
Kaloyan Raev
3c73d3a33c
Fixes intermitent failures in storage/ec TestPut ( #1239 )
2019-02-05 18:49:52 +02:00
Egon Elbre
39c1e5ccec
Ensure checker tests run against postgres ( #1229 )
2019-02-05 18:00:52 +02:00
Egon Elbre
94ee200e35
Ensure cursor tests run against postgres ( #1228 )
2019-02-05 17:45:09 +02:00
Egon Elbre
6564357e13
Ensure that tally tests run against postgres ( #1227 )
2019-02-05 16:23:45 +02:00
Kaloyan Raev
dd76829d10
Improve logic for cutting the long tail during upload ( #909 )
2019-02-05 12:54:25 +02:00
Egon Elbre
87d6410b50
Revert "Remove node package and simplify DHT interface."
...
This reverts commit 03ec1ff92d
2019-02-05 10:38:48 +02:00
Egon Elbre
03ec1ff92d
Remove node package and simplify DHT interface.
2019-02-05 10:37:24 +02:00
Dylan Lott
21a58d0ece
node graveyard resurrection ( #1186 )
...
* deletes nodes from cache if unresponsive
* update error messages
* WIP restructuring discovery tickers
* passes tests
* Removes log messages
* Passes up error in graveyard ticker
2019-02-04 14:52:06 -07:00
Egon Elbre
c37e0c1b6d
testplanet.Run ( #1222 )
2019-02-04 22:37:46 +02:00
Michal Niewrzal
2f8b3da1af
Upload/download for testplanet Uplink ( #1183 )
...
* Upload/download for testplanet Uplink
* check error in tests
* cleanup
* refactor node -> uplink
* add missing test file
* rest of refactoring
* workaround to resolve cycles in tests
* rename method
* add missing comments
* review comments
* use KiB
2019-02-04 17:56:10 +01:00
Michal Niewrzal
53c11dfc5d
Stop storagenode on ctrl+c ( #1220 )
...
* Stop storagenode on ctrl+c
* Cancel grpc server
* handle error
* handle error
* use errgroup
* fix check-travis-tidy
* remove pipefail
2019-02-04 15:50:55 +01:00
paul cannon
0238a7785b
make a map object before inserting into it ( #1217 )
...
I think this is causing panics.
2019-02-03 18:42:20 -06:00
Bill Thorp
0f662b8e38
sql based tally ( #1166 )
...
* WIP
* wacky changes
* more
* it builds.... IT BUILDS!!!
* fixed SQL, broke out saving tallies for testing
* shorter lines
* fixed SQL, moved tally_test to testplanet
* lint
* WIP logic error preventing PUT and GETs to same serialnum
* fixed BWA test
* fixed temporary brain failure
* eliminated magic numbers
* cleaned up satellite uplink stats
* use errs.Combine instead
* thrashing
* fixed tally erroneous error msg
* fixed tally test
* lint
* SQL syntax attempt to fix
* spelling error
* made bwa db test resist old postgres data
* postgres pk error msg hunting
* postgres pk error msg hunting
* postgres
* err might be nil?
* fixed error logging bug
* hopefully solved postgres issue
* using rebind
* moved tests to _test package
* fixing test dirs
* finally made sense of Egons package name feedback
* UTC, array fixes
2019-02-01 13:50:12 -05:00
Egon Elbre
4117600f32
Cleanup repository a bit ( #1203 )
2019-02-01 20:21:00 +02:00
Kaloyan Raev
5c2b67db63
Fix EOF error on audits (storage node) ( #1208 )
...
* Fix EOF error on audits
* Specify the correct number of bytes to CopyN instead of checking for EOF
* Fix imports order
2019-02-01 10:57:11 -07:00
Kaloyan Raev
a90aa18582
Fix panic in PointerDB.Get ( #1209 )
...
* Fix panic in PointerDB.Get
* Allow nil nodes, check if v is nil instead
* More checks like this
* Remove unnecessary check
* More checks
2019-02-01 10:55:47 -07:00
JT Olio
41d19f3c3d
miniogw: fix buffer size alignment ( #1213 )
...
Change-Id: I47e0e67e385d236ac0338fe36f11cc7c00e23dba
2019-02-01 10:42:30 -07:00
Dylan Lott
c77a647174
Updates transport dialopts to use WithBlock() ( #1201 )
...
* Updates transport opts to use WithBlock()
* Removes unnecessary withBlock's and adds WithBlock to DialAddress
2019-02-01 10:40:35 -07:00
Egon Elbre
5a63c00442
Fix issues with blocking during startup ( #1212 )
2019-02-01 19:28:40 +02:00
Bryan White
3e3088e37c
improve identity create ux ( #1211 )
2019-02-01 11:58:33 -05:00
Egon Elbre
7a85c5c1c6
Fix audit node ids ( #1207 )
2019-02-01 16:48:57 +02:00
Egon Elbre
00570df1f9
Add top flag to tparse ( #1205 )
2019-02-01 16:35:13 +02:00
Egon Elbre
b91d77436f
Test merging planets ( #1181 )
2019-02-01 15:32:28 +02:00
Egon Elbre
0c366c1ed2
Clearer code for node selection ( #1173 )
2019-01-31 13:49:00 -05:00
Kaloyan Raev
8c357ac571
Correctly update uptime when ping fails ( #1190 )
2019-01-31 16:51:38 +02:00
Dylan Lott
6025f9f19e
deletes nodes from cache if unresponsive ( #1185 )
...
* deletes nodes from cache if unresponsive
* update error messages
2019-01-30 14:15:24 -07:00
Egon Elbre
97714d150f
Remove unused kademlia.Disconnect method ( #1178 )
2019-01-30 22:57:13 +02:00
Egon Elbre
d5346982c2
Delete provider package ( #1177 )
2019-01-30 22:47:21 +02:00
Egon Elbre
19de56bf83
Fix race in discovery test ( #1184 )
2019-01-30 22:29:33 +02:00
Alexander Leitner
7c741c16d1
Readable sizes in dashboard ( #1180 )
...
* Readable sizes
* Use tabwriter
* Added more addresses to dashboard stats
* Use kad to look up addresses
* Move dashboard command to dashboard.go
2019-01-30 14:49:25 -05:00
Michal Niewrzal
14434594c4
Re-enable repair checker tests ( #1172 )
...
* Re-enable repair checker tests
* simpler NodeID construction
* adjust benchmark
2019-01-30 19:34:44 +01:00
Dylan Lott
5e1ab841cf
Cache refresher ( #1171 )
...
* got tests passed
* wire up paginate function for cache node retrieval
* Add tests for paginate but they're failing
* fix the test arguments
* Updates paginate function to return more variable
* Updates
* Some test and logic tweaks
* improves config handling in discovery
* adds refresh offset to discovery struct
2019-01-30 11:29:18 -05:00
Egon Elbre
6132ce86b7
Remove utils.LogClose ( #1169 )
2019-01-29 22:42:27 +02:00
Natalie Villasana
8d7944bcf8
new node selection in overlay ( #1136 )
2019-01-29 14:42:43 -05:00
Egon Elbre
1df81b1460
Separate garbage collect logic from psdb ( #1167 )
2019-01-29 17:41:01 +02:00
Egon Elbre
54eb1b099c
don't display message unless logger has been specified ( #1165 )
2019-01-29 14:12:24 +02:00
Egon Elbre
e1a8bbdcb6
Kademlia flags cleanup ( #1137 )
2019-01-29 08:51:07 +02:00
Dylan Lott
bb2588e9ae
Refactors kademlia inspector to use FindNear ( #1157 )
...
* WIP
* update identity paths to get dump-nodes working
* Remove commented out code
* remove comment
* updates from code reviews
* updates linter
2019-01-28 15:53:37 -07:00
Bill Thorp
60946c2024
make bandwidth agreements sensible: without []byte's ( #1152 )
...
removed []byte's from bandwidth agreement protocol buffers
2019-01-28 14:45:25 -05:00
Dylan Lott
df903ea215
Decreases refresh time per bucket in Kademlia ( #1150 )
2019-01-28 12:36:20 -07:00
Egon Elbre
d50c07e56c
Implement WorkGroup ( #1151 )
2019-01-28 21:04:42 +02:00
Egon Elbre
cecd4b0816
Remove server aliases ( #1154 )
2019-01-28 17:04:53 +02:00
Bryan White
2b20acbec9
identity cleanup ( #1145 )
2019-01-26 15:59:53 +01:00
Egon Elbre
49dacb662c
Improve bwagreement tests ( #1149 )
2019-01-26 01:06:38 +02:00
Egon Elbre
f7e18882c4
Kademlia closing race ( #1139 )
2019-01-26 00:33:20 +02:00
Bill Thorp
8ba4b61e59
Make Bandwidth Agreements Secure / Trustable ( #1117 )
...
* Added cert chains and nodeid verification to bandwidth agreement
2019-01-25 13:05:21 -05:00
Bryan White
7bed8050aa
Improve identity cli ux: ( #1142 )
2019-01-25 17:55:45 +01:00
Jennifer Li Johnson
5397efe5c1
removes standalone payments service ( #1134 )
...
* remove standalone payments service - will integrate within satellite directly
* regenerate proto files
2019-01-25 11:23:25 -05:00
Egon Elbre
85b43926b4
Separate identity from server config ( #1138 )
2019-01-25 16:54:54 +02:00
Egon Elbre
a1027c7836
Make inmemory a separate command ( #1127 )
2019-01-24 23:04:29 +02:00
Egon Elbre
78b0414e6c
internal/sync2: fix file handle leak ( #1124 )
2019-01-24 22:48:54 +02:00
Cameron
f5b1e77cf2
rollup timestamp fix ( #1132 )
...
* create timestamp if isNew
* pass isNil into SaveRollup
2019-01-24 15:41:22 -05:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 ( #1133 )
2019-01-24 15:15:10 -05:00
Jennifer Li Johnson
030d294399
adds logs in rollup and tally run ( #1131 )
2019-01-24 15:05:53 -05:00
Jennifer Li Johnson
70940b8710
Making sure payments works ( #1122 )
...
* byte hours
* updates comment
* removes payment test data methods
* wip
* adds tally and rollup to peer
* remove sim test for payment (for now)
* lint error
2019-01-24 13:44:14 -05:00
Alexander Leitner
c4afec9328
quick log cleanup for storage node ( #1121 )
...
* Clean up logs
2019-01-24 11:09:37 -05:00
Egon Elbre
28427e7fad
Change identity command ( #1128 )
2019-01-24 17:41:16 +02:00
Egon Elbre
5de7f8af7f
Satellite Peer ( #1119 )
2019-01-23 21:58:44 +02:00
Cameron
95d2d54fc3
storagenode approvedSatelliteIDs ( #1116 )
...
* add config fields for satellite restriction on psserver
* add whitelistedSatIDs to psserver Server struct
* check pbwa satellite ID against whitelist
* add whitelist to psserver tests
* reword help message, make approved() a method on server
2019-01-23 12:56:12 -05:00
Michal Niewrzal
3fdb47e31c
Try accessing bootstrap node for verification ( #1089 )
2019-01-23 17:48:46 +02:00
Egon Elbre
8677c68b68
Print successfully connected message every 100 times ( #1118 )
2019-01-23 14:07:22 +02:00
Egon Elbre
05b96f95a3
Change default difficulty to 30 ( #1113 )
2019-01-23 13:36:19 +02:00
Egon Elbre
b6c61cdd55
Use storagenode.Peer for storagenode ( #1107 )
2019-01-23 12:39:03 +02:00
Egon Elbre
99d3b7a3c8
Fix import grouping ( #1111 )
2019-01-22 17:48:23 +02:00
Egon Elbre
ad49f4b994
Enable linters ( #1105 )
2019-01-22 17:09:02 +02:00
Bryan White
322a2813f7
oops ( #1110 )
2019-01-22 15:34:40 +01:00
Egon Elbre
6c48d69c38
Disable mounting for bsd-like systems ( #1109 )
...
* Disable mounting for bsd-like
* Remove debug code
2019-01-22 15:28:57 +02:00
Bryan White
8edfd43abb
certificates config and --certs-dir fixes ( #1093 )
2019-01-22 13:35:48 +01:00
Michal Niewrzal
eb2d858ffc
Use pointerdb.Service instead of pointerdb.Server ( #1099 )
2019-01-19 13:58:53 -05:00
Alexander Leitner
aabcceef52
Don't create identity when running dash ( #1098 )
...
* Don't create identity when running dash
* only use configs for connecting with dash, delete unused client example
* use server address
2019-01-18 18:08:45 +01:00
Cameron
ecd704692e
wip testing and timestamp fix ( #1100 )
...
* wip testing and timestamp fix
* fixed import cycle
2019-01-18 11:53:23 -05:00
Michal Niewrzal
cc1bdef8b7
Move pointerdb logic to a service ( #1037 )
...
* test
* Move pointerdb logic to service
* tset
* reorder constructor params
* restore field
2019-01-18 10:10:21 -05:00
JT Olio
2d53a33bc5
pkg/node: fix connection logging ( #1096 )
...
* pkg/node: fix connection logging
Change-Id: Iac552ac5eb126642fc1d01bae5f44720fafad536
* debug logging
Change-Id: Ibb95bc7baf1d4d0f07640ae807b2eb2f943aaab0
2019-01-18 10:03:19 -05:00
Egon Elbre
2d00f882a3
Kademlia Dialer ( #1092 )
2019-01-18 10:00:56 -05:00
Egon Elbre
78dc02b758
Satellite Peer ( #1034 )
...
* add satellite peer
* Add overlay
* reorganize kademlia
* add RunRefresh
* add refresh to storagenode.Peer
* add discovery
* add agreements and metainfo
* rename
* add datarepair checker
* add repair
* add todo notes for audit
* add testing interface
* add into testplanet
* fixes
* fix compilation errors
* fix compilation errors
* make testplanet run
* remove audit refrences
* ensure that audit tests run
* dev
* checker tests compilable
* fix discovery
* fix compilation
* fix
* fix
* dev
* fix
* disable auth
* fixes
* revert go.mod/sum
* fix linter errors
* fix
* fix copyright
* Add address param for SN dashboard (#1076 )
* Rename storj-sdk to storj-sim (#1078 )
* Storagenode logs and config improvements (#1075 )
* Add more info to SN logs
* remove config-dir from user config
* add output where config was stored
* add message for successful connection
* fix linter
* remove storage.path from user config
* resolve config path
* move success message to info
* log improvements
* Remove captplanet (#1070 )
* pkg/server: include production cert (#1082 )
Change-Id: Ie8e6fe78550be83c3bd797db7a1e58d37c684792
* Generate Payments Report (#1079 )
* memory.Size: autoformat sizes based on value entropy (#1081 )
* Jj/bytes (#1085 )
* run tally and rollup
* sets dev default tally and rollup intervals
* nonessential storj-sim edits (#1086 )
* Closing context doesn't stop storage node (#1084 )
* Print when cancelled
* Close properly
* Don't log nil
* Don't print error when closing dashboard
* Fix panic in inspector if ping fails (#1088 )
* Consolidate identity management to identity cli commands (#1083 )
* Consolidate identity management:
Move identity cretaion/signing out of storagenode setup command.
* fixes
* linters
* Consolidate identity management:
Move identity cretaion/signing out of storagenode setup command.
* fixes
* sava backups before saving signed certs
* add "-prebuilt-test-cmds" test flag
* linters
* prepare cli tests for travis
* linter fixes
* more fixes
* linter gods
* sp/sdk/sim
* remove ca.difficulty
* remove unused difficulty
* return setup to its rightful place
* wip travis
* Revert "wip travis"
This reverts commit 56834849dcf066d3cc0a4f139033fc3f6d7188ca.
* typo in travis.yaml
* remove tests
* remove more
* make it only create one identity at a time for consistency
* add config-dir for consitency
* add identity creation to storj-sim
* add flags
* simplify
* fix nolint and compile
* prevent overwrite and pass difficulty, concurrency, and parent creds
* goimports
2019-01-18 08:54:08 -05:00
Dylan Lott
7bc1944a08
Dashboard updates ( #1087 )
2019-01-18 05:41:25 -05:00
Egon Elbre
bbf81f2479
Consolidate identity management to identity cli commands ( #1083 )
...
* Consolidate identity management:
Move identity cretaion/signing out of storagenode setup command.
* fixes
* linters
* Consolidate identity management:
Move identity cretaion/signing out of storagenode setup command.
* fixes
* sava backups before saving signed certs
* add "-prebuilt-test-cmds" test flag
* linters
* prepare cli tests for travis
* linter fixes
* more fixes
* linter gods
* sp/sdk/sim
* remove ca.difficulty
* remove unused difficulty
* return setup to its rightful place
* wip travis
* Revert "wip travis"
This reverts commit 56834849dcf066d3cc0a4f139033fc3f6d7188ca.
* typo in travis.yaml
* remove tests
* remove more
* make it only create one identity at a time for consistency
* add config-dir for consitency
* add identity creation to storj-sim
* add flags
* simplify
* fix nolint and compile
* prevent overwrite and pass difficulty, concurrency, and parent creds
* goimports
2019-01-18 11:36:58 +01:00
Alexander Leitner
944daeab02
Closing context doesn't stop storage node ( #1084 )
...
* Print when cancelled
* Close properly
* Don't log nil
* Don't print error when closing dashboard
2019-01-17 16:08:05 -05:00
Jennifer Li Johnson
c4b876ddc5
Jj/bytes ( #1085 )
...
* run tally and rollup
* sets dev default tally and rollup intervals
2019-01-17 14:39:32 -05:00
JT Olio
a255e385df
memory.Size: autoformat sizes based on value entropy ( #1081 )
2019-01-17 20:47:57 +02:00
Jennifer Li Johnson
e6fbf63620
Generate Payments Report ( #1079 )
2019-01-17 13:34:13 -05:00
JT Olio
b2e69d847f
pkg/server: include production cert ( #1082 )
...
Change-Id: Ie8e6fe78550be83c3bd797db7a1e58d37c684792
2019-01-17 12:36:45 -05:00
Michal Niewrzal
f9abb4584c
Storagenode logs and config improvements ( #1075 )
...
* Add more info to SN logs
* remove config-dir from user config
* add output where config was stored
* add message for successful connection
* fix linter
* remove storage.path from user config
* resolve config path
* move success message to info
* log improvements
2019-01-16 21:41:43 -05:00
Dylan Lott
82b9b54695
storagenode dashboard is working and passing tests locally ( #1072 )
...
* storagenode dashboard is working and passing tests locally
* linter fixes
* linter fixes
* moar linter fixes god
* review fixes and updates
* linter fixes
2019-01-16 16:09:13 -05:00
paul cannon
a2333c4809
trivial: log bootstrap node address used to DEBUG ( #1063 )
2019-01-16 15:43:00 -05:00
Bill Thorp
342dc857f5
rollup query ( #1056 )
...
* implemention notes
* more notes
* starting rollup query
* not working yet
* fixed build
* fixed cfg bug
* change context cancelled errs to debugs
* using byte hours for at rest tally
* revert changes to go.mod
* comment fixes
* prevent double recording tallies in rollup
* linting
* stop leaking dbx
* nodeid changes
* fix build
2019-01-16 14:30:33 -05:00
paul cannon
b3713b7d92
trivial: log "connection to node failed" at DEBUG level ( #1064 )
...
instead of ERROR, because errors contacting other nodes in the network
are entirely normal and expected, and they don't need urgent attention.
2019-01-16 13:01:59 -05:00
Michal Niewrzal
01e4811a6f
Fix ignored overridden flag in user config ( #1059 )
...
* Fix ignored overridden flag in user config
* rename variable
2019-01-15 16:03:24 -05:00
Egon Elbre
f8906ce000
Change overlay cache database interface ( #1047 )
2019-01-15 11:08:45 -05:00
Michal Niewrzal
ab0c18de3a
Fix SN overrided params ( #1049 )
...
* Fix SN overrided params
* remove one more override
* fix missing var
* overriding default for storage node
* restore removed code
* fix docker
2019-01-15 10:43:10 -05:00
paul cannon
248ee6438f
use Node ID for metrics ID ( #1052 )
...
This change removes automatic metrics reporting for everything going
through process.Exec(), and re-adds metrics reporting for those commands
which are expected to be long-lived. Other commands (which may have been
intermittently sending metrics before this, if they ran unusually long)
will no longer send any metrics.
For commands where it makes sense, a node ID is used as the metrics ID.
2019-01-15 10:02:54 -05:00
Kaloyan Raev
a63abf8fab
Save only user-specific flags to storage node's config.yaml ( #1051 )
2019-01-15 15:55:33 +02:00
Yehor Butko
e38cf8f50d
Renaming and moving pkg/satellite to satellite/console ( #1054 )
...
* [WIP] V3-853 Merge the satellite DB into the master database
* Removing consoleDB from satelliteDB
* Fixing tests for satellite/console
* fixing linter
* sorting imports in satellite/console
* fixing console config
* fixing linter
2019-01-15 15:03:24 +02:00
littleskunk
2aebbeed34
add more bwagreement unit tests ( #1046 )
...
* add more bwagreement unit tests
2019-01-15 05:44:56 +01:00
Alexander Leitner
bfde515391
Clean up Storage node setup ( #1013 )
...
* Edit config on Setup
* Default to 1TiB storage space and 500GiB bandwidth
* Use human readable formats
* Use memory
* units of 1024 are measured with KiB/MiB etc
* pkg/cfgstruct: allow values to be configured with human readable sizes
Change-Id: Ic4e9ae461516d1d26fb81f6e44c5ac5cfccf777f
* Modify tests
* Removed comments
* More merge conflict stuff resolved
* Fix lint
* test fixin
Change-Id: I3a008206bf03a4446da19f642a2f9c1f9acaae36
* Remove commented code but secretly leave it in the histroy forever
* Move flag definition to struct
2019-01-14 16:19:15 -05:00
JT Olio
87925789de
pkg/inspector: split functionality to be per-service ( #1029 )
...
now kad inspector features exist on every server that has
kademlia running. likewise, overlay and statdb.
this means kad inspection features are now available on
storage nodes
Change-Id: I343c873552341de13302bfb7a5d79cccc84fc6b8
2019-01-14 11:47:22 -07:00
Bill Thorp
8b82114b7f
agreement sender bug fix ( #1044 )
2019-01-11 19:31:34 -05:00
Bill Thorp
273c3924ca
continue agreement sender loop on error ( #1020 )
...
* continue agreement sender loop on error
* added REJECTED grpc status
2019-01-11 13:15:49 -05:00
Bryan White
9e1ec633d8
Improve cert signing integration test+: ( #997 )
2019-01-11 15:59:35 +01:00
