Commit Graph

417 Commits

Author SHA1 Message Date
Egon Elbre
94e79eda6d
remove overlay endpoint (#1521) 2019-03-23 10:06:11 +02:00
Jess G
9236ac4bdf
change default to true to enforce sa whitelist (#1549)
* change default to true to enable whitelist

* add devDefault
2019-03-22 07:21:16 -07:00
Egon Elbre
1d96d25f3f
kademlia ping tracking (#1538) 2019-03-22 15:27:59 +02:00
Kaloyan Raev
30dfc2b20c
Remove PointerDB client (#1520) 2019-03-22 11:01:49 +02:00
paul cannon
1d78ddc3df
Test that network stalls don't cause indefinite hangs in uplink (#1530)
* test-network-stalls tests... network stalls!

in particular, right now, it just tests whether an uplink correctly
times out after some amount of time when one of the nodes it's talking
to suddenly goes offline.

This tool is meant to be run under `storj-sim network test`.

Also included here:

* fix storj-sim-related test scripts on Mac

the default storj config dir on Mac has a space in it
('~/Library/Application Support/Storj'), which breaks everywhere it
shows up in an unquoted variable in a sh/bash script. easy enough to fix
as a one-off, but quoting bash vars avoids a dozen other potential
problems too.

change a few things using `head -c` to use `dd`. `head -c` works,
but is not as widely understood (as evidenced by each of these scripts
getting through code review, one at a time, with the comments not
matching the numbers actually used).

* storj-sim reports PIDs of worker processes to test

so that the tests can cause unfortunate "accidents" to befall the worker
processes in the course of the test, and find out whether everything
reacts correctly.
2019-03-20 08:58:07 -06:00
Yehor Butko
c6f8d82978
V3-1319 Add an authorization token field for account creation backend (#1492)
* V3-1319 Add an authorization token field for account creation on the satellite UI backend
2019-03-19 19:55:43 +02:00
Egon Elbre
636d73934e Fix storage node path handling (#1519) 2019-03-19 11:10:23 +02:00
Egon Elbre
05d148aeb5
Storage node and upload/download protocol refactor (#1422)
refactor storage node server
refactor upload and download protocol
2019-03-18 12:55:06 +02:00
Cameron
c7ffbe1c28
Add ability to view irreparable segments on satellite (#1448)
* define irreparable inspector protobuf

* add IrreparableDB method GetLimited

* fill out irreparable inspector API

* add IrreparableInspector server to satellite, fix small error

* refactor IrreparableDB to use pb.IrreparableSegment instead of irreparable.RemoteSegmentInfo
2019-03-15 16:21:52 -04:00
Alexander Leitner
af889f1554
Move iswriteable to fpath (#1464) 2019-03-12 12:13:40 -04:00
Alexander Leitner
bb77d9b4a6
Warn about permissions when creating identity (#1384)
* Warn about permissions when creating identity

* Function to determine if directory is writeable

* Check if writable before authorizing

* Remove redeclatarion

* remove windows specific utils

* Nat nits

* Actually test if directory is writeable with file creation
2019-03-12 10:42:38 -04:00
Bill Thorp
66718cc5e6
Development defaults for configuration (#1430)
added --dev command line option, cfgstruct.DevFlag(), and cfgstruct.SetupFlag()
2019-03-12 08:51:06 -04:00
Yehor Butko
3ab696164f
Satellite console new activation flow (#1421)
Satellite console new activation flow
2019-03-08 16:01:11 +02:00
Jess G
193a70f0a6
add private listener to grpc server (#1398)
* add private listener to grpc server

* add changes per init CR

* fix server.close

* add insecure grpc connection, update logs msg

* fix tests, move insecure client

* add private ports to storj-sim, add insecure client to other inspectors

* add ports to test so there arent conflicts

* fix lint err

* fix node started log msg, close public listener

* remove commented out line
2019-03-07 13:19:37 -05:00
Jess G
3c9d83dbfe
convert psserver dashboard into an inspector (#1407)
* Convert psserver dashboard into an inspector

* remove dashboard stream, update ps.pb.mock

* fixes for lint errs
2019-03-05 15:48:37 -05:00
Yehor Butko
3e2c101bd8
V3-1152 Node bootstrap web backend (#1327)
* V3-1152 Node bootstrap
2019-03-05 12:38:21 +02:00
Benjamin Sirb
7e2e4b5397 Update README.md (#1400) 2019-03-05 09:54:49 +02:00
Bryan White
675e0ef683 [V3-1320] fix empty node ID verification non-error (#1395)
* small identity refactor:

+ Optimize? iterative cert chain methods to use array instead of slice
+ Add `ToChain` helper for converting 1d to 2d cert chain
  TODO: replace literal declarations with this
+ rename `ChainRaw/RestChainRaw` to `RawChain/RawRestChain`
  (adjective noun, instead of nound adjective)

* add regression tests for V3-1320

* fix V3-1320

* separate `DialUnverifiedIDOption` from `DialOption`

* separate `PingNode` and `DialNode` from `PingAddress` and `DialAddress`

* update node ID while bootstrapping

* goimports & fix comment

* add test case
2019-03-04 15:03:33 -05:00
Yaroslav Vorobiov
a30ba4eca8
Add mail service to the satellite (#1302) 2019-03-02 17:22:20 +02:00
Kaloyan Raev
54f68347ad
Use DialNode for requesting node info (#1394) 2019-03-02 09:34:08 +02:00
Dylan Lott
9abf191b01
Change payments to use statDB for wallet addresses rather than Overlay cache (#1368)
* Wiring up DumpNodes response for Inspector

* Finalize everything and test that it works

* Get Count and DumpNodes working for Overlay Cache

* WIP updating payment rollup to check statDB instead of overlay

* FIrst pass at updating statDB to take wallet and email

* Passing tests

* use pb.NodeOperator instead of Meta struct

* remove TODO

* revert go.mod

* Get SQL migration working correctly

* Changes Meta to Operator in NodeStats struct

* Adds update operator logic for statDB

* Fix db migrate tests - added v5 snapshot

* User friendly msg for missing snapshot version

* Passing tests

* Change node update to happen in discovery instead of in overlay

* Fix logic and update function calls

* Update comment on UpdateOperator interface method

* Update name of parameter

* Change type of argument to UpdateOperator

* Updates statDB tests
2019-03-01 10:46:34 -07:00
Philip Hutchins
1f452e67dc Adding dockerfile for running the web UI for Satellite (#1366)
* Adding dockerfile for running the web UI for Satellite

* Updating to work with Makefile and from root directory of repo

* Updating satellite ui build process to run in a more production like mode by generating the assets the pulling those into the satellite container

* Updates to allow external traffic to UI, updates to storagenode for identity creation, and logging for bug tracking

* Adding auto cert generation for storagenode

* removing satellite-ui-image from main images flow in Makefile and adding latest tag to docker build for it

* Adding solid defaults, tuning dockerfiles, and moving to standard logging methods

* Updating logging to be more standard

* Updating to logger.Debug

* Removing unused library and unused identity creation code

Change-Id: I956453037e303693ea37f94318180af0ab7984d5
2019-02-28 13:18:53 -07:00
Cameron
018cdd4037
uplink ttl (#1365)
* add expires flag to uplink cp

* fix ttlDBs delete expired query
2019-02-28 12:51:24 -05:00
Jennifer Li Johnson
6ecda190cb
Jj/payments cleanup (#1379)
* Removes date from payments csv and re-query of wallet addr

* removes dbx get wallet addr

* switch order in csv
2019-02-28 12:13:59 -05:00
Jess G
e82d3367f0
rename payment command to node usage (#1378) 2019-02-27 16:55:19 -05:00
Egon Elbre
d0b5b9b51b
reorder graphite metric string for easier parsing (#1362) 2019-02-26 15:47:31 +02:00
Bryan White
03747bad51
some certificates cmd cleanup: (#1361) 2019-02-26 09:55:52 +01:00
Bryan White
62a3749139
fix claims delete config (#1360) 2019-02-25 22:29:19 +01:00
Kaloyan Raev
1ec17653d4 Endpoint for local node info (#1355)
Adds a new `Info` method to the Kademlia endpoint that returns the following local node info:
* ID
* Type
* Metadata (email and wallet)
* Restrictions (free storage and bandwidth)

The new endpoint is exposed as `inspector kad node-info` command too.
2019-02-25 19:41:51 +01:00
Michal Niewrzal
3512952656
[storj-sim] storage node must wait for satellite start (#1348) 2019-02-22 15:35:51 +01:00
JT Olio
6f3ef3f683 cmd/identity: set default CONFDIR (#1346)
this bandaid-fixes the identity tool issues (revocation
db defaults to /revocations.db due to the missing CONFDIR)

Change-Id: Ibdc8d1e9b64ec9a545bea592b0bd167ff5138117
2019-02-22 13:56:13 +01:00
Michal Niewrzal
8d685217e4
Storagenode migrations (#1299)
* creates initial migration for psdb
* add test mechanism to validate migration to every version
* fix few small issues in versions.go and context.go
2019-02-19 10:39:04 +01:00
Egon Elbre
46a19d5186 show nicer time in storage node dashboard (#1325) 2019-02-18 16:16:48 +01:00
Egon Elbre
12261c0624
Satellite migrations (#1301) 2019-02-14 23:55:21 +02:00
Bryan White
d93c57cffa add certificates claim delete & refactor (#1317)
* add \`certificates claim delete\` & refactor

* quickfix

* goimports
2019-02-14 20:17:26 +01:00
Michal Niewrzal
58792533b9
Fix invalid config key msg for dashbord (#1314) 2019-02-14 14:06:15 +01:00
JT Olio
2a59679766 pkg/transport: require tls configuration for dialing (#1286)
* separate TLS options from server options (because we need them for dialing too)
* stop creating transports in multiple places
* ensure that we actually check revocation, whitelists, certificate signing, etc, for all connections.
2019-02-11 13:17:32 +02:00
Yehor Butko
8156d911fa
Updating account activation flow (#1251)
* Updating account activation flow

* Updated integration tests, createUserMutation updated

* removing redundant index

* removed redundant testing code

* review comments fixed
2019-02-11 12:33:56 +02:00
Michal Niewrzal
c7ad427d95
Move metainfo config (#1234)
* Move metainfo config

* move config to uplink

* reorganize code

* use satellite apikey

* comment update
2019-02-08 13:57:35 +01:00
paul cannon
c35b93766d
Unite all cryptographic signing and verifying (#1244)
this change removes the cryptopasta dependency.

a couple possible sources of problem with this change:

 * the encoding used for ECDSA signatures on SignedMessage has changed.
   the encoding employed by cryptopasta was workable, but not the same
   as the encoding used for such signatures in the rest of the world
   (most particularly, on ECDSA signatures in X.509 certificates). I
   think we'll be best served by using one ECDSA signature encoding from
   here on, but if we need to use the old encoding for backwards
   compatibility with existing nodes, that can be arranged.

 * since there's already a breaking change in SignedMessage, I changed
   it to send and receive public keys in raw PKIX format, instead of
   PEM. PEM just adds unhelpful overhead for this case.
2019-02-07 14:39:20 -06:00
paul cannon
ef61c170b1
Consolidate key/cert/signature encoding and decoding (#1243) 2019-02-07 12:40:28 -06:00
Kaloyan Raev
0ffdaeebce
Disable colors in dashboard by default. --color flag to enable them (#1264) 2019-02-07 19:54:46 +02:00
paul cannon
32882daf38
SHA256Hash() and TBSBytes() needn't return error (#1242) 2019-02-07 11:08:52 -06:00
Stefan Benten
f5ab9e34e6 fix difficulty display (#1256) 2019-02-07 11:36:19 +02:00
paul cannon
0032147665 regroup things related to public-key cryptography (#1241) 2019-02-07 10:04:29 +01:00
Bryan White
3b54cf0e15
identity improvements: (#1215) 2019-02-06 17:40:55 +01:00
JT Olio
0c96353078
storagenode: fix port in entrypoint (#1252)
Change-Id: I07fa4cec447f2d6a3a3e308ad9b63bdc9dc1b161
2019-02-06 09:19:58 -07:00
Egon Elbre
60b0275a44
Enable ipv6 tests (#1204) 2019-02-06 14:47:00 +02:00
Bryan White
00554f0876
add certificates claims+ (#1216) 2019-02-06 12:00:26 +01:00
Bryan White
7b7e6c43f8
better batch-generation (#1219) 2019-02-06 09:04:12 +01:00
Yaroslav Vorobiov
2ff0d9d435
Add bucket to project relationship on satellite (#1143)
* integrate console api keys with buckets in pointerdb

* fix test

* fix tests kvmetainfo

* linter fix

* disable account activation

* fix test

* review fixes

* fix comments

* little refactoring

* remove debug println

* fix typo

* disable activation in a propper way

* fix test

* fix imports

* fix uplink count in testplanet

* move key creation to planet.newUplink
2019-02-05 19:22:17 +02:00
Michal Niewrzal
383cb05e8e
Divide uplink and gateway params set (#1026)
* Divide uplink and gateway params set

* attempt to fix docker

* attempt to fix all in one

* test

* more reorganization

* fix compilation error

* fix imports order

* fix dependency

* rename structs

* keep minio params for now

* review comments

* remove manual flag check
2019-02-04 16:50:06 +01:00
Bill Thorp
0f662b8e38 sql based tally (#1166)
* WIP

* wacky changes

* more

* it builds.... IT BUILDS!!!

* fixed SQL, broke out saving tallies for testing

* shorter lines

* fixed SQL, moved tally_test to testplanet

* lint

* WIP logic error preventing PUT and GETs to same serialnum

* fixed BWA test

* fixed temporary brain failure

* eliminated magic numbers

* cleaned up satellite uplink stats

* use errs.Combine instead

* thrashing

* fixed tally erroneous error msg

* fixed tally test

* lint

* SQL syntax attempt to fix

* spelling error

* made bwa db test resist old postgres data

* postgres pk error msg hunting

* postgres pk error msg hunting

* postgres

* err might be nil?

* fixed error logging bug

* hopefully solved postgres issue

* using rebind

* moved tests to _test package

* fixing test dirs

* finally made sense of Egons package name feedback

* UTC, array fixes
2019-02-01 13:50:12 -05:00
Egon Elbre
f7c692f844 Use base10 for all values (#1187) 2019-02-01 11:29:37 -07:00
Egon Elbre
4117600f32
Cleanup repository a bit (#1203) 2019-02-01 20:21:00 +02:00
Bryan White
2a88642c55 yea, that's really all it takes to fix this (#1202) 2019-01-31 22:57:14 -05:00
Kaloyan Raev
5ae47c0de4
V3-1127 Properly initialize default config dir in storagenode binary (#1193) 2019-01-31 18:57:44 +02:00
Cameron
1403b15cc7
payments accesses database directly (#1135)
* move payments command into satellite/main.go

* flag for db connection string in paymentsCmd

* refactor payments to satellite subcommand

* reports command, add payments arg descriptions

* report data prints to stdout unless --out is set

* fix small error in csv columns
2019-01-30 16:44:50 -05:00
Egon Elbre
d5346982c2
Delete provider package (#1177) 2019-01-30 22:47:21 +02:00
Alexander Leitner
7c741c16d1
Readable sizes in dashboard (#1180)
* Readable sizes

* Use tabwriter

* Added more addresses to dashboard stats

* Use kad to look up addresses

* Move dashboard command to dashboard.go
2019-01-30 14:49:25 -05:00
Egon Elbre
471ed4ab2c
Remove empty file (#1175) 2019-01-30 12:40:53 +02:00
Jennifer Li Johnson
6355ae7a35
closes db in satellite, storage node and bootstrap cmdRun (#1159) 2019-01-30 00:22:58 -05:00
Egon Elbre
6132ce86b7
Remove utils.LogClose (#1169) 2019-01-29 22:42:27 +02:00
Egon Elbre
1df81b1460
Separate garbage collect logic from psdb (#1167) 2019-01-29 17:41:01 +02:00
Egon Elbre
f49436268a
storj-sim: read in config values for gateway (#1162) 2019-01-29 16:23:30 +02:00
Dylan Lott
bb2588e9ae
Refactors kademlia inspector to use FindNear (#1157)
* WIP

* update identity paths to get dump-nodes working

* Remove commented out code

* remove comment

* updates from code reviews

* updates linter
2019-01-28 15:53:37 -07:00
Bill Thorp
60946c2024
make bandwidth agreements sensible: without []byte's (#1152)
removed []byte's from bandwidth agreement protocol buffers
2019-01-28 14:45:25 -05:00
Egon Elbre
04d3699107
storj-sim configs (#1140) 2019-01-28 16:48:49 +02:00
Timothy Adams
2dbc06f7e1
fixing entrypoint script looking for invalid flag (#1147)
* fixing entrypoint script looking for invalid flag

* fixed other entrypoints

* removed ca_difficulty entries from compose.yaml
2019-01-28 07:53:16 -05:00
Bryan White
2b20acbec9
identity cleanup (#1145) 2019-01-26 15:59:53 +01:00
Dylan Lott
1385a2d6bd
Storagenode online confirmation ping (#1144)
* initial commit of dashboard connection tools

* Currently working against default configs

* Update configs and remove hardcoded addresses

* Update config handling

* adds external address param for tunneling nodes

* remove unnecessary address check logic
2019-01-25 10:46:09 -07:00
Bryan White
7bed8050aa
Improve identity cli ux: (#1142) 2019-01-25 17:55:45 +01:00
Jennifer Li Johnson
5397efe5c1
removes standalone payments service (#1134)
* remove standalone payments service - will integrate within satellite directly

* regenerate proto files
2019-01-25 11:23:25 -05:00
Egon Elbre
85b43926b4
Separate identity from server config (#1138) 2019-01-25 16:54:54 +02:00
Egon Elbre
a1027c7836
Make inmemory a separate command (#1127) 2019-01-24 23:04:29 +02:00
Egon Elbre
187e9b2138
Code consistency between peers (#1126) 2019-01-24 22:28:06 +02:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Jennifer Li Johnson
70940b8710
Making sure payments works (#1122)
* byte hours

* updates comment

* removes payment test data methods

* wip

* adds tally and rollup to peer

* remove sim test for payment (for now)

* lint error
2019-01-24 13:44:14 -05:00
JT Olio
d87aa11fd0
Identity cleanups (#1129)
* identity: fix unexpected control-c nil-dereference stacktrace

Change-Id: I3366c26908736f96b3c0828d0fd7b5586f3cc2c2

* identity: newlines

Change-Id: Ia31b1a0066b7f9d84043f3fc5669c6efc710b546

* identity: make default signer address

Change-Id: Ibb051397756cb76721b98851fcfa3d9e62b580db
2019-01-24 10:23:45 -07:00
Yaroslav Vorobiov
f437effaef
Add console to satellite peer (#1120) 2019-01-24 18:26:36 +02:00
Egon Elbre
28427e7fad
Change identity command (#1128) 2019-01-24 17:41:16 +02:00
Egon Elbre
5de7f8af7f
Satellite Peer (#1119) 2019-01-23 21:58:44 +02:00
Michal Niewrzal
3fdb47e31c Try accessing bootstrap node for verification (#1089) 2019-01-23 17:48:46 +02:00
Egon Elbre
05b96f95a3
Change default difficulty to 30 (#1113) 2019-01-23 13:36:19 +02:00
Egon Elbre
b6c61cdd55
Use storagenode.Peer for storagenode (#1107) 2019-01-23 12:39:03 +02:00
Egon Elbre
99d3b7a3c8
Fix import grouping (#1111) 2019-01-22 17:48:23 +02:00
Bryan White
322a2813f7
oops (#1110) 2019-01-22 15:34:40 +01:00
Egon Elbre
6c48d69c38
Disable mounting for bsd-like systems (#1109)
* Disable mounting for bsd-like
* Remove debug code
2019-01-22 15:28:57 +02:00
Bryan White
8edfd43abb
certificates config and --certs-dir fixes (#1093) 2019-01-22 13:35:48 +01:00
Alexander Leitner
aabcceef52 Don't create identity when running dash (#1098)
* Don't create identity when running dash

* only use configs for connecting with dash, delete unused client example

* use server address
2019-01-18 18:08:45 +01:00
Dennis Coyle
369ca7d6fe
Coyle/cert inspector (#1094)
* change default BootstrapAddr

* load cert files

* goimports

* linting gods appeased
2019-01-18 10:02:35 -05:00
paul cannon
d6c2737521
fix some email list parsing in certificates tool (#1095)
this avoids getting an "userID cannot be empty" error when using an
emails list that has a newline at the end.
2019-01-18 10:01:55 -05:00
Dylan Lott
7bc1944a08 Dashboard updates (#1087) 2019-01-18 05:41:25 -05:00
Egon Elbre
bbf81f2479 Consolidate identity management to identity cli commands (#1083)
* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* linters

* Consolidate identity management:

Move identity cretaion/signing out of storagenode setup command.

* fixes

* sava backups before saving signed certs

* add "-prebuilt-test-cmds" test flag

* linters

* prepare cli tests for travis

* linter fixes

* more fixes

* linter gods

* sp/sdk/sim

* remove ca.difficulty

* remove unused difficulty

* return setup to its rightful place

* wip travis

* Revert "wip travis"

This reverts commit 56834849dcf066d3cc0a4f139033fc3f6d7188ca.

* typo in travis.yaml

* remove tests

* remove more

* make it only create one identity at a time for consistency

* add config-dir for consitency

* add identity creation to storj-sim

* add flags

* simplify

* fix nolint and compile

* prevent overwrite and pass difficulty, concurrency, and parent creds

* goimports
2019-01-18 11:36:58 +01:00
Kaloyan Raev
82f5bb6072 Fix panic in inspector if ping fails (#1088) 2019-01-17 16:14:34 -05:00
Natalie Villasana
3bf7c7d43b
nonessential storj-sim edits (#1086) 2019-01-17 15:59:26 -05:00
Jennifer Li Johnson
c4b876ddc5
Jj/bytes (#1085)
* run tally and rollup

* sets dev default tally and rollup intervals
2019-01-17 14:39:32 -05:00
Jennifer Li Johnson
e6fbf63620
Generate Payments Report (#1079) 2019-01-17 13:34:13 -05:00
Egon Elbre
c89609abbe
Remove captplanet (#1070) 2019-01-17 04:35:22 -05:00
Michal Niewrzal
f9abb4584c
Storagenode logs and config improvements (#1075)
* Add more info to SN logs

* remove config-dir from user config

* add output where config was stored

* add message for successful connection

* fix linter

* remove storage.path from user config

* resolve config path

* move success  message to info

* log improvements
2019-01-16 21:41:43 -05:00