Commit Graph

635 Commits

Author SHA1 Message Date
Michal Niewrzal
ead8af3c16
storagenode/updater: recover command in Service Manager (#3425) 2019-11-15 06:37:29 -08:00
Egon Elbre
ee6c1cac8a
private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00
Michal Niewrzal
4912922631
cmd: skeleton for segment-reaper detect command (#3562) 2019-11-14 07:57:27 -08:00
Bryan White
b56cc21710 cmd/storagenode-updater: make updater test windows compatible (#3542) 2019-11-14 16:03:49 +02:00
Egon Elbre
1e64006e32 lint: add staticcheck as a separate step (#3569) 2019-11-14 10:31:30 +02:00
Bryan White
4c822b630d {certificates,pkg/rpcstatus}: improve error logging (#3475) 2019-11-13 11:07:21 +01:00
Bryan White
7cc4217fef
cmd/storagenode-updater: simplify and reorder update sqeuence (#3487) 2019-11-12 13:31:57 +01:00
Kaloyan Raev
20623fdc96
Increase min required difficulty to 36 in signing service (#3535) 2019-11-11 12:27:09 +02:00
Bryan White
53a060cdfa
cmd/storagenode-updater: TODO cleanup (#3486) 2019-11-07 09:42:07 +01:00
Caleb Case
3b78addb2d
Metadata Access from Uplink CLI (#3310) 2019-11-06 11:04:12 -05:00
Michal Niewrzal
2f13c0adf3
storj-sim: add rollout seed to avoid warnings (#3508) 2019-11-06 06:06:10 -08:00
littleskunk
7eb6724c92
logging: unify logging around satellite ID, node ID and piece ID (#3491)
* logging: unify logging around satellite ID, node ID and piece ID

* unify segment index
2019-11-05 22:04:07 +01:00
Yaroslav Vorobiov
35edc2bcc3 satellite/payments: invoice creation (#3468) 2019-11-05 15:16:02 +02:00
Jess G
5abb91afcf
satellite: change the Peer name to Core (#3472)
* change satellite.Peer name to Core

* change to Core in testplanet

* missed a few places

* keep shared stuff in peer.go to stay consistent with storj/docs
2019-11-04 11:01:02 -08:00
Jennifer Li Johnson
0621830f05 cmd/storj-sim: allow user to designate which redis db they want to start at (#3458) 2019-11-04 14:55:02 +02:00
Kaloyan Raev
84fea5820f
Fix error when storagenode-updater service is stopped (#3440) 2019-11-04 11:22:08 +02:00
Kaloyan Raev
857c7f3cd7
storagenode/updater: disable self-autoupdate (#3459) 2019-11-04 09:33:39 +02:00
Jess G
8d92c288e2
satellitedb: separate migration into subcommand (#3436)
* separate sadb migration, add version check

* update checkversion to do same validation as migration

* changes per CR

* add sa migration to storj-sim

* add different debug port in storj-sim for migration

* add wait for exit for storj-sim migration

* update sa docker entrypoint to support migration

* storj-sim satellite parts all wait for migration

* upgrade golang-migrate/migrate to v4 because bug

* fix go mod tidy
2019-11-02 13:09:07 -07:00
Egon Elbre
be2dd1ca72 cmd/storj-sim: add --redis flag (#3451) 2019-11-01 20:48:37 +01:00
Jennifer Li Johnson
76b64b79ba
cmd/identity: allow using redis for RevocationDB (#3259) 2019-11-01 13:27:47 -04:00
Kaloyan Raev
3c35339f02 Increase default difficulty in identity CLI to 36 (#3428) 2019-10-31 16:49:24 +01:00
Michal Niewrzal
015350e230
storagenode-updater: add autoupdating (#3422) 2019-10-31 05:27:53 -07:00
Jess G
e96d615013
satellite: remove satellite API code from peer (#3414)
* rm dup api code from sa peer, update storj-sim

* fix for backwards compat tests

* use env var instead of localhost

* changes per CR

* fix env var name

* skip peer for setup
2019-10-30 12:23:09 -07:00
Egon Elbre
95cd3fc750 cmd/storj-sim: use less ports (#3412) 2019-10-29 11:25:25 -04:00
Cameron
b2ff13f1fa
{cmd/satellite, storj/satellite}: create command to run repair process in isolation (#3341)
* set up satellite repair run command

* add separated repair process to storj-sim

* add repairer peer to satellite in testplanet

* move api run cmd into api.go

* add satellite run repair to entrypoint
2019-10-29 10:55:57 -04:00
Egon Elbre
93353df4d6
internal/sync2: make Fence accept context (#3393) 2019-10-28 16:04:31 +02:00
Bryan White
f17b34d02e
cmd/storagenode-updater: add "...up to date" log line if shoudn't update (#3392) 2019-10-28 11:19:19 +01:00
Kaloyan Raev
b10d444560
Rename --log-path back to --log for storagenode-updater (#3391) 2019-10-28 10:30:27 +02:00
paul cannon
1469f7f41f
storagenode/contact: wait for UpdateSelf before start (#3332)
When the contact chore starts running before the monitor service has
provided any useful capacity data, the first outgoing contact has
not-very-helpful data for the satellite. This change causes the contact
chore to wait until capacity data is available. The wait should be quite
short in all reasonable cases: even when a node starts with a lot of
stored pieces and no cached spaceUsedDB data, new data will have been
calculated and cached by the call to
`peer.Storage2.CacheService.Init(ctx)` in `storagenode.cmdRun()` before
`peer.Run(ctx)`.

Change-Id: Ibc26d5c1fc10a23006c00bc3f13ff6cf71f8bf1d
2019-10-26 12:16:25 -05:00
Maximillian von Briesen
abb567f6ae
cmd/satellite: add graceful exit reports command to satellite CLI (#3300)
* update lock file and add comment

* add created at and bytes transferred

* cleanup

* rename db func to GetGracefulExitNodesByTimeFrame

* fix flag

* split into two overlay functions

* := to =

* fix test

* add node not found error class

* fix overlay test

* suggested test changes

* review suggestions

* get exit status from overlay.Get()

* check rows.Err

* fix panic when ExitFinishedAt is nil

* fix comments in cmdGracefulExit
2019-10-22 21:06:01 -04:00
JT Olio
2c6fa3c5f8
pkg/rpc: remove read/write deadlines as a mechanism for request timeouts (#3335)
libuplink was incorrectly setting timeouts to 10 seconds still, but
should have been at least 10 minutes. the order sender was setting them
to 1 hour. we don't want timeouts in uplink-side logic as it establishes
a minimum rate on tcp streams.

instead of all of this, just use tcp keep alive. tcp keep alive packets are
sent every 15 seconds and if the peer stops responding the connection
dies. this is enabled by default with go. this will kill tcp connections
when they stop working.

Change-Id: I3d7ad49f71950b3eb43044eedf4b17993116045b
2019-10-22 17:57:24 -06:00
Jess G
1014d5a7d1
cmd/satellite: add API run command to satellite dockerfile entrypoint (#3319)
* update satellite entrypoint to add API run command

* make true a string for clarity
2019-10-21 09:16:59 -07:00
Bryan White
f468816f13
{internal/version,versioncontrol,cmd/storagenode-updater}: add rollout to storagenode updater (#3276) 2019-10-21 12:50:59 +02:00
Bryan White
243ba1cb17
{versioncontrol,internal/version,cmd/*}: refactor version control (#3253) 2019-10-20 09:56:23 +02:00
Egon Elbre
89ed997706
satellite/satellitedb: switch to postgres only (#3320) 2019-10-18 22:03:10 +03:00
Egon Elbre
44bf98ee9c
cmd/storj-sim: make postgres default to STORJ_SIM_POSTGRES (#3317) 2019-10-18 19:12:30 +03:00
Yehor Butko
26cc625dc6
satellite/console: payments api (#3297) 2019-10-17 17:42:18 +03:00
Jess G
34764e5c97
cmd/satellite: create API subcommand (#3280)
* create sa api subcommand

* add api to storj-sim

* require postgres

* pass live accounting cache into satellite.NewAPI
2019-10-16 13:34:25 -07:00
Cameron
76ad83f12c
satellite/accounting: add redis support to live accounting (#3213)
* set up redis support in live accounting

* move live.Service interface into accounting package and rename to Cache, pass into satellite

* refactor Cache to store one int64 total, add IncrBy method to redis client implementation

* add monkit tracing to live accounting
2019-10-16 12:50:29 -04:00
Yingrong Zhao
87e3764390
storagenode/cmd: add exit-status command for graceful exit (#3264)
* add exit-status command

* remove todo and fix format

* fix status display

* change startExit to exit progress

* fix linting error

* add successful column in exit progress

* fix test

* remove extra new line

* fix TYPOS

* format the percentage better
2019-10-15 18:07:32 -04:00
Jennifer Li Johnson
b185dbbee2
satellite/discovery: remove discovery related code (#3175) 2019-10-14 10:57:01 -04:00
Yingrong Zhao
743a0fc38b storagenode/cmd: create start graceful exit CLI (#3202) 2019-10-11 09:58:12 -04:00
Egon Elbre
e9c36d560f
satellite: make PointerDB an argument to satellite.New (#3233) 2019-10-10 21:06:26 +03:00
Bill Thorp
9cbb0c4374
cmd/storj-sim: expose GATEWAY_X_API_KEY (#3225)
* expose API_KEY
2019-10-09 14:33:53 -04:00
Cameron
10b364a2da
cmd/storagenode: enable migration of configs of different types (#3189)
* move deprecated flags code to deprecated.go, refactor to allow migration of other flags
* hide deprecated flags
2019-10-08 14:26:53 -04:00
JT Olio
37491d0d32 storagenode: embed the console into the binary and makefile (#3164)
* web/storagenode: add package-lock.json
* storagenode: compile console into binary
2019-10-08 10:52:19 +02:00
Jennifer Li Johnson
7ceaabb18e
Delete Bootstrap and Kademlia (#2974) 2019-10-04 16:48:41 -04:00
Michal Niewrzal
0911b7d1fb
uplink/gateway: wizard message update (#3150) 2019-10-04 07:43:26 -07:00
littleskunk
b2e328f118 storagenode/dashboard: update online status (#3168) 2019-10-03 20:31:39 +02:00
Jennifer Li Johnson
755cbd4dce
storagenode/main: map aliases for kademlia config values (#3118) 2019-09-30 19:33:00 -04:00
Bryan White
c8aa821ccb
pkg/certificates: move certificate package to root (#3107) 2019-09-26 09:11:05 -07:00
Jeff Wendling
098cbc9c67 all: use pkg/rpc instead of pkg/transport
all of the packages and tests work with both grpc and
drpc. we'll probably need to do some jenkins pipelines
to run the tests with drpc as well.

most of the changes are really due to a bit of cleanup
of the pkg/transport.Client api into an rpc.Dialer in
the spirit of a net.Dialer. now that we don't need
observers, we can pass around stateless configuration
to everything rather than stateful things that issue
observations. it also adds a DialAddressID for the
case where we don't have a pb.Node, but we do have an
address and want to assert some ID. this happened
pretty frequently, and now there's no more weird
contortions creating custom tls options, etc.

a lot of the other changes are being consistent/using
the abstractions in the rpc package to do rpc style
things like finding peer information, or checking
status codes.

Change-Id: Ief62875e21d80a21b3c56a5a37f45887679f9412
2019-09-25 15:37:06 -06:00
Egon Elbre
e0d5cbbbd7
jenkins: update golangci to v1.19.1 (#3119) 2019-09-25 19:35:56 +03:00
Michal Niewrzal
366a0be6aa
storagenode: avoid starting command on service stop (#3105) 2019-09-24 12:17:22 -07:00
Kaloyan Raev
8d16349b4c
storagenode/winsvc: avoid the 15s sleep when stopping the service (#3098) 2019-09-24 13:01:36 -06:00
Bryan White
515799267f fix certificates auth export command (#3110)
* fix certificates auth export command

* actually fix command config logic

* fix test-certificates.sh

* simplify
2019-09-24 10:38:18 -06:00
Isaac Hess
580e511b4c
storagenode/storagenodedb: Migrate to separate dbs (#3081)
* storagenode/storagenodedb: Migrate to separate dbs

* storagenode/storagenodedb: Add migration to drop versions tables

* Put drop table statements into a transaction.

* Fix CI errors.

* Fix CI errors.

* Changes requested from PR feedback.

* storagenode/storagenodedb: fix tx commit
2019-09-23 12:36:46 -07:00
Jennifer Li Johnson
d2502bb51b Adds tests for kad replacement and restores kad operator configs (#3094)
* test that all nodes can check in with all satellites

* keep kademlia config

* add untrusted satellite test

* use getversion

* remove kademlia config changes in test-sim-backwards.sh

* add kademlia flags back to storj-sim storagenode

* reset kademlia flags in storagenode entrypoint
2019-09-20 16:02:23 -04:00
Michal Niewrzal
68d281db44 autoupdater: use blang/semver (#3063) 2019-09-20 17:22:40 +03:00
Jennifer Li Johnson
724bb44723
Remove Kademlia dependencies from Satellite and Storagenode (#2966)
What:

cmd/inspector/main.go: removes kad commands
internal/testplanet/planet.go: Waits for contact chore to finish
satellite/contact/nodesservice.go: creates an empty nodes service implementation
satellite/contact/service.go: implements Local and FetchInfo methods & adds external address config value
satellite/discovery/service.go: replaces kad.FetchInfo with contact.FetchInfo in Refresh() & removes Discover()
satellite/peer.go: sets up contact service and endpoints
storagenode/console/service.go: replaces nodeID with contact.Local()
storagenode/contact/chore.go: replaces routing table with contact service
storagenode/contact/nodesservice.go: creates empty implementation for ping and request info nodes service & implements RequestInfo method
storagenode/contact/service.go: creates a service to return the local node and update its own capacity
storagenode/monitor/monitor.go: uses contact service in place of routing table
storagenode/operator.go: moves operatorconfig from kad into its own setup
storagenode/peer.go: sets up contact service, chore, pingstats and endpoints
satellite/overlay/config.go: changes NodeSelectionConfig.OnlineWindow default to 4hr to allow for accurate repair selection
Removes kademlia setups in:

cmd/storagenode/main.go
cmd/storj-sim/network.go
internal/testplane/planet.go
internal/testplanet/satellite.go
internal/testplanet/storagenode.go
satellite/peer.go
scripts/test-sim-backwards.sh
scripts/testdata/satellite-config.yaml.lock
storagenode/inspector/inspector.go
storagenode/peer.go
storagenode/storagenodedb/database.go
Why: Replacing Kademlia

Please describe the tests:
• internal/testplanet/planet_test.go:

TestBasic: assert that the storagenode can check in with the satellite without any errors
TestContact: test that all nodes get inserted into both satellites' overlay cache during testplanet setup
• satellite/contact/contact_test.go:

TestFetchInfo: Tests that the FetchInfo method returns the correct info
• storagenode/contact/contact_test.go:

TestNodeInfoUpdated: tests that the contact chore updates the node information
TestRequestInfoEndpoint: tests that the Request info endpoint returns the correct info
Please describe the performance impact: Node discovery should be at least slightly more performant since each node connects directly to each satellite and no longer needs to wait for bootstrapping. It probably won't be faster in real time on start up since each node waits a random amount of time (less than 1 hr) to initialize its first connection (jitter).
2019-09-19 15:56:34 -04:00
Kaloyan Raev
45df0c5340
storagenode/process: respond to Windows Service events (#3025) 2019-09-19 19:37:40 +03:00
Michal Niewrzal
d860353603
storagenode-updater: respond to Windows Service events (#3077) 2019-09-19 04:00:26 -07:00
Michal Niewrzal
43846f2074
cmd: rename auto-updater to storagenode-updater (#3089) 2019-09-19 02:33:56 -07:00
Michal Niewrzal
d65386f69e
auto-updater: unpack/check binary, restart service (#2968) 2019-09-17 00:19:56 -07:00
Matt Robinson
0c2ae7786e storagenode/dashboard: Enable storagenode dashboard in docker images (#3024) 2019-09-16 15:09:59 +02:00
Andrew Harding
f550ab5d1c
Uplink "import" command (#2981)
* uplink import cmd

* pkg/process: fix import order

* fix golangci-lint failures

* remove "help" from the satellite config lock file
2019-09-13 12:33:30 -06:00
littleskunk
1d8cd526e0 storj-sim: correct storagenode dashboard config (#3010) 2019-09-12 15:20:52 +03:00
paul cannon
c139ed8ea1 storagenode/console: remove kademlia (#2942)
this is a trivial operation for storagenode/console, as it doesn't
really need or use kademlia in the first place.

What:

Removes kademlia from storagenode/console

Why:

We are in the process of getting rid of kademlia, and this is one place where it's particularly easy.

Please describe the tests:

Existing tests exercise storagenode/console behavior; if they continue to work, everything here should be tested satisfactorily.
Please describe the performance impact:

None
2019-09-11 16:41:43 -04:00
Michal Niewrzal
40ff56f6c7
versioncontrol: add new version schema (#2991) 2019-09-11 07:01:36 -07:00
Bryan White
6c80f01bf0
pkg/certificates: add authorization endpoint and refactor (#2971) 2019-09-11 10:36:44 +02:00
paul cannon
7cf5650560 pkg/certificate: properly close certificateclient.Client (#2986) 2019-09-10 19:24:41 +03:00
Egon Elbre
0d4fd6ceeb
cmd/uplink: fix progressbar data race (#2987) 2019-09-10 11:51:49 +03:00
Egon Elbre
7589ca796f
cmd/storj-sim: allow overriding executables (#2976)
The backward compatibility test upgrades only half of the storage nodes and
tests with both the release and new uplink.
2019-09-09 22:13:38 +03:00
Michal Niewrzal
587be8f206
auto-updater: download versions and archive with binary (#2922) 2019-09-05 23:10:05 +02:00
Bryan White
1fc0c63a1d
{cmd,pkg}/certificates: service refactor (#2938) 2019-09-05 17:11:21 +02:00
Bryan White
b222936d19
cmd/certificates: separate subcommand configs (#2932) 2019-09-05 16:56:57 +02:00
Bryan White
62df8ddb0c cmd/certificates: auth export improvements & certificates test script (#2897) 2019-09-04 19:58:38 +02:00
Yingrong Zhao
10a896bf73
web/marketing: static asset path (#2872)
* use relative path instead of absolute path

* add template func baseURL

* add a method

* update storj-sim

* add comment
2019-08-30 18:43:53 -04:00
Michal Niewrzal
c680970e03
Storage node auto-updater skeleton (#2902) 2019-08-30 11:02:36 +02:00
littleskunk
7e37452abb open port for storage node dashboard API (#2899) 2019-08-29 11:47:43 -04:00
Stefan Benten
d0ab3c03ec cmd/*: Change loglevel from error to warn (#2876) 2019-08-27 11:24:47 +02:00
Egon Elbre
00b2e1a7d7 all: enable staticcheck (#2849)
* by having megacheck in disable it also disabled staticcheck

* fix closing body

* keep interfacer disabled

* hide bodies

* don't use deprecated func

* fix dead code

* fix potential overrun

* keep stylecheck disabled

* don't pass nil as context

* fix infinite recursion

* remove extraneous return

* fix data race

* use correct func

* ignore unused var

* remove unused consts
2019-08-22 13:40:15 +02:00
Egon Elbre
9ec0ceddf3
pkg/revocation: ensure we close revocation databases (#2825) 2019-08-20 18:04:17 +03:00
Isaac Hess
25154720bd
lib/uplink: remove redis and bolt dependencies (#2812)
* identity: remove redis and bolt dependencies

* identity: move revDB creation to main files
2019-08-19 16:10:38 -06:00
Egon Elbre
fec10ccbd5
cmd/certificates: ensure we can bind config values (#2779) 2019-08-14 17:26:45 +03:00
Jess G
022f5d2e14
storagenode: add space used cache for pieces (#2753)
* add cache, update cache w/piece create/delete

* add service w/loop to cache to recalculate space used cache

* add piecestore cache to other sn svcs to use

* add table to persist the total space used

* rm cache where not needed

* rm stuff from sn svcs

* start fixing tests, changes per comments

* update commits

* add unit tests

* fix commiting before we write header bytes

* fix cache create test

* copy cache map, add started back to recalc

* fix test

* add test, update comments
2019-08-12 14:43:05 -07:00
Bryan White
13900140e9
remove identity setup from uplink instructions (#2761) 2019-08-12 15:40:16 +02:00
Bryan White
34c9285142 Uplink setup: respect tls args in libuplink config (#2738) 2019-08-08 15:35:43 +03:00
Egon Elbre
c8edeb0257
satellite/overlay: rename overlay.Cache to overlay.Service (#2717) 2019-08-06 19:35:59 +03:00
Jeff Wendling
21a3bf89ee cmd/uplink: use scopes to open (#2501)
What: Change cmd/uplink to use scopes

It moves the fields that will be subsumed by scopes into an explicit legacy section and hides their configuration flags.

Why: So that it can read scopes in from files and stuff
2019-08-05 11:01:20 -06:00
Stefan Benten
8532625f85
Add 24 hours to account for the entire rest of the end day (#2698) 2019-08-02 10:27:36 -04:00
Egon Elbre
369a51ed00 lib/uplink: ensure it's silent by default (#2676) 2019-08-01 07:14:09 -04:00
Egon Elbre
4f0d39cc64
don't use global loggers (#2675) 2019-07-31 17:38:44 +03:00
Egon Elbre
ec3d5c0bdd
don't use global loggers (#2671)
* pkg/server: don't use global logger
* satellite/overlay: use correct logger
* pkg/kademlia: use correct logger
* linksharing: use conventional way to pass in logger
* use zaptest in tests
2019-07-31 15:09:45 +03:00
Kaloyan Raev
4b8820230d
Initialize correctly libuplink for uplink setup and gateway setup (#2662) 2019-07-30 20:49:01 +03:00
Kaloyan Raev
175c30048d
Use SaltedKeyFromPassphrase in Uplink CLI and Gateway (#2637)
Co-authored-by: Jeff Wendling <leterip@gmail.com>
2019-07-29 10:17:49 +03:00
Egon Elbre
5d0816430f
rename all the things (#2531)
* rename pkg/linksharing to linksharing
* rename pkg/httpserver to linksharing/httpserver
* rename pkg/eestream to uplink/eestream
* rename pkg/stream to uplink/stream
* rename pkg/metainfo/kvmetainfo to uplink/metainfo/kvmetainfo
* rename pkg/auth/signing to pkg/signing
* rename pkg/storage to uplink/storage
* rename pkg/accounting to satellite/accounting
* rename pkg/audit to satellite/audit
* rename pkg/certdb to satellite/certdb
* rename pkg/discovery to satellite/discovery
* rename pkg/overlay to satellite/overlay
* rename pkg/datarepair to satellite/repair
2019-07-28 08:55:36 +03:00
Egon Elbre
0cdeae1922 add missing error handling (#2630) 2019-07-25 17:01:44 +02:00
Stefan Benten
d84b987717 Disable CAWhitelist config when signing identities(#2628)
Ensure we don't enforce a signed Peer Identity during the Authorize command
2019-07-25 08:57:21 +02:00
Stefan Benten
6e5e261244
Fix missing Version config parameter (#2627) 2019-07-24 16:11:17 -04:00