Commit Graph

219 Commits

Author SHA1 Message Date
Cameron Ayer
5a337c48ec {cmd,private,storagenode}: create storage dir verification during setup
Previously, we created a new file to use for directory verification
every time the storage node starts. This is not helpful if the storage node
points to the wrong directory when restarting. Now we will only create the file
on setup. Now the file should be created only once and will be verified at
runtime.

Change-Id: Id529f681469138d368e5ea3c63159befe62b1a5b
2020-11-11 11:01:36 -05:00
Cameron Ayer
07acf0e574 cmd/storagenode: add docker env variable to toggle running setup
Previously, we ran setup if no config file was found in the expected dir.
However, there may be situations where a previously set up node's files
may be unreachable. In this case, we would prefer to exit with an error
rather than assume this node needs to be initialized.

The solution here is to add a new env variable to call the setup command.
If SETUP == true, the node will setup, but not run. If SETUP != true,
the node will run and not setup.

If a previously set up node runs with SETUP, it will return an error.
If a node runs without an initial SETUP, it will return an error.

Change-Id: Id2c796ec3d43f2add5e5f34fb777a563eae59f2f
2020-11-11 13:11:19 +00:00
Egon Elbre
1903b15474 storagenode/internalpb: move gracefulexit.proto
Change-Id: Ia3614846ed49a39c8f39331516d16d45a695240b
2020-10-30 15:24:56 +02:00
Egon Elbre
cda67a659a storagenode/internalpb: move inspector.proto
Change-Id: I951379c3b2ff00d1bc09d6a49c026a7e723432d6
2020-10-30 14:51:26 +02:00
Egon Elbre
89ce1fe626 storagenode/storagenodedb: add ctx to OpenNew and OpenExisting
Database opening usually dial and hence we should pass ctx to them.

Change-Id: I9160ae95829f22f347bd525904898a47279a7427
2020-10-29 09:52:37 +02:00
Egon Elbre
d0beaa4a87 pkg/revocation: pass ctx into opening the database
Opening a databases requires ctx, this is first step to passing ctx
to the appropriate level.

Change-Id: I12700f39a320206d8a2a4e054452319f8585b44b
2020-10-29 07:15:36 +00:00
Egon Elbre
cf2dd76db7 cmd/satellite: proper log usage
log.Fatal immediately terminates the program without running any defers.
We should properly close all the services and databases.

Change-Id: I5e959cef3eafedeacb3a2062e3da47e8d04e8e75
2020-10-13 16:56:35 +03:00
Egon Elbre
2268cc1df3 all: fix linter complaints
Change-Id: Ia01404dbb6bdd19a146fa10ff7302e08f87a8c95
2020-10-13 15:59:01 +03:00
Yaroslav Vorobiov
a840cb71e7 storagenode: check db version before run
Change-Id: I912f63fd62f2bff10341346c28dfb92fcd683806
2020-09-30 10:58:09 +00:00
Cameron Ayer
ca0c1a5f0c storagenode/{monitor,pieces}, storage/filestore: add loop to check storage directory writability
periodically create and delete a temp file in the storage directory
to verify writability. If this check fails, shut the node down.

Change-Id: I433e3a8d1d775fc779ae78e7cf3144a05ffd0574
2020-08-31 21:20:49 +00:00
Qweder93
6e90ca8b0e cmd/storagenode: CLI dashboard last contact renamed to status
Change-Id: I29bcd62d0d2970d6294dc642b442b60a3331bbd6
2020-08-06 13:54:07 +00:00
JT Olio
c89746a383 storagenode: use hardcoded metric application name
Change-Id: If11f226d03bc4bf70e2465569f8d578750718ae7
2020-07-28 16:24:15 +00:00
Yaroslav Vorobiov
4d2a505788 storagenode/db: explicitly open and create dbs
To prevent storagenode from implicitly recreating missing dbs and storage,
as such behaviour leads to audit failures. Do not allow storagenode to
start if any of dbs or storage is missing, corrupted, or dedicated storage disk is
unmounted, to get downtime instead.

Change-Id: Ic64e1f0ff4d8ef5b2fddbe7a7e53df4f4bd8652e
2020-07-24 14:08:47 +03:00
Egon Elbre
b67d7ecbc5 cmd/storagenode,storage/cockroachkv: better error handling
Change-Id: I6646aa046dc365c0dee38f23041be4fc2defb759
2020-07-16 20:03:50 +03:00
Egon Elbre
d8dcae3075 all: fix error checking
Change-Id: Ia0da1bbd6ce695139922f94096c2419281905e32
2020-07-16 19:13:14 +03:00
Egon Elbre
080ba47a06 all: fix dots
Change-Id: I6a419c62700c568254ff67ae5b73efed2fc98aa2
2020-07-16 14:58:28 +00:00
Ivan Fraixedes
37cfc01ac6 cmd/storagenode: Add long description to ge command
Add a long description to the graceful exit command to clearly mention
that the command is interactive asking which satellite the SNO wants to
exit.

Change-Id: Icd4056a470e707322f600133e63d9dc56eb877b7
2020-06-24 08:47:52 +00:00
Qweder93
e52809d53e cmd/storagenode: add check if satellites available to gracefulexit
Change-Id: I8747507593d810bbdec0d140de0600ee147011c3
2020-06-10 13:38:36 +00:00
Egon Elbre
0de1b5fa9f ci: fix golangci-lint failures
Change-Id: Ia8c4dd03439ce4e3f85890d45255b8ea1c94b50a
2020-06-05 12:55:35 +00:00
Ethan
159df8b2e4 Add logging listener for retrieving and setting log levels
See https://storjlabs.atlassian.net/browse/SM-752

These changes allow us to change the log level at runtime through a handler off of the debug endpoint.

Examples of changing the log level on storj-sim

To get the current level for the satellite api process:
curl -XGET 'http://127.0.0.1:10009/logging' --header 'Content-Type: text/plain'

To change the log level:
curl -XPUT 'http://127.0.0.1:10009/logging' --header 'Content-Type: text/plain' --data-raw '{"level":"error"}'

Change-Id: I05d164b290929fa06b6d78c01075ee41f8238044
2020-05-12 16:38:06 -04:00
Egon Elbre
ec589a8289 all: fix comments about grpc
Change-Id: Id830fbe2d44f083c88765561b6c07c5689afe5bd
2020-05-11 13:05:34 +03:00
Egon Elbre
c630cf2490 storagenode/pieces: implement buffering for writing
Currently uploads can cause a lot of IOPS, reduce this by introducing a
in-memory buffer on-top of the file.

Change-Id: I5f4e3e01c0a36258271d180b922107de447bcb59
2020-05-04 06:01:32 +00:00
Egon Elbre
8928399d02 all: rename CreateTables to MigrateToLatest
CreateTables hasn't been quite true for a while now, rename to
MigrateToLatest to be clearer in it's behavior.

Change-Id: Ida48e95122a5d9b7a814e922d3698e00024a2ba7
2020-04-30 07:21:17 +00:00
Michal Niewrzal
b2acd93a78 storagenode/inspector: adjust Uptime type change in protobuf
Change-Id: I4f5110b534ac4f419b74f1a3dd72f8600e0a53a8
2020-04-29 09:28:37 +00:00
Yingrong Zhao
8375a09c89 cmd: remove InitTracing from satellite and storagenode main.go file
Change-Id: I4addbe7d0645f66abfb3e98d74d17035e9624e69
2020-04-20 14:06:26 -04:00
Kaloyan Raev
a2ce836761 remove sugar logging
Change-Id: I6b6ca9704837cb3f5f5449ba7f55661487814d9f
2020-04-15 12:37:47 +00:00
Moby von Briesen
68946c86b4 cmd/storagenode: update colored storagenode dashboard to have correct indendation
Change-Id: I8f8fc2322c42e7e1098437abf4aad28bd754e79c
2020-04-14 13:21:17 +00:00
Yingrong Zhao
96e58d21b4 cmd;pkg/server: init tracing collector in all processes
Add tracing handler in drpc server.
Initializing tracing collector in admin, satellite api, garbage
collection, satellite core, repaier, storagenode.
Change-Id: Ie98420e35dfc6913836ebd82b517d9d12877aefc

Change-Id: I91057b6265a4ac8bde033dfde692b8a28acca99f
2020-04-07 17:20:59 -04:00
sixcorners
4f8cf53c9e
Remove VOLUME /root/.local/share/storj/storagenode () 2020-04-03 00:05:20 +02:00
Egon Elbre
644df8dcdc private/version: minimal fix for tag-release.sh
Previous split to a storj.io/private repository broke tag-release.sh
script. This is the minimal temporary fix to make things work.

This links the build information to specified variables and sets them
inline. This approach, of course, is very fragile.

Change-Id: I73db2305e6c304146e5a14b13f1d917881a7455c
2020-04-01 13:46:45 +00:00
Yingrong Zhao
b7b19289d1 bump storj.io/common to latest
Change-Id: I16e337660ce8e1ef332cc842dbf4cfa067b9b98b
2020-03-25 09:08:40 -04:00
Michal Niewrzal
fdf40a7526 storj: remove storj/private/version package which was moved to
`storj/private` repo

Change-Id: I81c3f5b9d5e4fe7bca760999eb045ee9734e5e2e
2020-03-24 14:31:33 +00:00
Michal Niewrzal
f0aeda3091 storj: remove from storj/pkg packages moved to storj/private repo
* debug
* traces
* cfgstruct
* process

Package `storj/private/version` will be removed as a separate change.

Change-Id: Iadc40faa782e6225513b28218952f02d9c240a9f
2020-03-24 09:56:29 +01:00
Jennifer Johnson
140e2f0045 web/storagenode/src: removes references to available and remaining bandwidth
Change-Id: I3945a94ddd6d80d0282f65e5e9939952e39ba571
2020-03-11 14:17:33 +00:00
Jennifer Johnson
1c1750e6be removes bandwidth limiting
On satellite, remove all references to free_bandwidth column in nodes table.
On storage node, remove references to AllocatedBandwidth and MinimumBandwidth and mark as deprecated.

Protobuf message, NodeCapacity, is left intact for backwards compatibility.
Once this is released to all satellites, we can drop the column from the DB.

Change-Id: I2ff6c6537fc9008a0c5588e951afea58ede85838
2020-03-04 14:04:00 +00:00
Qweder93
484ec7463a storagenode: notifications on outdated software version
Change-Id: If19b075c78a7b2c441e11b783c3c09fed55060c7
2020-03-02 16:48:02 +00:00
Egon Elbre
9752d01884 private/prompt: remove dependency to go-prompt
Change-Id: Ida8ef731ce806cec076343dc77d72a3b0d7736b4
2020-02-25 13:09:41 +02:00
Egon Elbre
f237d70098 storagenode,satellite: use pkg/debug
Use debug.Server in storage node and satellite for customizing debug server.

Change-Id: I7979412376d028cadf29656d838ab94f18e2aa99
2020-01-29 16:30:31 -05:00
Moby von Briesen
273eb66fae cmd/storagenode,storagenode/preflight: add config flag to disable
storagenode database preflight check.

Disable preflight database check by default, and have the option to
enable it. This will allow us to enable it once it is definitely
working.

Also change the name of the config flag for preflight  time sync.

Change-Id: Ie2e20f9e25dcb38794eafa7e1505e7c6ff287c99
2020-01-17 17:53:17 +00:00
Moby von Briesen
e115bc1903 cmd/storagenode;storagenode/storagenodedb: add preflight database check
for storagenode

Ensure that database schema matches latest test migration schema before
allowing the node to start up.

Ensure minimal read/write functionality for each storagenode database
before allowing the node to start up.

This will eliminate many unhandled audit errors we are seeing.

Change-Id: Ic0e628b04a9c35b7a8243f6a81d4683918170ba9
2020-01-16 18:44:46 +00:00
Egon Elbre
6615ecc9b6 common: separate repository
Change-Id: Ibb89c42060450e3839481a7e495bbe3ad940610a
2019-12-27 14:11:15 +02:00
Egon Elbre
d55288cf68 pkg/rpc: replace methods with direct calls to pb
Change-Id: I8bd015d8d316a2c12c1daceca1d9fd257f6f57bc
2019-12-22 17:12:43 +02:00
Yingrong Zhao
66f1a1680f
add completion receipt to exit-status cli command on storage node () 2019-11-26 12:32:26 -05:00
Egon Elbre
ee6c1cac8a
private: rename internal to private () 2019-11-14 21:46:15 +02:00
Egon Elbre
1e64006e32 lint: add staticcheck as a separate step () 2019-11-14 10:31:30 +02:00
littleskunk
7eb6724c92
logging: unify logging around satellite ID, node ID and piece ID ()
* logging: unify logging around satellite ID, node ID and piece ID

* unify segment index
2019-11-05 22:04:07 +01:00
Yingrong Zhao
87e3764390
storagenode/cmd: add exit-status command for graceful exit ()
* add exit-status command

* remove todo and fix format

* fix status display

* change startExit to exit progress

* fix linting error

* add successful column in exit progress

* fix test

* remove extra new line

* fix TYPOS

* format the percentage better
2019-10-15 18:07:32 -04:00
Yingrong Zhao
743a0fc38b storagenode/cmd: create start graceful exit CLI () 2019-10-11 09:58:12 -04:00
Cameron
10b364a2da
cmd/storagenode: enable migration of configs of different types ()
* move deprecated flags code to deprecated.go, refactor to allow migration of other flags
* hide deprecated flags
2019-10-08 14:26:53 -04:00
JT Olio
37491d0d32 storagenode: embed the console into the binary and makefile ()
* web/storagenode: add package-lock.json
* storagenode: compile console into binary
2019-10-08 10:52:19 +02:00
Jennifer Li Johnson
7ceaabb18e
Delete Bootstrap and Kademlia () 2019-10-04 16:48:41 -04:00
littleskunk
b2e328f118 storagenode/dashboard: update online status () 2019-10-03 20:31:39 +02:00
Jennifer Li Johnson
755cbd4dce
storagenode/main: map aliases for kademlia config values () 2019-09-30 19:33:00 -04:00
Jeff Wendling
098cbc9c67 all: use pkg/rpc instead of pkg/transport
all of the packages and tests work with both grpc and
drpc. we'll probably need to do some jenkins pipelines
to run the tests with drpc as well.

most of the changes are really due to a bit of cleanup
of the pkg/transport.Client api into an rpc.Dialer in
the spirit of a net.Dialer. now that we don't need
observers, we can pass around stateless configuration
to everything rather than stateful things that issue
observations. it also adds a DialAddressID for the
case where we don't have a pb.Node, but we do have an
address and want to assert some ID. this happened
pretty frequently, and now there's no more weird
contortions creating custom tls options, etc.

a lot of the other changes are being consistent/using
the abstractions in the rpc package to do rpc style
things like finding peer information, or checking
status codes.

Change-Id: Ief62875e21d80a21b3c56a5a37f45887679f9412
2019-09-25 15:37:06 -06:00
Michal Niewrzal
366a0be6aa
storagenode: avoid starting command on service stop () 2019-09-24 12:17:22 -07:00
Kaloyan Raev
8d16349b4c
storagenode/winsvc: avoid the 15s sleep when stopping the service () 2019-09-24 13:01:36 -06:00
Isaac Hess
580e511b4c
storagenode/storagenodedb: Migrate to separate dbs ()
* storagenode/storagenodedb: Migrate to separate dbs

* storagenode/storagenodedb: Add migration to drop versions tables

* Put drop table statements into a transaction.

* Fix CI errors.

* Fix CI errors.

* Changes requested from PR feedback.

* storagenode/storagenodedb: fix tx commit
2019-09-23 12:36:46 -07:00
Jennifer Li Johnson
d2502bb51b Adds tests for kad replacement and restores kad operator configs ()
* test that all nodes can check in with all satellites

* keep kademlia config

* add untrusted satellite test

* use getversion

* remove kademlia config changes in test-sim-backwards.sh

* add kademlia flags back to storj-sim storagenode

* reset kademlia flags in storagenode entrypoint
2019-09-20 16:02:23 -04:00
Jennifer Li Johnson
724bb44723
Remove Kademlia dependencies from Satellite and Storagenode ()
What:

cmd/inspector/main.go: removes kad commands
internal/testplanet/planet.go: Waits for contact chore to finish
satellite/contact/nodesservice.go: creates an empty nodes service implementation
satellite/contact/service.go: implements Local and FetchInfo methods & adds external address config value
satellite/discovery/service.go: replaces kad.FetchInfo with contact.FetchInfo in Refresh() & removes Discover()
satellite/peer.go: sets up contact service and endpoints
storagenode/console/service.go: replaces nodeID with contact.Local()
storagenode/contact/chore.go: replaces routing table with contact service
storagenode/contact/nodesservice.go: creates empty implementation for ping and request info nodes service & implements RequestInfo method
storagenode/contact/service.go: creates a service to return the local node and update its own capacity
storagenode/monitor/monitor.go: uses contact service in place of routing table
storagenode/operator.go: moves operatorconfig from kad into its own setup
storagenode/peer.go: sets up contact service, chore, pingstats and endpoints
satellite/overlay/config.go: changes NodeSelectionConfig.OnlineWindow default to 4hr to allow for accurate repair selection
Removes kademlia setups in:

cmd/storagenode/main.go
cmd/storj-sim/network.go
internal/testplane/planet.go
internal/testplanet/satellite.go
internal/testplanet/storagenode.go
satellite/peer.go
scripts/test-sim-backwards.sh
scripts/testdata/satellite-config.yaml.lock
storagenode/inspector/inspector.go
storagenode/peer.go
storagenode/storagenodedb/database.go
Why: Replacing Kademlia

Please describe the tests:
• internal/testplanet/planet_test.go:

TestBasic: assert that the storagenode can check in with the satellite without any errors
TestContact: test that all nodes get inserted into both satellites' overlay cache during testplanet setup
• satellite/contact/contact_test.go:

TestFetchInfo: Tests that the FetchInfo method returns the correct info
• storagenode/contact/contact_test.go:

TestNodeInfoUpdated: tests that the contact chore updates the node information
TestRequestInfoEndpoint: tests that the Request info endpoint returns the correct info
Please describe the performance impact: Node discovery should be at least slightly more performant since each node connects directly to each satellite and no longer needs to wait for bootstrapping. It probably won't be faster in real time on start up since each node waits a random amount of time (less than 1 hr) to initialize its first connection (jitter).
2019-09-19 15:56:34 -04:00
Kaloyan Raev
45df0c5340
storagenode/process: respond to Windows Service events () 2019-09-19 19:37:40 +03:00
Matt Robinson
0c2ae7786e storagenode/dashboard: Enable storagenode dashboard in docker images () 2019-09-16 15:09:59 +02:00
paul cannon
c139ed8ea1 storagenode/console: remove kademlia ()
this is a trivial operation for storagenode/console, as it doesn't
really need or use kademlia in the first place.

What:

Removes kademlia from storagenode/console

Why:

We are in the process of getting rid of kademlia, and this is one place where it's particularly easy.

Please describe the tests:

Existing tests exercise storagenode/console behavior; if they continue to work, everything here should be tested satisfactorily.
Please describe the performance impact:

None
2019-09-11 16:41:43 -04:00
littleskunk
7e37452abb open port for storage node dashboard API () 2019-08-29 11:47:43 -04:00
Stefan Benten
d0ab3c03ec cmd/*: Change loglevel from error to warn () 2019-08-27 11:24:47 +02:00
Egon Elbre
9ec0ceddf3
pkg/revocation: ensure we close revocation databases () 2019-08-20 18:04:17 +03:00
Isaac Hess
25154720bd
lib/uplink: remove redis and bolt dependencies ()
* identity: remove redis and bolt dependencies

* identity: move revDB creation to main files
2019-08-19 16:10:38 -06:00
Jess G
022f5d2e14
storagenode: add space used cache for pieces ()
* add cache, update cache w/piece create/delete

* add service w/loop to cache to recalculate space used cache

* add piecestore cache to other sn svcs to use

* add table to persist the total space used

* rm cache where not needed

* rm stuff from sn svcs

* start fixing tests, changes per comments

* update commits

* add unit tests

* fix commiting before we write header bytes

* fix cache create test

* copy cache map, add started back to recalc

* fix test

* add test, update comments
2019-08-12 14:43:05 -07:00
Jeff Wendling
21a3bf89ee cmd/uplink: use scopes to open ()
What: Change cmd/uplink to use scopes

It moves the fields that will be subsumed by scopes into an explicit legacy section and hides their configuration flags.

Why: So that it can read scopes in from files and stuff
2019-08-05 11:01:20 -06:00
Egon Elbre
4f0d39cc64
don't use global loggers () 2019-07-31 17:38:44 +03:00
Egon Elbre
0cdeae1922 add missing error handling () 2019-07-25 17:01:44 +02:00
Stefan Benten
4ebaae26d1
Disabling Dashboard Link, as its localhost and not working yet () 2019-07-16 17:03:03 -04:00
Alexander Leitner
88732188cb
Update inspector timestamp to time.time ()
* Update inspector timestamp
2019-07-08 10:06:12 -04:00
Yaroslav Vorobiov
7aca0eb284 storagenode/dashboard: show console address () 2019-07-06 15:40:58 +02:00
Egon Elbre
c7679b9b30
Fix some leaks and add notes about close handling () 2019-06-25 23:00:51 +03:00
Jeff Wendling
e74cac52ab
Command line flags features and cleanup ()
* change BindSetup to be an option to Bind
* add process.Bind to allow composite structures
* hack fix for noprefix flags
* used tagged version of structs

Before this PR, some flags were created by calling `cfgstruct.Bind` and having their fields create a flag. Once the flags were parsed, `viper` was used to acquire all the values from them and config files, and the fields in the struct were set through the flag interface.

This doesn't work for slices of things on config structs very well, since it can only set strings, and for a string slice, it turns out that the implementation in `pflag` appends an entry rather than setting it.

This changes three things:

1. Only have a `Bind` call instead of `Bind` and `BindSetup`, and make `BindSetup` an option instead.
2. Add a `process.Bind` call that takes in a `*cobra.Cmd`, binds the struct to the command's flags, and keeps track of that struct in a global map keyed by the command.
3. Use `viper` to get the values and load them into the bound configuration structs instead of using the flags to propagate the changes.

In this way, we can support whatever rich configuration we want in the config yaml files, while still getting command like flags when important.
2019-05-29 17:56:22 +00:00
aligeti
ac2dc8684e
added warning info when total BW < used BW () 2019-05-24 09:53:29 -04:00
JT Olio
32b3f8fef0 cmd/storagenode: pull more things into releaseDefaults () 2019-05-21 13:48:47 +02:00
sanderr
a795ec93ac typo () 2019-05-07 22:00:20 +02:00
Matt Robinson
8e09c30e2b
Whitelist Saturn (121R...) and Jupiter (12L9...) () 2019-04-26 08:56:58 -04:00
Egon Elbre
60c4c10c79
storagenode: delete psserver () 2019-04-26 08:17:18 +03:00
Matt Robinson
92c0ab5b8d Merge alpha directory with main storagenode directory () 2019-04-23 17:51:04 +02:00
JT Olio
2744a26b60
pkg/cfgstruct: tie defaults to releases ()
* tie defaults to releases

this change makes it so that by default, the flag defaults are
chosen based on whether the build was built as a release build or
an ordinary build. release builds by default get release defaults,
whereas ordinary builds by default get dev defaults.

any binary can have its defaults changed by specifying

 --defaults=dev

or

 --defaults=release

Change-Id: I6d216aa345d211c69ad913159d492fac77b12c64

* make release defaults more clear

this change extends cfgstruct structs to support either
a 'default' tag, or a pair of 'devDefault' and 'releaseDefault'
tags, but not both, for added clarity

Change-Id: Ia098be1fa84b932fdfe90a4a4d027ffb95e249c6

* clarify cfgstruct.DefaultsFlag

Change-Id: I55f2ff9080ebbc0ce83abf956e085242a92f883e
2019-04-19 12:17:30 -06:00
JT Olio
ae855fd6eb
Forward-port release-alpha8 build script issues ()
* releases: update alpha dockerfiles to embed release tags

Change-Id: I2978d3ed77d1b768be24ec9194bc404cd92e4700

* release script use plain shell

Change-Id: Ie4ca6633014e3bf00ac0c82e28905174a6ade95b

* removed /scripts from .dockerignore

Change-Id: Ib555236d62fb3dc354689284ac4a032d5b4b3aa8

* switch back to bash

Change-Id: Ie0958cde71cd396f44ac28c177f48df0bea7ba22

* fix go1.12 modules

Change-Id: If4c8364e7271f09febd1a6c8bfc731d723d344d9

* make sure the work tree is clean

Change-Id: Ie6e52af28e2bc52880d92a3b574d1f859fc8e88e
2019-04-09 23:01:10 -06:00
Matt Robinson
e478d0855c
Build multiarch ARM32v6 and AMD64 images () 2019-04-08 16:01:20 -04:00
Matt Robinson
1adf2febf2
Add LE CA to satellite and storagenode images () 2019-04-08 13:06:01 -04:00
littleskunk
063fcf3bb8 dashboard change last contact never into offline () 2019-04-08 11:30:29 -04:00
JT Olio
09be9964eb internal/version: do version checks much earlier in the process initialization, take 2 ()
* internal/version: do version checks much earlier in the process initialization, take 2

Change-Id: Ida8c7e3757e0deea0ec7aea867d3d27ce97dc134

* linter and test failures

Change-Id: I45b02a16ec1c0f0981227dc842e68dbdf67fdbf4
2019-04-04 17:40:07 +02:00
Fadila
3d2ee890c3
Add since date label in dashboard () 2019-04-04 12:08:03 +02:00
Dennis Coyle
88f0d63f33 Adds the AMD64 and ARM Storagenode Dockerfiles and setup scripts ()
* add alpha dockerfile structure

* adds Stefans satelitte to white list
2019-04-03 14:17:55 -06:00
Stefan Benten
2cf86703a3
Add Versioning Server ()
* Initial Webserver Draft for Version Controlling

* Rename type to avoid confusion

* Move Function Calls into Version Package

* Fix Linting and Language Typos

* Fix Linting and Spelling Mistakes

* Include Copyright

* Include Copyright

* Adjust Version-Control Server to return list of Versions

* Linting

* Improve Request Handling and Readability

* Add Configuration File Option
Add Systemd Service file

* Add Logging to File

* Smaller Changes

* Add Semantic Versioning and refuses outdated Software from Startup ()

* implements internal Semantic Version library

* adds version logging + reporting to process

* Advance SemVer struct for easier handling

* Add Accepted Version Store

* Fix Function

* Restructure

* Type Conversion

* Handle Version String properly

* Add Note about array index

* Set temporary Default Version

* Add Copyright

* Adding Version to Dashboard

* Adding Version Info Log

* Renaming and adding CheckerProcess

* Iteration Sync

* Iteration V2

* linting

* made LogAndReportVersion a go routine

* Refactor to Go Routine

* Add Context to Go Routine and allow Operation if Lookup to Control Server fails

* Handle Unmarshal properly

* Linting

* Relocate Version Checks

* Relocating Version Check and specified default Version for now

* Linting Error Prevention

* Refuse Startup on outdated Version

* Add Startup Check Function

* Straighten Logging

* Dont force Shutdown if --dev flag is set

* Create full Service/Peer Structure for ControlServer

* Linting

* Straighting Naming

* Finish VersionControl Service Layout

* Improve Error Handling

* Change Listening Address

* Move Checker Function

* Remove VersionControl Peer

* Linting

* Linting

* Create VersionClient Service

* Renaming

* Add Version Client to Peer Definitions

* Linting and Renaming

* Linting

* Remove Transport Checks for now

* Move to Client Side Flag

* Remove check

* Linting

* Transport Client Version Intro

* Adding Version Client to Transport Client

* Add missing parameter

* Adding Version Check, to set Allowed = true

* Set Default to true, testing

* Restructuring Code

* Uplink Changes

* Add more proper Defaults

* Renaming of Version struct

* Dont pass Service use Pointer

* Set Defaults for Versioning Checks

* Put HTTP Server in go routine

* Add Versioncontrol to Storj-Sim

* Testplanet Fixes

* Linting

* Add Error Handling and new Server Struct

* Move Lock slightly

* Reduce Race Potentials

* Remove unnecessary files

* Linting

* Add Proper Transport Handling

* small fixes

* add fence for allowed check

* Add Startup Version Check and Service Naming

* make errormessage private

* Add Comments about VersionedClient

* Linting

* Remove Checks that refuse outgoing connections

* Remove release cmd

* Add Release Script

* Linting

* Update to use correct Values

* Move vars private and set minimum default versions for testing builds

* Remove VersionedClient

* Better Error Handling and naked return removal

* Straighten the Regex and string conversion

* Change Check to allows testplanet and storj-sim to run without the
need to pass an LDFlag

* Cosmetic Change to Dashboard

* Cleanup Returns and remove commented code

* Remove Version Check if no build options are passed in

* Pass in Config Values instead of Pointers

* Handle missed Error

* Update Endpoint URL

* Change Type of Release Flag

* Add additional Logging

* Remove Versions Logging of other Services

* minor fixes

Change-Id: I5cc04a410ea6b2008d14dffd63eb5f36dd348a8b
2019-04-03 21:13:39 +02:00
Cameron
cac55a29e4
Add used egress/ingress to storage node dashboard ()
* add egress and ingress to StatSummaryResponse

* print egress and ingress to storagenode dashboard
2019-03-27 15:44:18 -04:00
Egon Elbre
1d96d25f3f
kademlia ping tracking () 2019-03-22 15:27:59 +02:00
Egon Elbre
636d73934e Fix storage node path handling () 2019-03-19 11:10:23 +02:00
Egon Elbre
05d148aeb5
Storage node and upload/download protocol refactor ()
refactor storage node server
refactor upload and download protocol
2019-03-18 12:55:06 +02:00
Bill Thorp
66718cc5e6
Development defaults for configuration ()
added --dev command line option, cfgstruct.DevFlag(), and cfgstruct.SetupFlag()
2019-03-12 08:51:06 -04:00
Jess G
193a70f0a6
add private listener to grpc server ()
* add private listener to grpc server

* add changes per init CR

* fix server.close

* add insecure grpc connection, update logs msg

* fix tests, move insecure client

* add private ports to storj-sim, add insecure client to other inspectors

* add ports to test so there arent conflicts

* fix lint err

* fix node started log msg, close public listener

* remove commented out line
2019-03-07 13:19:37 -05:00
Jess G
3c9d83dbfe
convert psserver dashboard into an inspector ()
* Convert psserver dashboard into an inspector

* remove dashboard stream, update ps.pb.mock

* fixes for lint errs
2019-03-05 15:48:37 -05:00
Philip Hutchins
1f452e67dc Adding dockerfile for running the web UI for Satellite ()
* Adding dockerfile for running the web UI for Satellite

* Updating to work with Makefile and from root directory of repo

* Updating satellite ui build process to run in a more production like mode by generating the assets the pulling those into the satellite container

* Updates to allow external traffic to UI, updates to storagenode for identity creation, and logging for bug tracking

* Adding auto cert generation for storagenode

* removing satellite-ui-image from main images flow in Makefile and adding latest tag to docker build for it

* Adding solid defaults, tuning dockerfiles, and moving to standard logging methods

* Updating logging to be more standard

* Updating to logger.Debug

* Removing unused library and unused identity creation code

Change-Id: I956453037e303693ea37f94318180af0ab7984d5
2019-02-28 13:18:53 -07:00
Michal Niewrzal
8d685217e4
Storagenode migrations ()
* creates initial migration for psdb
* add test mechanism to validate migration to every version
* fix few small issues in versions.go and context.go
2019-02-19 10:39:04 +01:00
Egon Elbre
46a19d5186 show nicer time in storage node dashboard () 2019-02-18 16:16:48 +01:00