Commit Graph

75 Commits

Author SHA1 Message Date
JT Olio
ccb158c99b
pkg/auth: add monkit task to missing places (#2123)
What: add monkit.Task to a bunch of functions that are missing it

Why: this will significantly help our instrumentation, data collection, and tracing about what's going on in the network
2019-06-05 07:47:01 -06:00
JT Olio
679cdfda9e storage/filestore: add monkit task to missing places (#2124)
Change-Id: I61f8056617a8d0596a00c232a6d0b330909e08f9
2019-06-05 15:06:06 +02:00
JT Olio
d02427e41a db: set max open conns, conn max lifetime, add db stat monitoring (#2117) 2019-06-04 23:30:21 +02:00
JT Olio
d6c02fc657 storagenode/piecestore: add meters to upload/download rates (#2106) 2019-06-04 15:22:00 +02:00
JT Olio
c4bb84f209 storagenode: add monkit task to missing places (#2107) 2019-06-04 14:31:38 +02:00
Kaloyan Raev
2ab95b533e
Check errors for possible outcomes from audit's DownloadShares (#2072) 2019-06-03 12:17:09 +03:00
Maximillian von Briesen
04c20b0ac0
Add monkit stats to piecestore upload/download (#2078) 2019-05-30 11:44:47 -04:00
Brandon Iglesias
771271e7b8 updating help message on storage node for AllocatedBandwidth (#2074) 2019-05-29 13:40:04 -04:00
ethanadams
268dc6b7e4
Enable gocritic linter (#2051)
* first round cleanup based on go-critic

* more issues resolved for ifelsechain and unlambda checks

* updated from master and gocritic found a new ifElseChain issue

* disable appendAssign. i reports false positives

* re-enabled go-critic appendAssign and disabled lint check at code line level

* fixed go-critic lint error

* fixed // nolint add gocritic specifically
2019-05-29 09:14:25 -04:00
Egon Elbre
9c23c2d427 db: set max idle connections higher to avoid redialing all the time (#1991) 2019-05-21 17:30:06 +03:00
JT Olio
32b3f8fef0 cmd/storagenode: pull more things into releaseDefaults (#1980) 2019-05-21 13:48:47 +02:00
littleskunk
910eb5d2c7 improve logging (#1987) 2019-05-17 19:02:39 +02:00
Marc Schubert
2509a4c98a Update readwrite.go (#1950)
Correcting wrong doc string.
2019-05-13 13:53:14 +02:00
Egon Elbre
a2b61fd67c
storage node collector (#1913) 2019-05-08 14:11:59 +03:00
Stefan Benten
ac452a5819
Add Network Wipe for Storagenodes (#1909)
* Add Network Wipe Migration to InfoDB

* Remove New Data Section
2019-05-07 22:05:50 +02:00
Stefan Benten
eeb4f6541e Update Calculation to include the used space for the allocation (#1899) 2019-05-06 14:59:30 -04:00
Bill Thorp
6ece4f11ad
moved invalid/offline back into SQL (#1838)
* moved invalid/offline back into SQL, removed GetAll()
2019-05-01 09:45:52 -04:00
Egon Elbre
60c4c10c79
storagenode: delete psserver (#1837) 2019-04-26 08:17:18 +03:00
Egon Elbre
7ba1a2bc53
storagenode: getting signee cert always asked the kademlia network (#1823) 2019-04-24 11:13:48 +03:00
Egon Elbre
f7ed63a119
handle database error checks properly (#1796) 2019-04-23 14:13:57 +03:00
Kaloyan Raev
8fc5fe1d6f
Refactor pb.Node protobuf (#1785) 2019-04-22 12:07:50 +03:00
Egon Elbre
5b3c146d8a
Check context cancellation more nicely (#1752) 2019-04-17 13:09:44 +03:00
Michal Niewrzal
e922f71f61 Piecestore space and bandwidth allocation check (#1527)
* Space and bandwidth allocation check

* use proper config flag

* one more check + tests

* use monitor to check space and bandwidth

* remove unused field

* check during read/write

* fix linter

* fix pieceid

* remove unused methods

* revert unneeded change
2019-04-15 12:12:22 +02:00
Egon Elbre
a1fd7cb6b4 don't cache failed satellite certificate fetch attempt (#1745)
* don't cache failing error

* fix test

* fix linter errors
2019-04-12 16:28:27 +02:00
Natalie Villasana
a73fd1aa9b
no longer includes totalUsedSpaceOld in used space calculation (#1736) 2019-04-10 11:55:06 -04:00
Stefan Benten
bae4c820ee
Add Version Information into KAD Network and SatelliteDB & Change Selection Process (#1648)
* Initial Webserver Draft for Version Controlling

* Rename type to avoid confusion

* Move Function Calls into Version Package

* Fix Linting and Language Typos

* Fix Linting and Spelling Mistakes

* Include Copyright

* Include Copyright

* Adjust Version-Control Server to return list of Versions

* Linting

* Improve Request Handling and Readability

* Add Configuration File Option
Add Systemd Service file

* Add Logging to File

* Smaller Changes

* Add Semantic Versioning and refuses outdated Software from Startup (#1612)

* implements internal Semantic Version library

* adds version logging + reporting to process

* Advance SemVer struct for easier handling

* Add Accepted Version Store

* Fix Function

* Restructure

* Type Conversion

* Handle Version String properly

* Add Note about array index

* Set temporary Default Version

* Add Copyright

* Adding Version to Dashboard

* Adding Version Info Log

* Renaming and adding CheckerProcess

* Iteration Sync

* Iteration V2

* linting

* made LogAndReportVersion a go routine

* Refactor to Go Routine

* Add Context to Go Routine and allow Operation if Lookup to Control Server fails

* Handle Unmarshal properly

* Linting

* Relocate Version Checks

* Relocating Version Check and specified default Version for now

* Linting Error Prevention

* Refuse Startup on outdated Version

* Add Startup Check Function

* Straighten Logging

* Dont force Shutdown if --dev flag is set

* Create full Service/Peer Structure for ControlServer

* Linting

* Straighting Naming

* Finish VersionControl Service Layout

* Improve Error Handling

* Change Listening Address

* Move Checker Function

* Remove VersionControl Peer

* Linting

* Linting

* Create VersionClient Service

* Renaming

* Add Version Client to Peer Definitions

* Linting and Renaming

* Linting

* Remove Transport Checks for now

* Move to Client Side Flag

* Remove check

* Linting

* Transport Client Version Intro

* Adding Version Client to Transport Client

* Add missing parameter

* Adding Version Check, to set Allowed = true

* Set Default to true, testing

* Restructuring Code

* Uplink Changes

* Add more proper Defaults

* Renaming of Version struct

* Dont pass Service use Pointer

* Set Defaults for Versioning Checks

* Put HTTP Server in go routine

* Add Versioncontrol to Storj-Sim

* Testplanet Fixes

* Linting

* Add Error Handling and new Server Struct

* Move Lock slightly

* Reduce Race Potentials

* Remove unnecessary files

* Linting

* Add Proper Transport Handling

* small fixes

* add fence for allowed check

* Add Startup Version Check and Service Naming

* make errormessage private

* Add Comments about VersionedClient

* Linting

* Remove Checks that refuse outgoing connections

* Remove release cmd

* Add Release Script

* Linting

* Update to use correct Values

* Change Timestamp handling

* Adding Protobuf changes back in

* Adding SatelliteDB Changes and adding Storj Node Version to PB

* Add Migration Table

* Add Default Stats for Creation

* Move to BigInt

* Proper SQL Migration

* Ensure minimum Version is passed to the node selection

* Linting...

* Remove VersionedClient and adjust smaller changes from prior merge

* Linting

* Fix PB Message Handling and Query for Node Selection

* some future-proofing type changes

Change-Id: I3cb5018dcccdbc9739fe004d859065992720caaf

* fix a compiler error

Change-Id: If66bb92d8b98e31cd618ecec9c6448ab9b037fa5

* Comment on Constant for Overlay

* Remove NOT NULL and add epoch call as function

* add versions to bootstrap and satellites

Change-Id: I436944589ea5f21600cdd997742a84fe0b16e47b

* Change Update Migration

* Fix DB Migration

* Increase Timeout temporarily, to see whats going on

* Remove unnecessary const and vars
Cleanup Function calls from deprecated NodeVersion struct

* Updated Protopuf, removed depcreated Code from Inspector

* Implement NodeVersion into InfoResponse

* Regenerated locked.go

* Linting

* Fix Tests

* Remove unnecessary constant

* Update Function and Flag Description

* Remove Empty Stat Creation

* return properly with error

* Remove unnecessary struct

* simplify migration step

* Update Inspector to return Version Info

* Update local Endpoint Version Handling

* Reset Travis Timeout

* Add Default for CommitHash

* single quotes
2019-04-10 08:04:24 +02:00
littleskunk
a3caa8e00d
upload, download and delete success message on log level info (#1704) 2019-04-09 01:14:09 +02:00
Egon Elbre
1c87a53eb8 Don't cancel fetching peer identity context when request is cancelled (#1699)
* don't cancel fetching peer identity context when request is cancelled

* fix typo

* add tests

* fix typo
2019-04-08 22:31:20 +02:00
Bryan White
faf5fae3f9
Identity versioning (#1389) 2019-04-08 20:15:19 +02:00
Maximillian von Briesen
bb3b4e4816 Data repair integration test (#1582) 2019-04-08 13:33:47 -04:00
aligeti
6a1d343abd Delete expired pieces on storage nodes (#1629) 2019-04-08 18:46:38 +02:00
Bill Thorp
255b92b4b8
fixed test on windows (#1667)
* fixed test on windows
2019-04-04 12:56:42 -04:00
JT Olio
09be9964eb internal/version: do version checks much earlier in the process initialization, take 2 (#1666)
* internal/version: do version checks much earlier in the process initialization, take 2

Change-Id: Ida8c7e3757e0deea0ec7aea867d3d27ce97dc134

* linter and test failures

Change-Id: I45b02a16ec1c0f0981227dc842e68dbdf67fdbf4
2019-04-04 17:40:07 +02:00
Stefan Benten
2cf86703a3
Add Versioning Server (#1576)
* Initial Webserver Draft for Version Controlling

* Rename type to avoid confusion

* Move Function Calls into Version Package

* Fix Linting and Language Typos

* Fix Linting and Spelling Mistakes

* Include Copyright

* Include Copyright

* Adjust Version-Control Server to return list of Versions

* Linting

* Improve Request Handling and Readability

* Add Configuration File Option
Add Systemd Service file

* Add Logging to File

* Smaller Changes

* Add Semantic Versioning and refuses outdated Software from Startup (#1612)

* implements internal Semantic Version library

* adds version logging + reporting to process

* Advance SemVer struct for easier handling

* Add Accepted Version Store

* Fix Function

* Restructure

* Type Conversion

* Handle Version String properly

* Add Note about array index

* Set temporary Default Version

* Add Copyright

* Adding Version to Dashboard

* Adding Version Info Log

* Renaming and adding CheckerProcess

* Iteration Sync

* Iteration V2

* linting

* made LogAndReportVersion a go routine

* Refactor to Go Routine

* Add Context to Go Routine and allow Operation if Lookup to Control Server fails

* Handle Unmarshal properly

* Linting

* Relocate Version Checks

* Relocating Version Check and specified default Version for now

* Linting Error Prevention

* Refuse Startup on outdated Version

* Add Startup Check Function

* Straighten Logging

* Dont force Shutdown if --dev flag is set

* Create full Service/Peer Structure for ControlServer

* Linting

* Straighting Naming

* Finish VersionControl Service Layout

* Improve Error Handling

* Change Listening Address

* Move Checker Function

* Remove VersionControl Peer

* Linting

* Linting

* Create VersionClient Service

* Renaming

* Add Version Client to Peer Definitions

* Linting and Renaming

* Linting

* Remove Transport Checks for now

* Move to Client Side Flag

* Remove check

* Linting

* Transport Client Version Intro

* Adding Version Client to Transport Client

* Add missing parameter

* Adding Version Check, to set Allowed = true

* Set Default to true, testing

* Restructuring Code

* Uplink Changes

* Add more proper Defaults

* Renaming of Version struct

* Dont pass Service use Pointer

* Set Defaults for Versioning Checks

* Put HTTP Server in go routine

* Add Versioncontrol to Storj-Sim

* Testplanet Fixes

* Linting

* Add Error Handling and new Server Struct

* Move Lock slightly

* Reduce Race Potentials

* Remove unnecessary files

* Linting

* Add Proper Transport Handling

* small fixes

* add fence for allowed check

* Add Startup Version Check and Service Naming

* make errormessage private

* Add Comments about VersionedClient

* Linting

* Remove Checks that refuse outgoing connections

* Remove release cmd

* Add Release Script

* Linting

* Update to use correct Values

* Move vars private and set minimum default versions for testing builds

* Remove VersionedClient

* Better Error Handling and naked return removal

* Straighten the Regex and string conversion

* Change Check to allows testplanet and storj-sim to run without the
need to pass an LDFlag

* Cosmetic Change to Dashboard

* Cleanup Returns and remove commented code

* Remove Version Check if no build options are passed in

* Pass in Config Values instead of Pointers

* Handle missed Error

* Update Endpoint URL

* Change Type of Release Flag

* Add additional Logging

* Remove Versions Logging of other Services

* minor fixes

Change-Id: I5cc04a410ea6b2008d14dffd63eb5f36dd348a8b
2019-04-03 21:13:39 +02:00
Egon Elbre
fba9a5f945 migration tests for storagenodedb infodb (#1628) 2019-04-02 09:54:09 +02:00
Michal Niewrzal
f80750693c Store bandwidth from orders on satellite (#1586) 2019-04-01 16:14:58 -04:00
Stefan Benten
c50a21d4cf
Fix Dashboard calculation for used space and bandwidth usage (#1615)
* Fix Dashboard calculation for used space and bandwidth usage

* Copy+Paste Err
2019-03-30 14:16:08 +01:00
Egon Elbre
63737e350f
Delete psserver and unused mocks (#1605) 2019-03-29 16:40:06 +02:00
Cameron
cac55a29e4
Add used egress/ingress to storage node dashboard (#1565)
* add egress and ingress to StatSummaryResponse

* print egress and ingress to storagenode dashboard
2019-03-27 15:44:18 -04:00
Michal Niewrzal
bfdfebbde2
Satellite orders receiving (#1564)
This change adds satellite endpoint for receiving OrderLimits sent by storage node.
Change includes:
* wire up orders sender in storage node (also in testplanet)
* saving serial number for OrderLimit in serial_numbers table
* satellite endpoint for receiving, verifying and storing OrderLimit and Order serial number
* initial implementation for Orders DB
* basic test for sending orders to satellite
2019-03-27 11:24:35 +01:00
Natalie Villasana
0fa1d536e7
removes pingbackTimeout (#1556) 2019-03-22 16:06:57 -04:00
Egon Elbre
1d96d25f3f
kademlia ping tracking (#1538) 2019-03-22 15:27:59 +02:00
Egon Elbre
2c5c2c29da
storage node order sending (#1535) 2019-03-21 15:24:26 +02:00
Michal Niewrzal
d7feafe56b Move psserver tests (#1522) 2019-03-20 23:12:00 +02:00
Natalie Villasana
61ee04d363
adds pingbackTimeout to kademlia endpoint (#1518) 2019-03-19 14:30:27 -04:00
Michal Niewrzal
b05cf05649
Restrict slash in bucket name (#1524) 2019-03-19 15:37:28 +01:00
Egon Elbre
7961bcbc92 remove free disk check since it's unreliable (#1516) 2019-03-18 16:18:44 -04:00
Egon Elbre
a24c74c502 fix message formatting (#1512) 2019-03-18 18:02:37 +01:00
Egon Elbre
80916ffb53
storagenode/pieces: ensure we can call Commit or Cancel only once (#1511) 2019-03-18 16:29:54 +02:00
Egon Elbre
117edec54c
Add serial number type (#1508) 2019-03-18 15:08:24 +02:00