Commit Graph

99 Commits

Author SHA1 Message Date
Bryan White
faf5fae3f9
Identity versioning (#1389) 2019-04-08 20:15:19 +02:00
Maximillian von Briesen
bb3b4e4816 Data repair integration test (#1582) 2019-04-08 13:33:47 -04:00
Maximillian von Briesen
887b20c31b Enable uplink TestDownloadWithSomeNodesOffline (#1580) 2019-04-04 13:35:19 -04:00
Kaloyan Raev
bfdee78f05
Introduce NodeDossier type and cleanup overlay.DB interface (#1626)
Co-authored-by: Natalie Villasana <navillasa@gmail.com>
Co-authored-by: Bill Thorp <bill3000@hotmail.com>
2019-04-04 19:34:36 +03:00
Stefan Benten
2cf86703a3
Add Versioning Server (#1576)
* Initial Webserver Draft for Version Controlling

* Rename type to avoid confusion

* Move Function Calls into Version Package

* Fix Linting and Language Typos

* Fix Linting and Spelling Mistakes

* Include Copyright

* Include Copyright

* Adjust Version-Control Server to return list of Versions

* Linting

* Improve Request Handling and Readability

* Add Configuration File Option
Add Systemd Service file

* Add Logging to File

* Smaller Changes

* Add Semantic Versioning and refuses outdated Software from Startup (#1612)

* implements internal Semantic Version library

* adds version logging + reporting to process

* Advance SemVer struct for easier handling

* Add Accepted Version Store

* Fix Function

* Restructure

* Type Conversion

* Handle Version String properly

* Add Note about array index

* Set temporary Default Version

* Add Copyright

* Adding Version to Dashboard

* Adding Version Info Log

* Renaming and adding CheckerProcess

* Iteration Sync

* Iteration V2

* linting

* made LogAndReportVersion a go routine

* Refactor to Go Routine

* Add Context to Go Routine and allow Operation if Lookup to Control Server fails

* Handle Unmarshal properly

* Linting

* Relocate Version Checks

* Relocating Version Check and specified default Version for now

* Linting Error Prevention

* Refuse Startup on outdated Version

* Add Startup Check Function

* Straighten Logging

* Dont force Shutdown if --dev flag is set

* Create full Service/Peer Structure for ControlServer

* Linting

* Straighting Naming

* Finish VersionControl Service Layout

* Improve Error Handling

* Change Listening Address

* Move Checker Function

* Remove VersionControl Peer

* Linting

* Linting

* Create VersionClient Service

* Renaming

* Add Version Client to Peer Definitions

* Linting and Renaming

* Linting

* Remove Transport Checks for now

* Move to Client Side Flag

* Remove check

* Linting

* Transport Client Version Intro

* Adding Version Client to Transport Client

* Add missing parameter

* Adding Version Check, to set Allowed = true

* Set Default to true, testing

* Restructuring Code

* Uplink Changes

* Add more proper Defaults

* Renaming of Version struct

* Dont pass Service use Pointer

* Set Defaults for Versioning Checks

* Put HTTP Server in go routine

* Add Versioncontrol to Storj-Sim

* Testplanet Fixes

* Linting

* Add Error Handling and new Server Struct

* Move Lock slightly

* Reduce Race Potentials

* Remove unnecessary files

* Linting

* Add Proper Transport Handling

* small fixes

* add fence for allowed check

* Add Startup Version Check and Service Naming

* make errormessage private

* Add Comments about VersionedClient

* Linting

* Remove Checks that refuse outgoing connections

* Remove release cmd

* Add Release Script

* Linting

* Update to use correct Values

* Move vars private and set minimum default versions for testing builds

* Remove VersionedClient

* Better Error Handling and naked return removal

* Straighten the Regex and string conversion

* Change Check to allows testplanet and storj-sim to run without the
need to pass an LDFlag

* Cosmetic Change to Dashboard

* Cleanup Returns and remove commented code

* Remove Version Check if no build options are passed in

* Pass in Config Values instead of Pointers

* Handle missed Error

* Update Endpoint URL

* Change Type of Release Flag

* Add additional Logging

* Remove Versions Logging of other Services

* minor fixes

Change-Id: I5cc04a410ea6b2008d14dffd63eb5f36dd348a8b
2019-04-03 21:13:39 +02:00
Jess G
d51bdf14df
project usage limiting (#1561)
* reorg uplink cmd files for consistency

* init implementation of usage limiting

* Revert "reorg uplink cmd files for consistency"

This reverts commit 91ced7639bf36fc8af1db237b01e233ca92f1890.

* add changes per CR comments

* fix custom query to use rebind

* updates per convo about what to limit on

* changes per comments

* fix syntax and comments

* add integration test, add db methods for test

* update migration, add rebind to query

* update testdata for psql

* remove unneeded drop index statement

* fix migrations, fix calculate usage limit

* fix comment

* add audit test back

* change methods to use bucketName/projectID, fix tests

* add changes per CR comments

* add test for uplink upload and err ssg

* changes per CR comments

* check get/put limit separately
2019-04-02 11:21:18 -07:00
Simon Guindon
d3885b7b78
Change crypto/rand to use math/rand in pgutil (#1589)
* Added retry logic and tests for handling if crypto/rand fails.

* Added retry logic and tests for handling if crypto/rand fails.

* Fixing linting error.

* Removing use of `crypto/rand` for use of `math/rand`.

* Changing code comment about why we ignore this error.
2019-04-02 12:52:25 -04:00
Egon Elbre
872bd5d7c1
Remove pointerdb.Server (#1609) 2019-03-30 13:21:49 +02:00
Michal Niewrzal
bfdfebbde2
Satellite orders receiving (#1564)
This change adds satellite endpoint for receiving OrderLimits sent by storage node.
Change includes:
* wire up orders sender in storage node (also in testplanet)
* saving serial number for OrderLimit in serial_numbers table
* satellite endpoint for receiving, verifying and storing OrderLimit and Order serial number
* initial implementation for Orders DB
* basic test for sending orders to satellite
2019-03-27 11:24:35 +01:00
Natalie Villasana
435bd690cd
skips TestDownloadWithSomeNodesOffline (#1579) 2019-03-26 14:45:23 -04:00
Natalie Villasana
5b48a48a79
adds test for correct download with non-critical amount of nodes offline (#1574) 2019-03-26 14:09:44 -04:00
Bryan White
ef99c1657f
TLS extension handling overhaul (#1458) 2019-03-25 22:52:12 +01:00
Yaroslav Vorobiov
cce6fa8fd3
added LOGIN auth scheme to satellite mail service (#1552) 2019-03-24 02:08:41 +02:00
Egon Elbre
694b6dc1da
make tests run faster (#1553) 2019-03-22 15:14:17 +02:00
Kaloyan Raev
30dfc2b20c
Remove PointerDB client (#1520) 2019-03-22 11:01:49 +02:00
Maximillian von Briesen
db64d6590b Add repairer tests (#1494) 2019-03-21 16:26:56 +02:00
Michal Niewrzal
56251570ef Remove overlay client (#1510) 2019-03-18 21:46:10 +02:00
Egon Elbre
05d148aeb5
Storage node and upload/download protocol refactor (#1422)
refactor storage node server
refactor upload and download protocol
2019-03-18 12:55:06 +02:00
Bill Thorp
66718cc5e6
Development defaults for configuration (#1430)
added --dev command line option, cfgstruct.DevFlag(), and cfgstruct.SetupFlag()
2019-03-12 08:51:06 -04:00
Egon Elbre
0f784279bc
Ensure we only listen on 127.0.0.1 (#1433) 2019-03-08 18:24:50 +02:00
Egon Elbre
ead4a0ae95
Add new piece id definition (#1411) 2019-03-06 11:17:35 +02:00
Yehor Butko
3e2c101bd8
V3-1152 Node bootstrap web backend (#1327)
* V3-1152 Node bootstrap
2019-03-05 12:38:21 +02:00
Yaroslav Vorobiov
a30ba4eca8
Add mail service to the satellite (#1302) 2019-03-02 17:22:20 +02:00
Bryan White
fde0020c68
dial node/address with bad server cert (#1342) 2019-02-26 19:35:16 +01:00
Bryan White
b652b6a522 use peer ca whitelist in testplanet (#1337) 2019-02-25 09:38:03 +02:00
Natalie Villasana
c3d3f41d30 removes some SignedMessage use (#1258)
Removes most instances of pb.SignedMessage (there's more to take out but they shouldn't hurt anyone as is).

There used to be places in psserver where a PieceID was hmac'd with the SatelliteID, which was gotten from a SignedMessage. This PR makes it so some functions access the SatelliteID from the Payer Bandwidth Allocation instead.

This requires passing a SatelliteID into psserver functions where they weren't before, so the following proto messages have been changed:

 * PieceId - satellite_id field added
   This is so the psserver.Piece function has access to the SatelliteID when it needs to get the namespaced pieceID.
   This proto message should probably be renamed to PieceRequest, or a new PieceRequest message should be created so this isn't misnamed.

 * PieceDelete - satellite_id field added
   This is so the psserver.Delete function has access to the SatelliteID when receiving a request to Delete.
2019-02-19 23:36:08 -06:00
Michal Niewrzal
8d685217e4
Storagenode migrations (#1299)
* creates initial migration for psdb
* add test mechanism to validate migration to every version
* fix few small issues in versions.go and context.go
2019-02-19 10:39:04 +01:00
Egon Elbre
12261c0624
Satellite migrations (#1301) 2019-02-14 23:55:21 +02:00
Egon Elbre
497fb756fe
create dbutil package for migration testing (#1305) 2019-02-13 18:06:34 +02:00
JT Olio
2a59679766 pkg/transport: require tls configuration for dialing (#1286)
* separate TLS options from server options (because we need them for dialing too)
* stop creating transports in multiple places
* ensure that we actually check revocation, whitelists, certificate signing, etc, for all connections.
2019-02-11 13:17:32 +02:00
JT Olio
ea28a9a581 pkg/transport: fail on bad certificates (#1285) 2019-02-11 13:02:49 +02:00
Egon Elbre
17a7f9d002
Enable planet Merging test (#1281) 2019-02-08 22:35:59 +02:00
Egon Elbre
12bdc0ef21
Disable revocation table for testplanet. (#1283) 2019-02-08 21:27:31 +02:00
Egon Elbre
e37e0c1b5f
Fix server config usage (#1282) 2019-02-08 20:57:17 +02:00
Michal Niewrzal
c7ad427d95
Move metainfo config (#1234)
* Move metainfo config

* move config to uplink

* reorganize code

* use satellite apikey

* comment update
2019-02-08 13:57:35 +01:00
Egon Elbre
9c1e299f3c
Ensure everyone sees everyone else (#1275) 2019-02-08 11:25:13 +02:00
paul cannon
ef61c170b1
Consolidate key/cert/signature encoding and decoding (#1243) 2019-02-07 12:40:28 -06:00
Egon Elbre
bb11d83ed0
Proper planet shutdown (#1249) 2019-02-06 15:19:14 +02:00
Egon Elbre
fdbe2db273
Remove node package and simplify DHT interface (#1233) 2019-02-06 14:37:17 +02:00
Egon Elbre
ab83f0f077
Fix parallel execution sharing (#1247) 2019-02-06 11:16:05 +02:00
Egon Elbre
b5d86aa0b4
Make testplanet.Run and satellitedbtest.Run parallel by default (#1231) 2019-02-05 21:44:00 +02:00
Egon Elbre
07412698a9
Use lower bcrypt cost for console.Service when running tests and disable flaky tests (#1236) 2019-02-05 19:31:53 +02:00
Yaroslav Vorobiov
2ff0d9d435
Add bucket to project relationship on satellite (#1143)
* integrate console api keys with buckets in pointerdb

* fix test

* fix tests kvmetainfo

* linter fix

* disable account activation

* fix test

* review fixes

* fix comments

* little refactoring

* remove debug println

* fix typo

* disable activation in a propper way

* fix test

* fix imports

* fix uplink count in testplanet

* move key creation to planet.newUplink
2019-02-05 19:22:17 +02:00
Egon Elbre
6564357e13
Ensure that tally tests run against postgres (#1227) 2019-02-05 16:23:45 +02:00
Egon Elbre
87d6410b50 Revert "Remove node package and simplify DHT interface."
This reverts commit 03ec1ff92d.
2019-02-05 10:38:48 +02:00
Egon Elbre
03ec1ff92d Remove node package and simplify DHT interface. 2019-02-05 10:37:24 +02:00
Dylan Lott
21a58d0ece
node graveyard resurrection (#1186)
* deletes nodes from cache if unresponsive

* update error messages

* WIP restructuring discovery tickers

* passes tests

* Removes log messages

* Passes up error in graveyard ticker
2019-02-04 14:52:06 -07:00
Egon Elbre
c37e0c1b6d
testplanet.Run (#1222) 2019-02-04 22:37:46 +02:00
Michal Niewrzal
2f8b3da1af
Upload/download for testplanet Uplink (#1183)
* Upload/download for testplanet Uplink

* check error in tests

* cleanup

* refactor node -> uplink

* add missing test file

* rest of refactoring

* workaround to resolve cycles in tests

* rename method

* add missing comments

* review comments

* use KiB
2019-02-04 17:56:10 +01:00
Egon Elbre
d1cc0dd85b
testplanet: add capability to stop peers (#1221) 2019-02-04 17:40:37 +02:00