Commit Graph

59 Commits

Author SHA1 Message Date
Egon Elbre
fe2bf7d4a6 fix saving uplink public key on satellite (#1797)
* fix save into cert records

* fix transaction logic

* test shouldn't be flaky anymore

* add missing cause method

* restart jenkins
2019-04-23 15:48:57 -04:00
Egon Elbre
f7ed63a119
handle database error checks properly (#1796) 2019-04-23 14:13:57 +03:00
Bill Thorp
17a227e6e9
refactor injuredsegments db so that we can't have duplicates (#1717)
made repairqueue not use a true queue, forbid duplicates
2019-04-16 14:14:09 -04:00
Stefan Benten
bae4c820ee
Add Version Information into KAD Network and SatelliteDB & Change Selection Process (#1648)
* Initial Webserver Draft for Version Controlling

* Rename type to avoid confusion

* Move Function Calls into Version Package

* Fix Linting and Language Typos

* Fix Linting and Spelling Mistakes

* Include Copyright

* Include Copyright

* Adjust Version-Control Server to return list of Versions

* Linting

* Improve Request Handling and Readability

* Add Configuration File Option
Add Systemd Service file

* Add Logging to File

* Smaller Changes

* Add Semantic Versioning and refuses outdated Software from Startup (#1612)

* implements internal Semantic Version library

* adds version logging + reporting to process

* Advance SemVer struct for easier handling

* Add Accepted Version Store

* Fix Function

* Restructure

* Type Conversion

* Handle Version String properly

* Add Note about array index

* Set temporary Default Version

* Add Copyright

* Adding Version to Dashboard

* Adding Version Info Log

* Renaming and adding CheckerProcess

* Iteration Sync

* Iteration V2

* linting

* made LogAndReportVersion a go routine

* Refactor to Go Routine

* Add Context to Go Routine and allow Operation if Lookup to Control Server fails

* Handle Unmarshal properly

* Linting

* Relocate Version Checks

* Relocating Version Check and specified default Version for now

* Linting Error Prevention

* Refuse Startup on outdated Version

* Add Startup Check Function

* Straighten Logging

* Dont force Shutdown if --dev flag is set

* Create full Service/Peer Structure for ControlServer

* Linting

* Straighting Naming

* Finish VersionControl Service Layout

* Improve Error Handling

* Change Listening Address

* Move Checker Function

* Remove VersionControl Peer

* Linting

* Linting

* Create VersionClient Service

* Renaming

* Add Version Client to Peer Definitions

* Linting and Renaming

* Linting

* Remove Transport Checks for now

* Move to Client Side Flag

* Remove check

* Linting

* Transport Client Version Intro

* Adding Version Client to Transport Client

* Add missing parameter

* Adding Version Check, to set Allowed = true

* Set Default to true, testing

* Restructuring Code

* Uplink Changes

* Add more proper Defaults

* Renaming of Version struct

* Dont pass Service use Pointer

* Set Defaults for Versioning Checks

* Put HTTP Server in go routine

* Add Versioncontrol to Storj-Sim

* Testplanet Fixes

* Linting

* Add Error Handling and new Server Struct

* Move Lock slightly

* Reduce Race Potentials

* Remove unnecessary files

* Linting

* Add Proper Transport Handling

* small fixes

* add fence for allowed check

* Add Startup Version Check and Service Naming

* make errormessage private

* Add Comments about VersionedClient

* Linting

* Remove Checks that refuse outgoing connections

* Remove release cmd

* Add Release Script

* Linting

* Update to use correct Values

* Change Timestamp handling

* Adding Protobuf changes back in

* Adding SatelliteDB Changes and adding Storj Node Version to PB

* Add Migration Table

* Add Default Stats for Creation

* Move to BigInt

* Proper SQL Migration

* Ensure minimum Version is passed to the node selection

* Linting...

* Remove VersionedClient and adjust smaller changes from prior merge

* Linting

* Fix PB Message Handling and Query for Node Selection

* some future-proofing type changes

Change-Id: I3cb5018dcccdbc9739fe004d859065992720caaf

* fix a compiler error

Change-Id: If66bb92d8b98e31cd618ecec9c6448ab9b037fa5

* Comment on Constant for Overlay

* Remove NOT NULL and add epoch call as function

* add versions to bootstrap and satellites

Change-Id: I436944589ea5f21600cdd997742a84fe0b16e47b

* Change Update Migration

* Fix DB Migration

* Increase Timeout temporarily, to see whats going on

* Remove unnecessary const and vars
Cleanup Function calls from deprecated NodeVersion struct

* Updated Protopuf, removed depcreated Code from Inspector

* Implement NodeVersion into InfoResponse

* Regenerated locked.go

* Linting

* Fix Tests

* Remove unnecessary constant

* Update Function and Flag Description

* Remove Empty Stat Creation

* return properly with error

* Remove unnecessary struct

* simplify migration step

* Update Inspector to return Version Info

* Update local Endpoint Version Handling

* Reset Travis Timeout

* Add Default for CommitHash

* single quotes
2019-04-10 08:04:24 +02:00
Natalie Villasana
2533aa1ab1
logs dbx constraint violation error (#1712) 2019-04-08 18:03:48 -04:00
Jennifer Li Johnson
8549421385
Remove bw from tally service + query bandwidth in rollup service (#1618) 2019-04-04 11:20:59 -04:00
Yaroslav Vorobiov
b38b87cb14
Console usage rollup api (#1664) 2019-04-04 17:56:20 +03:00
Jess G
d51bdf14df
project usage limiting (#1561)
* reorg uplink cmd files for consistency

* init implementation of usage limiting

* Revert "reorg uplink cmd files for consistency"

This reverts commit 91ced7639bf36fc8af1db237b01e233ca92f1890.

* add changes per CR comments

* fix custom query to use rebind

* updates per convo about what to limit on

* changes per comments

* fix syntax and comments

* add integration test, add db methods for test

* update migration, add rebind to query

* update testdata for psql

* remove unneeded drop index statement

* fix migrations, fix calculate usage limit

* fix comment

* add audit test back

* change methods to use bucketName/projectID, fix tests

* add changes per CR comments

* add test for uplink upload and err ssg

* changes per CR comments

* check get/put limit separately
2019-04-02 11:21:18 -07:00
Michal Niewrzal
f80750693c Store bandwidth from orders on satellite (#1586) 2019-04-01 16:14:58 -04:00
Kaloyan Raev
034c283eda Set last_contact_success when a node is initially added to overlay cache (#1632) 2019-04-01 12:42:06 -06:00
Cameron
6d43832c4f
record bucket data into bucket_storage_tally table (#1595)
* add MetadataSize to stats

* add logic for accumulating bucket stats in calculateAtRestData

* rename stats to BucketTally, move to accounting package

* define method on accountingDB for inserting bucketTallies

* insert bucketTallies into bucket_storage_tally table
2019-04-01 09:42:17 -04:00
Kaloyan Raev
f9ba935286
Merge overlay_cache_nodes into nodes table (#1581) 2019-03-29 10:53:43 +02:00
Jennifer Li Johnson
b3a8beb1bc
Updates accounting tables models and migration (#1584)
* go gen

* undo changes to bucket usage

* update locked

* spacing

* moves changes to migration v11

* minor changes to fix lint and test err

* change sql to fix errs
2019-03-27 22:46:49 -04:00
Yehor Butko
86bf3dee9f
V3-1307 combine first and last name to full name (#1569)
* V3-1307 combine first and last name to full name
2019-03-27 14:33:32 +02:00
Michal Niewrzal
bfdfebbde2
Satellite orders receiving (#1564)
This change adds satellite endpoint for receiving OrderLimits sent by storage node.
Change includes:
* wire up orders sender in storage node (also in testplanet)
* saving serial number for OrderLimit in serial_numbers table
* satellite endpoint for receiving, verifying and storing OrderLimit and Order serial number
* initial implementation for Orders DB
* basic test for sending orders to satellite
2019-03-27 11:24:35 +01:00
Stefan Benten
3db903fe40
Move from Unique to Index on SatelliteDB (#1563)
* Move from Unique to Index

* Remove Index

* Make some more Indexes Unique and adjust migration

* Fix Migration Statements

* Fix Typo

* Fix Migration of older Table

* Exchange DROP statement

* Remove "if not exists"

* Revert Change in old Migration
2019-03-26 11:34:30 +01:00
Egon Elbre
297b393dfd database for tracking information without storing orders / agreements (#1551)
* initial schema declaration

* fix typo

* split database

* fix syntax errors and generate code

* add queries for serial number manipulation

* update comment

* add migration script

* fix comments

* add new data

* fix bucket id
2019-03-22 11:54:22 -07:00
Egon Elbre
694b6dc1da
make tests run faster (#1553) 2019-03-22 15:14:17 +02:00
Yehor Butko
c6f8d82978
V3-1319 Add an authorization token field for account creation backend (#1492)
* V3-1319 Add an authorization token field for account creation on the satellite UI backend
2019-03-19 19:55:43 +02:00
paul cannon
cd91a22e0f better way to check if err is ErrorCode_NoRows (#1453)
* better way to check if err is ErrorCode_NoRows
2019-03-18 20:15:27 -04:00
Cameron
c7ffbe1c28
Add ability to view irreparable segments on satellite (#1448)
* define irreparable inspector protobuf

* add IrreparableDB method GetLimited

* fill out irreparable inspector API

* add IrreparableInspector server to satellite, fix small error

* refactor IrreparableDB to use pb.IrreparableSegment instead of irreparable.RemoteSegmentInfo
2019-03-15 16:21:52 -04:00
Bill Thorp
52e829c6de
framework for deleting expired Orders from Satellite (#1436)
framework for deleting old bandwidth agreements
2019-03-12 16:57:21 -04:00
Bill Thorp
1f4e1a3ab0
framework to delete raw accounting tallies (#1439)
framework to delete raw accounting tallies, renamed Query
2019-03-12 09:08:23 -04:00
Yaroslav Vorobiov
9cf56c03fa
Bucket usage info rollup table creation and implementation (#1340) 2019-03-06 17:54:48 +02:00
Dylan Lott
9abf191b01
Change payments to use statDB for wallet addresses rather than Overlay cache (#1368)
* Wiring up DumpNodes response for Inspector

* Finalize everything and test that it works

* Get Count and DumpNodes working for Overlay Cache

* WIP updating payment rollup to check statDB instead of overlay

* FIrst pass at updating statDB to take wallet and email

* Passing tests

* use pb.NodeOperator instead of Meta struct

* remove TODO

* revert go.mod

* Get SQL migration working correctly

* Changes Meta to Operator in NodeStats struct

* Adds update operator logic for statDB

* Fix db migrate tests - added v5 snapshot

* User friendly msg for missing snapshot version

* Passing tests

* Change node update to happen in discovery instead of in overlay

* Fix logic and update function calls

* Update comment on UpdateOperator interface method

* Update name of parameter

* Change type of argument to UpdateOperator

* Updates statDB tests
2019-03-01 10:46:34 -07:00
Jennifer Li Johnson
6ecda190cb
Jj/payments cleanup (#1379)
* Removes date from payments csv and re-query of wallet addr

* removes dbx get wallet addr

* switch order in csv
2019-02-28 12:13:59 -05:00
Egon Elbre
3beaeebdb6
Update dbx (#1300) 2019-02-12 21:46:03 +02:00
Yehor Butko
8156d911fa
Updating account activation flow (#1251)
* Updating account activation flow

* Updated integration tests, createUserMutation updated

* removing redundant index

* removed redundant testing code

* review comments fixed
2019-02-11 12:33:56 +02:00
Bill Thorp
0b35762105
Convert Payments to use SQL, for SUM() and Wallet (#1266)
* payments query no longer DBX, using SQL

* sum in SQL

* removed old function

* fixed rollup test

* wrap errors

* removed DBX code
2019-02-07 15:26:55 -05:00
aligeti
b736ae4823
Store the uplinks public key on the satellite so that it can verify bandwidth requests in the future (#1042)
* integrated with bwagreement & psserver

* integrated with pointerdb

* code review updates

* refactor after code review

* uplinkdb rename to certdb

* Code review changes
2019-02-07 14:22:49 -05:00
Cameron
bfe8060f4d
Rollup tests (#1224)
* createdAt field no longer autoinsert

* rollup tests
2019-02-06 23:16:24 -05:00
Yaroslav Vorobiov
2ff0d9d435
Add bucket to project relationship on satellite (#1143)
* integrate console api keys with buckets in pointerdb

* fix test

* fix tests kvmetainfo

* linter fix

* disable account activation

* fix test

* review fixes

* fix comments

* little refactoring

* remove debug println

* fix typo

* disable activation in a propper way

* fix test

* fix imports

* fix uplink count in testplanet

* move key creation to planet.newUplink
2019-02-05 19:22:17 +02:00
Michal Niewrzal
383cb05e8e
Divide uplink and gateway params set (#1026)
* Divide uplink and gateway params set

* attempt to fix docker

* attempt to fix all in one

* test

* more reorganization

* fix compilation error

* fix imports order

* fix dependency

* rename structs

* keep minio params for now

* review comments

* remove manual flag check
2019-02-04 16:50:06 +01:00
Bill Thorp
0f662b8e38 sql based tally (#1166)
* WIP

* wacky changes

* more

* it builds.... IT BUILDS!!!

* fixed SQL, broke out saving tallies for testing

* shorter lines

* fixed SQL, moved tally_test to testplanet

* lint

* WIP logic error preventing PUT and GETs to same serialnum

* fixed BWA test

* fixed temporary brain failure

* eliminated magic numbers

* cleaned up satellite uplink stats

* use errs.Combine instead

* thrashing

* fixed tally erroneous error msg

* fixed tally test

* lint

* SQL syntax attempt to fix

* spelling error

* made bwa db test resist old postgres data

* postgres pk error msg hunting

* postgres pk error msg hunting

* postgres

* err might be nil?

* fixed error logging bug

* hopefully solved postgres issue

* using rebind

* moved tests to _test package

* fixing test dirs

* finally made sense of Egons package name feedback

* UTC, array fixes
2019-02-01 13:50:12 -05:00
Yehor Butko
19bc01c19a
V3-1091 Extend Users table with IsActive functionality (#1170)
* V3-1091 Extend Users table with IsActive functionality

* fixed review comments
2019-01-30 17:04:40 +02:00
Bill Thorp
e94f6408ff
bw agreement db refactor (#1158) 2019-01-28 16:16:21 -05:00
Bogdan Artemenko
b75f06e229
Removed TermsAccepted in project from satellite console and web. (#1155)
* Removed TermsAccepted from satellite console and web.
2019-01-28 20:20:33 +02:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Jennifer Li Johnson
e6fbf63620
Generate Payments Report (#1079) 2019-01-17 13:34:13 -05:00
Yaroslav Vorobiov
d832789481
ConsoleDB merge and buckets table creation (#1066)
* init

* db merge

* bucket table renamed to bucket info

* remove id and fix comments

* fix imports
2019-01-16 15:23:28 -05:00
Bill Thorp
342dc857f5 rollup query (#1056)
* implemention notes

* more notes

* starting rollup query

* not working yet

* fixed build

* fixed cfg bug

* change context cancelled errs to debugs

* using byte hours for at rest tally

* revert changes to go.mod

* comment fixes

* prevent double recording tallies in rollup

* linting

* stop leaking dbx

* nodeid changes

* fix build
2019-01-16 14:30:33 -05:00
Egon Elbre
f8906ce000
Change overlay cache database interface (#1047) 2019-01-15 11:08:45 -05:00
Cameron
ef50bbf8b6
satellites reject expired BWAs (#1015)
* add 45 day expiration to PBAs

* add expiration field to relevant areas, DeleteExpired placeholder

* reject expired BWAs

* test for expired BWAs

* add BwExpiration config value
2019-01-10 13:30:55 -05:00
aligeti
457f909a74 Satellite accepts manipulated paychecks V3-1023 (#972)
* fix - Satellite crashing on receiving a manipulated bandwidthagreement

* provider.PeerIdentityFromContext called twice. Remove one

* add storage node ID to serial number

* remove serialNum query and transaction

* add uuid to GeneratePayerBandwidthAllocation for testing

* enable expected failure on duplicate serialnum cases

* Revert "enable expected failure on duplicate serialnum cases"

This reverts commit 5948f43ed1741c280f0bb34a86c1c490365417bc.

* enable expected failure on duplicate serialnum cases
2019-01-09 10:02:03 -05:00
Egon Elbre
0ca03b41e2
Use generated locking database implementation (#947) 2019-01-02 19:53:27 +02:00
Michal Niewrzal
8c00b2109e
Format satellite.dbx (#930)
* Format satellite.dbx

* remove empty lines
2018-12-27 22:21:50 +01:00
Egon Elbre
4346cd060f
Implement mutex around satellitedb (#932) 2018-12-27 11:56:25 +02:00
Cameron
f70b826fd4
repair queue masterDB support (#865)
* add injuredsegment model to satellitedb.dbx

* add context to queue.RepairQueue interface

* use queue.RepairQueue interface, use masterdb
2018-12-21 10:11:19 -05:00
Michal Niewrzal
bbe1dbb92c
Fix overlay commands after master db merge (#895)
* Fix overlay commands after master db merge

* remove unused var

* remove unused comment
2018-12-18 21:41:31 +01:00
Jennifer Li Johnson
01df85704c
Updates accounting tables and allows tally to update at raw table for at rest data (#874) 2018-12-18 12:18:42 -05:00