Commit Graph

177 Commits

Author SHA1 Message Date
Stefan Benten
0cefb12a28
Set Default Node Type to invalid, to avoid cache issues (#1746) 2019-04-12 15:30:19 +02:00
Egon Elbre
0eee46524d update linter to v1.16 (#1741)
* update to v1.16

* use full version number

* fix linter issues

* fix comment

* nicer comment

* restart travis

* restart travis
2019-04-11 13:32:40 -04:00
Bogdan Artemenko
6a50b187eb
Implemented password reset on satellite console web. (#1665) 2019-04-10 22:16:10 +03:00
Stefan Benten
bae4c820ee
Add Version Information into KAD Network and SatelliteDB & Change Selection Process (#1648)
* Initial Webserver Draft for Version Controlling

* Rename type to avoid confusion

* Move Function Calls into Version Package

* Fix Linting and Language Typos

* Fix Linting and Spelling Mistakes

* Include Copyright

* Include Copyright

* Adjust Version-Control Server to return list of Versions

* Linting

* Improve Request Handling and Readability

* Add Configuration File Option
Add Systemd Service file

* Add Logging to File

* Smaller Changes

* Add Semantic Versioning and refuses outdated Software from Startup (#1612)

* implements internal Semantic Version library

* adds version logging + reporting to process

* Advance SemVer struct for easier handling

* Add Accepted Version Store

* Fix Function

* Restructure

* Type Conversion

* Handle Version String properly

* Add Note about array index

* Set temporary Default Version

* Add Copyright

* Adding Version to Dashboard

* Adding Version Info Log

* Renaming and adding CheckerProcess

* Iteration Sync

* Iteration V2

* linting

* made LogAndReportVersion a go routine

* Refactor to Go Routine

* Add Context to Go Routine and allow Operation if Lookup to Control Server fails

* Handle Unmarshal properly

* Linting

* Relocate Version Checks

* Relocating Version Check and specified default Version for now

* Linting Error Prevention

* Refuse Startup on outdated Version

* Add Startup Check Function

* Straighten Logging

* Dont force Shutdown if --dev flag is set

* Create full Service/Peer Structure for ControlServer

* Linting

* Straighting Naming

* Finish VersionControl Service Layout

* Improve Error Handling

* Change Listening Address

* Move Checker Function

* Remove VersionControl Peer

* Linting

* Linting

* Create VersionClient Service

* Renaming

* Add Version Client to Peer Definitions

* Linting and Renaming

* Linting

* Remove Transport Checks for now

* Move to Client Side Flag

* Remove check

* Linting

* Transport Client Version Intro

* Adding Version Client to Transport Client

* Add missing parameter

* Adding Version Check, to set Allowed = true

* Set Default to true, testing

* Restructuring Code

* Uplink Changes

* Add more proper Defaults

* Renaming of Version struct

* Dont pass Service use Pointer

* Set Defaults for Versioning Checks

* Put HTTP Server in go routine

* Add Versioncontrol to Storj-Sim

* Testplanet Fixes

* Linting

* Add Error Handling and new Server Struct

* Move Lock slightly

* Reduce Race Potentials

* Remove unnecessary files

* Linting

* Add Proper Transport Handling

* small fixes

* add fence for allowed check

* Add Startup Version Check and Service Naming

* make errormessage private

* Add Comments about VersionedClient

* Linting

* Remove Checks that refuse outgoing connections

* Remove release cmd

* Add Release Script

* Linting

* Update to use correct Values

* Change Timestamp handling

* Adding Protobuf changes back in

* Adding SatelliteDB Changes and adding Storj Node Version to PB

* Add Migration Table

* Add Default Stats for Creation

* Move to BigInt

* Proper SQL Migration

* Ensure minimum Version is passed to the node selection

* Linting...

* Remove VersionedClient and adjust smaller changes from prior merge

* Linting

* Fix PB Message Handling and Query for Node Selection

* some future-proofing type changes

Change-Id: I3cb5018dcccdbc9739fe004d859065992720caaf

* fix a compiler error

Change-Id: If66bb92d8b98e31cd618ecec9c6448ab9b037fa5

* Comment on Constant for Overlay

* Remove NOT NULL and add epoch call as function

* add versions to bootstrap and satellites

Change-Id: I436944589ea5f21600cdd997742a84fe0b16e47b

* Change Update Migration

* Fix DB Migration

* Increase Timeout temporarily, to see whats going on

* Remove unnecessary const and vars
Cleanup Function calls from deprecated NodeVersion struct

* Updated Protopuf, removed depcreated Code from Inspector

* Implement NodeVersion into InfoResponse

* Regenerated locked.go

* Linting

* Fix Tests

* Remove unnecessary constant

* Update Function and Flag Description

* Remove Empty Stat Creation

* return properly with error

* Remove unnecessary struct

* simplify migration step

* Update Inspector to return Version Info

* Update local Endpoint Version Handling

* Reset Travis Timeout

* Add Default for CommitHash

* single quotes
2019-04-10 08:04:24 +02:00
Yehor Butko
ba891538af
V3-1399 Error messages (#1715)
* V3-1399 Error messages
2019-04-10 03:15:12 +03:00
Yaroslav Vorobiov
71843632a0
Console add bucket usage report (#1706) 2019-04-09 19:14:19 -04:00
Cameron
32192aca10
convert times to UTC before entering as utimestamp (#1708) 2019-04-09 15:12:58 -04:00
Jess G
55bfe507e5
add tests for tally and accounting db (#1675)
* add tally calcAtRestData test

* add account db tests

* move tests from sadb to accounting pkg

* add func to calcEncSize

* remove unneeded pad changes

* update comment

* add changes to comments
2019-04-09 09:48:35 -04:00
Michal Niewrzal
a684b7b953
Test number of pieces lower than repair threshold (#1683) 2019-04-09 15:31:19 +02:00
Yaroslav Vorobiov
2218793c55 Added error logging to account activation handler (#1647) 2019-04-09 08:20:29 -04:00
Natalie Villasana
2533aa1ab1
logs dbx constraint violation error (#1712) 2019-04-08 18:03:48 -04:00
Jess G
5dfe28a8c3
use allocated instead of settled (#1700)
* use allocated instead of settled

* add expansion factor

* changes per CR
2019-04-08 14:35:54 -07:00
Bill Thorp
061deb6add
only showing online node count (#1702) 2019-04-08 16:55:04 -04:00
Natalie Villasana
0d006661f4
returns an error if tx.Get_Node_By_Id returns nil node (#1698) 2019-04-08 13:52:53 -04:00
Maximillian von Briesen
bb3b4e4816 Data repair integration test (#1582) 2019-04-08 13:33:47 -04:00
Yehor Butko
f4430f9063
Satellite frontned empty states (#1682)
* V3-1421 Empty State Pages in no Project Case
2019-04-05 18:08:14 +03:00
Kaloyan Raev
92c1121072
Fix "Number of valid pieces is lower then success threshold" (#1679) 2019-04-05 13:19:20 +03:00
Michal Niewrzal
5ea797889b
Store inline bandiwdth (#1619) 2019-04-05 09:42:56 +02:00
Alexander Leitner
06e2e7a1e0
Filehealth cli (#1649)
* print CSV formatted data for checking object and segment health
2019-04-04 13:11:19 -04:00
Kaloyan Raev
bfdee78f05
Introduce NodeDossier type and cleanup overlay.DB interface (#1626)
Co-authored-by: Natalie Villasana <navillasa@gmail.com>
Co-authored-by: Bill Thorp <bill3000@hotmail.com>
2019-04-04 19:34:36 +03:00
Jess G
1638d96b16
update ProjectStorageTotals func to get all records for project (#1657)
* update ProjectStorageTotals func to get all records for project

* fix var name

* update test to catch bug

* fix spelling

* modify query so we only have to make 1
2019-04-04 09:01:04 -07:00
JT Olio
09be9964eb internal/version: do version checks much earlier in the process initialization, take 2 (#1666)
* internal/version: do version checks much earlier in the process initialization, take 2

Change-Id: Ida8c7e3757e0deea0ec7aea867d3d27ce97dc134

* linter and test failures

Change-Id: I45b02a16ec1c0f0981227dc842e68dbdf67fdbf4
2019-04-04 17:40:07 +02:00
Jennifer Li Johnson
8549421385
Remove bw from tally service + query bandwidth in rollup service (#1618) 2019-04-04 11:20:59 -04:00
Yaroslav Vorobiov
b38b87cb14
Console usage rollup api (#1664) 2019-04-04 17:56:20 +03:00
Michal Niewrzal
6d86610bcc
Fix UseSerialNumber panic (#1663)
This change fixes UseSerialNumber and case when given serial number doesn't exists in DB.
2019-04-04 16:42:01 +02:00
Stefan Benten
2cf86703a3
Add Versioning Server (#1576)
* Initial Webserver Draft for Version Controlling

* Rename type to avoid confusion

* Move Function Calls into Version Package

* Fix Linting and Language Typos

* Fix Linting and Spelling Mistakes

* Include Copyright

* Include Copyright

* Adjust Version-Control Server to return list of Versions

* Linting

* Improve Request Handling and Readability

* Add Configuration File Option
Add Systemd Service file

* Add Logging to File

* Smaller Changes

* Add Semantic Versioning and refuses outdated Software from Startup (#1612)

* implements internal Semantic Version library

* adds version logging + reporting to process

* Advance SemVer struct for easier handling

* Add Accepted Version Store

* Fix Function

* Restructure

* Type Conversion

* Handle Version String properly

* Add Note about array index

* Set temporary Default Version

* Add Copyright

* Adding Version to Dashboard

* Adding Version Info Log

* Renaming and adding CheckerProcess

* Iteration Sync

* Iteration V2

* linting

* made LogAndReportVersion a go routine

* Refactor to Go Routine

* Add Context to Go Routine and allow Operation if Lookup to Control Server fails

* Handle Unmarshal properly

* Linting

* Relocate Version Checks

* Relocating Version Check and specified default Version for now

* Linting Error Prevention

* Refuse Startup on outdated Version

* Add Startup Check Function

* Straighten Logging

* Dont force Shutdown if --dev flag is set

* Create full Service/Peer Structure for ControlServer

* Linting

* Straighting Naming

* Finish VersionControl Service Layout

* Improve Error Handling

* Change Listening Address

* Move Checker Function

* Remove VersionControl Peer

* Linting

* Linting

* Create VersionClient Service

* Renaming

* Add Version Client to Peer Definitions

* Linting and Renaming

* Linting

* Remove Transport Checks for now

* Move to Client Side Flag

* Remove check

* Linting

* Transport Client Version Intro

* Adding Version Client to Transport Client

* Add missing parameter

* Adding Version Check, to set Allowed = true

* Set Default to true, testing

* Restructuring Code

* Uplink Changes

* Add more proper Defaults

* Renaming of Version struct

* Dont pass Service use Pointer

* Set Defaults for Versioning Checks

* Put HTTP Server in go routine

* Add Versioncontrol to Storj-Sim

* Testplanet Fixes

* Linting

* Add Error Handling and new Server Struct

* Move Lock slightly

* Reduce Race Potentials

* Remove unnecessary files

* Linting

* Add Proper Transport Handling

* small fixes

* add fence for allowed check

* Add Startup Version Check and Service Naming

* make errormessage private

* Add Comments about VersionedClient

* Linting

* Remove Checks that refuse outgoing connections

* Remove release cmd

* Add Release Script

* Linting

* Update to use correct Values

* Move vars private and set minimum default versions for testing builds

* Remove VersionedClient

* Better Error Handling and naked return removal

* Straighten the Regex and string conversion

* Change Check to allows testplanet and storj-sim to run without the
need to pass an LDFlag

* Cosmetic Change to Dashboard

* Cleanup Returns and remove commented code

* Remove Version Check if no build options are passed in

* Pass in Config Values instead of Pointers

* Handle missed Error

* Update Endpoint URL

* Change Type of Release Flag

* Add additional Logging

* Remove Versions Logging of other Services

* minor fixes

Change-Id: I5cc04a410ea6b2008d14dffd63eb5f36dd348a8b
2019-04-03 21:13:39 +02:00
Maximillian von Briesen
6028d8c3de
Fix repair order limit creation (#1650)
* fix repair order limit creation (use piece size instead of share size)
* fix counter in CreateGetRepairOrderLimits
2019-04-03 09:17:32 -04:00
Jess G
d51bdf14df
project usage limiting (#1561)
* reorg uplink cmd files for consistency

* init implementation of usage limiting

* Revert "reorg uplink cmd files for consistency"

This reverts commit 91ced7639bf36fc8af1db237b01e233ca92f1890.

* add changes per CR comments

* fix custom query to use rebind

* updates per convo about what to limit on

* changes per comments

* fix syntax and comments

* add integration test, add db methods for test

* update migration, add rebind to query

* update testdata for psql

* remove unneeded drop index statement

* fix migrations, fix calculate usage limit

* fix comment

* add audit test back

* change methods to use bucketName/projectID, fix tests

* add changes per CR comments

* add test for uplink upload and err ssg

* changes per CR comments

* check get/put limit separately
2019-04-02 11:21:18 -07:00
Simon Guindon
d3885b7b78
Change crypto/rand to use math/rand in pgutil (#1589)
* Added retry logic and tests for handling if crypto/rand fails.

* Added retry logic and tests for handling if crypto/rand fails.

* Fixing linting error.

* Removing use of `crypto/rand` for use of `math/rand`.

* Changing code comment about why we ignore this error.
2019-04-02 12:52:25 -04:00
Alexander Leitner
59b3eb190e
API endpoint checking object and segment health from the satellite (#1546)
* Generate protobuf for checking object and segment health

* Initial cmd client for health check

* Register the health inspector on the satellite peer
2019-04-02 10:55:58 -04:00
Michal Niewrzal
f80750693c Store bandwidth from orders on satellite (#1586) 2019-04-01 16:14:58 -04:00
Kaloyan Raev
034c283eda Set last_contact_success when a node is initially added to overlay cache (#1632) 2019-04-01 12:42:06 -06:00
Cameron
6d43832c4f
record bucket data into bucket_storage_tally table (#1595)
* add MetadataSize to stats

* add logic for accumulating bucket stats in calculateAtRestData

* rename stats to BucketTally, move to accounting package

* define method on accountingDB for inserting bucketTallies

* insert bucketTallies into bucket_storage_tally table
2019-04-01 09:42:17 -04:00
Egon Elbre
872bd5d7c1
Remove pointerdb.Server (#1609) 2019-03-30 13:21:49 +02:00
paul cannon
3b12b5e85c
non-racy CreateEntryIfNotExists() (#1611)
This changes semantics slightly! with this change,
CreateEntryIfNotExists() will do a cache Update with every node passed
in, whether it exists or not. Update() already does a race-free upsert
operation, so that change removes the problematic race in
CreateEntryIfNotExists(). As far as I can tell, this semantic change
doesn't break any expectations of callers, and shouldn't affect
performance in a significant way, as we already have an awful lot of
round-trips to the db either way. But if I've misunderstood the
intention of the method, someone ought to catch it during review.
2019-03-29 15:07:07 -06:00
Yehor Butko
66a7d6bf3a
Satellite password check simplification (#1610) 2019-03-29 19:42:20 +02:00
Egon Elbre
9dbc7c719e
fix error handling in GetByProjectID (#1604) 2019-03-29 14:55:05 +02:00
Egon Elbre
de15a4fdcf
remove utils.CombineErrors and utils.ErrorGroup (#1603) 2019-03-29 14:30:23 +02:00
Yaroslav Vorobiov
0ce6d4ab81
Project api auth update (#1572) 2019-03-29 14:13:37 +02:00
Kaloyan Raev
9fb99c8484
Ensure serial number is not saved if error during order limit creation (#1602) 2019-03-29 11:53:53 +02:00
Kaloyan Raev
f9ba935286
Merge overlay_cache_nodes into nodes table (#1581) 2019-03-29 10:53:43 +02:00
Egon Elbre
be06fdfd6c Create orders.Service (#1593) 2019-03-28 22:09:23 +02:00
Jennifer Li Johnson
b3a8beb1bc
Updates accounting tables models and migration (#1584)
* go gen

* undo changes to bucket usage

* update locked

* spacing

* moves changes to migration v11

* minor changes to fix lint and test err

* change sql to fix errs
2019-03-27 22:46:49 -04:00
Yehor Butko
86bf3dee9f
V3-1307 combine first and last name to full name (#1569)
* V3-1307 combine first and last name to full name
2019-03-27 14:33:32 +02:00
Michal Niewrzal
bfdfebbde2
Satellite orders receiving (#1564)
This change adds satellite endpoint for receiving OrderLimits sent by storage node.
Change includes:
* wire up orders sender in storage node (also in testplanet)
* saving serial number for OrderLimit in serial_numbers table
* satellite endpoint for receiving, verifying and storing OrderLimit and Order serial number
* initial implementation for Orders DB
* basic test for sending orders to satellite
2019-03-27 11:24:35 +01:00
Natalie Villasana
5b48a48a79
adds test for correct download with non-critical amount of nodes offline (#1574) 2019-03-26 14:09:44 -04:00
Yaroslav Vorobiov
b2b86bee56
[V3-1366] Email project name bug (#1571) 2019-03-26 17:56:16 +02:00
Stefan Benten
3db903fe40
Move from Unique to Index on SatelliteDB (#1563)
* Move from Unique to Index

* Remove Index

* Make some more Indexes Unique and adjust migration

* Fix Migration Statements

* Fix Typo

* Fix Migration of older Table

* Exchange DROP statement

* Remove "if not exists"

* Revert Change in old Migration
2019-03-26 11:34:30 +01:00
Kaloyan Raev
d1639c4157 Merge statdb pkg into overlay pkg (#1570) 2019-03-25 18:25:09 -04:00
Yaroslav Vorobiov
cce6fa8fd3
added LOGIN auth scheme to satellite mail service (#1552) 2019-03-24 02:08:41 +02:00