Commit Graph

325 Commits

Author SHA1 Message Date
Ethan Adams
f3dccb56b1
satellite/gracefulexit: Check if pointer has been overwritten or deleted before sending transfer message. (#3481) 2019-11-07 11:13:05 -05:00
Yaroslav Vorobiov
35edc2bcc3 satellite/payments: invoice creation (#3468) 2019-11-05 15:16:02 +02:00
Yehor Butko
0c2e498f09
satellite/satellitedb: console tables archview comments updated (#3465) 2019-11-04 16:37:39 +02:00
Yaroslav Vorobiov
30a3205745
satellite/payments: update account balance (#3379) 2019-10-29 18:04:34 +02:00
Egon Elbre
2e44a9fa6b
satellite/satellitedb/dbx: remove sqlite.sql file (#3409) 2019-10-29 12:35:27 +02:00
Ethan Adams
9905f2c61e add piece num to transfer queue PK (#3390) 2019-10-28 11:08:33 -04:00
Yaroslav Vorobiov
1a304f5ef9
satellite/payments: add payments loop, update pending transactions (#3318) 2019-10-23 15:04:54 +03:00
Egon Elbre
3c438f31bd
satellite/satellitedb: remove sqlite support (#3296) 2019-10-19 00:27:57 +03:00
Natalie Villasana
45c35d7c3f
satellite/satellitedb: add exit_status column to nodes table (#3301) 2019-10-17 11:01:39 -04:00
Yaroslav Vorobiov
24e72f35d3
satellite/payments: token deposit (#3283) 2019-10-17 17:04:50 +03:00
Yehor Butko
a5f4bbee22
satellite/payments: dbx scheme renamed, userID placed on Account level (#3281) 2019-10-15 21:05:45 +03:00
Ethan Adams
a1275746b4
satellite/gracefulexit: Implement the 'process' endpoint on the satellite (#3223) 2019-10-11 17:18:05 -04:00
Yehor Butko
451909b3ec
satellite/payments: account balance (#3242) 2019-10-11 18:00:35 +03:00
Yehor Butko
0cc23add5b
satellite/payments - payment account setup (#3187) 2019-10-10 20:12:23 +03:00
Bogdan Artemenko
5f775b9e46
satellite/console: Added error for adding api key with existing name attempt. (#3185) 2019-10-10 16:28:35 +03:00
Ethan Adams
9edfb6efe0
satellite/satellitedb: Initial GE Satellite DB Implementation (#3049)
Initial GE Satellite DB impl
Add basic CRUD operations for graceful_exit_progress and graceful_exit_transfer_queue tables.
2019-09-25 11:12:44 -06:00
Ethan Adams
886041e0ba
satellite/satellitedb: add new graceful exit tables and add graceful exit fields to nodes (#3033)
DB schema changes for satellite to support Graceful Exit
2019-09-13 12:57:32 -04:00
Egon Elbre
8ef57a2af3
satellite/satellitedb: use noreturn (#3022) 2019-09-12 20:31:50 +03:00
Bogdan Artemenko
2c7813d40d
satellite/console: Added email normalization to users table (#2586) 2019-09-10 17:00:33 +03:00
Bryan White
a33106df1c
satellite/satellitedb: persist piece counts to/from db (#2803) 2019-08-27 14:37:42 +02:00
aligeti
33aff71959 satellitedb/overlay: add database for storing peer identities (#2764) 2019-08-26 19:49:42 +03:00
Bryan White
6400d63a6c
satellite/satellitedb: Add piece count column to nodes table (#2795) 2019-08-19 12:58:13 +02:00
aligeti
32f95a14fd
satellite/certdb: remove certdb that was used to store uplink certificates (#2760)
* satellitedb/certDB: refactors of the node certificate storage DB table

The existing implementation doesnt allow to store the complete certificate chain of uplinkIDs or storagenodeIDs, so the current table is dropped and new table will be added which addresses the storage and retrieval of certificates

pkg/identity: fixes spelling mistakes that I missed on PR#2754

Fixes V3-1992/V3-2388
2019-08-12 10:41:34 -04:00
Yehor Butko
b0e5964718
satellite/console: create owner_id column for project table (#2706)
* satellite/console: create owner_id column for project table
2019-08-07 15:28:13 +03:00
aligeti
65932ad692
Updating the certdb to support storage of multiple public keys for same node ID (#2692)
* updating the certdb to support storage of multiple public keys for same node ID
2019-08-02 13:47:35 -04:00
Yingrong Zhao
51833d0650
satellite/satellitedb: get active offer for partners (#2664)
* get partner offer

* fix lint

* fix deleting user account

* fix sqlite query

* add comments

* fix migration

* fix query_test

* add error logs

* add tests for user credits
2019-08-01 13:46:33 -04:00
Faris Huskovic
2144181c99
satellite/marketingweb: Add Partner Offer Type (#2540) 2019-07-25 18:06:23 -04:00
aligeti
29b576961f
value attribution merge fix and more test cases (#2588)
* value attribution merge fix and more test cases
2019-07-19 11:17:34 -04:00
Bill Thorp
a7cc940776
Nodes should not be able to fail the same audit multiple times (#2404)
update pointer on audit failure
2019-07-18 14:08:15 -04:00
Jess G
3af9250659
update irreparableDB.GetLimited query to use where instead of offset (#2585)
* update query to use where instead of offset, update tests

* update cmd/inspector irreparable

* add comment for offset
2019-07-18 09:21:21 -07:00
aligeti
daa3b32ee2
Add Attribution Columns to appropriate tables for OSPP referral link (#2516)
* adds "partner_id" column to user, project, api_key & bucket_metainfo tables
2019-07-12 13:59:19 -04:00
Maximillian von Briesen
b590e53d64
Order by attempted time in injured segments select (#2533) 2019-07-12 13:35:20 -04:00
Alexander Leitner
64b2769de3
discovery: parallelize refresh (#2535)
* parallelize discovery refresh

* add paginateQualifiedtest, address pr comments

* Remove duplicate uptime update

* Lower concurrency in Testplanet for discovery
2019-07-12 10:35:48 -04:00
Yingrong Zhao
d887ffec62
satellite/satellitedb: add default offer for offers table (#2522)
* add default offer for offers table

* fix migration test

* Trigger Jenkins

* set the default value to be correct type

* skip soon will deleted test

* fix test data

* add orderby for ListAll

* change durations, redeemable cap to be a nullable field

* remove unecessary code
2019-07-12 10:19:38 -04:00
Yingrong Zhao
6e57b102c5
satellite/satellitedb: remove num_redeemed in offers table (#2510)
* remove num_redeemed

* remove the field frol Offer struct

* remove the field in the UI templates

* fix merge conflict

* fix migration

* fix merge conflict

* fix typo
2019-07-12 09:19:18 -04:00
Jess G
f11bf46a11
Jg/1967 mv bucket metadata uplink (#2505)
* add bucketstore, add init uplink bucket

* update uplink to use bucket rpc

* fix tests

* wrap metainfo client errors

* add allowedBucket struct, fix tests

* update comment

* add paging

* updates per CR

* add test for pagination

* fix lint

* fix uplink test so its easier tyo understand

* fix gateway pagination bug

* changes per cr

* fix bug w/allowedBuckets, add test to catch
2019-07-12 08:57:02 -04:00
Michal Niewrzal
268c629ba8
Replace base64 encoding for path segments (#2345) 2019-07-11 13:26:07 -04:00
Bogdan Artemenko
32e0227c45
Project Payment methods (#2037) 2019-07-10 23:29:26 +03:00
Faris Huskovic
0d294103e9
satellite/rewards: nicer offers handling (#2390)
* organize offers

* revert changes to go.mod and go.sum

* change OfferStatus enums back to original

* revert modified auto-gen files

* don't render empty row if offers is empty

* change return val of ListAll to Offers

* fix build

* add method to check for empty offer when rendering template

* fix typo

* fix lint and typos

* lean out IsEmpty

* dont use named return vals

* better clarify offer statuses

* change back order of setting offer.Status

* lint

* satellite/marketingweb: allow disabling rewards (#2392)

* implement handler for stop offer endpoint

* use proper text and fix data-target for free-credit stop modal
2019-07-10 13:12:40 -04:00
Jess G
f9696d6c5e
satellite/metainfo: add buckets RPC and database (#2460)
* add db interface and methods, add sa metainfo endpoints and svc

* add bucket metainfo svc funcs

* add sadb bucekts

* bucket list gets all buckets

* filter buckets list on macaroon restrictions

* update pb cipher suite to be enum

* add conversion funcs

* updates per comments

* bucket settings should say default

* add direction to list buckets, add tests

* fix test bucket names

* lint err

* only support forward direction

* add comments

* minor refactoring

* make sure list up to limit

* update test

* update protolock file

* fix lint

* change per PR
2019-07-08 15:32:18 -07:00
Yingrong Zhao
e36f43c479
satellite/rewards: replace iota with hardcoded OfferType value (#2432)
* use 0 as the invalid offer type value

* add comment

* Update satellitedb.dbx.go

* Update satellitedb.dbx.go

* update comment
2019-07-08 14:39:56 -04:00
Jess G
0587dd79d6 change bucket table to fix conflict with pointerDB (#2452)
* change bucket table to fix conflict with pointerDB

* update init query

* update migration for dbx psql

* fix migration testdata
2019-07-04 00:03:56 +02:00
Jess G
0158b1eb5a
add bucket metadata table in SA masterDB (#2369)
* add bucket metadata table in SA masterDB

* fix indentation

* update db model per CR comments

* update testdata

* add missing field on sql testdata

* fix args to testdata

* unique bucket name

* fix fkey constraint for test

* fix one too many commas

*  update timestamp type

* Trigger Jenkins

* Trigger Jenkins yet again
2019-07-01 13:45:21 -07:00
Stefan Benten
01beaa289a
Mask IP Addresses to subnets (#2305) 2019-06-24 17:33:18 +02:00
Bill Thorp
8f47fca5d3
Remove audit / uptime ratio fields (#2247)
* removed ratios
2019-06-21 13:14:53 -04:00
Natalie Villasana
9386187fe6
add disqualification and new reputation system into overlay cache (#2227) 2019-06-20 09:56:04 -04:00
aligeti
7b450927ff
code updates with comments (#2245) 2019-06-19 08:50:50 -04:00
Egon Elbre
8f4a6afc8a Fixes for value attribution (#2238)
* Move value attribution to satellite/

* remove unnecessary conversion
2019-06-19 08:02:37 -04:00
Bill Thorp
81134e97cc
nodes db - adds alpha and beta reputation fields (#2215)
* node dbx and migrate adding alpha and beta reputation
2019-06-18 09:45:02 -04:00
aligeti
09940d4e0b
value attribution DB interface Insert & Get method support (#2200)
* value attribution DB interface methods support
2019-06-18 09:06:33 -04:00
Ivan Fraixedes
35c8648330
[v3-1914] Storage node disqualification: Change type from bool to timestamp (#2212)
* satellite/satellitedb: Alter nodes disqualification column
  Change the type of the 'disqualification' column of the nodes table from
  boolean to timestamp.
* overlay/cache: Change Disqualified field type
  Change the Disqualified field type the NodeDossier struct type from bool
  to time.Time to match with the disqualified type used by the DB layer.
* satellite/satellitedb: Update queries uses disqualified
  Update the queries which uses the disqualified column due to the column
  type has been changed from boolean to nullable timestamp.
* docs/design: Update disqualification due impl changes
  Update the disqualification design document to contain the architectural
  change required to be able to restore unfair disqualified nodes in case
  of an unexpected cause (bug, mistake, hard network disconnection, etc.).
2019-06-18 11:14:31 +02:00
Faris Huskovic
c0fe061a67 add user credits table (#2085)
* add user credits table

* change primary key, change type for credit_type, and change relation kind of foreign keys from cascade to restrict

* modify table and query methods

* modify schema

* add dbx queries

* add migration file

* add orderby to read available credit entries
2019-06-13 09:52:33 -04:00
ethanadams
380684e766 Removing bucket_id and adding project_id and bucket_name as primary key (#2182)
* removing bucket_id and adding project_id and bucket_name as primary key

* cleanup
2019-06-13 08:28:55 -04:00
Yaroslav Vorobiov
51db703b2b
Console add payments to service (#2100) 2019-06-06 19:07:14 +03:00
Egon Elbre
28a1201590 pkg/bwagreements: remove service (#2138) 2019-06-06 15:57:58 +02:00
Dennis Coyle
b9d586901e
Value Attribution DBX Model (#2116)
* adds model to satellite dbx

* cleans up model spacing

* generated golang from dbx

* added migration steps

* Added testdata

* changed node_id -> bucket_id

* adds -- NEW DATA -- to testdata

* more testdata changes

* adds -- NEW DATA -- line

* dbx makes the table plural

* missed a singular value_attribution

* restart jenkins

* Update satellitedb.dbx

* adjust to PR comments

* autogenerated dbx models

* restart jenkins
2019-06-05 12:06:14 -04:00
Yingrong Zhao
09b0c2a630
create db implementation for offer table (#2031)
* init marketing service

Fix linting error

Create offerdb implementation

Create offers service

Add update method

Create offer table and migration

Fix linting error

fix conflicts

Insert new data

Change duration to have clear indication to be based on days

add error wrapper

Change from using uuid to int for id field

* Create Marketing service

* make error virable name more readable

* add condition in update service method to check offer status

* generate lock file

Change get to listAllOffers

* Add method for getting current offer

wip

* add check for expires_at in update method

* Fix conflicts

* add copyright header

* Fix linting error

* only allow update to active offers

* add isDefault argument to GetCurrent

* Update lock file

* add migration file

* finish migrate for adding credit_in_cents for both award and invitee

* save 100 years as expiration date for default offers

* create crud test for offers

* add GetCurrent test

* modify doc

* Fix GetCurrent to work with default offer

* fix linting issue

* add more tests and address feedbacks

* fix migration file

* add type column back to match with mockup design

* add type column back to match with mockup design

* move doc changes to new pr

* add comments

* change GetCurrent to GetCurrentByType

* fix typo
2019-06-04 15:17:01 -04:00
Natalie Villasana
6db9388082 add disqualified column to nodes table (#2086)
* add disqualified column to nodes table, update migrate script and testdata

* fix crazy formatting of postgres.v25.sql
2019-05-30 17:38:23 -04:00
Michal Niewrzal
f731267e8c Per-project usage limiting (#2036)
What: Changes to support custom usage limit for the project. With this implementation by default project usage limit is taken from configuration flag. If project DB field usage_limit will be set to value larger than 0 it will become custom usage limit and we will be used to verify is limit was exceeded.

Whats changed:

usage_limit (bigint) field added to projects table (with migration)
things related to project usage moved from metainfo endpoint to project usage type
accounting.ProjectAccounting extended with GetProjectUsageLimits() method
Why: We need to have different usage limits per project. https://storjlabs.atlassian.net/browse/V3-1814
2019-05-28 09:36:52 -06:00
Jeff Wendling
1bd52b9f90 server side macaroons (#1945)
What: Adds macaroon support to the server side

Why: So that api keys are now macaroons
2019-05-24 10:51:27 -06:00
Yingrong Zhao
f9045f8385 Create offer table in satellitedb for free credit program (#1975) 2019-05-22 23:41:55 +02:00
Cameron
4058c29ca4
filter duplicate node IPs (#1890)
* add last_ip field to dbx model node, generate dbx

* add last_ip to node proto, generate pb

* migrate

* resolve address in transport.DialNode, update lastIp in cache.UpdateAddress

* use net.SplitHostPort to isolate host address from port

* define DistinctIPs flag

* add test for GetIP

* select last_ip when querying for nodes

* if distinctIPs flag == true, query for nodes with distinct IPs

* some basic tests

* change last_ip to field 14 in proto

* remove comments

* check err

* change distinctIPs to distinctIP

* exclude IPs from newNodes in query for reputable nodes

* add index on last_ip

* only add to excludedIPs if flag is true

* test half new nodes returns distinct IPs

* fix alignment

* add test

* rework ip filter query, add retry logic, add switch for database driver

* add retry to SelectNewNodes

* change discovery intervals so IPs don't get overwritten

* remove TestGetIP

* edit updating node stats in test

* split exclude into nodeIDs and IPs

* separate non-distinct IP query into other function

* trigger checks

* remove else block
2019-05-22 16:06:27 -04:00
Maximillian von Briesen
cc020dfdea
Create containment mode database table and migrate scripts (#1970) 2019-05-16 10:11:15 -04:00
Bogdan Artemenko
f46487b015
ResetPassword Table and all CRUD methods. (#1916) 2019-05-13 18:53:52 +03:00
Jennifer Li Johnson
5395ff5fe6
Refactor accountingdb interface (#1897)
* splits accounting db into storagenodeaccounting and projectaccounting interfaces and renames methods to match
2019-05-10 15:05:42 -04:00
Egon Elbre
ecde1bd251 jenkins: use -race for check-imports and fix dbx check (#1873)
What: Use -race for check-imports, this means it will use the cached build files.

Why:
2019-05-01 09:44:12 -06:00
Egon Elbre
fe2bf7d4a6 fix saving uplink public key on satellite (#1797)
* fix save into cert records

* fix transaction logic

* test shouldn't be flaky anymore

* add missing cause method

* restart jenkins
2019-04-23 15:48:57 -04:00
Egon Elbre
f7ed63a119
handle database error checks properly (#1796) 2019-04-23 14:13:57 +03:00
Bill Thorp
17a227e6e9
refactor injuredsegments db so that we can't have duplicates (#1717)
made repairqueue not use a true queue, forbid duplicates
2019-04-16 14:14:09 -04:00
Stefan Benten
bae4c820ee
Add Version Information into KAD Network and SatelliteDB & Change Selection Process (#1648)
* Initial Webserver Draft for Version Controlling

* Rename type to avoid confusion

* Move Function Calls into Version Package

* Fix Linting and Language Typos

* Fix Linting and Spelling Mistakes

* Include Copyright

* Include Copyright

* Adjust Version-Control Server to return list of Versions

* Linting

* Improve Request Handling and Readability

* Add Configuration File Option
Add Systemd Service file

* Add Logging to File

* Smaller Changes

* Add Semantic Versioning and refuses outdated Software from Startup (#1612)

* implements internal Semantic Version library

* adds version logging + reporting to process

* Advance SemVer struct for easier handling

* Add Accepted Version Store

* Fix Function

* Restructure

* Type Conversion

* Handle Version String properly

* Add Note about array index

* Set temporary Default Version

* Add Copyright

* Adding Version to Dashboard

* Adding Version Info Log

* Renaming and adding CheckerProcess

* Iteration Sync

* Iteration V2

* linting

* made LogAndReportVersion a go routine

* Refactor to Go Routine

* Add Context to Go Routine and allow Operation if Lookup to Control Server fails

* Handle Unmarshal properly

* Linting

* Relocate Version Checks

* Relocating Version Check and specified default Version for now

* Linting Error Prevention

* Refuse Startup on outdated Version

* Add Startup Check Function

* Straighten Logging

* Dont force Shutdown if --dev flag is set

* Create full Service/Peer Structure for ControlServer

* Linting

* Straighting Naming

* Finish VersionControl Service Layout

* Improve Error Handling

* Change Listening Address

* Move Checker Function

* Remove VersionControl Peer

* Linting

* Linting

* Create VersionClient Service

* Renaming

* Add Version Client to Peer Definitions

* Linting and Renaming

* Linting

* Remove Transport Checks for now

* Move to Client Side Flag

* Remove check

* Linting

* Transport Client Version Intro

* Adding Version Client to Transport Client

* Add missing parameter

* Adding Version Check, to set Allowed = true

* Set Default to true, testing

* Restructuring Code

* Uplink Changes

* Add more proper Defaults

* Renaming of Version struct

* Dont pass Service use Pointer

* Set Defaults for Versioning Checks

* Put HTTP Server in go routine

* Add Versioncontrol to Storj-Sim

* Testplanet Fixes

* Linting

* Add Error Handling and new Server Struct

* Move Lock slightly

* Reduce Race Potentials

* Remove unnecessary files

* Linting

* Add Proper Transport Handling

* small fixes

* add fence for allowed check

* Add Startup Version Check and Service Naming

* make errormessage private

* Add Comments about VersionedClient

* Linting

* Remove Checks that refuse outgoing connections

* Remove release cmd

* Add Release Script

* Linting

* Update to use correct Values

* Change Timestamp handling

* Adding Protobuf changes back in

* Adding SatelliteDB Changes and adding Storj Node Version to PB

* Add Migration Table

* Add Default Stats for Creation

* Move to BigInt

* Proper SQL Migration

* Ensure minimum Version is passed to the node selection

* Linting...

* Remove VersionedClient and adjust smaller changes from prior merge

* Linting

* Fix PB Message Handling and Query for Node Selection

* some future-proofing type changes

Change-Id: I3cb5018dcccdbc9739fe004d859065992720caaf

* fix a compiler error

Change-Id: If66bb92d8b98e31cd618ecec9c6448ab9b037fa5

* Comment on Constant for Overlay

* Remove NOT NULL and add epoch call as function

* add versions to bootstrap and satellites

Change-Id: I436944589ea5f21600cdd997742a84fe0b16e47b

* Change Update Migration

* Fix DB Migration

* Increase Timeout temporarily, to see whats going on

* Remove unnecessary const and vars
Cleanup Function calls from deprecated NodeVersion struct

* Updated Protopuf, removed depcreated Code from Inspector

* Implement NodeVersion into InfoResponse

* Regenerated locked.go

* Linting

* Fix Tests

* Remove unnecessary constant

* Update Function and Flag Description

* Remove Empty Stat Creation

* return properly with error

* Remove unnecessary struct

* simplify migration step

* Update Inspector to return Version Info

* Update local Endpoint Version Handling

* Reset Travis Timeout

* Add Default for CommitHash

* single quotes
2019-04-10 08:04:24 +02:00
Natalie Villasana
2533aa1ab1
logs dbx constraint violation error (#1712) 2019-04-08 18:03:48 -04:00
Jennifer Li Johnson
8549421385
Remove bw from tally service + query bandwidth in rollup service (#1618) 2019-04-04 11:20:59 -04:00
Yaroslav Vorobiov
b38b87cb14
Console usage rollup api (#1664) 2019-04-04 17:56:20 +03:00
Jess G
d51bdf14df
project usage limiting (#1561)
* reorg uplink cmd files for consistency

* init implementation of usage limiting

* Revert "reorg uplink cmd files for consistency"

This reverts commit 91ced7639bf36fc8af1db237b01e233ca92f1890.

* add changes per CR comments

* fix custom query to use rebind

* updates per convo about what to limit on

* changes per comments

* fix syntax and comments

* add integration test, add db methods for test

* update migration, add rebind to query

* update testdata for psql

* remove unneeded drop index statement

* fix migrations, fix calculate usage limit

* fix comment

* add audit test back

* change methods to use bucketName/projectID, fix tests

* add changes per CR comments

* add test for uplink upload and err ssg

* changes per CR comments

* check get/put limit separately
2019-04-02 11:21:18 -07:00
Michal Niewrzal
f80750693c Store bandwidth from orders on satellite (#1586) 2019-04-01 16:14:58 -04:00
Kaloyan Raev
034c283eda Set last_contact_success when a node is initially added to overlay cache (#1632) 2019-04-01 12:42:06 -06:00
Cameron
6d43832c4f
record bucket data into bucket_storage_tally table (#1595)
* add MetadataSize to stats

* add logic for accumulating bucket stats in calculateAtRestData

* rename stats to BucketTally, move to accounting package

* define method on accountingDB for inserting bucketTallies

* insert bucketTallies into bucket_storage_tally table
2019-04-01 09:42:17 -04:00
Kaloyan Raev
f9ba935286
Merge overlay_cache_nodes into nodes table (#1581) 2019-03-29 10:53:43 +02:00
Jennifer Li Johnson
b3a8beb1bc
Updates accounting tables models and migration (#1584)
* go gen

* undo changes to bucket usage

* update locked

* spacing

* moves changes to migration v11

* minor changes to fix lint and test err

* change sql to fix errs
2019-03-27 22:46:49 -04:00
Yehor Butko
86bf3dee9f
V3-1307 combine first and last name to full name (#1569)
* V3-1307 combine first and last name to full name
2019-03-27 14:33:32 +02:00
Michal Niewrzal
bfdfebbde2
Satellite orders receiving (#1564)
This change adds satellite endpoint for receiving OrderLimits sent by storage node.
Change includes:
* wire up orders sender in storage node (also in testplanet)
* saving serial number for OrderLimit in serial_numbers table
* satellite endpoint for receiving, verifying and storing OrderLimit and Order serial number
* initial implementation for Orders DB
* basic test for sending orders to satellite
2019-03-27 11:24:35 +01:00
Stefan Benten
3db903fe40
Move from Unique to Index on SatelliteDB (#1563)
* Move from Unique to Index

* Remove Index

* Make some more Indexes Unique and adjust migration

* Fix Migration Statements

* Fix Typo

* Fix Migration of older Table

* Exchange DROP statement

* Remove "if not exists"

* Revert Change in old Migration
2019-03-26 11:34:30 +01:00
Egon Elbre
297b393dfd database for tracking information without storing orders / agreements (#1551)
* initial schema declaration

* fix typo

* split database

* fix syntax errors and generate code

* add queries for serial number manipulation

* update comment

* add migration script

* fix comments

* add new data

* fix bucket id
2019-03-22 11:54:22 -07:00
Egon Elbre
694b6dc1da
make tests run faster (#1553) 2019-03-22 15:14:17 +02:00
Yehor Butko
c6f8d82978
V3-1319 Add an authorization token field for account creation backend (#1492)
* V3-1319 Add an authorization token field for account creation on the satellite UI backend
2019-03-19 19:55:43 +02:00
paul cannon
cd91a22e0f better way to check if err is ErrorCode_NoRows (#1453)
* better way to check if err is ErrorCode_NoRows
2019-03-18 20:15:27 -04:00
Cameron
c7ffbe1c28
Add ability to view irreparable segments on satellite (#1448)
* define irreparable inspector protobuf

* add IrreparableDB method GetLimited

* fill out irreparable inspector API

* add IrreparableInspector server to satellite, fix small error

* refactor IrreparableDB to use pb.IrreparableSegment instead of irreparable.RemoteSegmentInfo
2019-03-15 16:21:52 -04:00
Bill Thorp
52e829c6de
framework for deleting expired Orders from Satellite (#1436)
framework for deleting old bandwidth agreements
2019-03-12 16:57:21 -04:00
Bill Thorp
1f4e1a3ab0
framework to delete raw accounting tallies (#1439)
framework to delete raw accounting tallies, renamed Query
2019-03-12 09:08:23 -04:00
Yaroslav Vorobiov
9cf56c03fa
Bucket usage info rollup table creation and implementation (#1340) 2019-03-06 17:54:48 +02:00
Dylan Lott
9abf191b01
Change payments to use statDB for wallet addresses rather than Overlay cache (#1368)
* Wiring up DumpNodes response for Inspector

* Finalize everything and test that it works

* Get Count and DumpNodes working for Overlay Cache

* WIP updating payment rollup to check statDB instead of overlay

* FIrst pass at updating statDB to take wallet and email

* Passing tests

* use pb.NodeOperator instead of Meta struct

* remove TODO

* revert go.mod

* Get SQL migration working correctly

* Changes Meta to Operator in NodeStats struct

* Adds update operator logic for statDB

* Fix db migrate tests - added v5 snapshot

* User friendly msg for missing snapshot version

* Passing tests

* Change node update to happen in discovery instead of in overlay

* Fix logic and update function calls

* Update comment on UpdateOperator interface method

* Update name of parameter

* Change type of argument to UpdateOperator

* Updates statDB tests
2019-03-01 10:46:34 -07:00
Jennifer Li Johnson
6ecda190cb
Jj/payments cleanup (#1379)
* Removes date from payments csv and re-query of wallet addr

* removes dbx get wallet addr

* switch order in csv
2019-02-28 12:13:59 -05:00
Egon Elbre
3beaeebdb6
Update dbx (#1300) 2019-02-12 21:46:03 +02:00
Yehor Butko
8156d911fa
Updating account activation flow (#1251)
* Updating account activation flow

* Updated integration tests, createUserMutation updated

* removing redundant index

* removed redundant testing code

* review comments fixed
2019-02-11 12:33:56 +02:00
Bill Thorp
0b35762105
Convert Payments to use SQL, for SUM() and Wallet (#1266)
* payments query no longer DBX, using SQL

* sum in SQL

* removed old function

* fixed rollup test

* wrap errors

* removed DBX code
2019-02-07 15:26:55 -05:00
aligeti
b736ae4823
Store the uplinks public key on the satellite so that it can verify bandwidth requests in the future (#1042)
* integrated with bwagreement & psserver

* integrated with pointerdb

* code review updates

* refactor after code review

* uplinkdb rename to certdb

* Code review changes
2019-02-07 14:22:49 -05:00
Cameron
bfe8060f4d
Rollup tests (#1224)
* createdAt field no longer autoinsert

* rollup tests
2019-02-06 23:16:24 -05:00
Yaroslav Vorobiov
2ff0d9d435
Add bucket to project relationship on satellite (#1143)
* integrate console api keys with buckets in pointerdb

* fix test

* fix tests kvmetainfo

* linter fix

* disable account activation

* fix test

* review fixes

* fix comments

* little refactoring

* remove debug println

* fix typo

* disable activation in a propper way

* fix test

* fix imports

* fix uplink count in testplanet

* move key creation to planet.newUplink
2019-02-05 19:22:17 +02:00
Michal Niewrzal
383cb05e8e
Divide uplink and gateway params set (#1026)
* Divide uplink and gateway params set

* attempt to fix docker

* attempt to fix all in one

* test

* more reorganization

* fix compilation error

* fix imports order

* fix dependency

* rename structs

* keep minio params for now

* review comments

* remove manual flag check
2019-02-04 16:50:06 +01:00
Bill Thorp
0f662b8e38 sql based tally (#1166)
* WIP

* wacky changes

* more

* it builds.... IT BUILDS!!!

* fixed SQL, broke out saving tallies for testing

* shorter lines

* fixed SQL, moved tally_test to testplanet

* lint

* WIP logic error preventing PUT and GETs to same serialnum

* fixed BWA test

* fixed temporary brain failure

* eliminated magic numbers

* cleaned up satellite uplink stats

* use errs.Combine instead

* thrashing

* fixed tally erroneous error msg

* fixed tally test

* lint

* SQL syntax attempt to fix

* spelling error

* made bwa db test resist old postgres data

* postgres pk error msg hunting

* postgres pk error msg hunting

* postgres

* err might be nil?

* fixed error logging bug

* hopefully solved postgres issue

* using rebind

* moved tests to _test package

* fixing test dirs

* finally made sense of Egons package name feedback

* UTC, array fixes
2019-02-01 13:50:12 -05:00
Yehor Butko
19bc01c19a
V3-1091 Extend Users table with IsActive functionality (#1170)
* V3-1091 Extend Users table with IsActive functionality

* fixed review comments
2019-01-30 17:04:40 +02:00
Bill Thorp
e94f6408ff
bw agreement db refactor (#1158) 2019-01-28 16:16:21 -05:00
Bogdan Artemenko
b75f06e229
Removed TermsAccepted in project from satellite console and web. (#1155)
* Removed TermsAccepted from satellite console and web.
2019-01-28 20:20:33 +02:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 (#1133) 2019-01-24 15:15:10 -05:00
Jennifer Li Johnson
e6fbf63620
Generate Payments Report (#1079) 2019-01-17 13:34:13 -05:00
Yaroslav Vorobiov
d832789481
ConsoleDB merge and buckets table creation (#1066)
* init

* db merge

* bucket table renamed to bucket info

* remove id and fix comments

* fix imports
2019-01-16 15:23:28 -05:00
Bill Thorp
342dc857f5 rollup query (#1056)
* implemention notes

* more notes

* starting rollup query

* not working yet

* fixed build

* fixed cfg bug

* change context cancelled errs to debugs

* using byte hours for at rest tally

* revert changes to go.mod

* comment fixes

* prevent double recording tallies in rollup

* linting

* stop leaking dbx

* nodeid changes

* fix build
2019-01-16 14:30:33 -05:00
Egon Elbre
f8906ce000
Change overlay cache database interface (#1047) 2019-01-15 11:08:45 -05:00
Cameron
ef50bbf8b6
satellites reject expired BWAs (#1015)
* add 45 day expiration to PBAs

* add expiration field to relevant areas, DeleteExpired placeholder

* reject expired BWAs

* test for expired BWAs

* add BwExpiration config value
2019-01-10 13:30:55 -05:00
aligeti
457f909a74 Satellite accepts manipulated paychecks V3-1023 (#972)
* fix - Satellite crashing on receiving a manipulated bandwidthagreement

* provider.PeerIdentityFromContext called twice. Remove one

* add storage node ID to serial number

* remove serialNum query and transaction

* add uuid to GeneratePayerBandwidthAllocation for testing

* enable expected failure on duplicate serialnum cases

* Revert "enable expected failure on duplicate serialnum cases"

This reverts commit 5948f43ed1741c280f0bb34a86c1c490365417bc.

* enable expected failure on duplicate serialnum cases
2019-01-09 10:02:03 -05:00
Egon Elbre
0ca03b41e2
Use generated locking database implementation (#947) 2019-01-02 19:53:27 +02:00
Michal Niewrzal
8c00b2109e
Format satellite.dbx (#930)
* Format satellite.dbx

* remove empty lines
2018-12-27 22:21:50 +01:00
Egon Elbre
4346cd060f
Implement mutex around satellitedb (#932) 2018-12-27 11:56:25 +02:00
Cameron
f70b826fd4
repair queue masterDB support (#865)
* add injuredsegment model to satellitedb.dbx

* add context to queue.RepairQueue interface

* use queue.RepairQueue interface, use masterdb
2018-12-21 10:11:19 -05:00
Michal Niewrzal
bbe1dbb92c
Fix overlay commands after master db merge (#895)
* Fix overlay commands after master db merge

* remove unused var

* remove unused comment
2018-12-18 21:41:31 +01:00
Jennifer Li Johnson
01df85704c
Updates accounting tables and allows tally to update at raw table for at rest data (#874) 2018-12-18 12:18:42 -05:00
Michal Niewrzal
a819d819e3
Overlay Cache master db merge (#877)
* Overlay Cache master db merge

* test update

* review comments

* Fix transaction usage

* removed unused variable

* added better limit handling

* better error handling
2018-12-17 21:14:16 +01:00
aligeti
5e1b02ca8b
Statdb master db v3 848 (#830)
* intial changes to migrate statdb to masterdb framework

* statdb refactor compiles

* added TestCreateDoesNotExist testcase

* Initial port of statdb to masterdb framework working

* refactored statdb proto def to pkg/statdb

* removed statdb/proto folder

* moved pb.Node to storj.NodeID

* CreateEntryIfNotExistsRequest moved pd.Node to storj.NodeID

* moved the fields from pb.Node to statdb.UpdateRequest

ported TestUpdateExists, TestUpdateUptimeExists, TestUpdateAuditSuccessExists TestUpdateBatchExists
2018-12-14 15:17:30 -05:00
Bill Thorp
f9845e7e92
changes to masterdb to support accountingDB (#846)
* added rollup to captplanet, moved accountingDB to masterdb
2018-12-14 09:27:21 -05:00
aligeti
d27bcddaab
Merge the irreparable DB into the master database (#777)
* initial refactoring to merge irreparabledb to master db

* refactor

* code review comment updates
2018-12-10 14:08:45 -05:00
Kevin Leffew
4a128e65d9 Removed unnecessary comment (#804) 2018-12-10 13:54:52 -05:00
Egon Elbre
c7cf97cc23
Fix sqlite registration (#806) 2018-12-10 10:32:49 +02:00
JT Olio
1c96db01ba
better database error handling (#784)
* better database error handling

Change-Id: I28dbd69cf6c2fa268e02405521ff6e6c1a68a702

* missing comments added

* missing comment added
2018-12-07 07:46:42 -07:00
Michal Niewrzal
034a34069a
Merge bwagreement db into satellite master db (#770)
* Merge bwagreement db into satellite master db

* adjust to recent tally changes

* linter problems

* linter problems

* returning db structs in more optimal way

* added pointer for assignment

* error message changed

* better param message
2018-12-07 10:59:31 +01:00
Michal Niewrzal
44a6cfa1c1
Initial satellite master database setup (#735)
* Initial satellite master database setup

* fixed unsed ctx and /nolint added for temp change

* tests for two db drivers + connection param renamed
2018-12-05 10:35:50 +01:00