Cameron
32192aca10
convert times to UTC before entering as utimestamp ( #1708 )
2019-04-09 15:12:58 -04:00
Jess G
55bfe507e5
add tests for tally and accounting db ( #1675 )
...
* add tally calcAtRestData test
* add account db tests
* move tests from sadb to accounting pkg
* add func to calcEncSize
* remove unneeded pad changes
* update comment
* add changes to comments
2019-04-09 09:48:35 -04:00
Jess G
5dfe28a8c3
use allocated instead of settled ( #1700 )
...
* use allocated instead of settled
* add expansion factor
* changes per CR
2019-04-08 14:35:54 -07:00
Jess G
1638d96b16
update ProjectStorageTotals func to get all records for project ( #1657 )
...
* update ProjectStorageTotals func to get all records for project
* fix var name
* update test to catch bug
* fix spelling
* modify query so we only have to make 1
2019-04-04 09:01:04 -07:00
Jennifer Li Johnson
8549421385
Remove bw from tally service + query bandwidth in rollup service ( #1618 )
2019-04-04 11:20:59 -04:00
Cameron
34a439a99c
Delete accounting raws after rollup ( #1646 )
...
* enable deletion of raws
* assert deletion of raws during TestRollupRaws
* add some unit tests for DeleteRawBefore
2019-04-02 23:55:24 -04:00
Jess G
d51bdf14df
project usage limiting ( #1561 )
...
* reorg uplink cmd files for consistency
* init implementation of usage limiting
* Revert "reorg uplink cmd files for consistency"
This reverts commit 91ced7639bf36fc8af1db237b01e233ca92f1890.
* add changes per CR comments
* fix custom query to use rebind
* updates per convo about what to limit on
* changes per comments
* fix syntax and comments
* add integration test, add db methods for test
* update migration, add rebind to query
* update testdata for psql
* remove unneeded drop index statement
* fix migrations, fix calculate usage limit
* fix comment
* add audit test back
* change methods to use bucketName/projectID, fix tests
* add changes per CR comments
* add test for uplink upload and err ssg
* changes per CR comments
* check get/put limit separately
2019-04-02 11:21:18 -07:00
Cameron
6d43832c4f
record bucket data into bucket_storage_tally table ( #1595 )
...
* add MetadataSize to stats
* add logic for accumulating bucket stats in calculateAtRestData
* rename stats to BucketTally, move to accounting package
* define method on accountingDB for inserting bucketTallies
* insert bucketTallies into bucket_storage_tally table
2019-04-01 09:42:17 -04:00
Kaloyan Raev
d1639c4157
Merge statdb pkg into overlay pkg ( #1570 )
2019-03-25 18:25:09 -04:00
Egon Elbre
94e79eda6d
remove overlay endpoint ( #1521 )
2019-03-23 10:06:11 +02:00
Egon Elbre
26497df0e2
Faster agreement sender ( #1507 )
2019-03-19 14:56:59 +02:00
Egon Elbre
05d148aeb5
Storage node and upload/download protocol refactor ( #1422 )
...
refactor storage node server
refactor upload and download protocol
2019-03-18 12:55:06 +02:00
Bill Thorp
665fd33e3c
Repair queue isolation level fix ( #1466 )
...
Implemented custom SQLite and Postgres Repairqueue Dequeue handlers
2019-03-14 17:12:47 -04:00
Bill Thorp
52e829c6de
framework for deleting expired Orders from Satellite ( #1436 )
...
framework for deleting old bandwidth agreements
2019-03-12 16:57:21 -04:00
Bill Thorp
1f4e1a3ab0
framework to delete raw accounting tallies ( #1439 )
...
framework to delete raw accounting tallies, renamed Query
2019-03-12 09:08:23 -04:00
Bill Thorp
66718cc5e6
Development defaults for configuration ( #1430 )
...
added --dev command line option, cfgstruct.DevFlag(), and cfgstruct.SetupFlag()
2019-03-12 08:51:06 -04:00
Yaroslav Vorobiov
9cf56c03fa
Bucket usage info rollup table creation and implementation ( #1340 )
2019-03-06 17:54:48 +02:00
aligeti
588e2a51d2
Handles zero bucket count and bucket with no files ( #1391 )
...
* handles zero bucket count and bucket with no files
2019-03-04 12:47:08 -05:00
Stefan Benten
5f57d2c906
Reduce Log Messages on Production Settings ( #1396 )
...
* Reduce Tally Log Messages on Production Settings
* Reduce Logging in Piecestore Client as well
2019-03-04 11:12:43 +01:00
Cameron
3fd55c7034
remove unused date field in csv row ( #1388 )
2019-03-01 08:23:27 -05:00
aligeti
c6ad7644d2
Total file count through Monkit ( #1351 )
...
* segment, file, byte stats, total and per-bucket; checker: report segment health stats; reports the total num of lost files
* code review updates
2019-02-26 10:17:51 -05:00
Bill Thorp
373b301736
BWA aliases ( #1333 )
...
aliased RBAs and PBAs
2019-02-22 16:17:35 -05:00
Egon Elbre
9c1e299f3c
Ensure everyone sees everyone else ( #1275 )
2019-02-08 11:25:13 +02:00
paul cannon
c35b93766d
Unite all cryptographic signing and verifying ( #1244 )
...
this change removes the cryptopasta dependency.
a couple possible sources of problem with this change:
* the encoding used for ECDSA signatures on SignedMessage has changed.
the encoding employed by cryptopasta was workable, but not the same
as the encoding used for such signatures in the rest of the world
(most particularly, on ECDSA signatures in X.509 certificates). I
think we'll be best served by using one ECDSA signature encoding from
here on, but if we need to use the old encoding for backwards
compatibility with existing nodes, that can be arranged.
* since there's already a breaking change in SignedMessage, I changed
it to send and receive public keys in raw PKIX format, instead of
PEM. PEM just adds unhelpful overhead for this case.
2019-02-07 14:39:20 -06:00
Bill Thorp
0b35762105
Convert Payments to use SQL, for SUM() and Wallet ( #1266 )
...
* payments query no longer DBX, using SQL
* sum in SQL
* removed old function
* fixed rollup test
* wrap errors
* removed DBX code
2019-02-07 15:26:55 -05:00
aligeti
b736ae4823
Store the uplinks public key on the satellite so that it can verify bandwidth requests in the future ( #1042 )
...
* integrated with bwagreement & psserver
* integrated with pointerdb
* code review updates
* refactor after code review
* uplinkdb rename to certdb
* Code review changes
2019-02-07 14:22:49 -05:00
Cameron
bfe8060f4d
Rollup tests ( #1224 )
...
* createdAt field no longer autoinsert
* rollup tests
2019-02-06 23:16:24 -05:00
Bill Thorp
690e8b2061
storage node cert cache ( #1226 )
...
* draft
* still errors
* double close fix
* added tests
* weird, goimports must not be working
* renames
* missed one
* forgot to save:
2019-02-05 12:57:56 -05:00
Egon Elbre
6564357e13
Ensure that tally tests run against postgres ( #1227 )
2019-02-05 16:23:45 +02:00
paul cannon
0238a7785b
make a map object before inserting into it ( #1217 )
...
I think this is causing panics.
2019-02-03 18:42:20 -06:00
Bill Thorp
0f662b8e38
sql based tally ( #1166 )
...
* WIP
* wacky changes
* more
* it builds.... IT BUILDS!!!
* fixed SQL, broke out saving tallies for testing
* shorter lines
* fixed SQL, moved tally_test to testplanet
* lint
* WIP logic error preventing PUT and GETs to same serialnum
* fixed BWA test
* fixed temporary brain failure
* eliminated magic numbers
* cleaned up satellite uplink stats
* use errs.Combine instead
* thrashing
* fixed tally erroneous error msg
* fixed tally test
* lint
* SQL syntax attempt to fix
* spelling error
* made bwa db test resist old postgres data
* postgres pk error msg hunting
* postgres pk error msg hunting
* postgres
* err might be nil?
* fixed error logging bug
* hopefully solved postgres issue
* using rebind
* moved tests to _test package
* fixing test dirs
* finally made sense of Egons package name feedback
* UTC, array fixes
2019-02-01 13:50:12 -05:00
Bill Thorp
60946c2024
make bandwidth agreements sensible: without []byte's ( #1152 )
...
removed []byte's from bandwidth agreement protocol buffers
2019-01-28 14:45:25 -05:00
Egon Elbre
49dacb662c
Improve bwagreement tests ( #1149 )
2019-01-26 01:06:38 +02:00
Bill Thorp
8ba4b61e59
Make Bandwidth Agreements Secure / Trustable ( #1117 )
...
* Added cert chains and nodeid verification to bandwidth agreement
2019-01-25 13:05:21 -05:00
Cameron
f5b1e77cf2
rollup timestamp fix ( #1132 )
...
* create timestamp if isNew
* pass isNil into SaveRollup
2019-01-24 15:41:22 -05:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 ( #1133 )
2019-01-24 15:15:10 -05:00
Jennifer Li Johnson
030d294399
adds logs in rollup and tally run ( #1131 )
2019-01-24 15:05:53 -05:00
Jennifer Li Johnson
70940b8710
Making sure payments works ( #1122 )
...
* byte hours
* updates comment
* removes payment test data methods
* wip
* adds tally and rollup to peer
* remove sim test for payment (for now)
* lint error
2019-01-24 13:44:14 -05:00
Egon Elbre
5de7f8af7f
Satellite Peer ( #1119 )
2019-01-23 21:58:44 +02:00
Michal Niewrzal
eb2d858ffc
Use pointerdb.Service instead of pointerdb.Server ( #1099 )
2019-01-19 13:58:53 -05:00
Cameron
ecd704692e
wip testing and timestamp fix ( #1100 )
...
* wip testing and timestamp fix
* fixed import cycle
2019-01-18 11:53:23 -05:00
Michal Niewrzal
cc1bdef8b7
Move pointerdb logic to a service ( #1037 )
...
* test
* Move pointerdb logic to service
* tset
* reorder constructor params
* restore field
2019-01-18 10:10:21 -05:00
Jennifer Li Johnson
c4b876ddc5
Jj/bytes ( #1085 )
...
* run tally and rollup
* sets dev default tally and rollup intervals
2019-01-17 14:39:32 -05:00
Jennifer Li Johnson
e6fbf63620
Generate Payments Report ( #1079 )
2019-01-17 13:34:13 -05:00
Bill Thorp
342dc857f5
rollup query ( #1056 )
...
* implemention notes
* more notes
* starting rollup query
* not working yet
* fixed build
* fixed cfg bug
* change context cancelled errs to debugs
* using byte hours for at rest tally
* revert changes to go.mod
* comment fixes
* prevent double recording tallies in rollup
* linting
* stop leaking dbx
* nodeid changes
* fix build
2019-01-16 14:30:33 -05:00
littleskunk
2aebbeed34
add more bwagreement unit tests ( #1046 )
...
* add more bwagreement unit tests
2019-01-15 05:44:56 +01:00
Michal Niewrzal
e13f7c7a7d
Fix unstarted pointerdb responsibility ( #1033 )
2019-01-11 13:45:11 +01:00
Cameron
ef50bbf8b6
satellites reject expired BWAs ( #1015 )
...
* add 45 day expiration to PBAs
* add expiration field to relevant areas, DeleteExpired placeholder
* reject expired BWAs
* test for expired BWAs
* add BwExpiration config value
2019-01-10 13:30:55 -05:00
Bill Thorp
fb4a11ebb0
Differentiate GET and PUT operations in the tally service ( #965 )
...
* draft of new bandwidth agreement types
* updated storagenode report for new types
* use correct pba types
2019-01-10 06:41:57 -05:00
aligeti
457f909a74
Satellite accepts manipulated paychecks V3-1023 ( #972 )
...
* fix - Satellite crashing on receiving a manipulated bandwidthagreement
* provider.PeerIdentityFromContext called twice. Remove one
* add storage node ID to serial number
* remove serialNum query and transaction
* add uuid to GeneratePayerBandwidthAllocation for testing
* enable expected failure on duplicate serialnum cases
* Revert "enable expected failure on duplicate serialnum cases"
This reverts commit 5948f43ed1741c280f0bb34a86c1c490365417bc.
* enable expected failure on duplicate serialnum cases
2019-01-09 10:02:03 -05:00
