Jess G
daf166d423
fix tally test flakiness ( #2250 )
...
* fix test flakiness
* update tests to rm uplink.upload
* rm metainfo change
* add test case of no objects in bucket
2019-06-20 12:15:13 -07:00
Jess G
f0f59a5577
Always encrypt inline segments without padding ( #2183 )
2019-06-19 11:11:27 +03:00
Jennifer Li Johnson
d177761f51
Adds test coverage to CalculateBucketAtRestData ( #2203 )
2019-06-17 22:20:40 -04:00
ethanadams
4e1cc37a4c
Fix bucket_storage_tally data ( #2165 )
2019-06-13 12:58:40 -04:00
JT Olio
f1641af802
storage: add monkit task to missing places ( #2122 )
...
* storage: add monkit task to missing places
Change-Id: I9e17a6b14f7c25bbf698eeecf32785e9add3f26e
* fix tests
Change-Id: Id078276fa3de61a28eb3d01d4e751732ecbb173f
* import order
Change-Id: I814e33755b9f10b5219af37cd828cd75eb3da1a4
* remove part of other commit
Change-Id: Idaa4c95cd65e97567fb466de49718db8203cfbe1
2019-06-05 16:23:10 +02:00
JT Olio
9c5708da32
pkg/*: add monkit task to missing places ( #2109 )
2019-06-04 13:36:27 +02:00
ethanadams
16e3b77cf5
Enable Scopelint Linter ( #2049 )
...
* added scopelint and correcte issues found
* corrected scopelint issue
* made updates based on Ivan's suggestions
Most were around naming conventions
Some were false positives, but I kept them since the test.Run could eventually be changed to run in parallel, which could cause a bug
Others were false positives. Added // nolint: scopelint
2019-05-29 09:30:16 -04:00
Jennifer Li Johnson
5395ff5fe6
Refactor accountingdb interface ( #1897 )
...
* splits accounting db into storagenodeaccounting and projectaccounting interfaces and renames methods to match
2019-05-10 15:05:42 -04:00
paul cannon
02be91b029
real-time tracking of space used per project ( #1910 )
...
Ran into difficulties trying to find the ideal solution for sharing
these counts between multiple satellite servers, so for now this is a
dumb solution storing recent space-usage changes in a big dumb in-memory
map with a big dumb lock around it. The interface used, though, should
allow us to swap out the implementation without much difficulty
elsewhere once we know what we want it to be.
2019-05-09 20:39:21 -05:00
Cameron
60abab1176
move nodeData length check after latestTally is updated in CalculateAtRestData ( #1841 )
...
* mode nodeData length check after latestTally is updated
* add test to verify unique constraint issue is fixed
2019-04-29 13:46:38 -04:00
Michal Niewrzal
fe3dfc1587
Move pointerdb.Service to satellite ( #1826 )
2019-04-25 10:46:32 +02:00
JT Olio
2744a26b60
pkg/cfgstruct: tie defaults to releases ( #1787 )
...
* tie defaults to releases
this change makes it so that by default, the flag defaults are
chosen based on whether the build was built as a release build or
an ordinary build. release builds by default get release defaults,
whereas ordinary builds by default get dev defaults.
any binary can have its defaults changed by specifying
--defaults=dev
or
--defaults=release
Change-Id: I6d216aa345d211c69ad913159d492fac77b12c64
* make release defaults more clear
this change extends cfgstruct structs to support either
a 'default' tag, or a pair of 'devDefault' and 'releaseDefault'
tags, but not both, for added clarity
Change-Id: Ia098be1fa84b932fdfe90a4a4d027ffb95e249c6
* clarify cfgstruct.DefaultsFlag
Change-Id: I55f2ff9080ebbc0ce83abf956e085242a92f883e
2019-04-19 12:17:30 -06:00
Cameron
32192aca10
convert times to UTC before entering as utimestamp ( #1708 )
2019-04-09 15:12:58 -04:00
Jess G
55bfe507e5
add tests for tally and accounting db ( #1675 )
...
* add tally calcAtRestData test
* add account db tests
* move tests from sadb to accounting pkg
* add func to calcEncSize
* remove unneeded pad changes
* update comment
* add changes to comments
2019-04-09 09:48:35 -04:00
Jennifer Li Johnson
8549421385
Remove bw from tally service + query bandwidth in rollup service ( #1618 )
2019-04-04 11:20:59 -04:00
Cameron
34a439a99c
Delete accounting raws after rollup ( #1646 )
...
* enable deletion of raws
* assert deletion of raws during TestRollupRaws
* add some unit tests for DeleteRawBefore
2019-04-02 23:55:24 -04:00
Cameron
6d43832c4f
record bucket data into bucket_storage_tally table ( #1595 )
...
* add MetadataSize to stats
* add logic for accumulating bucket stats in calculateAtRestData
* rename stats to BucketTally, move to accounting package
* define method on accountingDB for inserting bucketTallies
* insert bucketTallies into bucket_storage_tally table
2019-04-01 09:42:17 -04:00
Egon Elbre
94e79eda6d
remove overlay endpoint ( #1521 )
2019-03-23 10:06:11 +02:00
Egon Elbre
26497df0e2
Faster agreement sender ( #1507 )
2019-03-19 14:56:59 +02:00
Egon Elbre
05d148aeb5
Storage node and upload/download protocol refactor ( #1422 )
...
refactor storage node server
refactor upload and download protocol
2019-03-18 12:55:06 +02:00
Bill Thorp
665fd33e3c
Repair queue isolation level fix ( #1466 )
...
Implemented custom SQLite and Postgres Repairqueue Dequeue handlers
2019-03-14 17:12:47 -04:00
Bill Thorp
52e829c6de
framework for deleting expired Orders from Satellite ( #1436 )
...
framework for deleting old bandwidth agreements
2019-03-12 16:57:21 -04:00
Bill Thorp
66718cc5e6
Development defaults for configuration ( #1430 )
...
added --dev command line option, cfgstruct.DevFlag(), and cfgstruct.SetupFlag()
2019-03-12 08:51:06 -04:00
aligeti
588e2a51d2
Handles zero bucket count and bucket with no files ( #1391 )
...
* handles zero bucket count and bucket with no files
2019-03-04 12:47:08 -05:00
Stefan Benten
5f57d2c906
Reduce Log Messages on Production Settings ( #1396 )
...
* Reduce Tally Log Messages on Production Settings
* Reduce Logging in Piecestore Client as well
2019-03-04 11:12:43 +01:00
aligeti
c6ad7644d2
Total file count through Monkit ( #1351 )
...
* segment, file, byte stats, total and per-bucket; checker: report segment health stats; reports the total num of lost files
* code review updates
2019-02-26 10:17:51 -05:00
Bill Thorp
373b301736
BWA aliases ( #1333 )
...
aliased RBAs and PBAs
2019-02-22 16:17:35 -05:00
paul cannon
c35b93766d
Unite all cryptographic signing and verifying ( #1244 )
...
this change removes the cryptopasta dependency.
a couple possible sources of problem with this change:
* the encoding used for ECDSA signatures on SignedMessage has changed.
the encoding employed by cryptopasta was workable, but not the same
as the encoding used for such signatures in the rest of the world
(most particularly, on ECDSA signatures in X.509 certificates). I
think we'll be best served by using one ECDSA signature encoding from
here on, but if we need to use the old encoding for backwards
compatibility with existing nodes, that can be arranged.
* since there's already a breaking change in SignedMessage, I changed
it to send and receive public keys in raw PKIX format, instead of
PEM. PEM just adds unhelpful overhead for this case.
2019-02-07 14:39:20 -06:00
aligeti
b736ae4823
Store the uplinks public key on the satellite so that it can verify bandwidth requests in the future ( #1042 )
...
* integrated with bwagreement & psserver
* integrated with pointerdb
* code review updates
* refactor after code review
* uplinkdb rename to certdb
* Code review changes
2019-02-07 14:22:49 -05:00
Cameron
bfe8060f4d
Rollup tests ( #1224 )
...
* createdAt field no longer autoinsert
* rollup tests
2019-02-06 23:16:24 -05:00
Bill Thorp
690e8b2061
storage node cert cache ( #1226 )
...
* draft
* still errors
* double close fix
* added tests
* weird, goimports must not be working
* renames
* missed one
* forgot to save:
2019-02-05 12:57:56 -05:00
Egon Elbre
6564357e13
Ensure that tally tests run against postgres ( #1227 )
2019-02-05 16:23:45 +02:00
paul cannon
0238a7785b
make a map object before inserting into it ( #1217 )
...
I think this is causing panics.
2019-02-03 18:42:20 -06:00
Bill Thorp
0f662b8e38
sql based tally ( #1166 )
...
* WIP
* wacky changes
* more
* it builds.... IT BUILDS!!!
* fixed SQL, broke out saving tallies for testing
* shorter lines
* fixed SQL, moved tally_test to testplanet
* lint
* WIP logic error preventing PUT and GETs to same serialnum
* fixed BWA test
* fixed temporary brain failure
* eliminated magic numbers
* cleaned up satellite uplink stats
* use errs.Combine instead
* thrashing
* fixed tally erroneous error msg
* fixed tally test
* lint
* SQL syntax attempt to fix
* spelling error
* made bwa db test resist old postgres data
* postgres pk error msg hunting
* postgres pk error msg hunting
* postgres
* err might be nil?
* fixed error logging bug
* hopefully solved postgres issue
* using rebind
* moved tests to _test package
* fixing test dirs
* finally made sense of Egons package name feedback
* UTC, array fixes
2019-02-01 13:50:12 -05:00
Bill Thorp
60946c2024
make bandwidth agreements sensible: without []byte's ( #1152 )
...
removed []byte's from bandwidth agreement protocol buffers
2019-01-28 14:45:25 -05:00
Egon Elbre
49dacb662c
Improve bwagreement tests ( #1149 )
2019-01-26 01:06:38 +02:00
Bill Thorp
8ba4b61e59
Make Bandwidth Agreements Secure / Trustable ( #1117 )
...
* Added cert chains and nodeid verification to bandwidth agreement
2019-01-25 13:05:21 -05:00
Jennifer Li Johnson
856b98997c
updates copyright 2018 to 2019 ( #1133 )
2019-01-24 15:15:10 -05:00
Jennifer Li Johnson
030d294399
adds logs in rollup and tally run ( #1131 )
2019-01-24 15:05:53 -05:00
Jennifer Li Johnson
70940b8710
Making sure payments works ( #1122 )
...
* byte hours
* updates comment
* removes payment test data methods
* wip
* adds tally and rollup to peer
* remove sim test for payment (for now)
* lint error
2019-01-24 13:44:14 -05:00
Egon Elbre
5de7f8af7f
Satellite Peer ( #1119 )
2019-01-23 21:58:44 +02:00
Michal Niewrzal
eb2d858ffc
Use pointerdb.Service instead of pointerdb.Server ( #1099 )
2019-01-19 13:58:53 -05:00
Cameron
ecd704692e
wip testing and timestamp fix ( #1100 )
...
* wip testing and timestamp fix
* fixed import cycle
2019-01-18 11:53:23 -05:00
Michal Niewrzal
cc1bdef8b7
Move pointerdb logic to a service ( #1037 )
...
* test
* Move pointerdb logic to service
* tset
* reorder constructor params
* restore field
2019-01-18 10:10:21 -05:00
Jennifer Li Johnson
c4b876ddc5
Jj/bytes ( #1085 )
...
* run tally and rollup
* sets dev default tally and rollup intervals
2019-01-17 14:39:32 -05:00
Bill Thorp
342dc857f5
rollup query ( #1056 )
...
* implemention notes
* more notes
* starting rollup query
* not working yet
* fixed build
* fixed cfg bug
* change context cancelled errs to debugs
* using byte hours for at rest tally
* revert changes to go.mod
* comment fixes
* prevent double recording tallies in rollup
* linting
* stop leaking dbx
* nodeid changes
* fix build
2019-01-16 14:30:33 -05:00
littleskunk
2aebbeed34
add more bwagreement unit tests ( #1046 )
...
* add more bwagreement unit tests
2019-01-15 05:44:56 +01:00
Michal Niewrzal
e13f7c7a7d
Fix unstarted pointerdb responsibility ( #1033 )
2019-01-11 13:45:11 +01:00
Cameron
ef50bbf8b6
satellites reject expired BWAs ( #1015 )
...
* add 45 day expiration to PBAs
* add expiration field to relevant areas, DeleteExpired placeholder
* reject expired BWAs
* test for expired BWAs
* add BwExpiration config value
2019-01-10 13:30:55 -05:00
Bill Thorp
fb4a11ebb0
Differentiate GET and PUT operations in the tally service ( #965 )
...
* draft of new bandwidth agreement types
* updated storagenode report for new types
* use correct pba types
2019-01-10 06:41:57 -05:00
