storj/satellite/console
Cameron Ayer 4424697d7f satellite/accounting: refactor live accounting to hold current estimated totals
live accounting used to be a cache to store writes before they are picked up during
the tally iteration, after which the cache is cleared. This created a window in which
users could potentially exceed the storage limit. This PR refactors live accounting to
hold current estimations of space used per project. This should also reduce DB load
since we no longer need to query the satellite DB when checking space used for limiting.

The mechanism by which the new live accounting system works is as follows:

During the upload of any segment, the size of that segment is added to its respective
project total in live accounting. At the beginning of the tally iteration we record
the current values in live accounting as `initialLiveTotals`. At the end of the tally
iteration we again record the current totals in live accounting as `latestLiveTotals`.
The metainfo loop observer in tally allows us to get the project totals from what it
observed in metainfo DB which are stored in `tallyProjectTotals`. However, for any
particular segment uploaded during the metainfo loop, the observer may or may not
have seen it. Thus, we take half of the difference between `latestLiveTotals` and
`initialLiveTotals`, and add that to the total that was found during tally and set that
as the new live accounting total.

Initially, live accounting was storing the total stored amount across all nodes rather than
the segment size, which is inconsistent with how we record amounts stored in the project
accounting DB, so we have refactored live accounting to record segment size

Change-Id: Ie48bfdef453428fcdc180b2d781a69d58fd927fb
2020-01-16 10:26:49 -05:00
..
consoleauth common: separate repository 2019-12-27 14:11:15 +02:00
consoleweb satellite/accounting: refactor live accounting to hold current estimated totals 2020-01-16 10:26:49 -05:00
apikeys_test.go common: separate repository 2019-12-27 14:11:15 +02:00
apikeys.go satellite/console: Added error for adding api key with existing name attempt. (#3185) 2019-10-10 16:28:35 +03:00
auth.go web/satellite: auth error messages fixed (#3426) 2019-11-05 13:55:26 +02:00
billinghistoryitem.go satellite/payments: coupons, coupons usage, invoice generation with pricing model applied 2020-01-14 12:45:00 +00:00
database.go satellite/console: use transaction helpers in consoledb 2020-01-07 17:59:10 +00:00
projectmembers_test.go common: separate repository 2019-12-27 14:11:15 +02:00
projectmembers.go web/satellite: API keys paged backend (#2839) 2019-09-12 17:19:30 +03:00
projects_test.go common: separate repository 2019-12-27 14:11:15 +02:00
projects.go satellite/accounting: separate project limit from project entity (#3632) 2019-11-25 16:18:04 +02:00
projectusagelimits.go satellite/console: project usage limits api (#3702) 2019-12-12 14:58:15 +02:00
registrationtoken.go all: add archview annotations (#2964) 2019-09-10 16:24:16 +03:00
resetpasswordtoken_test.go common: separate repository 2019-12-27 14:11:15 +02:00
resetpasswordtoken.go all: add archview annotations (#2964) 2019-09-10 16:24:16 +03:00
service.go satellite/payments: coupon addition removed 2020-01-14 16:24:48 +00:00
sortdirection.go web/satellite: API keys paged backend (#2839) 2019-09-12 17:19:30 +03:00
usercredits_test.go common: separate repository 2019-12-27 14:11:15 +02:00
usercredits.go private: rename internal to private (#3573) 2019-11-14 21:46:15 +02:00
users_test.go satellite,storage: clean global ctx usage in tests 2020-01-09 10:37:21 +00:00
users.go satellite/referrals: set up referrals service and http endpoints (#3566) 2019-11-25 16:36:36 -05:00
validation.go satellite/referrals: set up referrals service and http endpoints (#3566) 2019-11-25 16:36:36 -05:00