Commit Graph

77 Commits

Author SHA1 Message Date
Jeff Wendling
f671eb2beb satellite/satellitedb: use queue for orders to get back fast billing
This change adds two new tables to process orders as fast as we used
to but in an asynchronous manner and with hopefully less storage
usage. This should help scale on cockroach, but limits us to one
worker. It lays the groundwork for the order processing pipeline to
be queue rather than database driven.

For more details, see the added fast billing changes blueprint.

It also fixes the orders db so that all the timestamps that are
passed to columns that do not contain a time zone are converted to
UTC at the last possible opportunity, making it less likely to use
the APIs incorrectly. We really should migrate to include timezones
on all of our timestamp columns.

Change-Id: Ibfda8e7a3d5972b7798fb61b31ff56419c64ea35
2020-02-24 17:07:07 +00:00
Maximillian von Briesen
a5afa4834a
docs/blueprints: Add blueprint for creating a new state for storagenodes, suspension (#3778)
This document describes a new state, suspension, which nodes will
be placed in if they respond to too many audits with "unknown" errors.
When in this state, node operators will have to fix their node within a
certain amount of time, or they will be disqualified. While nodes are
suspended, they will not be able to receive new data.

Change-Id: I543f27ada66d8b40fc52ca8ead8ecc8479326bad
2020-02-19 14:29:48 -05:00
Andrew Harding
cb89496569 storagenode/trust: wire up list into pool
- also updated ping chore to pick up trust changes
- fixed small typo in blueprint
- fixed flags for storj-sim
- wired up changes to testplanet

Change-Id: I02982f3a63a1b4150b82a009ee126b25ed51917d
2019-12-13 20:32:50 +00:00
Ivan Fraixedes
46ad27c5f2
docs/blueprints: Change adapt endpoint by creating (#3601)
Replace a sentence which indicates that we're creating and new endpoint
in the storage node side rather than adapting the current one.
2019-11-19 14:28:19 +01:00
Ivan Fraixedes
b5b4543df3
docs/blueprints: Deletion performance (#3520)
Creates the blueprint document for changing the mechanism to delete pieces and improve its performance.
2019-11-19 08:16:35 +01:00
Michal Niewrzal
89efd17f4d
docs/design: zombie segments cleaner (#3461) 2019-11-12 00:25:51 -08:00
Andrew Harding
0885eaa183 Storage Node Satellite Selection design (#3353) 2019-11-05 18:07:08 -05:00
Jess G
5abb91afcf
satellite: change the Peer name to Core (#3472)
* change satellite.Peer name to Core

* change to Core in testplanet

* missed a few places

* keep shared stuff in peer.go to stay consistent with storj/docs
2019-11-04 11:01:02 -08:00
Maximillian von Briesen
472672d52f
docs/storage-node-downtime-tracking: add open issue regarding satellite outages (#3370) 2019-10-25 11:39:11 -04:00
Brandon Iglesias
84156d6912 docs/blueprints: updating term to held amount (#3248) 2019-10-11 21:05:50 +03:00
Ethan Adams
4c4519f0be
satellite/gracefulexit: add transfer queue for pieces (#3174)
initial impl of transfer queue
updated docs represent the new design how we handle durability during exit
2019-10-07 16:38:05 -04:00
Yingrong Zhao
d29946d3d7
docs/blueprints: referral manager v1 (#3038)
* archive old referral offer doc

* init referral manager

* add Referral Manager CLI definition

* add draft implementation steps

* use existing registration_token table for storing invitation tokens

* fix numbering

* add open question about auth

* wip pseudocode

* add user interface design

* separate token generation step from getting users step

* change button to be a referral tab on the UI

* simplify some sentences. clarified CLI referral manager interaction

* add prevent two processes running at the same time

* add cli code

* clear users when CLI exites earlier or referral manager finishes sending tokens to satellites

* use transaction for saving tokens on satellite so we can roll back if error occurs

* add auth token and GetSatelliteURLs

* add schema for satellites table

* update to only have satelite to talk to referral manager

* add definition for eligible users and owner's satellite

* fix indentation

* add saving user info during redeem pseudocode

* wip

* address reviews

* add unredeemed token in users table

* address comments
2019-10-07 09:38:05 -04:00
Ethan Adams
9edfb6efe0
satellite/satellitedb: Initial GE Satellite DB Implementation (#3049)
Initial GE Satellite DB impl
Add basic CRUD operations for graceful_exit_progress and graceful_exit_transfer_queue tables.
2019-09-25 11:12:44 -06:00
Ethan Adams
886041e0ba
satellite/satellitedb: add new graceful exit tables and add graceful exit fields to nodes (#3033)
DB schema changes for satellite to support Graceful Exit
2019-09-13 12:57:32 -04:00
Yingrong Zhao
9f2f1527c5
storagenode/storagenodedb: add new tables for graceful exit (#3008)
* add database schema

* add migration

* change table name and update blueprint
2019-09-11 18:57:53 -04:00
Ethan Adams
eac20e977f
docs/design: Graceful Exit (#2866)
Splits original Graceful Exit document into 5 documents: Overview, Piece Gathering, Protocol, Reports & UI
2019-09-11 08:26:43 -04:00
Benjamin Sirb
86f4b41a70 docs: add reputation documentation (#2982) 2019-09-10 14:24:58 +03:00
JT Olio
c81e4fcb9e
design/blueprints (#2927)
we've decided to rename design docs to blueprints to indicate that we don't intend to keep blueprints up to date once completed. instead, the final step of blueprint implementation is to archive the blueprint and update actual documentation elsewhere (potentially copying and pasting most of the blueprint). this PR updates the template and readme
2019-09-09 08:55:05 -06:00
Ivan Fraixedes
4d65fdaea5
docs/design: Adapt SN Downtime tracking to be a blueprint (#2931)
Bluepints are replacing the design documents.
A blueprint has a wrapup section, this commit adds the section to the
document and replace any mention to design document by blueprint.
2019-09-03 16:44:10 +02:00
Egon Elbre
842c7118c5
docs/design: update audit v2 (#2901) 2019-08-28 22:47:43 +03:00
Ivan Fraixedes
c7bd75b032
docs/design: Storage Node downtime tracking (#2857)
Create the Storage Node downtime tracking design document, using the current template and revision workflow approval.
2019-08-28 21:05:18 +02:00
Jess G
3e121b13e0
docs/design: satellite service separation (#2815)
* add md extension to dd process doc

* rm old format of dd process doc

* wip design doc

* add diagram, add implementation

* adjust format

* fix format

* add updates per CR and arch review meeting

* update diagram sizes

* make image smaller

* try using svg instead of png

* replace png with svg, mv to docs/images dir

* updates per CR comments

* more CR comment changes

* update sa design img

* grammar fixes

* a few more updates

* fixings nits

* fix spelling err

* fix spelling, change binary to process, rm process doc

* add changes to design imagees per CR comments
2019-08-28 09:28:53 -07:00
Ivan Fraixedes
46a495fbaf docs/design: Fix typos & remove not applicable title (#2879) 2019-08-28 15:55:36 +03:00
Ivan Fraixedes
ec715cba9e docs/design: Remove subtitle part not applicable (#2880) 2019-08-28 15:38:08 +03:00
Jennifer Li Johnson
c8405f6c2b
docs/design/archive: moves archive directory (#2885) 2019-08-27 14:59:52 -04:00
Jennifer Li Johnson
9abc3e9d69
moves audit gating design draft to doc archive (#2883) 2019-08-27 10:58:02 -04:00
Kaloyan Raev
106a21ebe0
docs/design: Use WiX toolset directly instead of go-msi for Windows installer (#2851) 2019-08-27 16:06:00 +03:00
Alexander Leitner
462640a9fe docs/design: automatic updater (#2789) 2019-08-26 17:45:18 +03:00
Egon Elbre
36c9d569ff
design/docs: add successful pingback to kademlia removal document (#2837) 2019-08-26 15:34:04 +03:00
Kaloyan Raev
cff6dc2b2a
Rename design-doc-process.md to README.md (#2848)
This way the design doc process is directly visible when browsing the doc/design folder in the GitHub Web UI. As the number of docs grows it would be harder to find the doc with the design process.
2019-08-22 11:17:18 +03:00
Jennifer Li Johnson
9df2ec6a37
docs/design: Kademlia Removal Design Proposal (#2686) 2019-08-20 15:48:35 -04:00
Egon Elbre
2e1fc4bc52
docs/design: describe process in more detail (#2817) 2019-08-20 12:17:09 +03:00
Bryan White
8832a393e1
formatting fix (#2823) 2019-08-19 23:51:06 +02:00
Alexander Leitner
9c4af157f5
docs/design: windows installer (#2780)
* Add design doc for windows installer
2019-08-15 08:10:55 -04:00
Brandon Iglesias
d28c8aad8f
design doc for design doc process (#2782) 2019-08-14 12:16:18 -04:00
Jennifer Li Johnson
c57aaee664
docs/design: update kademlia audit gating 2019-08-13 11:21:40 -04:00
Bill Thorp
260f648275
Update repair-with-hashes.md (#2756) 2019-08-09 16:34:11 -04:00
Bill Thorp
05e99a78d9
docs/design: use piece hashes for repair (#2723)
* Update for new piece signing logic
2019-08-08 20:54:20 -04:00
Natalie Villasana
af69dcc6d9
docs/design: auditing 2.0 (#2703) 2019-08-08 13:22:13 -04:00
Egon Elbre
c8edeb0257
satellite/overlay: rename overlay.Cache to overlay.Service (#2717) 2019-08-06 19:35:59 +03:00
Ivan Fraixedes
a13008198c docs/design: Remove "Title" word from title header (#2693) 2019-08-02 13:42:53 +03:00
Cameron
5f096a3eab
Remove GetValid, add GetAll to vouchers DB (#2594)
* refactor GetValid to GetAll

* update design doc
2019-07-19 10:52:44 -04:00
Natalie Villasana
6aefa1ae80 adds metainfo rpc doc (#1957) 2019-07-09 21:44:01 +02:00
Andrew Harding
7212423c52
Link Sharing service design (#2360) 2019-07-09 09:36:08 -06:00
Jeff Wendling
671648012a Update password based key derivation design doc (#2367) 2019-07-09 00:07:03 -04:00
Jeff Wendling
c0097bbc39 design doc: path component encoding (#2386)
* design doc: path component encoding

in talks with a number of people on the team, it seems there is some
opportunity for coordination on how paths should be encoded so that
everyone works works well with each other. in particular, we care
about matching s3 semantics for list for unencrypted paths which
contain empty path components. that makes it hard to increment
and decrement keys for seek operations. additionally, there is
work to reduce our metadata overhead for storage of paths. this
proposal takes most of the ideas from that work and ensures that
it supports the ability to efficiently seek through prefixes of
paths.

* fix following and discuss c strings
2019-07-03 19:01:52 +02:00
nerdatwork
74e12a0bb4 Fixed typo in disqualification.md (#2396)
* Update .clabot

* Update .clabot

* Revert "Update .clabot"

This reverts commit de8cb5efa9a5e0bf43ec2f519e3324a485f78b01.

* Revert "Revert "Update .clabot""

This reverts commit 4a6e88df5e3d1a92ae38cd9cf5036a14917369db.

* Update .clabot

* Update sender.go

* Update .clabot

* Update disqualification.md
2019-07-01 14:03:29 -04:00
Egon Elbre
05a283f331 jenkins: disallow files over 600KB (#2323) 2019-06-26 08:44:52 +02:00
ethanadams
0e528bc56e
Add attribution report to the satellite CLI (#2288)
* added satalite partner value attribution report. WIP

* WIP

* basic attribution report test completed. still a WIP

* cleanup

* fixed projectID conversion

* report display cleanup

* cleanup .added more test data

* added partnerID to query results

* fixed lint issues

* fix import order

* suggestions from PR review

* updated doc to reflect implementation

* clarification comments in the report SQL

* Changed based on PR suggestion

* More changes based on PR suggestions

* Changes based on PR suggestions

* reordered tests to make consistant with previous 2

* small comments cleanup

* More PR suggestions

* fixed lint issue and removed printf

* fixed var name

* Updates based on PR suggestions

* fixed message

* fixed test

* changes required after merge from master
2019-06-25 16:58:38 -04:00
Brandon Iglesias
fc11a17df7
initial design doc for SN graceful exit (#2077)
* initial design doc for SN graceful exit

* updates to the document mostly syntax and typos

* updates based on the first acceptance meeting

* SN will get paid for download data from clients

* updates based on our second acceptance meeting

* typo updates

* updates to things and stuff based on comments

* updated based on comment

* syntax changes/ typo fix
2019-06-24 10:28:53 -04:00