storj/docs/blueprints
paul cannon 0c3dd44490 docs: audit-scaling: clarify process structure
and clarify some related implementation details.

Most notably, this change clarifies that the verification audit workers
and reverification audit workers are meant to live in a process or
processes separate from the satellite core, and outlines an extra queue
that will be used for communication with the core.

It's not entirely clear to me that this is the right approach; we would
save some fairly significant implementation time by leaving both types
of worker in the core. That would make it necessary to reconfigure and
restart the core when we wanted to change the number of verification
and/or reverification workers, and scaling would be limited to the
computational capacity of the core vm, but maybe those are acceptable
conditions.

Another option would be to leave the Verifier workers in the core, and
having a separate process for Reverifiers. That would be sort of a
middle way between the two above.

Change-Id: Ida12e423b94ef6088733b13d5cc58bdb78f2e93f
2022-10-27 20:00:36 +00:00
..
archive all: switch from master to main 2020-12-28 22:59:06 +01:00
byte-range-multipart-copy docs/blueprints: byte range multipart copy 2022-10-13 12:03:17 +00:00
images docs/blueprints: design doc for changes needed to allow for scaling audit workers 2022-09-28 19:47:41 +00:00
storagenode-graceful-exit chore: fix typos in the documentation (#3959) 2020-11-09 22:00:34 +02:00
access-revocation.md chore: fix typos in the documentation (#3959) 2020-11-09 22:00:34 +02:00
api-renovation.md docs/blueprints: Create API renovation blueprint 2022-01-24 14:24:31 +00:00
audit-containment.md design/blueprints (#2927) 2019-09-09 08:55:05 -06:00
audit-scaling.md docs: audit-scaling: clarify process structure 2022-10-27 20:00:36 +00:00
audit-suspend.md docs/blueprints: Add blueprint for creating a new state for storagenodes, suspension (#3778) 2020-02-19 14:29:48 -05:00
audit-v2.md design/blueprints (#2927) 2019-09-09 08:55:05 -06:00
cache-node-selection.md chore: fix typos in the documentation (#3959) 2020-11-09 22:00:34 +02:00
coupon-codes.md docs/blueprints: Create coupon codes blueprint (#4024) 2021-06-21 19:24:01 -04:00
deletion-performance.md docs/blueprints: Change adapt endpoint by creating (#3601) 2019-11-19 14:28:19 +01:00
disqualification.md design/blueprints (#2927) 2019-09-09 08:55:05 -06:00
distributed-tracing.md docs/blueprints: Add design doc for distributed tracing. 2020-02-25 20:29:05 +00:00
fast-billing-changes.md satellite/satellitedb: use queue for orders to get back fast billing 2020-02-24 17:07:07 +00:00
garbage-collection.md design/blueprints (#2927) 2019-09-09 08:55:05 -06:00
ge_refactoring.md docs/blueprints: graceful exit initial refactoring (#3938) 2020-10-19 23:34:48 -05:00
geofencing.md Geofencing and advanced placement-constraint support (#4227) 2021-11-15 15:23:41 +02:00
horizontal-scale-garbage-collection.md all: switch from master to main 2020-12-28 22:59:06 +01:00
kademlia-removal.md design/blueprints (#2927) 2019-09-09 08:55:05 -06:00
layer2-support.md blueprint: layer2 support for zksync 2021-03-01 23:41:11 +00:00
link-sharing-service.md design/blueprints (#2927) 2019-09-09 08:55:05 -06:00
linux-setup.md chore: fix typos in the documentation (#3959) 2020-11-09 22:00:34 +02:00
metainfo-rpc.md design/blueprints (#2927) 2019-09-09 08:55:05 -06:00
mountable-drive.md design/blueprints (#2927) 2019-09-09 08:55:05 -06:00
node-selection.md docs: add reputation documentation (#2982) 2019-09-10 14:24:58 +03:00
notification-system.md all: fix comments about grpc 2020-05-11 13:05:34 +03:00
password-key-derivation.md chore: fix typos in the documentation (#3959) 2020-11-09 22:00:34 +02:00
path-component-encoding.md design/blueprints (#2927) 2019-09-09 08:55:05 -06:00
README.md storj/storj: more domain changes 2021-04-15 20:51:43 +00:00
referral-manager-v1.md docs/blueprints: referral manager v1 (#3038) 2019-10-07 09:38:05 -04:00
repair-with-hashes.md chore: fix typos in the documentation (#3959) 2020-11-09 22:00:34 +02:00
satellite-billing-system.md chore: fix typos in the documentation (#3959) 2020-11-09 22:00:34 +02:00
satellite-side-async-packing.md docs/blueprints: add async packing 2022-04-23 06:34:17 +00:00
satellite-svc-separation.md all: switch from master to main 2020-12-28 22:59:06 +01:00
server-side-copy.md docs/blueprints: server-side copy 2022-02-16 17:26:23 +00:00
server-side-rename.md design doc: server-side move 2021-05-07 07:48:48 +00:00
slow-down-and-retry.md docs/blueprints: slow down and retry (#3826) 2020-04-15 00:25:08 +02:00
sparse-order-storage-rollout.md Blueprint: Sparse Order Storage 2020-12-21 18:04:54 +00:00
sparse-order-storage.md Blueprint: Sparse Order Storage 2020-12-21 18:04:54 +00:00
storage-node-automatic-updater.md Update automatic updater blueprint to catch missing requirement 2020-06-05 16:24:23 -06:00
storage-node-downtime-tracking-with-audits.md docs: change 'offline score' to 'online score' in downtime tracking doc 2020-12-29 15:31:20 +00:00
storage-node-satellite-selection.md chore: fix typos in the documentation (#3959) 2020-11-09 22:00:34 +02:00
storage-node-windows-installer.md design/blueprints (#2927) 2019-09-09 08:55:05 -06:00
TEMPLATE.md design/blueprints (#2927) 2019-09-09 08:55:05 -06:00
trusted-delegated-repair.md docs/blueprints: trusted-delegated-repair.md 2020-06-04 18:22:25 +00:00
uplink-scopes.md design/blueprints (#2927) 2019-09-09 08:55:05 -06:00
uplink-telemetry.md docs: Add uplink telemetry doc 2020-02-25 17:52:34 +00:00
value-attribution.md all: fix comments about grpc 2020-05-11 13:05:34 +03:00
webapp-session-management.md docs/blueprints: Add session management blueprint 2022-04-06 21:10:33 +00:00
zombie-segments-cleaner.md docs/design: zombie segments cleaner (#3461) 2019-11-12 00:25:51 -08:00

Blueprint Process

Design docs are now known as blueprints. We do not intend to keep blueprints up to date once the implementation has been completed. The final step of a blueprint is to update documentation elsewhere, such as:

Blueprints will be checked in as Markdown files in docs/blueprints folder.

Blueprints should have an editor, at least one discussion meeting, and at least two reviewers from the architecture board.

The editor is responsible for:

  • soliciting authors or authoring the document themselves,
  • scheduling discussion meetings,
  • finding reviewers,
  • posting the document in our forum, and
  • getting the document finalized and merged

The editor is also responsible for making a list of epics and tickets after the blueprint is finalized and merged. These should include archiving the blueprint when completed and updating appropriate documentation.

The reviewers are responsible for reviewing the clarity and reasonableness of the document.

The discussion meeting should have at least three people present. Invitees should familiarize with the document prior to the discussion. The reviewers should be present. If there are open problems after the meeting, then the meeting should be repeated.

One of the reviewers must be an architecture owner, currently:

  • Egon (@egonelbre),
  • Kaloyan (@kaloyan-raev),
  • JT (@jtolds),
  • Jens (@littleskunk),
  • unless agreed otherwise

The other reviewer should be someone with significant distributed systems expertise, currently:

  • Paul (@thepaul),
  • Simon (@simongui),
  • Jeff (@zeebo),
  • Matt (@brimstone),
  • unless agreed otherwise.

However, it is expected that there should be feedback from the engineering team, DevOps team, data science team, UX team, QA team, and the community.

Template

The blueprint uses TEMPLATE.md as a guide. However, it can be modified when necessary.

The template has the following sections:

  • Abstract gives an overview of the blueprint and what it accomplishes. It does not go into details about the problem.

  • Background gives an overview of the background why this blueprint exists. It describes the problems the blueprint tries to solve. It describes the goals of the design.

  • Design contains the solution and its parts. The level of detail should correspond to the level of risk. The more problems a wrong solution would cause, the more detailed should be the description. The design should describe the solution to the degree it is usable by the end-user.

  • Rationale section describes the alternate approaches and trade-offs. It should be clear why the proposed design was chosen among the alternate solutions.

  • Implementation describes the steps to complete this blueprint. It should contain a rough outline of tasks. If necessary, there can be additional details about the changes to the codebase and process.

  • Wrap up describes the steps needed

  • Open Issues contains open questions that the author did not know how to solve.

Format

The blueprint is intended to be read by developer contributors outside of Storj, but not the wider public. It is okay to assume some familiarity with the Storj project. Try to avoid acronyms that aren't common in the general developer community. Prefer simple sentences. Active voice is usually easier to read. Overall, be clear.