Commit Graph

229 Commits

Author SHA1 Message Date
pennae
2ab8e742a5 nixos-render-docs: move recursive manual parsing to base class
the html renderer will need all of these functions as well. some
extensions will be needed, but we'll add those as they become necessary.
2023-02-21 18:26:40 +01:00
pennae
5b8be28e66 nixos-render-docs: don't render options during manual parsing
we should really be rendering options at *rendering* time, not at parse
time. currently this is just an academic exercise, but the html renderer
will have to inspect the options.json data after the entire document has
been parsed, but before anything gets rendered.
2023-02-21 18:26:40 +01:00
pennae
068916ae8f nixos-render-docs: keep revision in renderer, not converter
ultimately it's the renderer that needs it, for the options rendering
that will be simplified in a bit.
2023-02-21 18:26:39 +01:00
pennae
6f253fc70b nixos-render-docs: drop options, env parameters
these weren't used for anything. options never was (and does not contain
any information for the renderer that we *want* to honor), and env is
not used because typed renderer state is much more useful for all our cases.
2023-02-21 18:26:39 +01:00
pennae
0236dcb59f nixos-render-docs: don't use markdown-it RendererProtocol
our renderers carry significantly more state than markdown-it wants to
easily cater for, and the html renderer will need even more state still.
relying on the markdown-it-provided rendering functions has already
proven to be a nuisance, and since parsing and rendering are split well
enough we can just replace the rendering part with our own stuff outright.

this also frees us from the tyranny of having to set instance variables
before calling super().__init__ just to make sure that the renderer
creation callback has access to everything it needs.
2023-02-21 18:26:39 +01:00
pennae
417dd2ad16 nixos-render-docs: add options asciidoc converter
same reasoning as for the earlier commonmark converter.
2023-02-21 18:19:00 +01:00
pennae
4d3aef762f nixos-render-docs: add options commonmark converter
the old method of pasting parts of options.json into a markdown document
and hoping for the best no longer works now that options.json contains
more than just docbook. given the infrastructure we have now we can
actually render options.md properly, so we may as well do that.
2023-02-21 18:19:00 +01:00
pennae
6c182075bb nixos-render-docs: forbid attrspans and examples in options
inline anchors are not allowed in option docs per the manual, and the
sole class we current have (.keycap) is never used anyway. disallow them
for now to avoid future surprises.

the same goes for examples, which aren't even documented in the manual yet.
2023-02-21 18:19:00 +01:00
pennae
82d066ffe3 nixos-render-docs: refactor option docs restrictions
move the restrictions we care about into a mixin class. a few more
restrictions will appear soon and a few new converters as well, the
renderers of which need not have these restrictions already baked in by
accident (like the manpage renderer does right now).
2023-02-21 18:19:00 +01:00
pennae
45619b3c4a nixos-render-docs: extend md_make_code
add the ability to set the info string for a newly created fenced code
block, and a flag to always emit a fenced block. the commonmark
converter will need this to faithfully recreate fenced and indented code
blocks.
2023-02-21 18:19:00 +01:00
pennae
895d9e69dd nixos-render-docs: extract md code block factory
the commonmark exporter can make good use once it appears.
2023-02-21 18:19:00 +01:00
pennae
00bffb84da nixos-render-docs: drop frozendict
with mypy type checking and Mapping types this is a lot less useful than
anticipated. let's drop it for simplicity and having fewer dependencies.

frozendict 2.3.5 also broke the mypy checks.
2023-02-21 18:19:00 +01:00
pennae
81cf173256 nixos-render-docs: use multiprocessing for options
options processing is pretty slow right now, mostly because the
markdown-it-py parser is pure python (and with performance
pessimizations at that). options parsing *is* embarassingly parallel
though, so we can just fork out all the work to worker processes and
collect the results.

multiprocessing probably has a greater benefit on linux than on darwin
since the worker spawning method darwin uses is less efficient than
fork() on linux. this hasn't been tested on darwin, only on linux, but
if anything darwin will be faster with its preferred method.
2023-02-18 13:17:57 +01:00
Robert Hensing
b72592ee04
Merge pull request #215973 from pennae/nrd-manual-structure
nixos/manual: specify manual structure in markdown
2023-02-17 18:01:23 +01:00
Felix Buehler
bc3d5934d7 treewide: use lib.optionals 2023-02-14 19:11:59 +01:00
pennae
1229e735ac nixos-render-docs: add structural includes, use for manual
this adds support for structural includes to nixos-render-docs.
structural includes provide a way to denote the (sub)structure of the
nixos manual in the markdown source files, very similar to how we used
literal docbook blocks before, and are processed by nixos-render-docs
without involvement of xml tooling. this will ultimately allow us to
emit the nixos manual in other formats as well, e.g. html, without going
through docbook at all.

alternatives to this source layout were also considered:

a parallel structure using e.g. toml files that describe the document
tree and links to each part is possible, but much more complicated to
implement than the solution chosen here and makes it harder to follow
which files have what substructure. it also makes it much harder to
include a substructure in the middle of a file.

much the same goes for command-line arguments to the converter, only
that command-lined arguments are even harder to specify correctly and
cannot be reasonably pulled together from many places without involving
another layer of tooling. cli arguments would also mean that the manual
structure would be fixed in default.nix, which is also not ideal.
2023-02-12 13:02:42 +01:00
pennae
d30da4d9cd nixos-render-docs: add support for <part>
<part> is different from all other blocks we care about in that it
requires textual content to be wrapped in <partintro>. add support for
this to the generic docbook renderer, which will just assume that a part
is the whole document start to finish. we do make provision for the
manual renderer to close a partintro tag early though.
2023-02-12 11:55:09 +01:00
pennae
ad2b150af7 nixos-render-docs: use Mapping for options converter
this way we can pass in frozendicts from other converters.
2023-02-12 11:55:09 +01:00
pennae
d004105003 nixos-render-docs: print exception trees by __cause__
__context__ is always set to the prior exception, even when not using
the raise from form. __cause__ is only set during raise from. use
__cause__ so we can override a leaf exception (eg KeyError to something
more meaningful).
2023-02-12 11:55:09 +01:00
pennae
652a283e51 nixos-render-docs: render manual chapters during manual build
render all manual chapters to docbook from scratch every time the manual
is built. nixos-render-docs is quick enough at this to not worry about
the cost (needing only about a second), and it means we can remove
md-to-db.sh in the next commit.

no changes to the rendered html manual except for replacements and smartquotes.
2023-02-10 06:40:02 +01:00
pennae
67917ac102 nixos-render-docs: rename manual docbook converter to docbook-section
we'll soon add another docbook converter that does not emit a section as
a collection of chapters, but sections or chapters on their own. this
should clarify naming a bit before there can be any confusion.
2023-02-10 06:40:02 +01:00
pennae
b59b0230ae nixos-render-docs: add example blocks
this is currently only supported by the docbook exporter, and even the
docbook exporter doesn't do much with them. we mirror the conversion
pandoc did for consistency with the previous manual chapter conversion,
which is to add just an anchor with the given id. future exporters that
go directly to html might want to do more.
2023-02-10 06:40:02 +01:00
pennae
bb6526e0de nixos-render-docs: add generic attributed-block parsing
this is a subset of pandoc's fenced divs. currently we only use this for
admonitions (which get a new name to differentiate them from other kinds
of blocks), but more users will appear soon.
2023-02-10 06:40:02 +01:00
pennae
36b0f53f85 nixos-render-docs: promote compact-list attrs to core rule
rules are a better place for this. since _post_parse is now empty (and
presumably will never grow) we'll remove that as well.
2023-02-10 06:40:02 +01:00
pennae
6cd368870b nixos-render-docs: allow dots in heading ids
this is used by release notes (and we don't want to break links to
those), and is also technically allowed anyway. we will *not* extend the
regex to allow more characters just yet due to a mozilla recommendation
against it (cf https://developer.mozilla.org/en-US/docs/Web/HTML/Global_attributes/id)
2023-02-10 06:40:02 +01:00
pennae
fd9f6c7501 nixos-render-docs: promote heading id extraction to a core rule
this should've been a core rule from the beginning. not being a core
rule made it always run after smartquotes and replacements, which
could've wrecked the id.
2023-02-10 06:40:02 +01:00
pennae
4b06b82130 nixos-render-docs: add the .keycap class
this lets us parse the `[F12]{.keycap}` syntax we recently introduced to
the nixos manual markdown sources. the docbook renderer emits the keycap
element for this class, the manpage renderer will reject it because it's
not entirely clear what to do with it: while html has <kbd> mandoc has
nothing of the sort, and with no current occurences in options doc we
don't have to settle on a (potentially bad) way to render these.
2023-02-10 06:40:01 +01:00
pennae
67086639e0 nixos-render-docs: add support for full attributed spans
this is pretty much what pandoc calls bracketed spans. since we only
want to support ids and classes it doesn't seem fair to copy the name,
so we'll call them "attributed span" for now. renderers are expected to
know about *all* classes they could encounter and act appropriately, and
since there are currently no classes with any defined behavior the most
appropriate thing to do for now is to reject all classes.
2023-02-10 06:40:01 +01:00
pennae
702e1fc743 nixos-render-docs: add all-features manpage renderer test
now that the renderer produces the output we want to keep for the future
we can add a test that checks all of its features. this test notably
does not include markdown headings since we don't want to have those in
manpages (at least right now), but tests for other converters may add
headings for themselves.
2023-02-08 15:23:34 +01:00
pennae
78052a22cb nixos-render-docs: track links in manpages
for the longest time we completely dropped link targets in
configuration.nix.5.  let's stop doing this now and instead provide a
footnote for each link in a given option, numbered locally per option.

we will currently duplicate the link for <labelless-links> because it
makes it easier to get the collection of all links in a given option.
this may not be useful enough, so over time we might decide to drop the
footnotes for such links.
2023-02-08 15:23:34 +01:00
pennae
3c7fd940ba nixos-render-docs: indent and embolden list item heads in manpages
this matches what html outputs do more closely, and feels like it'll be
easier to read because it looks less like just another paragraph.
2023-02-08 15:23:34 +01:00
pennae
f47adfcb6f nixos-render-docs: make manpage deflists a little nicer
indent the entire list by 4, just like each definition is already
indented by 4. this matches rendering in html, which indents terms once
and indents definitions twice.
2023-02-08 15:23:34 +01:00
pennae
1e4bafdbc5 nixos-render-docs: style file literals in manpages
similar to inline code these were indistinguishale from other text.
render then in italic font instead, like mdoc .Pa does.
2023-02-08 15:23:34 +01:00
pennae
29252d1477 nixos-render-docs: add quotes to inline code in manpages
other output types already have markings for inline code, manpages do
not. this can be somewhat confusing, so we'll do the least intrusive
thing: surrounding inline code blocks in ‘’. doing so separates inline
code from the rest of the text and is unlikely to collide with the
quoted contents. it's also what mdoc does with its Ql macro.
2023-02-08 15:23:34 +01:00
pennae
f33e360f67 nixos-render-docs: remove the ... escape in manpages
this is a holdover from docbook stylesheets. not really sure why they
did that.
2023-02-08 15:23:34 +01:00
pennae
3a3274231e nixos-render-docs: always render links bold in manpages
no reason to differentiate between links by source of their label. this
feature seems to be mostly used to change labels of links to other
options, but this should ultimately be done by auto-linking from
{option}`...`. at some point we may want to introduce a warning when
this pattern is encountered, but there's a lot to work out still before
we can do that.
2023-02-08 15:23:34 +01:00
pennae
5c5dadd382 nixos-render-docs: support compact lists in manpages
most of the lists in option docs are actually compact, but docbook to
manpage processing always rendered them as non-compact. compactifying
these lists improves readability somewhat since most lists and their
contents are pretty short.
2023-02-08 15:23:34 +01:00
pennae
10a4f0daca nixos-render-docs: add options manpage converter
mdoc is just too slow to render on groff, and semantic markup doesn't
help us any for generated pages.

this produces a lot of changes to configuration.nix.5, but only few
rendering changes. most of those seem to be place losing a space where
docbook emitted roff code that did not faithfully represent the input
text, though a few places also gained space where docbook dropped them.
notably we also don't need the compatibility code docbook-xsl emitted
because that problem was fixed over a decade ago.

this will handle block quotes, which the docbook stylesheets turned into
a mess of roff requests that ended up showing up in the output instead
of being processed.
2023-02-08 15:23:34 +01:00
pennae
56f1d99b16 nixos-render-docs: factor out sorting of options list 2023-02-08 15:23:34 +01:00
pennae
b2a5b4d789 nixos-render-docs: move list-is-compact attr to meta
Token.attr is a dict[str, str | int | float], meta has no restriction on
the value type. attrs is ostensibly meant for html attributes, meta for
any information whatsoever.
2023-02-08 15:23:34 +01:00
pennae
09411102f6 nixos-render-docs: add option block separators
this will be necessary for manpages, which separate option declarations
not with external tags but by interspersing mandoc spacing instructions.
2023-02-08 15:23:34 +01:00
pennae
32136b1b01 nixos-render-docs: don't render empty descriptions at all 2023-02-08 15:23:34 +01:00
pennae
11daebd2d9 nixos-render-docs: add block and inline joiners
these work together with render and renderInline to produce an output
from either of the two. rendering manpages will need both: to join
blocks with newlines, and to run some postprocessing and the rendered inlines.
2023-02-08 15:23:34 +01:00
pennae
5a5255983b nixos-render-docs: calculate list end indices
that'll be useful to calculate the width of list item heads, which we'll
ned to render manpages.
2023-02-08 15:23:34 +01:00
pennae
edccae739a nixos-render-docs: add a test for running mypy
pulling mypy into the build closure is unfortunately not reasonable, the
closure for mypy is rather large and takes a long time to build. if we
have the type checks hooked into CI we'll get most of the benefit though.
2023-02-08 15:23:34 +01:00
figsoda
acf0a7d9fd nix-init: 0.1.0 -> 0.1.1
Diff: https://github.com/nix-community/nix-init/compare/v0.1.0...v0.1.1

Changelog: https://github.com/nix-community/nix-init/blob/v0.1.1/CHANGELOG.md
2023-02-06 15:51:57 -05:00
pennae
9711de7b7e nixos-render-docs: improve error messages for multi-title manual chapters 2023-02-03 00:20:59 +01:00
Nick Cao
a74f695255
Merge pull request #210864 from peng1999/nix-info-patch
nix-info: fix error when no channel installed
2023-01-31 18:10:55 +08:00
Peng Guanwen
ffe9031129
Use buitins.currentSystem 2023-01-31 17:56:23 +08:00
pennae
5b6dcece88
Merge pull request #212684 from pennae/nixos-render-docs
nixos-render-docs: init, use for some manual rendering to docbook
2023-01-30 19:26:07 +01:00