Principle

We found this approach convenient for lighter projects, i.e. ones comprising a limited number of pages. This is the case of this page and more generally of the whole Ceylan-HOWTO website.

We chose to rely on the reStructuredText syntax and tools, also known as RST, a part of the Docutils project. Here we do not specifically rely on elements related to Python or the Sphinx toolchain, as our more heavy-duty approach does.

We augmented reStructuredText with:

• a set of make-based defines and rules (automatic or explicit) that were aggregated in Ceylan-Myriad (see notably GNUmakerules-docutils.inc and the generate-docutils.sh generation script); this mechanism is layer-friendly, in the sense that all layers defined (directly or not) on top of Myriad are able by default to re-use these elements and to customise them if needed
• a template on which we rely for most documents, featuring notably a standard table (to specify usual metadata such as organisation, contact information, abstract, versions, etc.), a table of contents, conventions in terms of title hierarchy and, for the HTML output, a banner (a fixed, non-scrolling panel offering shortcuts, in the top-right corner of the page)
• a simple tag-based system to have the actual document markup (*.rst) directly generated from a higher-level source one (*.rst.template); in practice, if defined, only the latter element is edited by the user, and tags (such as *_VERSION_TAG, *_DATE_TAG, etc.) are automatically filled-in appropriately

Of course this website, and many others that we created, rely on this approach; as an example, one may look at the sources of the current document.

Specific Topics

$ pacman -Sy mathjax

$ cd /tmp && git clone https://github.com/mathjax/MathJax.git
$ mv MathJax/es5 ~/Software/MathJax

$ make create-mathjax-symlink

M = \begin{bmatrix}
       a11 & a12 & ... & a1n \\
       a21 & a22 & ... & a2n \\
       ... & ... & ... & ... \\
       am1 & am2 & ... & amn \\
    \end{bmatrix}

<img src="foobar.png" id="responsive-image-large"></img>

.. _`how to render equations`:

This algorithm can be:

.. code:: erlang
   [...]

Here is `some link <http://example.org/xxx>`__.

Principle

It applies to more ambitious projects, involving larger content with potentially many interlinked pages.

It is based on the Sphinx toolchain, and therefore shares many elements with our lightweight approach above, starting from the RST syntax.

It offers out of the box many useful mechanisms beyond the generation of a single-page website, from a smooth navigation between a set of pages based on foldable menus to a generated index and a local search engine. It moreover can be easily customised.

More types of outputs are readily supported: HTML, PDF, EPUB, man pages.

Installation

One may follow these guidelines. On Arch Linux, we install the python-sphinx package, thanks to: pacman -Sy python-sphinx gnu-free-fonts texlive-binextra (last package being needed now for latexmk, to generate PDFs).

Configuration

Running sphinx-quickstart is one's best route; in terms of choices:

• we strongly recommend separating the source and build directories
• the "Project name" shall not include "documentation", as this word will be added automatically wherever needed

Then make html should generate a base website (including makefiles) that can be browsed in the build/html directory (use make clean to force its erasure; run just make to list all targets of interest, including the linkcheck one, to check all external links for integrity).

The project settings can be edited in source/conf.py.

Theme Selection

The default (HTML) theme is the Alabaster one. Other themes may be preferred, whether they are Sphinx built-ins or external ones. Many can be customised.

As for us, we prefer mobile-compliant themes with a left column to navigate. This includes the popular Read the Docs theme, which will use here, but also the classic theme.

The documentation of the Read the Docs theme details everything needed. For an installation thereof on Arch, the simplest is to run pacman -Sy python-sphinx_rtd_theme.

Then it is just a matter of editing source/conf.py so that 'sphinx_rtd_theme' is listed in the extensions list, and that the html_theme is now set to 'sphinx_rtd_theme'.

Running make html again should be sufficient to take this new theme into account. The generated result is quite satisfying.

Here is one customisation thereof (theme options) that we like:

#html_theme = 'alabaster'
html_theme = 'sphinx_rtd_theme'

html_theme_options = {
   #'analytics_id': 'G-XXXXXXXXXX',  #  Provided by Google in your dashboard
   #'analytics_anonymize_ip': True,
   'logo_only': False,
   'display_version': True,
   'prev_next_buttons_location': 'both',
   'style_external_links': True,
   'vcs_pageview_mode': '',
   #'style_nav_header_background': 'white',
   #'style_nav_header_background': '#2980B9',
   'style_nav_header_background': 'black',
   # Toc options
   'collapse_navigation': False,
   'sticky_navigation': True,
   'navigation_depth': 4,
   'includehidden': True,
   'titles_only': False
}

html_logo = '../foobar-title.png'
html_favicon =  '../foobar-icon.png'
html_last_updated_fmt = ''
html_copy_source = False
html_show_sourcelink = False
html_show_sphinx = False

html_static_path = ['_static']

# So that inner cross-RST file references can be found:
default_role = 'any'

Adding Content

This is just a matter of adding *.rst files (each defining at least one title) in the source directory, and to reference them in a least one table of contents (e.g. in source/index.rst), like in:

.. toctree::
  :maxdepth: 2
  :caption: Contents:

  ./foobar.rst
  ./buz.rst

The result in a static website that can safely be transferred and served by any webserver of choice.

Inner Links with Sphinx

In our preferred approach, described here and there, such inner links - which are possibly defined and referenced in different RST files - are to be managed in a slightly different way than for Docutils's inner links:

• first, as shown in the conf.py example above, the default_role = 'any' setting should be specified
• then, if each inner link target shall be specified as usual (e.g. as .. _`brief answers`:), they should be referenced like with Docutils but without a trailing underscore (e.g. as `brief answers` - rather than as `brief answers`_)

The Problem of Nested Includes

We would have liked to organise a Sphinx document according to a filesystem tree, so that there is in each directory a RST file that comprises the content for that level and that just lists its direct local children as the RST files in its subdirectories, as relative files, like, in a a/b/c/c.rst file: .. include:: d/d.rst.

Strangely enough, it worked for 3-level nesting (a, a/b and a/b/c), but not for the next level: even though the RST files in c where included as d/d.rst, they were never found:

a/b/c/c.rst:4: CRITICAL: Problems with "include" directive path:
InputError: [Errno 2] No such file or directory: 'a/b/d/d.rst'.

(we can see that c is lacking; no way of adding it once; and trying there to specify .. include:: c/d/d.rst results in a/b/c/c/d/d.rst not being found...)

The only (unsatisfactory) solution we found it is specify paths that are "absolute" (i.e. relative to the root of the tree), for example as .. include:: /a/b/c/d/d.rst.

Validating / Checking HTML Content

In addition to the verification of the messages reported when the document is built, some tools allow to perform some checks on a generated document.

Notably an online HTML page, or set of pages, can be verified by third-party tools like this one, to detect dead links.

Fixing Permissions in Third-Party Content to Integrate in a Web Root

The objective is to ensure that a filesystem tree can be transferred as a whole without permission errors to a given server more than once, whereas the destination user and group (typically specialised and restricted on a server) differ from the source ones.

It is indeed often necessary to fix permissions in a third-party tree before it is transferred to a server (e.g. MathJax being copied from a client host through scp in a web root on a given server); otherwise the next transfers will stumble on the initial, inadequate group rights, typically preventing them to be overwritten by a process belonging to a different user yet being in the same group, like 700 instead of 770 - resulting in Permission denied errors.

For that we recommend executing our fix-www-metadata.sh script in the source root prior to transfer it to a webroot of choice. This typically applies to MathJax (see the Rendering Mathematical Elements section), which therefore should not be installed in the system tree thanks to a package manager (as our script will alter its permissions) but in the user account (e.g. as ~/Software/mathjax).

So typically this script shall be symlinked in each third-party root of interest, and be executed there at each update thereof.

However doing so does not solve all issues: the file entries created/updated by scp on the server will be owned by the user on the server implied by the scp command, for example stallone:users - not the desired web-srv-user:web-srv-group. Of course the fix-www-metadata.sh script can be run (by root) on the server to correct that. Yet the next update of this webroot will fail again with Permission denied errors, as the groups are not expected to match anymore (we cannot overwrite with our client-side users group a remote directory whose permission is 770 that is owned by group web-srv-group).

A solution is to ensure that the source content bears already the target group. As scp relies on user/group IDs, not on names (e.g. on a numerical GID like 1001, not a name like users or web-srv-group), the simplest solution is to determine the actual GID of the target group on the server (e.g. running, as root, grep web-srv-group /etc/group may tell us that the GID of web-srv-group is 1002 there) and to create on the client a group with the same GID (if ever possible - that is if there is not already another group happening to have be set to that GID) and to apply it to the source content to transfer, like in:

# We are on the "client", source host, as root:
# (web-srv-group-of-target-server clearer than web-srv-group)
$ groupadd --gid 1002 web-srv-group-of-target-server
$ usermod -a -G web-srv-group-of-target-server stallone
$ chgrp -R web-srv-group-of-target-server /home/stallone/mathjax

Then, afterwards, when the stallone user performs his scp repeatedly to transfer updated versions of his mathjax directory from the client to the server, he should be able to perform a flawless update of its files and directories.

Pointing to a Specific Moment in a Linked Video

It is as simple as designating, in an HTML link, the targeted second by suffixing the URL video filename with #t=DURATION_IN_SECONDS, like in some-video.mp4#t=1473 [1].

Conversion between Markup Formats

Pandoc is the tool of choice for such operations, as it often yields good results.

For example, in order to convert a page written in Mediawiki syntax, whose source content has been pasted in a old-content-in-mediawiki.txt file, into one that be specified in a GitLab wiki (hence in GFM markup, for GitLab Flavored Markdown) from a converted content, to be written in a converted-content.gfm file, one may use:

$ pandoc old-content-in-mediawiki.txt --from=mediawiki --to=gfm --standalone -o converted-content.gfm

# Or, for older versions of pandoc not supporting a gfm writer:
$ pandoc old-content-in-mediawiki.txt --from=mediawiki --to=markdown_github --standalone -o converted-content.gfm

Then the content in converted-content.gfm file can be pasted in the target GitLab wiki page.

Another example is the conversion of a GitLab wiki page into a RST document (e.g. then for a PDF generation):

$ pandoc my-gitlab-wiki-extract.gfm --from=gfm --to=rst --standalone -o my-converted-content.rst

# Or, for older versions of pandoc not supporting a gfm reader:
$ pandoc my-gitlab-wiki-extract.gfm --from=markdown_github --to=rst --standalone -o my-converted-content.rst

Finally, if really needing to generate a Word document, an example may be:

$ pandoc my-document.rst --from=rst --to=docx -o my-converted-document.docx

The lists of the input and output formats supported by Pandoc and of their corresponding command-line options is specified here.

These options are also returned by: pandoc --list-input-formats and pandoc --list-output-formats (or, for older versions of pandoc, thanks to pandoc --help).

An input file may not be encoded in UTF-8, which can result in:

pandoc: Cannot decode byte '\xe9': Data.Text.Internal.Encoding.Fusion.streamUtf8: Invalid UTF-8 stream

In this case, the actual encoding shall be determined, for example with:

$ file input.html
input.html: HTML document, ISO-8859 text

Then the encoding may be changed before calling pandoc, for example like:

$ iconv -f ISO-8859-1 -t utf-8 input.html | pandoc --from=html --to=markdown_github --standalone -o output.gfm

Transformation of PDF files

For that, one may use the pdftk tool, possibly with the convert one, which comes from ImageMagick (typically available thanks to a imagemagick package):

• to split all pages of a PDF in as many individual files (named pg_0001.pdf, pg_0002.pdf, etc.): pdftk document.pdf burst
• to extract a range of pages from a PDF: pdftk original-document.pdf cat 276-313 output my-extract.pdf
• to convert a PDF file (typically a single page) into a PNG one (typically in order to edit the PNG with The Gimp afterwards): convert pg_000x.pdf pg_000x.png
• to convert (possibly back) a PNG file to a PDF one: convert pg_000x-modified.png pg_000x-modified.pdf
• to concatenate PDFs: pdftk 1.pdf 2.pdf 3.pdf cat output 123.pdf

PDF documents may contain images/scans (possibly of texts) and/or actual, raw texts. If a PDF is a scan, OCR (Optical character recognition) can be used in order to convert the embedded scans into their actual text. Such a transformation can be done online, and we found PDF24 very useful for that. From such services, usually a PDF (thus including text instead of images) is generated. To obtain a text version thereof respecting its layout (typically to preserve the indentation of a scanned program), one may use: pdftotext -layout my-OCRed-document.pdf in order to enjoy a proper my-OCRed-document.txt.

Image Transformations

One may rely on:

• GIMP (GNU Image Manipulation Program; corresponding, on Arch, to the gimp package), for bitmap (e.g. PNG) graphics
• Inkscape, for vector-based (e.g. SVG) graphics
• or on command-line ImageMagick (on Arch Linux, install the imagemagick package, which provides notably the convert and display executables)

$ convert source.png -channel RGB -negate target.png

$ convert -density 1200 -size 1000 -background none latex-formula.svg target.png

$ magick source.png -bordercolor red -border 10x5 target.png

$ magick source.png -bordercolor black -border 2 target.png

$ magick composite my-formula.png my-overall-plot.png -geometry +100+150 target.png

$ magick composite my-formula.png my-overall-plot.png -gravity SouthEast -geometry +50+200 target.png

$ magick composite my-formula.png my-overall-plot.png -gravity Center target.png

Plot Generation

Refer to our data display section.

UML Diagrams

If SysML can also be of interest, we focus here on UML2 class diagrams (one of the 14 types of diagrams provided by UML2).

Quick UML Cheat Sheet

Multiplicities

A multiplicity is a definition of cardinality (i.e. number of elements) of some collection of elements.

It can be set for attributes, operations, and associations in a class diagram, and for associations in a use case diagram. The multiplicity is an indication of how many objects may participate in the given relationship.

It is defined as an inclusive interval based on non-negative integers, with * denoting an unlimited upper bound (not, for example, n).

Most common multiplicities are:

• no instance or one instance: 0..1
• any number of instances, including zero: * (shorthand for 0..*)
• exactly k instances: k (so, if k=5, 5)
• at least M instances: M..* (2..*)
• at least M instances, but no more than N (hence bounds included): M..N (e.g. 3..5)

For associations, the default multiplicity is automatically is 0..1, while new attributes and operations have a default multiplicity of 1.

Association

An association is a relation between two classes (binary association) or more (N-ary association) that describes structural relationships between their instances.

For example a polygon may be defined from at least 3 vertices that it would reference, whereas a point may take part to any number of polygones (including none):

(see the sources of this diagram)

The multiplicity of an endpoint denotes the number of instances of the corresponding class that may take part to this association. For example, at least 3 points are needed to form a polygon, whereas any number of polygons can include a given point.

In UML the direction of the association is easily ambiguous (here we have to rely on external knwoledge to determine whether a polygon is composed of points, of if a point is composed of polygons). Adding a chevron (like > or <, e.g. "references >" ; ideally this should be a small solid triangle) to the text is not a good solution either, as the layout may place the respective endpoints in any relative position. Adding an arrow to the end of the line segment cannot be done either, as it would denote the navigability of the association instead.

Aggregation

An aggregation is a specific association that denotes that an instance of a class (e.g. Library) is to loosely contain instances of another class (e.g. Book), in the sense that the lifecycle of the contained classes is not strongly dependent on the one of the container (e.g. books will still exist even if the library is dismantled).

Here a library may contain any number of books (possibly none), and a given book belongs to at most one library.

(see the sources of this diagram)

Composition

A composition is a specific association that denotes that an instance of a class (e.g. HumanBeing) is to own instances of another class (e.g. Leg), in the sense that the lifecycle of the contained classes fully depends on the one of the container (here, if a human being dies, his/her legs will not exist anymore either).

Here a human being has exactly 2 legs, and any given leg belongs to exactly one human being (therefore this model does not account for one-legged persons).

(see the sources of this diagram)

Inheritance

An inheritance relationship is a specific association that denotes that a class (e.g. HumanBeing) is a specific case of a more general one (e.g. Animal), and thus that an instance of the first one is also an instance of the second one ("is-a" relationship).

Here a human being is a specific animal.

(see the sources of this diagram)

Tooling

In a design phase, one may prefer lightweight tools like Graphviz, PlantUML or even Dia.

As long as the architecture of a framework is not stabilised, having one's tool determine by itself the layout of the rendering (rather than having to place manually one's graphical components) is surely preferable.

For that we use Graphviz, with our own build conventions.

For example, supposing this diagram example, i.e. a source file named uml_class_diagram_example.graph:

$ make uml_class_diagram_example.png
# or, to force a regeneration and a displaying of the result:
$ make clean uml_class_diagram_example.png VIEW_GRAPH=true

This example results in the following diagram:

(see the sources of this diagram)

Finding Usable Content

Assets can be found thanks to Creative Commons, which references, among others, OpenClipart, otherwise possibly publicdomainvectors.org.

General purposes elements:

• arrow icons for toolbar buttons

The simplest approach is to rely on SVG files, to edit them on Inkscape and possibly to export a selection of them as PNG files of the desired size.

Using Additional Fonts

One may use websites like Dafont in order to select, based on appearance and licence, a given TTF font.

At least on Arch, it is sufficient to copy the corresponding downloaded TTF file (as root) in /usr/share/fonts/ so that tools like The Gimp support it right afterwards (e.g. no need to run fc-cache beforehand).