Using the GNU/Linux Operating System

Organisation:Copyright (C) 2021-2024 Olivier Boudeville
Contact:about (dash) howtos (at) esperide (dot) com
Creation date:Sunday, December 19, 2021
Lastly updated:Sunday, January 7, 2024

Overview

GNU/Linux is our operating system of choice, for many reasons: it is in free software, it is efficient, trustable, reliable and controllable, its mode of operation does not change much over time so any time invested on it is well spent.

Over the years we tried many distributions, including Ubuntu, Debian, Gentoo, Mint.

Our personal all-time favorite is clearly Arch Linux, because it leaves much control to its end user (not attempting to hide details that have to be mastered anyway), it is a "clean" one, driven by a skilled and knowledgeable community, and also because it is a rolling distribution: it updates constantly its packages without needing to regularly upgrade the whole system, which would jeopardise it in the same movement (global system updates rarely complete successfully and tend to be postponed because of the many problems they trigger; we found preferable to deal with issues incrementally on a live system - rather than on one that may fail to reboot properly).

It ends up with a very stable, hassle-free distribution, with cutting-edge packages and higher uptimes (several months without needing to reboot), which is desirable for server-like usages.

Software Update

The setup that we use is to perform automatic nightly updates. For that we use our update-distro.sh script, run through root's crontab as:

$ crontab -l
# Each day at 5:17 AM, update the distro:
17  05   *   *  * /usr/local/bin/update-distro.sh -q

As a result, all packages, libraries, executables, etc. are transparently updated, for the best.

However, for a proper management of modules [1], the kernel-related packages shall be special-cased; otherwise after the first kernel update no more modules can be loaded (they will expect to link to that latest installed kernel version, not to the older one being running).

[1]We tried to rely on DKMS for that, but had still issues with some graphic-related modules, so we preferred managing updates by ourselves.

A first line of defense is to force the loading of the modules known to be of interest directly at boot-time, so that they can be for sure loaded and linked to the right kernel.

This may be done by populating /etc/modules-load.d/ with as many files listing the modules to auto-load, like in:

::::::::::::::
/etc/modules-load.d/for-3g-keys.conf
::::::::::::::
option
usb_wwan

::::::::::::::
/etc/modules-load.d/for-all-usb-keys.conf
::::::::::::::
# To be able to mount all kinds of USB keys:
vfat
uas
dm_crypt

::::::::::::::
/etc/modules-load.d/for-mobile-file-transfer.conf
::::::::::::::
# To be able to transfer files between this hosts and mobile phones by MTP:
nls_utf8
isofs
sr_mod
cdrom
# Maybe also: agpgart ahci wdat_wdt wmi_bmof xts

::::::::::::::
/etc/modules-load.d/for-tty-serial-on-usb.conf
::::::::::::::
# To be able to connect tty-like interfaces through a USB port:
ftdi_sio
usbserial

::::::::::::::
/etc/modules-load.d/for-usb-tethering.conf
::::::::::::::
# To enable an Internet access thanks to a smartphone via USB:
usbnet
# Implies:
#rndis_host
#cdc_ether

::::::::::::::
/etc/modules-load.d/for-vlan-support.conf
::::::::::::::
# To be able to manage VLANs:
8021q

This is not sufficient though (e.g. one cannot anticipate all modules needed after a while); disabling the automatic updates of kernels is also key to reduce issues; this can be done by specifying in /etc/pacman.conf:

IgnorePkg = linux linux-headers

LTS (Long Term Support) kernels are intentionally not listed here, as we prefer having them regularly updated in order to minimise the risk that the base and LTS kernels belong to too close versions (as then a problem in terms of hardware support is more likely to arise at the same time with both).

At least users of NVidia graphic cards may also list there their drivers, as apparently an hardware acceleration supported at boot may be lost after some time, presumably because of an update of its drivers (knowing that the update of the kernel itself was already disabled in that case) - so, if appropriate, better be safe than sorry:

IgnorePkg = linux linux-headers nvidia nvidia-utils

See also our section about operating system support for 3D.

Updating all packages but kernel-related ones is fine, but of course the latters shall still be also updated appropriately. The best moment for that is just prior to rebooting (knowing that your Linux box never crashes, isn't it?), so for that we use (as root) our shutdown-local-host.sh script, like in:

$ shutdown-local-host.sh --reboot

The kernel packages, and possibly driver-related ones, will then only be properly updated before the host is rebooted.

Package Management

Configuration

One may enable the multilib repository, which is useful to run 32-bit software on 64-bit hardware. This is useful for example if needing wine, knowing that its build from the AUR may fail.

To enable multilib, uncomment in /etc/pacman.conf:

[multilib]
Include = /etc/pacman.d/mirrorlist

then upgrade your system with pacman -Syu.

Interesting Packages

They might be lesser known:

  • cpulimit: the way of limiting CPU usage of a given process, for example to avoid overheat (nice just defines respective process priorities)
  • inotify-tools: to be able to monitor filesystem events (e.g. with inotifywait) from scripts
  • jq: for command-line JSON processing (e.g. jq . myfile.json to display it properly on a terminal)
  • mathjax: to generate LaTeX-like images for the web
  • most: a replacement for more
  • pdftk: to transform PDF files
  • pkgfile: to retrieve file information about packages

Process-related Post-Mortem Investigations

Sometimes a UNIX process crashes and, typically if one developed it, one wants to investigate the issue, based on a core dump produced by the operating system.

This Arch Linux article will give all relevant details.

In short, coredumpctl list will list all known core dumps from oldest to most recent, such as in:

$ coredumpctl list
TIME                        PID   UID  GID SIG     COREFILE EXE                SIZE
[...]
Tue 2021-12-21 20:53:02 CET 73873 1007 988 SIGSEGV present  [...]/bin/beam.smp 14.6M

The last core dump produced may be studied directly, thanks to coredumpctl debug, relying on gdb to fetch much lower-level information:

$ coredumpctl debug
          PID: 73873 (beam.smp)
          UID: 1007 (xxx)
          GID: 988 (users)
       Signal: 11 (SEGV)
    Timestamp: Tue 2021-12-21 20:53:01 CET (38min ago)
 Command Line: /home/xxx/Software/Erlang/Erlang-24.2/lib/erlang/erts-12.2/bin/beam.smp -W w -K true -A 128 [...]
   Executable: /home/xxx/Software/Erlang/Erlang-24.2/lib/erlang/erts-12.2/bin/beam.smp
Control Group: /user.slice/user-1007.slice/session-2.scope
         Unit: session-2.scope
        Slice: user-1007.slice
      Session: 2
    Owner UID: 1007 (xxx)
      Boot ID: f8abe9473f7e4fea8ba24944e35ce7d9
   Machine ID: c9413a71e7b4498f831e2df7a08e5f33
     Hostname: xxx
      Storage: /var/lib/systemd/coredump/core.beam\x2esmp.1007.f8abe9473f7e4fea8ba24944e35ce7d9.73873.1640116381000000.zst (present)
    Disk Size: 14.6M
      Message: Process 73873 (beam.smp) of user 1007 dumped core.

               Found module /home/xxx/Software/Erlang/Erlang-24.2/lib/erlang/erts-12.2/bin/beam.smp with build-id: 8cfbf76728dd7399444638f1ba124471181840e7
               Found module /home/xxx/Software/Erlang/Erlang-24.2/lib/erlang/lib/wx-2.1.1/priv/erl_gl.so with build-id: 0b96532e586839d3da9afd6e5f23aa76346a0e45
[...]
   Stack trace of thread 74039:
   #0  0x00007f6e5461a74b __memmove_avx_unaligned_erms (libc.so.6 + 0x16374b)
   #1  0x00007f6d8a204428 n/a (iris_dri.so + 0xd12428)
   #2  0x00007f6d89733207 n/a (iris_dri.so + 0x241207)
   #3  0x00007f6d89733c97 n/a (iris_dri.so + 0x241c97)
   #4  0x00007f6d898d8b0d n/a (iris_dri.so + 0x3e6b0d)
   #5  0x00007f6d898d8bf2 n/a (iris_dri.so + 0x3e6bf2)
   #6  0x00007f6d8b2f241c n/a (/home/xxx/Software/Erlang/Erlang-24.2/lib/erlang/lib/wx-2.1.1/priv/erl_gl.so + 0x5b41c)
[New LWP 74039]
[New LWP 73873]
[...]
Core was generated by `/home/xxx/Software/Erlang/Erlang-24.2/lib/erlang/erts-12.2/bin/beam.smp -'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f6e5461a74b in __memmove_avx_unaligned_erms () from /usr/lib/libc.so.6
[Current thread is 1 (Thread 0x7f6d900aa640 (LWP 74039))]

Then:

(gdb) bt
[...]
#0  0x00007f6e5461a74b in __memmove_avx_unaligned_erms () at /usr/lib/libc.so.6
#1  0x00007f6d8a204428 in  () at /usr/lib/dri/iris_dri.so
#2  0x00007f6d89733207 in  () at /usr/lib/dri/iris_dri.so
#3  0x00007f6d89733c97 in  () at /usr/lib/dri/iris_dri.so
#4  0x00007f6d898d8b0d in  () at /usr/lib/dri/iris_dri.so
#5  0x00007f6d898d8bf2 in  () at /usr/lib/dri/iris_dri.so
#6  0x00007f6d8b2f241c in ecb_glTexImage2D(ErlNifEnv*, ErlNifPid*, ERL_NIF_TERM*) (env=0x5642f62bdea0, self=0x5642f665e148, argv=0x5642f665e168) at gen/gl_nif.cpp:2844
[...]
#29 0x00007f6d92967188 in wxe_main_loop(void*) (_unused=<optimized out>) at wxe_main.cpp:138

(this example was an Erlang wx/OpenGL-oriented crash)

From there, standard gdb-fu shall be sufficient to give much insight. Once done, use q to quit.

Preparing Adequate USB Keys

Objective

The goal here is, once having purchased a basic yet robust (e.g. with a proper lid) USB key (preferably from a good brand, bought from a reliable seller in order to avoid counterfeits), to prepare it efficiently for everyday use.

Often capacity matters a bit, speed not so much, and the best value for money is met for the mid-range keys of the time.

Conventions, conventions

The name of our key (KEY_NAME) will be Kn, where n is a key counter (e.g. KEY_NAME=K7).

We will create here two (GPT) partitions on such a key:

  • a large, encrypted ("private"), Linux-friendly partition, as the main storage space of interest; based on EXT4 with dm-crypt and LUKS2, ciphered based on a 256-bit AES algorithm
  • a smaller, plain/unencrypted ("public"), Windows-friendly partition, for convenience (when you have files to transfer yet you do not remember the passphrase of the previous partition)

The name of a partition (P_NAME) is a disk label, often limited to 11 characters. We choose it as prefixed with the name of the key, followed by either pub (for public, unencrypted) or encr (for encrypted), then with the partition number. For example, K3-pub-1 or K11-encr-4.

The name of a filesystem (FS_NAME) of a partition is constrained as well, we specify it as: KEY_NAME-(pub|encr)-PHYSICAL_SIZE[-PARTITION_NUMBER], the partition number being useful to distinguish between any otherwise identical partitions of a given key. For instance K7-pub-2GB and K7-encr-14GB-4.

Any statically-defined mount point shall bear the same name as the associated filesystem: MOUNTPOINT=/mnt/${FS_NAME}. For example MOUNTPOINT="/mnt/K7-pub-2GB".

Targeting the Right Device

Each key shall be registered in one's referential, and one shall be careful not to format one's local hard disk instead of a key.

To identify for sure such a key, run lsblk --fs just before plugging it in, and just after, so that the difference can be easily spotted.

The device name and size should be checked.

For an increased security, environment variables will be associated here to such a key, for example with: export KEY_DEV=/dev/sdz.

Its characteristics can be recorded in one's referential:

$ fdisk -l ${KEY_DEV}
$ parted ${KEY_DEV} print

Creating the Partitions

As root:

$ export KEY_NAME="K7"
$ fdisk -l
$ export KEY_DEV=/dev/sdz
$ fdisk ${KEY_DEV}

Then:

  • print the partition table : (p)
  • delete if necessary any previously existing partition(s): (d) for each of them
  • create a GPT disklabel: (g) rather than a MBR one (i.e. not "dos" (o))
  • create each partition (first the encrypted one(s), to favor their use):
    • creation thanks to (n)
    • size (e.g."+55G")
    • type (e.g. "Linux filesystem", i.e. 20, or "Microsoft basic data", i.e. 11)
    • partition name (with GPT: switch to expert mode (x), then (n), then a name like K7-encr-part); back to the main menu (r)
  • check: (p), (F), (v)
  • write: (w)

So that the kernel updates its partition table, it may be necessary to unplug and plug again the key.

All information (obtained in expert mode) regarding the new partitions may be stored in one's referential.

Erasing a Target Partition

If feeling paranoid about the previous content and having quite a lot of time ahead, a low-level erasure of a partition can be performed.

For example:

$ export PUB_DEV_NUM=1
$ export PUB_DEV="${KEY_DEV}${PUB_DEV_NUM}"; echo "PUB_DEV: ${PUB_DEV}"

$ fdisk -l ${PUB_DEV}
$ parted ${PUB_DEV} print

# Remove the echo after serious verification:
$ echo dd bs=256K if=/dev/urandom of=${PUB_DEV}
# (wait for *very* long)
dd: error writing '/dev/sdz1': No space left on device
# (still blocks for very long, despite any CTRL-C; just wait)
16385+0 records in
16384+0 records out
2147483648 bytes (2.1 GB, 2.0 GiB) copied, 241.353 s, 8.9 MB/s

Creating Plain, Unencrypted Partitions

Some devices (e.g. printers) may be confused should there be multiple partitions, or some with non-FAT or encrypted filesystems. This may be a reason to create a single, overall FAT partition.

Formatting a Plain Partition

As FAT32 :

$ export PART_NUM=2
$ export PART_SIZE="2GB"

$ export FS_NAME="${KEY_NAME}-pub-${PART_NUM}-${PART_SIZE}"
# Or if a single partition is of that type:
$ export FS_NAME="${KEY_NAME}-pub-${PART_SIZE}"

$ echo ${FS_NAME}
$ export PUB_DEV="${KEY_DEV}${PART_NUM}"
# Remove the echo after serious verification:
$ echo mkdosfs -F 32 -n ${FS_NAME} ${PUB_DEV}
mkfs.fat 4.2 (2021-01-31)
mkfs.fat: Warning: lowercase labels might not work properly on some systems

Finalising and Testing a Plain Partition

We take this opportunity to, after the previous section, create its own mount point (typically to be referenced in /etc/fstab):

$ export MOUNT_POINT=/mnt/${FS_NAME}; mkdir ${MOUNT_POINT} && mount ${PUB_DEV} ${MOUNT_POINT}

# Should be not needed:
$ chown -R YOUR_USER:YOUR_GROUP ${MOUNT_POINT}

$ touch ${MOUNT_POINT}/WELCOME_TO_${KEY_NAME}_PUBLIC_${PART_NUM}_${PART_SIZE}_SPACE && ls -l ${MOUNT_POINT}

# Or if a single partition is of that type:
$ touch ${MOUNT_POINT}/WELCOME_TO_${KEY_NAME}_PUBLIC_${PART_SIZE}_SPACE && ls -l ${MOUNT_POINT}

If really wanting to register extraneous information:

$ mount | grep ${MOUNT_POINT}
/dev/sdb2 on /mnt/K5-pub-2GB type vfat (rw,relatime,fmask=0002,dmask=0002,allow_utime=0020,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)

$ df ${MOUNT_POINT}
Filesystem     1K-blocks  Used Available Use% Mounted on
/dev/sdb2        2774720     4   2774716   1% /mnt/K5-pub-2GB

$ blkid ${PUB_DEV}
/dev/sdb2: LABEL_FATBOOT="K5-pub-2GB" LABEL="K5-pub-2GB" UUID="8C2C-1849" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="K5-pub-part" PARTUUID="955a970a-9920-c448-ada1-3f523d6fded3"

$ lsblk --fs ${PUB_DEV}
NAME FSTYPE FSVER LABEL      UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sdb2 vfat   FAT32 K5-pub-2GB 8C2C-1849                               2.6G     0% /mnt/K5-pub-2GB

$ parted ${PUB_DEV} print
Model: Unknown (unknown)
Disk /dev/sdb2: 2847MB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
1      0.00B  2847MB  2847MB  fat32

$ umount ${MOUNT_POINT}

Creating Encrypted Partitions

After Partitioning

Such storage space is of course to be partitioned first like the plain ones (see Creating the Partitions), and can similarly be erased at a low level first (see Erasing a Target Partition).

For example we may end up with:

$ export ENCR_DEV_NUM=1

$ export ENCR_DEV="${KEY_DEV}${ENCR_DEV_NUM}"; echo "ENCR_DEV: ${ENCR_DEV}"
ENCR_DEV=/dev/sdb1

$ fdisk -l ${ENCR_DEV}
Disk /dev/sdb1: 12 GiB, 12884901888 bytes, 25165824 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

$ parted ${ENCR_DEV} print

$ export PART_SIZE="12GB"

Creating an encrypted LUKS container

We used to favor LUKS1 over LUKS2 for a better compatibility with ancient Linuces, yet it is no longer relevant, LUKS is widespread now.

In order to create such a container, there are many options to choose from; here are the ones that we prefer:

# Remove the echo after serious verification (enter YES then the main, daily
# passphrase to unlock that container):
#
$ echo cryptsetup --hash sha512 -c aes-xts-plain --key-size 512 luksFormat ${ENCR_DEV}
WARNING!
========
This will overwrite data on /dev/sdb1 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for /dev/sdb1:
Verify passphrase:

We strongly advise to add a second, "rescue" passphrase (longer, more difficult than the previous daily one, and potentially common to at least some keys), as a last chance:

# Remove the echo after serious verification (enter the previous passphrase, then
# the rescue one):
#
$ echo cryptsetup luksAddKey ${ENCR_DEV}
Enter any existing passphrase:
Enter new passphrase for key slot:
Verify passphrase:

Let's introduce then more variables:

$ export ENCR_DESIGNATOR="${KEY_NAME}-encr-${ENCR_DEV_NUM}-${PART_SIZE}"
# - OR -
$ export ENCR_DESIGNATOR="${KEY_NAME}-encr-${PART_SIZE}"

$ echo "ENCR_DESIGNATOR=${ENCR_DESIGNATOR}"
ENCR_DESIGNATOR=K5-encr-12GB

Now we unlock the LUKS container so that we can create an EXT4 partition in it.

A -fs suffix would not be very relevant (this is the name that will be used by to automounter):

$ export ENCR_FS_NAME="${ENCR_DESIGNATOR}"

# Remove the echo after serious verification:
$ echo cryptsetup config ${ENCR_DEV} --label ${ENCR_FS_NAME}
cryptsetup config /dev/sdb1 --label K5-encr-12GB
$ cryptsetup config ${ENCR_DEV} --label ${ENCR_FS_NAME}

$ export DM_NAME="my-${ENCR_DESIGNATOR}"

# Enter either of the two aforementioned passphrases:
$ cryptsetup luksOpen ${ENCR_DEV} ${DM_NAME}
Enter passphrase for /dev/sdb1:

Creating the In-Container EXT4 Filesystem

Deactivating journaling (with the -O ^has_journal option) could increase a bit the lifespan of the key, but would weaken its filesystem, whereas USB keys may be (unfortunately) unplugged while being still mounted; so we prefer keeping the journaling:

# Remove the echo after serious verification:
$ echo mkfs.ext4 /dev/mapper/${DM_NAME} -L ${ENCR_FS_NAME} -E discard
mkfs.ext4 /dev/mapper/my-K5-encr-12GB -L K5-encr-12GB -E discard

$ mkfs.ext4 /dev/mapper/${DM_NAME} -L ${ENCR_FS_NAME} -E discard
mke2fs 1.47.0 (5-Feb-2023)
Creating filesystem with 3141632 4k blocks and 786432 inodes
Filesystem UUID: 92c44f73-8614-44f9-a03c-cfd718aecb8e
Superblock backups stored on blocks:
   32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

# Should be needed (otherwise can be mounted yet not written by one's normal user):
$ chown -R YOUR_USER:YOUR_GROUP ${MOUNT_POINT}

So no more need here for tune2fs -O ^has_journal /dev/mapper/${DM_NAME}. One may still check the resulting settings with tune2fs -l /dev/mapper/${DM_NAME}.

Declaring keys in /etc/fstab and in /etc/crypttab is not necessary; moreover some care would be needed in order to ensure that the automounter would not freeze at the next reboot, expecting such a key to be available.

Finalising and Testing an Encrypted Partition

Here also one could take this opportunity to, after the previous section, create a dedicated mount point (see just above for a warning about referencing it through /etc/fstab and/or /etc/crypttab).

We consider that the previous LUKS container is still opened (otherwise: cryptsetup luksOpen ${ENCR_DEV} ${DM_NAME}).

Then:

$ export MOUNT_POINT=/mnt/${ENCR_DESIGNATOR}; echo "MOUNT_POINT = $MOUNT_POINT"
MOUNT_POINT = /mnt/K5-encr-12GB

$ mkdir ${MOUNT_POINT}

# For example DM_NAME=/dev/mapper/my-K5-encr-12GB:
$ mount /dev/mapper/${DM_NAME} ${MOUNT_POINT}

$ touch ${MOUNT_POINT}/WELCOME_TO_${KEY_NAME}_ENCRYPTED_${PART_NUM}_${PART_SIZE}_SPACE && ls -l ${MOUNT_POINT}
 or
$ touch ${MOUNT_POINT}/WELCOME_TO_${KEY_NAME}_ENCRYPTED_${PART_SIZE}_SPACE && ls -l ${MOUNT_POINT}
total 16
drwx------ 2 root root 16384 Apr 22 12:41 lost+found
-rw-rw-r-- 1 root root     0 Apr 22 16:14 WELCOME_TO_K5_ENCRYPTED_12GB_SPACE

# Record some information if wanted:
$ mount | grep ${MOUNT_POINT}
/dev/mapper/my-K5-encr-12GB on /mnt/K5-encr-12GB type ext4 (rw,relatime)

$ blkid ${ENCR_DEV}
/dev/sdb1: UUID="f00def37-529a-4400-aa8c-c7a36589c152" LABEL="K5-encr-12GB" TYPE="crypto_LUKS" PARTLABEL="K5-encr-part" PARTUUID="b4b72946-f1d1-1c45-976d-3bd389d23752"

$ lsblk --fs ${ENCR_DEV}
NAME              FSTYPE      FSVER LABEL        UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
sdb1              crypto_LUKS 2     K5-encr-12GB f00def37-529a-4400-aa8c-c7a36589c152
└─my-K5-encr-12GB ext4        1.0   K5-encr-12GB 92c45f73-8614-44f9-a03c-cfd718aecb8e   11.1G     0% /mnt/K5-encr-12GB

$ parted ${ENCR_DEV} print
Error: /dev/sdb1: unrecognised disk label
Model: Unknown (unknown)
Disk /dev/sdb1: 12.9GB
Sector size (logical/physical): 512B/512B
Partition Table: unknown
Disk Flags:

$ df ${MOUNT_POINT}
Filesystem                  1K-blocks  Used Available Use% Mounted on
/dev/mapper/my-K5-encr-12GB  12262536  2072  11615756   1% /mnt/K5-encr-12GB

Finally:

$ umount ${MOUNT_POINT}
$ cryptsetup close ${DM_NAME}

That's it! You should end up with a basic, inexpensive USB key - yet offering at least two satisfying partitions, one public (unencrypted), one adequately encrypted.

GRUB Ate My Distro Again: Fixing the Bootloader

So you tried recklessly to add some kernel parameters - presumably to prevent your laptop from freezing regularly for no apparent reason - and GRUB managed once again to replace a bootloader that used to work flawlessly with one that just cycles to the BIOS?

Here are a few guidelines (on Arch Linux, with UEFI and GPT - rather than any MBR) that could be useful, knowing that considerably more complete information can be found in this page.

Prerequisites

At least the following packages will be needed (anyway they are likely to be already available, either from a real install or from a rescue medium): grub, efibootmgr and os-prober.

You will also need a bootable (most probably removable) install/rescue medium, typically an USB stick whose content can be erased. Either you have access to another computer (Linux or not), or you have a multiboot and, after a smart journey in the BIOS menus, you will be able to nevertheless launch a Windows instance (generally GRUB just concentrates on wreaking havoc on unfortunate Linuces, rather than doing the same to Windows installations).

If having only a Windows instance at hand, one may just follow these guidelines, which mostly boil down to downloading and installing the (free software) Win32 Disk Imager tool, grabbing also a relevant ISO (e.g. archlinux-2023.04.01-x86_64.iso at the time of this writing) and writing in on said USB stick.

Running a Live Rescue Arch

Once having a proper rescue medium, ensure that it is inserted, and reboot your GRUB-affected host; possibly the BIOS configuration will have to be updated so that the host can boot on that medium.

Once done, you should end up with a root shell - albeit running from a live, rescue Arch Linux, whereas the one that you want to fix is of course the one of your host.

Preparing the chroot

First enforce any essential setting, such as loadkeys fr if you have a French keyboard. Then locate the /boot partition of the host. It may either be directly in the partition corresponding to the root / tree or, as per our conventions, in a separate one (so that all other partitions can be appropriately encrypted). For that, fdisk -l (and possibly lsblk as well) shall help finding out such a partition; for instance:

$ fdisk -l
Device              Start        End    Sectors   Size Type
/dev/nvme0n1p1       2048    1226751    1224704   598M EFI System
/dev/nvme0n1p2    1226752    1259519      32768    16M Microsoft reserved
/dev/nvme0n1p3    1259520  313274682  312015163 148.8G Microsoft basic data
/dev/nvme0n1p4  313276416  314570751    1294336   632M Windows recovery environment
/dev/nvme0n1p5  314572800  838860799  524288000   250G Microsoft basic data
/dev/nvme0n1p6  838860800  855638015   16777216     8G Linux swap
/dev/nvme0n1p7  855638016 1056964607  201326592    96G Linux filesystem
/dev/nvme0n1p8 1056964608 4000797326 2943832719   1.4T Linux filesystem

Here, one can guess that nvme0n1p1 is /boot (because of the EFI type, and the size), that nvme0n1p7 is / and nvme0n1p8 is /home.

Managing the chroot

A relevant chroot shall be performed, once all appropriate trees have been mounted; we do not need /home (on the contrary, it is safer if it remains locked and not even mounted), but we need the right /boot (the one on disk and that contains the previous bootloader elements) to be mounted, instead of the empty one that would be inherited by a mere mounting of the / root. So:

$ cd /mnt

# Intermediary directory usually useless but clearer:
$ mkdir my_chroot_tree

# Mount the whole system tree, "/":
$ mount /dev/nvme0n1p7 my_chroot_tree

# Take care of our separate "/boot" as well:
$ ls my_chroot_tree/boot
# (empty)

$ mount /dev/nvme0n1p1 my_chroot_tree/boot
$ ls my_chroot_tree/boot
. EFI initramfs-linux-fallback.img initramfs-linux-lts-fallback.img
intel-ucode.img  vmlinuz-linux .. grub initramfs-linux.img
initramfs-linux-lts.img 'System Volume Information' vmlinuz-linux-lts

# Switch to the actual host Arch system:
$ arch-chroot my_chroot_tree

(Re)Installing GRUB

As we understand it, installing GRUB (i.e. copying its relevant file elements in a proper location in /boot) and configuring it can be done in any order; we prefer anyway installing it first, with:

# Note that the 'EFI' directory is not directly specified here, only
# the /boot mount point:
$ grub-install --target=x86_64-efi --efi-directory=/boot --bootloader-id=MY_GRUB

Configuring GRUB

Note

Configuring GRUB means generating a settings file whose syntax may be more recent than the one supported by any already-installed GRUB - which would thus be bound to fail at boot. So any configuration update of GRUB shall be done together with the update of its installation.

Now is the right time to update one's /etc/default/grub, notably to ensure that GRUB_DISABLE_OS_PROBER=false is indeed defined, and is not commented out.

If os-prober is especially useful to auto-detect any Windows installation, we could see that it was not working from a chroot (and thus it has to be done later, directly from the final Arch host).

The last step is to generate the corresponding GRUB configuration file:

$ grub-mkconfig -o /boot/grub/grub.cfg

Finally

Rebooting and selecting a right entry menu (note that LTS kernels will be sorted first and thus be selected by default) should result in a restored boot proceeding as normal.

Moreover, from now, os-prober should be able to pick up any other system (especially Windows installations) and declare them appropriately: this should be just a matter of running grub-mkconfig -o /boot/grub/grub.cfg again (and hope for the best).

Quick Topics

Installing Wine

Install it, once enabling multilib has been done, with: pacman -S wine.

When run, this may lead wine-mono to be auto-installed.

The pseudo-Windows filesystem is then located mostly in ~/.wine/drive_c.

Overcoming Invalid PGP Signatures in Pacman Packages

This happens regularly, especially when updating a symptom being: File /var/cache/pacman/pkg/xxx.pkg.tar.zst is corrupted (invalid or corrupted package (PGP signature)).

Solution: pacman -S archlinux-keyring is at least often enough. Otherwise pacman-key --populate archlinux, or changing mirror might help.

Protecting Files and Directories

Beyond the basic chown / chmod commands (e.g. chmod 400 to have it read-only for its owner only), using chattr as well shall be considered for the most sensitive filesystem elements (like the ones related to ~/.ssh).

Notably, so that a file becomes immutable (even for root), one may use chattr +i MY_FILE.

Use chattr -i MY_FILE to clear that immutability, and lsattr to check it.

Converting Data Formats

  • to convert multimedia files, ffmpeg is very convenient; for example to convert an AIF file into a WAV one: ffmpeg -i sound.aif sound.wav
  • to read *.pck resource file, the Dragon UnPACKer (Windows) open source tool can be used

See also Ceylan-Hull's multimedia-related section.

Adding a Locale

On some hosts, issues in terms of a lacking locales may be reported, like in the following:

bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

This can be fixed by uncommenting the corresponding locale (en_US.UTF-8 here) in /etc/locale.gen, and then regenerating the system locales by running (as root) locale-gen:

$ locale-gen
Generating locales (this might take a while)...
 en_US.UTF-8... done
 fr_FR.UTF-8... done
 fr_FR.ISO-8859-15@euro... done
Generation complete.

Performing Searches and Replacements

Searches: to be made with grep

To search for a set of patterns, separate them by an (escaped) pipe; for example: grep 'hello\|goodbye\|farewell' Foobar.java.

Replacements: to be made with sed

  • multiple operations (typically replacements) can be made in a row by combining them with ";"; for example to uppercase only a set of letters: echo "aabbccddee" | sed 's|b|B|g;s|d|D|g' yields aaBBccDDee
  • to perform in-place replacements in a file: sed -i 's|MY_TAG|142|g' my_file.txt (if a suffix is specified, backups may be made)

Common to grep and sed

With grep and sed, [[:space:]] and \s are the same, they will both match any whitespace character spaces, tabs, etc.

Applying a XFCE4 configuration to a different user

The objective is to duplicate the configuration of the user_a desktop to the user_b one, for example if user_b is used to test an untrusted application.

A key point is to ensure that at least user_b has no ongoing graphical session (otherwise any new configuration may be overwritten with an older one at logout).

Then, typically as root:

$ rm -rf /home/user_b/.config/{xfce4,Thunar} /home/user_b/.local/share/xfce4
$ cp -rf /home/user_a/.config/{xfce4,Thunar} /home/user_b/.config
$ cp -rf /home/user_a/.local/share/xfce4 /home/user_b/.local/share
$ chmod -R /home/user_b/{.config,.local} user_b:users

Then a key point is to fully reset XFCE4. Unlogging and/or killall -HUP xfdesktop may not be sufficient. As a last resort, rebooting shall work.

Selecting a per-user wallpaper may be of help.

Using Emacs

Configuration

Our init.el, which may be stored in ~/.emacs.d/, currently relies on straight.el.

Installation

Quite surprisingly, Emacs still changes a lot (notably in terms of function names), and (elisp) scripts that work for an Emacs version may not work for the next one. So at least controlling one's version may be of use.

Of course using a package manager for that is the best option.

Otherwise, to perform a manual installation of Emacs on one's user account, it must be first downloaded; one may thus fetch for example emacs-28.2.tar.xz.

Prerequisites may be needed; running - unfortunately as root - apt-get build-dep emacs28 or alike may be of use, or at least having a package like libgtk-3-dev installed.

Then:

$ mkdir -p ~/Software/Emacs && cd ~/Software/Emacs
$ mv ~/Downloads/emacs-28.2.tar.xz .
$ tar xvJf emacs-28.2.tar.xz
$ cd emacs-28.2
# If not having these dependencies:
$ ./configure --with-xpm=ifavailable --with-gif=ifavailable \
--with-tiff=ifavailable --with-gnutls=ifavailable --prefix=${HOME}/Software/Emacs/emacs-28.2-install
$ make install
$ cd .. && ln -s emacs-28.2-install emacs-current-install

Then one's shell environment shall be updated once for all with:

$ export PATH="${HOME}/Software/Emacs/emacs-current-install/bin:${PATH}"

Hints

Here C corresponds to the "Ctrl" key, M the "Meta" one (i.e. "Alt", generally) and - is just a separator between a modifier (like Ctrl, Meta, etc.) and an actual key to press while the modifier is still held down.

For example C-x C-- means: press and hold the "Ctrl" key, and hit the "x" key, then release all, then press and hold the "Ctrl" key, and hit the "-" key, and release all.

Frequent actions needed:

  • to tune the font size:
    • temporarily: use C-x C-+ to increase, C-x C-- to decrease (or press Shift then click the first mouse button to select a relevant option; or, even better, use C-mousewheel)
    • permanently: one may add in one's Emacs configuration file for example: (set-face-attribute 'default nil :height 100)
  • to insert special characters (e.g. tab or newline) as raw characters in commands (e.g. in I-search or replace-string): use for example M-x quoted-insert <tab>, which is often bound (yet not in our conventions) to C-q; note that pasting a tab with the mouse in the minibuffer works as well
  • to insert in the current buffer the output of a shell command: C-u then M-!: enter that shell command, whose output will be pasted at the current cursor position
  • to prefix all lines of the selected region: C-x then r then t, then type the prefix then type Enter
  • to sort alphabetically the selected region: M-x sort-lines
  • to go to the matching delimiter (parenthesis, bracket, end of word, etc.): to go forward, use C-M-f, and to go backward use C-M-b
  • to re-select a region (e.g. to perform multiple substitutions in a row on the same region): C-x C-x
  • to abort the current entry in the minibuffer: C-g
  • to perform replacements based on regular expressions: M-x replace-regexp RET regexp RET newstring RET, with this REGEXP syntax (more information); for example, to remove, in each line, all characters from the first comma: M-x replace-regexp RET ,.* RET , RET

Other useful commands to trigger with M-x:

  • replace-string
  • query-replace
  • kill-rectangle

See also our Performing a Merge with Emacs section.

Troubleshooting

Sometimes problems arise due to older packages, this may be solved by removing the ~/.emacs.d/straight directory, relaunching Emacs and waiting for all related clones to be downloaded again.

Using E-mail Clients

Over time we were preferring Evolution to Thunderbird, notably regarding its far better OpenPGP support. Yet, at least for some IMAP servers (e.g. the ones of Mailo), errors (like: Error copying messages: Error) constantly happening (despite many attempts of configuration changes) and being reported ruined its use for us [2], so we switched back to Thunderbird.

[2]Moreover the fixed character-wrapping of Evolution is rather surprising and unfortunate.
  • for Thunderbird:
    • the message filters can be copied as a whole from a computer to another, knowing that with IMAP servers the emails and their folder structure should be readily available from all clients; transferring filters is as simple as copying, whereas Thunderbird instances are closed, the relevant msgFilterRules.dat file (located typically in a generated directly like ~/.thunderbird/6r77gatw.default-release/ImapMail/YOUR_IMAP_SERVER/msgFilterRules.dat)
    • we recommend using the DKIM Verifier add-on, that we found much useful to better detect spoofing attempts; it can be configured to display colored DKIM statuses for the sender of each email, which is very convenient
  • for Evolution: on Arch Linux, consider installing the evolution-bogofilter and/or evolution-spamassassin packages, otherwise spams (junk mails) do not seem to be managed properly (or at all)

See also Ceylan-Hull's email.sh and archive-emails.sh scripts.

Recording a Screencast

We rely on the impressive, feature-rich, free software SimpleScreenRecorder (pacman -Sy simplescreenrecorder).

This tool is run as simplescreenrecorder and offers many options and encoding choices; it is able to record OpenGL applications [3] and also the audio.

[3]In our case it was sufficient to select Record a fixed rectangle and Select window (which happened to be the one of an OpenGL application); it seemed a better option than selecting Record OpenGL, as afterwards no recording could be done due to Could not get the size of the OpenGL application. (maybe this could be alleviated by entering adequate settings).

Note that, as notified by the tool, the MP4 container type will produce unreadable files if the recording is interrupted (then the MKV container shall be preferred).

See also this comparison of screencasting software.

Other User Settings

See our preferences for Startpage; it can be set as homepage.

Shortcuts

Listing here the main keyboard shortcuts of interest for a few base tools.

For less

Mix of vi / more conventions, in order to:

  • forward-search for the N-th line containing the pattern (N defaults to 1): /pattern; use ?/pattern to backward-search
  • repeat previous search: n:
  • invoke an editor on the current file being viewed: v
  • go to next file: :n; for the previous one: :p
  • help: h
  • quit: q

For mplayer

  • pause/unpause the current playback: <space>
  • decrease the volume: /; to increase it: *
  • go backward/forward in the current playback: left and right arrow keys
  • jump to next playback: <Enter> or <Escape>
  • stop all playbacks: <CTRL-C>
  • display durations: o to toggle OSD (On-Screen Display), i.e. elapsed, elapsed / total, etc.
  • take a screenshot: s (notably if using our v script, as the -vf screenshot command-line option must have been specified); then a shot0001.png file will be silently created in the directory whence mplayer was launched, next screenshot will be shot0002.png, etc.

See Also

One may refer to our other mini-HOWTO regarding:

The Ceylan-Hull section system-related section might also be of interest.