---
title: "FreeBSD Email Server - Part 6: System Updates"
layout: post
tags: Tutorial FreeBSD Email
social:
  email: mailto:~tyil/public-inbox@lists.sr.ht&subject=FreeBSD Email Server
  mastodon: https://soc.fglt.nl/notice/9xF5VVpcK1NJR0kgOO
description: >
  Updates are important. This article aims to help you to learn from my
  mistakes, so your updates will go smooth.
---

Four years have past, and my FreeBSD email server has keps on running without
any problems. However, some people on IRC have recently been nagging me to
support TLSv1.3 on my mailserver. Since the installation was done 4 years ago,
it didn't do 1.3 yet, just 1.2. I set out to do a relatively simple system
update, which didn't go as smooth as I had hoped. This tutorial post should
help you avoid the mistakes I made, so your updates *will* go smooth.

{% admonition_md info %}
The rest of this tutorial assumes you're running as the `root` user.
{% endadmonition_md %}

## Preparations

Before we do anything wild, let's do the obvious first step: backups. Since
this is a FreeBSD server, it uses glorious
[ZFS](https://en.wikipedia.org/wiki/ZFS) as the filesystem, which allows us to
make use of
[snapshots](https://docs.oracle.com/cd/E23824_01/html/821-1448/gbciq.html).
Which subvolumes to make snapshots off depends on your particular setup. In my
case, my actual email data is stored on `zroot/srv`, and all the services and
their configurations are in `zroot/usr/local`. My database's data is stored on
`zroot/postgres/data96`. Additionally, I want to make a snapshot of
`zroot/usr/ports`.

{% highlight sh %}
zfs snapshot -r zroot/srv@`date +%Y%m%d%H%M%S`-11.0-final
zfs snapshot -r zroot/usr/local@`date +%Y%m%d%H%M%S`-11.0-final
zfs snapshot -r zroot/postgres@`date +%Y%m%d%H%M%S`-11.0-final
zfs snapshot -r zroot/usr/ports@`date +%Y%m%d%H%M%S`-11.0-final
{% endhighlight %}

This will make a snapshot of each of these locations, for easy restoration in
case any problems arise. You can list all your snapshots with `zfs list -t
snapshot`.

Your server is most likely hosted at a provider, not in your home. This means
you won't be able to just physically access it and retrieve the harddrive if
things go really bad. You might not be able to boot single-user mode either.
Because of this, you might not be able to restore the snapshots if things go
*really* bad. In this case, you should also make a local copy of the important
data.

The services and their configuration can be recreated, just follow the earlier
parts of this series again. The email data, however, cannot. This is the data
in `/srv/mail`. You can make a local copy of all this data using `rsync`.

{% highlight sh %}
rsync -av example.org:/srv/mail/ ~/mail-backup
{% endhighlight %}

There's one more thing to do, which I learned the hard way. Set your login
shell to a simple one, provided by the base system. The obvious choice is
`/bin/sh`, but some people may wrongly prefer `/bin/tcsh` as well. During a
major version update, the ABI changes, which will temporarily break most of
the user-installed packages, including your shell.

{% highlight sh %}
chsh
{% endhighlight %}

{% admonition_md warning %}
Be sure to change the shell for whatever user you're using to SSH into this
machine too, if any!
{% endadmonition_md %}

## Updating the Base System

With the preparations in place in case things get royally screwed up, the
actual updates can begin. FreeBSD has a dedicated program to handle updating
the base system, `freebsd-update`. First off, fetch any updates, and make sure
all the updates for your current version are applied.

{% highlight sh %}
freebsd-update fetch install
{% endhighlight %}

Afterwards, set the new system version you want to update to. In my case, this
is `12.1-RELEASE`, but if you're reading this in the future, you most certainly
want a newer version.

{% highlight sh %}
freebsd-update -r 12.1-RELEASE upgrade
{% endhighlight %}

This command will ask you to review the changes and confirm them as well. It
should generally be fine, but this is your last chance to make any backups or
perform other actions to secure your data! If you're ready to continue, install
the updates to the machine.

{% highlight sh %}
freebsd-update install
{% endhighlight %}

At this point, your kernel has been updated. Next you must reboot to start
using the new kernel.

{% highlight sh %}
reboot
{% endhighlight %}

Once the system is back online, you can continue installing the rest of the
updates.

{% highlight sh %}
freebsd-update install
{% endhighlight %}

When this command finishes, the base system has been updated and should be
ready for use. Next up is updating all the software you installed manually.

## Updating User-Installed Packages

Unlike GNU+Linux distributions, FreeBSD has a clear distinction between the
*base system* and *user installed software*. The base system has now been
updated, but everything installed through `pkg` or ports is still at the old
version. If you performed a major version upgrade (say, FreeBSD 11.x to 12.x),
the ABI has changed and few, if any, of the user-installed packages still work.

### Binary Packages using `pkg`

Binary packages are the most common packages used. These are the packages
installed through `pkg`. Currently, `pkg` itself doesn't even work. Luckily,
FreeBSD has `pkg-static`, which is a statically compiled version of `pkg`
intended to fix this very problem. Let's fix up `pkg` itself first.

{% highlight sh %}
pkg-static install -f pkg
{% endhighlight %}

That will make `pkg` itself work again. Now you can use `pkg` to update package
information, and upgrade all packages to a version that works under this
FreeBSD version.

{% highlight sh %}
pkg update
pkg upgrade
{% endhighlight %}

#### PostgreSQL

A particular package that was installed through `pkg`, PostgreSQL, just got
updated to the latest version. On FreeBSD, the data directory used by
PostgreSQL is dependent on the version you're running. If you try to list
databases now, you'll notice that the `mail` database used throughout the
tutorial is gone. The data directory is still there, so you *could* downgrade
PostgreSQL again, restart the database, run a `pgdump`, upgrade, restart and
import. However, I find it much cleaner to use FreeBSD jails to solve this
issue.

{% admonition_md info %}
My original installation used PostgreSQL 9.6, you may need to update some
version numbers accordingly!
{% endadmonition_md %}

I generally put my jails in a ZFS subvolume, so let's create one of those
first.

{% highlight sh %}
zfs create -o mountpoint=/usr/jails zroot/jails
zfs create zroot/jails/postgres96
{% endhighlight %}

This will create a new subvolume at `/usr/jails/postgres96`. Using
`bsdinstall`, a clean FreeBSD installation usable by the jail can be set up
here. This command will give you some popups you may remember from installing
FreeBSD initially. This time, you can uncheck *all* boxes, to get the most
minimal system.

{% highlight sh %}
bsdinstall jail /usr/jails/postgres96
{% endhighlight %}

When `bsdinstall` finishes, you can configure the jail. This is done in
`/etc/jail.conf`. If this file doesn't exist, you can create it. Make sure the
following configuration block is written to the file.

{% highlight cfg %}
postgres96 {
    # Init information
    exec.start = "/bin/sh /etc/rc";
    exec.stop  = "/bin/sh /etc/rc.shutdown";
    exec.clean;

    # Set the root path of the jail
    path = "/usr/jails/$name";

    # Mount /dev
    mount.devfs;

    # Set network information
    host.hostname = $name;
    ip4.addr = "lo0|127.1.1.1/32";
    ip6.addr = "lo0|fd00:1:1:1::1/64";

    # Required for PostgreSQL to function
    allow.raw_sockets;
    allow.sysvipc;
}
{% endhighlight %}

Now you can start up the jail, so it can be used.

{% highlight sh %}
service jail onestart postgres96
{% endhighlight %}

Using the host system's `pkg`, you can install PostgreSQL into the jail.

{% highlight sh %}
pkg -c /usr/jails/postgres96 install postgresql96-server
{% endhighlight %}

Now you just need to make the data directory available to the jail, which you
can most easily do using
[`nullfs`](https://www.freebsd.org/cgi/man.cgi?query=nullfs&sektion=&n=1).

{% highlight sh %}
mount -t nullfs /var/db/postgres/data96 /usr/jails/postgres96/var/db/postgres/data96
{% endhighlight %}

Now everything should be ready for use inside the jail. Let's head on in using
`jexec`.

{% highlight sh %}
jexec postgres96
{% endhighlight %}

Once inside the jail, you can start the PostgreSQL service, and dump the `mail`
database.

{% highlight sh %}
service postgresql onestart
su - postgres
pg_dump mail > ~/mail.sql
{% endhighlight %}

This will write the dump to `/usr/jails/postgres96/var/db/postgres/mail.sql` on
the host system. You can leave the jail and close it down again.

{% highlight sh %}
exit
exit
service jail onestop postgres96
{% endhighlight %}

This dump can be imported in your updated PostgreSQL on the host system.
Connect to the database first.

{% highlight sh %}
su - postgres
psql
{% endhighlight %}

Then, recreate the user, database and import the data from the dump.

{% highlight sql %}
CREATE USER postfix WITH PASSWORD 'incredibly-secret!';
CREATE DATABASE mail WITH OWNER postfix;
\c mail
\i /usr/jails/postgres96/var/db/postgres/mail.sql
\q
{% endhighlight %}

The `mail` database is now back, and ready for use!

### Packages from Ports

With all the binary packages out of the way, it's time to update packages from
ports. While it is very possible to just go to each port's directory and
manually update each one individually, I opted to use `portupgrade`. This will
need manual installation, but afterwards, we can rely on `portupgrade` to do
the rest. Before doing anything with the ports collection, it should be
updated, which is done using `portsnap`.

{% highlight sh %}
portsnap fetch extract
{% endhighlight %}

Once this is done, you can go to the `portupgrade` directory and install it.

{% highlight sh %}
cd /usr/ports/ports-mgmt/portupgrade
make install clean
{% endhighlight %}

Now, to upgrade all other ports.

{% highlight sh %}
portupgrade -a
{% endhighlight %}

Be sure to double-check the compilation options that you are prompted about! If
you're missing a certain option, you may miss an important feature that is
required for your mailserver to work appropriately. This can be easily fixed by
recompiling, but a few seconds checking now can save you an hour figuring it
out later!

## Tidying Up

Now that all user-installed software has been updated too, it's time to
finalize the update by running `freebsd-update` for a final time.

{% highlight sh %}
freebsd-update install
{% endhighlight %}

You can return to your favourite shell again.

{% highlight sh %}
chsh
{% endhighlight %}

And you can clean up the ports directories to get some wasted space back.

{% highlight sh %}
portsclean -C
{% endhighlight %}

I would suggest making a new snapshot as well, now that you're on a relatively
clean and stable state.

{% highlight sh %}
zfs snapshot -r zroot/srv@`date +%Y%m%d%H%M%S`-12.1-clean
zfs snapshot -r zroot/usr/local@`date +%Y%m%d%H%M%S`-12.1-clean
zfs snapshot -r zroot/postgres@`date +%Y%m%d%H%M%S`-12.1-clean
zfs snapshot -r zroot/usr/ports@`date +%Y%m%d%H%M%S`-12.1-clean
{% endhighlight %}

And that concludes your system update. Your mailserver is ready to be neglected
for years again!