summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorPatrick Spek <p.spek@tyil.nl>2020-07-19 13:03:54 +0200
committerPatrick Spek <p.spek@tyil.nl>2020-07-19 13:03:54 +0200
commit2df3ef84dcec99b4d03bf6b0f6eeac294e5525d7 (patch)
tree093af18c241e6793856c4fc67d57ad1ec0286d59
parent84f0a9be4c042e9733c0c3cb1270048f2ae27224 (diff)
Add tutorial on updating the FreeBSD email server
-rw-r--r--_posts/2020-07-19-freebsd-mailserver-part-6-system-updates.md341
1 files changed, 341 insertions, 0 deletions
diff --git a/_posts/2020-07-19-freebsd-mailserver-part-6-system-updates.md b/_posts/2020-07-19-freebsd-mailserver-part-6-system-updates.md
new file mode 100644
index 0000000..8e7d79d
--- /dev/null
+++ b/_posts/2020-07-19-freebsd-mailserver-part-6-system-updates.md
@@ -0,0 +1,341 @@
+---
+title: "FreeBSD Email Server - Part 6: System Updates"
+layout: post
+tags: Tutorial FreeBSD Email
+social:
+ email: mailto:~tyil/public-inbox@lists.sr.ht&subject=FreeBSD Email Server
+description: >
+ Updates are important. This article aims to help you to learn from my
+ mistakes, so your updates will go smooth.
+---
+
+Four years have past, and my FreeBSD email server has keps on running without
+any problems. However, some people on IRC have recently been nagging me to
+support TLSv1.3 on my mailserver. Since the installation was done 4 years ago,
+it didn't do 1.3 yet, just 1.2. I set out to do a relatively simple system
+update, which didn't go as smooth as I had hoped. This tutorial post should
+help you avoid the mistakes I made, so your updates *will* go smooth.
+
+{% admonition_md info %}
+The rest of this tutorial assumes you're running as the `root` user.
+{% endadmonition_md %}
+
+## Preparations
+
+Before we do anything wild, let's do the obvious first step: backups. Since
+this is a FreeBSD server, it uses glorious
+[ZFS](https://en.wikipedia.org/wiki/ZFS) as the filesystem, which allows us to
+make use of
+[snapshots](https://docs.oracle.com/cd/E23824_01/html/821-1448/gbciq.html).
+Which subvolumes to make snapshots off depends on your particular setup. In my
+case, my actual email data is stored on `zroot/srv`, and all the services and
+their configurations are in `zroot/usr/local`. My database's data is stored on
+`zroot/postgres/data96`. Additionally, I want to make a snapshot of
+`zroot/usr/ports`.
+
+{% highlight sh %}
+zfs snapshot -r zroot/srv@`date +%Y%m%d%H%M%S`-11.0-final
+zfs snapshot -r zroot/usr/local@`date +%Y%m%d%H%M%S`-11.0-final
+zfs snapshot -r zroot/postgres@`date +%Y%m%d%H%M%S`-11.0-final
+zfs snapshot -r zroot/usr/ports@`date +%Y%m%d%H%M%S`-11.0-final
+{% endhighlight %}
+
+This will make a snapshot of each of these locations, for easy restoration in
+case any problems arise. You can list all your snapshots with `zfs list -t
+snapshot`.
+
+Your server is most likely hosted at a provider, not in your home. This means
+you won't be able to just physically access it and retrieve the harddrive if
+things go really bad. You might not be able to boot single-user mode either.
+Because of this, you might not be able to restore the snapshots if things go
+*really* bad. In this case, you should also make a local copy of the important
+data.
+
+The services and their configuration can be recreated, just follow the earlier
+parts of this series again. The email data, however, cannot. This is the data
+in `/srv/mail`. You can make a local copy of all this data using `rsync`.
+
+{% highlight sh %}
+rsync -av example.org:/srv/mail/ ~/mail-backup
+{% endhighlight %}
+
+There's one more thing to do, which I learned the hard way. Set your login
+shell to a simple one, provided by the base system. The obvious choice is
+`/bin/sh`, but some people may wrongly prefer `/bin/tcsh` as well. During a
+major version update, the ABI changes, which will temporarily break most of
+the user-installed packages, including your shell.
+
+{% highlight sh %}
+chsh
+{% endhighlight %}
+
+{% admonition_md warning %}
+Be sure to change the shell for whatever user you're using to SSH into this
+machine too, if any!
+{% endadmonition_md %}
+
+## Updating the Base System
+
+With the preparations in place in case things get royally screwed up, the
+actual updates can begin. FreeBSD has a dedicated program to handle updating
+the base system, `freebsd-update`. First off, fetch any updates, and make sure
+all the updates for your current version are applied.
+
+{% highlight sh %}
+freebsd-update fetch install
+{% endhighlight %}
+
+Afterwards, set the new system version you want to update to. In my case, this
+is `12.1-RELEASE`, but if you're reading this in the future, you most certainly
+want a newer version.
+
+{% highlight sh %}
+freebsd-update -r 12.1-RELEASE upgrade
+{% endhighlight %}
+
+This command will ask you to review the changes and confirm them as well. It
+should generally be fine, but this is your last chance to make any backups or
+perform other actions to secure your data! If you're ready to continue, install
+the updates to the machine.
+
+{% highlight sh %}
+freebsd-update install
+{% endhighlight %}
+
+At this point, your kernel has been updated. Next you must reboot to start
+using the new kernel.
+
+{% highlight sh %}
+reboot
+{% endhighlight %}
+
+Once the system is back online, you can continue installing the rest of the
+updates.
+
+{% highlight sh %}
+freebsd-update install
+{% endhighlight %}
+
+When this command finishes, the base system has been updated and should be
+ready for use. Next up is updating all the software you installed manually.
+
+## Updating User-Installed Packages
+
+Unlike GNU+Linux distributions, FreeBSD has a clear distinction between the
+*base system* and *user installed software*. The base system has now been
+updated, but everything installed through `pkg` or ports is still at the old
+version. If you performed a major version upgrade (say, FreeBSD 11.x to 12.x),
+the ABI has changed and few, if any, of the user-installed packages still work.
+
+### Binary Packages using `pkg`
+
+Binary packages are the most common packages used. These are the packages
+installed through `pkg`. Currently, `pkg` itself doesn't even work. Luckily,
+FreeBSD has `pkg-static`, which is a statically compiled version of `pkg`
+intended to fix this very problem. Let's fix up `pkg` itself first.
+
+{% highlight sh %}
+pkg-static install -f pkg
+{% endhighlight %}
+
+That will make `pkg` itself work again. Now you can use `pkg` to update package
+information, and upgrade all packages to a version that works under this
+FreeBSD version.
+
+{% highlight sh %}
+pkg update
+pkg upgrade
+{% endhighlight %}
+
+#### PostgreSQL
+
+A particular package that was installed through `pkg`, PostgreSQL, just got
+updated to the latest version. On FreeBSD, the data directory used by
+PostgreSQL is dependent on the version you're running. If you try to list
+databases now, you'll notice that the `mail` database used throughout the
+tutorial is gone. The data directory is still there, so you *could* downgrade
+PostgreSQL again, restart the database, run a `pgdump`, upgrade, restart and
+import. However, I find it much cleaner to use FreeBSD jails to solve this
+issue.
+
+{% admonition_md info %}
+My original installation used PostgreSQL 9.6, you may need to update some
+version numbers accordingly!
+{% endadmonition_md %}
+
+I generally put my jails in a ZFS subvolume, so let's create one of those
+first.
+
+{% highlight sh %}
+zfs create -o mountpoint=/usr/jails zroot/jails
+zfs create zroot/jails/postgres96
+{% endhighlight %}
+
+This will create a new subvolume at `/usr/jails/postgres96`. Using
+`bsdinstall`, a clean FreeBSD installation usable by the jail can be set up
+here. This command will give you some popups you may remember from installing
+FreeBSD initially. This time, you can uncheck *all* boxes, to get the most
+minimal system.
+
+{% highlight sh %}
+bsdinstall jail /usr/jails/postgres96
+{% endhighlight %}
+
+When `bsdinstall` finishes, you can configure the jail. This is done in
+`/etc/jail.conf`. If this file doesn't exist, you can create it. Make sure the
+following configuration block is written to the file.
+
+{% highlight cfg %}
+postgres96 {
+ # Init information
+ exec.start = "/bin/sh /etc/rc";
+ exec.stop = "/bin/sh /etc/rc.shutdown";
+ exec.clean;
+
+ # Set the root path of the jail
+ path = "/usr/jails/$name";
+
+ # Mount /dev
+ mount.devfs;
+
+ # Set network information
+ host.hostname = $name;
+ ip4.addr = "lo0|127.1.1.1/32";
+ ip6.addr = "lo0|fd00:1:1:1::1/64";
+
+ # Required for PostgreSQL to function
+ allow.raw_sockets;
+ allow.sysvipc;
+}
+{% endhighlight %}
+
+Now you can start up the jail, so it can be used.
+
+{% highlight sh %}
+service jail onestart postgres96
+{% endhighlight %}
+
+Using the host system's `pkg`, you can install PostgreSQL into the jail.
+
+{% highlight sh %}
+pkg -c /usr/jails/postgres96 install postgresql96-server
+{% endhighlight %}
+
+Now you just need to make the data directory available to the jail, which you
+can most easily do using
+[`nullfs`](https://www.freebsd.org/cgi/man.cgi?query=nullfs&sektion=&n=1).
+
+{% highlight sh %}
+mount -t nullfs /var/db/postgres/data96 /usr/jails/postgres96/var/db/postgres/data96
+{% endhighlight %}
+
+Now everything should be ready for use inside the jail. Let's head on in using
+`jexec`.
+
+{% highlight sh %}
+jexec postgres96
+{% endhighlight %}
+
+Once inside the jail, you can start the PostgreSQL service, and dump the `mail`
+database.
+
+{% highlight sh %}
+service postgresql onestart
+su - postgres
+pg_dump mail > ~/mail.sql
+{% endhighlight %}
+
+This will write the dump to `/usr/jails/postgres96/var/db/postgres/mail.sql` on
+the host system. You can leave the jail and close it down again.
+
+{% highlight sh %}
+exit
+exit
+service jail onestop postgres96
+{% endhighlight %}
+
+This dump can be imported in your updated PostgreSQL on the host system.
+Connect to the database first.
+
+{% highlight sh %}
+su - postgres
+psql
+{% endhighlight %}
+
+Then, recreate the user, database and import the data from the dump.
+
+{% highlight sql %}
+CREATE USER postfix WITH PASSWORD 'incredibly-secret!';
+CREATE DATABASE mail WITH OWNER postfix;
+\c mail
+\i /usr/jails/postgres96/var/db/postgres/mail.sql
+\q
+{% endhighlight %}
+
+The `mail` database is now back, and ready for use!
+
+### Packages from Ports
+
+With all the binary packages out of the way, it's time to update packages from
+ports. While it is very possible to just go to each port's directory and
+manually update each one individually, I opted to use `portupgrade`. This will
+need manual installation, but afterwards, we can rely on `portupgrade` to do
+the rest. Before doing anything with the ports collection, it should be
+updated, which is done using `portsnap`.
+
+{% highlight sh %}
+portsnap fetch extract
+{% endhighlight %}
+
+Once this is done, you can go to the `portupgrade` directory and install it.
+
+{% highlight sh %}
+cd /usr/ports/ports-mgmt/portupgrade
+make install clean
+{% endhighlight %}
+
+Now, to upgrade all other ports.
+
+{% highlight sh %}
+portupgrade -a
+{% endhighlight %}
+
+Be sure to double-check the compilation options that you are prompted about! If
+you're missing a certain option, you may miss an important feature that is
+required for your mailserver to work appropriately. This can be easily fixed by
+recompiling, but a few seconds checking now can save you an hour figuring it
+out later!
+
+## Tidying Up
+
+Now that all user-installed software has been updated too, it's time to
+finalize the update by running `freebsd-update` for a final time.
+
+{% highlight sh %}
+freebsd-update install
+{% endhighlight %}
+
+You can return to your favourite shell again.
+
+{% highlight sh %}
+chsh
+{% endhighlight %}
+
+And you can clean up the ports directories to get some wasted space back.
+
+{% highlight sh %}
+portsclean -C
+{% endhighlight %}
+
+I would suggest making a new snapshot as well, now that you're on a relatively
+clean and stable state.
+
+{% highlight sh %}
+zfs snapshot -r zroot/srv@`date +%Y%m%d%H%M%S`-12.1-clean
+zfs snapshot -r zroot/usr/local@`date +%Y%m%d%H%M%S`-12.1-clean
+zfs snapshot -r zroot/postgres@`date +%Y%m%d%H%M%S`-12.1-clean
+zfs snapshot -r zroot/usr/ports@`date +%Y%m%d%H%M%S`-12.1-clean
+{% endhighlight %}
+
+And that concludes your system update. Your mailserver is ready to be neglected
+for years again!