From 2df3ef84dcec99b4d03bf6b0f6eeac294e5525d7 Mon Sep 17 00:00:00 2001 From: Patrick Spek Date: Sun, 19 Jul 2020 13:03:54 +0200 Subject: Add tutorial on updating the FreeBSD email server --- ...-19-freebsd-mailserver-part-6-system-updates.md | 341 +++++++++++++++++++++ 1 file changed, 341 insertions(+) create mode 100644 _posts/2020-07-19-freebsd-mailserver-part-6-system-updates.md (limited to '_posts') diff --git a/_posts/2020-07-19-freebsd-mailserver-part-6-system-updates.md b/_posts/2020-07-19-freebsd-mailserver-part-6-system-updates.md new file mode 100644 index 0000000..8e7d79d --- /dev/null +++ b/_posts/2020-07-19-freebsd-mailserver-part-6-system-updates.md @@ -0,0 +1,341 @@ +--- +title: "FreeBSD Email Server - Part 6: System Updates" +layout: post +tags: Tutorial FreeBSD Email +social: + email: mailto:~tyil/public-inbox@lists.sr.ht&subject=FreeBSD Email Server +description: > + Updates are important. This article aims to help you to learn from my + mistakes, so your updates will go smooth. +--- + +Four years have past, and my FreeBSD email server has keps on running without +any problems. However, some people on IRC have recently been nagging me to +support TLSv1.3 on my mailserver. Since the installation was done 4 years ago, +it didn't do 1.3 yet, just 1.2. I set out to do a relatively simple system +update, which didn't go as smooth as I had hoped. This tutorial post should +help you avoid the mistakes I made, so your updates *will* go smooth. + +{% admonition_md info %} +The rest of this tutorial assumes you're running as the `root` user. +{% endadmonition_md %} + +## Preparations + +Before we do anything wild, let's do the obvious first step: backups. Since +this is a FreeBSD server, it uses glorious +[ZFS](https://en.wikipedia.org/wiki/ZFS) as the filesystem, which allows us to +make use of +[snapshots](https://docs.oracle.com/cd/E23824_01/html/821-1448/gbciq.html). +Which subvolumes to make snapshots off depends on your particular setup. In my +case, my actual email data is stored on `zroot/srv`, and all the services and +their configurations are in `zroot/usr/local`. My database's data is stored on +`zroot/postgres/data96`. Additionally, I want to make a snapshot of +`zroot/usr/ports`. + +{% highlight sh %} +zfs snapshot -r zroot/srv@`date +%Y%m%d%H%M%S`-11.0-final +zfs snapshot -r zroot/usr/local@`date +%Y%m%d%H%M%S`-11.0-final +zfs snapshot -r zroot/postgres@`date +%Y%m%d%H%M%S`-11.0-final +zfs snapshot -r zroot/usr/ports@`date +%Y%m%d%H%M%S`-11.0-final +{% endhighlight %} + +This will make a snapshot of each of these locations, for easy restoration in +case any problems arise. You can list all your snapshots with `zfs list -t +snapshot`. + +Your server is most likely hosted at a provider, not in your home. This means +you won't be able to just physically access it and retrieve the harddrive if +things go really bad. You might not be able to boot single-user mode either. +Because of this, you might not be able to restore the snapshots if things go +*really* bad. In this case, you should also make a local copy of the important +data. + +The services and their configuration can be recreated, just follow the earlier +parts of this series again. The email data, however, cannot. This is the data +in `/srv/mail`. You can make a local copy of all this data using `rsync`. + +{% highlight sh %} +rsync -av example.org:/srv/mail/ ~/mail-backup +{% endhighlight %} + +There's one more thing to do, which I learned the hard way. Set your login +shell to a simple one, provided by the base system. The obvious choice is +`/bin/sh`, but some people may wrongly prefer `/bin/tcsh` as well. During a +major version update, the ABI changes, which will temporarily break most of +the user-installed packages, including your shell. + +{% highlight sh %} +chsh +{% endhighlight %} + +{% admonition_md warning %} +Be sure to change the shell for whatever user you're using to SSH into this +machine too, if any! +{% endadmonition_md %} + +## Updating the Base System + +With the preparations in place in case things get royally screwed up, the +actual updates can begin. FreeBSD has a dedicated program to handle updating +the base system, `freebsd-update`. First off, fetch any updates, and make sure +all the updates for your current version are applied. + +{% highlight sh %} +freebsd-update fetch install +{% endhighlight %} + +Afterwards, set the new system version you want to update to. In my case, this +is `12.1-RELEASE`, but if you're reading this in the future, you most certainly +want a newer version. + +{% highlight sh %} +freebsd-update -r 12.1-RELEASE upgrade +{% endhighlight %} + +This command will ask you to review the changes and confirm them as well. It +should generally be fine, but this is your last chance to make any backups or +perform other actions to secure your data! If you're ready to continue, install +the updates to the machine. + +{% highlight sh %} +freebsd-update install +{% endhighlight %} + +At this point, your kernel has been updated. Next you must reboot to start +using the new kernel. + +{% highlight sh %} +reboot +{% endhighlight %} + +Once the system is back online, you can continue installing the rest of the +updates. + +{% highlight sh %} +freebsd-update install +{% endhighlight %} + +When this command finishes, the base system has been updated and should be +ready for use. Next up is updating all the software you installed manually. + +## Updating User-Installed Packages + +Unlike GNU+Linux distributions, FreeBSD has a clear distinction between the +*base system* and *user installed software*. The base system has now been +updated, but everything installed through `pkg` or ports is still at the old +version. If you performed a major version upgrade (say, FreeBSD 11.x to 12.x), +the ABI has changed and few, if any, of the user-installed packages still work. + +### Binary Packages using `pkg` + +Binary packages are the most common packages used. These are the packages +installed through `pkg`. Currently, `pkg` itself doesn't even work. Luckily, +FreeBSD has `pkg-static`, which is a statically compiled version of `pkg` +intended to fix this very problem. Let's fix up `pkg` itself first. + +{% highlight sh %} +pkg-static install -f pkg +{% endhighlight %} + +That will make `pkg` itself work again. Now you can use `pkg` to update package +information, and upgrade all packages to a version that works under this +FreeBSD version. + +{% highlight sh %} +pkg update +pkg upgrade +{% endhighlight %} + +#### PostgreSQL + +A particular package that was installed through `pkg`, PostgreSQL, just got +updated to the latest version. On FreeBSD, the data directory used by +PostgreSQL is dependent on the version you're running. If you try to list +databases now, you'll notice that the `mail` database used throughout the +tutorial is gone. The data directory is still there, so you *could* downgrade +PostgreSQL again, restart the database, run a `pgdump`, upgrade, restart and +import. However, I find it much cleaner to use FreeBSD jails to solve this +issue. + +{% admonition_md info %} +My original installation used PostgreSQL 9.6, you may need to update some +version numbers accordingly! +{% endadmonition_md %} + +I generally put my jails in a ZFS subvolume, so let's create one of those +first. + +{% highlight sh %} +zfs create -o mountpoint=/usr/jails zroot/jails +zfs create zroot/jails/postgres96 +{% endhighlight %} + +This will create a new subvolume at `/usr/jails/postgres96`. Using +`bsdinstall`, a clean FreeBSD installation usable by the jail can be set up +here. This command will give you some popups you may remember from installing +FreeBSD initially. This time, you can uncheck *all* boxes, to get the most +minimal system. + +{% highlight sh %} +bsdinstall jail /usr/jails/postgres96 +{% endhighlight %} + +When `bsdinstall` finishes, you can configure the jail. This is done in +`/etc/jail.conf`. If this file doesn't exist, you can create it. Make sure the +following configuration block is written to the file. + +{% highlight cfg %} +postgres96 { + # Init information + exec.start = "/bin/sh /etc/rc"; + exec.stop = "/bin/sh /etc/rc.shutdown"; + exec.clean; + + # Set the root path of the jail + path = "/usr/jails/$name"; + + # Mount /dev + mount.devfs; + + # Set network information + host.hostname = $name; + ip4.addr = "lo0|127.1.1.1/32"; + ip6.addr = "lo0|fd00:1:1:1::1/64"; + + # Required for PostgreSQL to function + allow.raw_sockets; + allow.sysvipc; +} +{% endhighlight %} + +Now you can start up the jail, so it can be used. + +{% highlight sh %} +service jail onestart postgres96 +{% endhighlight %} + +Using the host system's `pkg`, you can install PostgreSQL into the jail. + +{% highlight sh %} +pkg -c /usr/jails/postgres96 install postgresql96-server +{% endhighlight %} + +Now you just need to make the data directory available to the jail, which you +can most easily do using +[`nullfs`](https://www.freebsd.org/cgi/man.cgi?query=nullfs&sektion=&n=1). + +{% highlight sh %} +mount -t nullfs /var/db/postgres/data96 /usr/jails/postgres96/var/db/postgres/data96 +{% endhighlight %} + +Now everything should be ready for use inside the jail. Let's head on in using +`jexec`. + +{% highlight sh %} +jexec postgres96 +{% endhighlight %} + +Once inside the jail, you can start the PostgreSQL service, and dump the `mail` +database. + +{% highlight sh %} +service postgresql onestart +su - postgres +pg_dump mail > ~/mail.sql +{% endhighlight %} + +This will write the dump to `/usr/jails/postgres96/var/db/postgres/mail.sql` on +the host system. You can leave the jail and close it down again. + +{% highlight sh %} +exit +exit +service jail onestop postgres96 +{% endhighlight %} + +This dump can be imported in your updated PostgreSQL on the host system. +Connect to the database first. + +{% highlight sh %} +su - postgres +psql +{% endhighlight %} + +Then, recreate the user, database and import the data from the dump. + +{% highlight sql %} +CREATE USER postfix WITH PASSWORD 'incredibly-secret!'; +CREATE DATABASE mail WITH OWNER postfix; +\c mail +\i /usr/jails/postgres96/var/db/postgres/mail.sql +\q +{% endhighlight %} + +The `mail` database is now back, and ready for use! + +### Packages from Ports + +With all the binary packages out of the way, it's time to update packages from +ports. While it is very possible to just go to each port's directory and +manually update each one individually, I opted to use `portupgrade`. This will +need manual installation, but afterwards, we can rely on `portupgrade` to do +the rest. Before doing anything with the ports collection, it should be +updated, which is done using `portsnap`. + +{% highlight sh %} +portsnap fetch extract +{% endhighlight %} + +Once this is done, you can go to the `portupgrade` directory and install it. + +{% highlight sh %} +cd /usr/ports/ports-mgmt/portupgrade +make install clean +{% endhighlight %} + +Now, to upgrade all other ports. + +{% highlight sh %} +portupgrade -a +{% endhighlight %} + +Be sure to double-check the compilation options that you are prompted about! If +you're missing a certain option, you may miss an important feature that is +required for your mailserver to work appropriately. This can be easily fixed by +recompiling, but a few seconds checking now can save you an hour figuring it +out later! + +## Tidying Up + +Now that all user-installed software has been updated too, it's time to +finalize the update by running `freebsd-update` for a final time. + +{% highlight sh %} +freebsd-update install +{% endhighlight %} + +You can return to your favourite shell again. + +{% highlight sh %} +chsh +{% endhighlight %} + +And you can clean up the ports directories to get some wasted space back. + +{% highlight sh %} +portsclean -C +{% endhighlight %} + +I would suggest making a new snapshot as well, now that you're on a relatively +clean and stable state. + +{% highlight sh %} +zfs snapshot -r zroot/srv@`date +%Y%m%d%H%M%S`-12.1-clean +zfs snapshot -r zroot/usr/local@`date +%Y%m%d%H%M%S`-12.1-clean +zfs snapshot -r zroot/postgres@`date +%Y%m%d%H%M%S`-12.1-clean +zfs snapshot -r zroot/usr/ports@`date +%Y%m%d%H%M%S`-12.1-clean +{% endhighlight %} + +And that concludes your system update. Your mailserver is ready to be neglected +for years again! -- cgit v1.1