summaryrefslogtreecommitdiff
path: root/src/_posts/2020-07-19-freebsd-mailserver-part-6-system-updates.md
blob: 5094eba7f4ed01d5e4fdf41f2d3f8c257aa65af6 (plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
---
title: "FreeBSD Email Server - Part 6: System Updates"
layout: post
tags: Tutorial FreeBSD Email
social:
  email: mailto:~tyil/public-inbox@lists.sr.ht&subject=FreeBSD Email Server
  mastodon: https://soc.fglt.nl/notice/9xF5VVpcK1NJR0kgOO
description: >
  Updates are important. This article aims to help you to learn from my
  mistakes, so your updates will go smooth.
---

Four years have past, and my FreeBSD email server has keps on running without
any problems. However, some people on IRC have recently been nagging me to
support TLSv1.3 on my mailserver. Since the installation was done 4 years ago,
it didn't do 1.3 yet, just 1.2. I set out to do a relatively simple system
update, which didn't go as smooth as I had hoped. This tutorial post should
help you avoid the mistakes I made, so your updates *will* go smooth.

{% admonition_md info %}
The rest of this tutorial assumes you're running as the `root` user.
{% endadmonition_md %}

## Preparations

Before we do anything wild, let's do the obvious first step: backups. Since
this is a FreeBSD server, it uses glorious
[ZFS](https://en.wikipedia.org/wiki/ZFS) as the filesystem, which allows us to
make use of
[snapshots](https://docs.oracle.com/cd/E23824_01/html/821-1448/gbciq.html).
Which subvolumes to make snapshots off depends on your particular setup. In my
case, my actual email data is stored on `zroot/srv`, and all the services and
their configurations are in `zroot/usr/local`. My database's data is stored on
`zroot/postgres/data96`. Additionally, I want to make a snapshot of
`zroot/usr/ports`.

{% highlight sh %}
zfs snapshot -r zroot/srv@`date +%Y%m%d%H%M%S`-11.0-final
zfs snapshot -r zroot/usr/local@`date +%Y%m%d%H%M%S`-11.0-final
zfs snapshot -r zroot/postgres@`date +%Y%m%d%H%M%S`-11.0-final
zfs snapshot -r zroot/usr/ports@`date +%Y%m%d%H%M%S`-11.0-final
{% endhighlight %}

This will make a snapshot of each of these locations, for easy restoration in
case any problems arise. You can list all your snapshots with `zfs list -t
snapshot`.

Your server is most likely hosted at a provider, not in your home. This means
you won't be able to just physically access it and retrieve the harddrive if
things go really bad. You might not be able to boot single-user mode either.
Because of this, you might not be able to restore the snapshots if things go
*really* bad. In this case, you should also make a local copy of the important
data.

The services and their configuration can be recreated, just follow the earlier
parts of this series again. The email data, however, cannot. This is the data
in `/srv/mail`. You can make a local copy of all this data using `rsync`.

{% highlight sh %}
rsync -av example.org:/srv/mail/ ~/mail-backup
{% endhighlight %}

There's one more thing to do, which I learned the hard way. Set your login
shell to a simple one, provided by the base system. The obvious choice is
`/bin/sh`, but some people may wrongly prefer `/bin/tcsh` as well. During a
major version update, the ABI changes, which will temporarily break most of
the user-installed packages, including your shell.

{% highlight sh %}
chsh
{% endhighlight %}

{% admonition_md warning %}
Be sure to change the shell for whatever user you're using to SSH into this
machine too, if any!
{% endadmonition_md %}

## Updating the Base System

With the preparations in place in case things get royally screwed up, the
actual updates can begin. FreeBSD has a dedicated program to handle updating
the base system, `freebsd-update`. First off, fetch any updates, and make sure
all the updates for your current version are applied.

{% highlight sh %}
freebsd-update fetch install
{% endhighlight %}

Afterwards, set the new system version you want to update to. In my case, this
is `12.1-RELEASE`, but if you're reading this in the future, you most certainly
want a newer version.

{% highlight sh %}
freebsd-update -r 12.1-RELEASE upgrade
{% endhighlight %}

This command will ask you to review the changes and confirm them as well. It
should generally be fine, but this is your last chance to make any backups or
perform other actions to secure your data! If you're ready to continue, install
the updates to the machine.

{% highlight sh %}
freebsd-update install
{% endhighlight %}

At this point, your kernel has been updated. Next you must reboot to start
using the new kernel.

{% highlight sh %}
reboot
{% endhighlight %}

Once the system is back online, you can continue installing the rest of the
updates.

{% highlight sh %}
freebsd-update install
{% endhighlight %}

When this command finishes, the base system has been updated and should be
ready for use. Next up is updating all the software you installed manually.

## Updating User-Installed Packages

Unlike GNU+Linux distributions, FreeBSD has a clear distinction between the
*base system* and *user installed software*. The base system has now been
updated, but everything installed through `pkg` or ports is still at the old
version. If you performed a major version upgrade (say, FreeBSD 11.x to 12.x),
the ABI has changed and few, if any, of the user-installed packages still work.

### Binary Packages using `pkg`

Binary packages are the most common packages used. These are the packages
installed through `pkg`. Currently, `pkg` itself doesn't even work. Luckily,
FreeBSD has `pkg-static`, which is a statically compiled version of `pkg`
intended to fix this very problem. Let's fix up `pkg` itself first.

{% highlight sh %}
pkg-static install -f pkg
{% endhighlight %}

That will make `pkg` itself work again. Now you can use `pkg` to update package
information, and upgrade all packages to a version that works under this
FreeBSD version.

{% highlight sh %}
pkg update
pkg upgrade
{% endhighlight %}

#### PostgreSQL

A particular package that was installed through `pkg`, PostgreSQL, just got
updated to the latest version. On FreeBSD, the data directory used by
PostgreSQL is dependent on the version you're running. If you try to list
databases now, you'll notice that the `mail` database used throughout the
tutorial is gone. The data directory is still there, so you *could* downgrade
PostgreSQL again, restart the database, run a `pgdump`, upgrade, restart and
import. However, I find it much cleaner to use FreeBSD jails to solve this
issue.

{% admonition_md info %}
My original installation used PostgreSQL 9.6, you may need to update some
version numbers accordingly!
{% endadmonition_md %}

I generally put my jails in a ZFS subvolume, so let's create one of those
first.

{% highlight sh %}
zfs create -o mountpoint=/usr/jails zroot/jails
zfs create zroot/jails/postgres96
{% endhighlight %}

This will create a new subvolume at `/usr/jails/postgres96`. Using
`bsdinstall`, a clean FreeBSD installation usable by the jail can be set up
here. This command will give you some popups you may remember from installing
FreeBSD initially. This time, you can uncheck *all* boxes, to get the most
minimal system.

{% highlight sh %}
bsdinstall jail /usr/jails/postgres96
{% endhighlight %}

When `bsdinstall` finishes, you can configure the jail. This is done in
`/etc/jail.conf`. If this file doesn't exist, you can create it. Make sure the
following configuration block is written to the file.

{% highlight cfg %}
postgres96 {
    # Init information
    exec.start = "/bin/sh /etc/rc";
    exec.stop  = "/bin/sh /etc/rc.shutdown";
    exec.clean;

    # Set the root path of the jail
    path = "/usr/jails/$name";

    # Mount /dev
    mount.devfs;

    # Set network information
    host.hostname = $name;
    ip4.addr = "lo0|127.1.1.1/32";
    ip6.addr = "lo0|fd00:1:1:1::1/64";

    # Required for PostgreSQL to function
    allow.raw_sockets;
    allow.sysvipc;
}
{% endhighlight %}

Now you can start up the jail, so it can be used.

{% highlight sh %}
service jail onestart postgres96
{% endhighlight %}

Using the host system's `pkg`, you can install PostgreSQL into the jail.

{% highlight sh %}
pkg -c /usr/jails/postgres96 install postgresql96-server
{% endhighlight %}

Now you just need to make the data directory available to the jail, which you
can most easily do using
[`nullfs`](https://www.freebsd.org/cgi/man.cgi?query=nullfs&sektion=&n=1).

{% highlight sh %}
mount -t nullfs /var/db/postgres/data96 /usr/jails/postgres96/var/db/postgres/data96
{% endhighlight %}

Now everything should be ready for use inside the jail. Let's head on in using
`jexec`.

{% highlight sh %}
jexec postgres96
{% endhighlight %}

Once inside the jail, you can start the PostgreSQL service, and dump the `mail`
database.

{% highlight sh %}
service postgresql onestart
su - postgres
pg_dump mail > ~/mail.sql
{% endhighlight %}

This will write the dump to `/usr/jails/postgres96/var/db/postgres/mail.sql` on
the host system. You can leave the jail and close it down again.

{% highlight sh %}
exit
exit
service jail onestop postgres96
{% endhighlight %}

This dump can be imported in your updated PostgreSQL on the host system.
Connect to the database first.

{% highlight sh %}
su - postgres
psql
{% endhighlight %}

Then, recreate the user, database and import the data from the dump.

{% highlight sql %}
CREATE USER postfix WITH PASSWORD 'incredibly-secret!';
CREATE DATABASE mail WITH OWNER postfix;
\c mail
\i /usr/jails/postgres96/var/db/postgres/mail.sql
\q
{% endhighlight %}

The `mail` database is now back, and ready for use!

### Packages from Ports

With all the binary packages out of the way, it's time to update packages from
ports. While it is very possible to just go to each port's directory and
manually update each one individually, I opted to use `portupgrade`. This will
need manual installation, but afterwards, we can rely on `portupgrade` to do
the rest. Before doing anything with the ports collection, it should be
updated, which is done using `portsnap`.

{% highlight sh %}
portsnap fetch extract
{% endhighlight %}

Once this is done, you can go to the `portupgrade` directory and install it.

{% highlight sh %}
cd /usr/ports/ports-mgmt/portupgrade
make install clean
{% endhighlight %}

Now, to upgrade all other ports.

{% highlight sh %}
portupgrade -a
{% endhighlight %}

Be sure to double-check the compilation options that you are prompted about! If
you're missing a certain option, you may miss an important feature that is
required for your mailserver to work appropriately. This can be easily fixed by
recompiling, but a few seconds checking now can save you an hour figuring it
out later!

## Tidying Up

Now that all user-installed software has been updated too, it's time to
finalize the update by running `freebsd-update` for a final time.

{% highlight sh %}
freebsd-update install
{% endhighlight %}

You can return to your favourite shell again.

{% highlight sh %}
chsh
{% endhighlight %}

And you can clean up the ports directories to get some wasted space back.

{% highlight sh %}
portsclean -C
{% endhighlight %}

I would suggest making a new snapshot as well, now that you're on a relatively
clean and stable state.

{% highlight sh %}
zfs snapshot -r zroot/srv@`date +%Y%m%d%H%M%S`-12.1-clean
zfs snapshot -r zroot/usr/local@`date +%Y%m%d%H%M%S`-12.1-clean
zfs snapshot -r zroot/postgres@`date +%Y%m%d%H%M%S`-12.1-clean
zfs snapshot -r zroot/usr/ports@`date +%Y%m%d%H%M%S`-12.1-clean
{% endhighlight %}

And that concludes your system update. Your mailserver is ready to be neglected
for years again!