1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
|
---
date: 2023-08-05
title: "My New Server Rack: Nouki"
tags:
- GNU+Linux
- Gentoo
- PostgreSQL
- Prometheus
- Systemd
- ZFS
---
After setting up [mieshu](/post/2023/07/23/my-new-server-rack-mieshu/), nouki is
the next server to work on in my home rack. Nouki is intended to live as my main
database server, mainly for PostgreSQL, but perhaps later on in life MySQL if I
ever want a service that doesn't support superiour databases.
The setup for nouki is much simpler in that regard, the base system is almost
identical. This server has ZFS with 2 NVMe disks running in a mirror
configuration. It is also a Gentoo based system, and again with systemd rather
than openrc. The experience of systemd with mieshu was much less painful than I
anticipated. It would seem that it has had time to mature, though I still
dislike how it kills diversity in init/service managers on GNU+Linux.
Both PostgreSQL and ZFS have received some tweaking to run more smoothly. I'm no
DBA, so if you see anything silly in here, do let me know so I can improve my
life.
For ZFS, tweaking was rather minimal. I've made a seperate dataset for
PostgreSQL to use, with `recordsize=8K` as option. For PostgreSQL, I've altered
a bit more. First and foremost, the `pg_hba.conf` to allow access from machines
in my tinc-based VPN.
```conf
host all all 10.57.0.0/16 scram-sha-256
```
The `postgresql.conf` file received the following treatment, based solely on the
guidance provided by [PGTune](https://pgtune.leopard.in.ua/).
```conf
listen_address = 10.57.101.20
max_connections = 200
shared_buffers = 8GB
effective_cache_size = 24GB
maintenance_work_mem = 2GB
checkpoint_completion_target = 0.9
wal_buffers = 16MB
default_statistics_target = 100
random_page_cost = 1.1
effective_io_concurrency = 200
work_mem = 5242kB
min_wal_size = 1GB
max_wal_size = 4GB
max_worker_processes = 12
max_parallel_workers_per_gather = 4
max_parallel_workers = 12
max_parallel_maintenance_workers = 4
```
With this, PostgreSQL seems to perform very well on this machine, applications
using it are noticably faster. Sadly I have no timings from when it all ran on
my desktop, so I cannot make an exact statement on how much faster everything
has become.
Additionally, I wanted to start gathering metrics of my machines and services,
so I can start thinking about dashboards and alerts. I've chosen to use the
current industry standard of Prometheus for this. Since I consider Prometheus to
be a database for metrics, it has been deployed on my database server as well.
Prometheus is currently set to scrape metrics from the `node_exporter` and
`postgresql_exporter`, and seems to work fine. I expect I may need to tweak it
in the future to configure how long I want metrics to be available, since I've
seen it use quite a large amount of memory when storing a large amount of
metrics for a very long time.
To actually see the metrics and have alerts, I currently intend to go with
Grafana. I already have ntfy running, and it appears relatively simple to mold
Grafana alerts into ntfy notifications. To do this properly, I will require some
machines to handle regular workloads. Most likely these will be Intel NUCs, or
similar machines, as they draw very little power for reasonable performance.
Raspberry Pi units would be cheaper, but also seem vastly less powerful, and I'd
need to ensure all my intended workloads can run on ARM which could become a
nuisance very quickly.
As I already have an Intel NUC to play with, that's what I'll be doing for the
coming few days to see if this can work for my desires. Perhaps I can try out a
highly available cluster setup of K3s in the near future!
|
