1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
|
---
date: 2023-07-23
title: "My New Server Rack: Mieshu"
tags:
- GNU+Linux
- Gentoo
- Systemd
- Garage
---
After saving up for a long while and thinking about what I want in my new home,
I have finally taken the leap and gotten myself a server rack for home use. Its
has a 15U capacity, which should be plenty to get started, but this same brand
has larger racks too, in case I do want to upgrade it and keep the same style.
That said, for now there's only two 4U units in them, one for (file) storage,
and one for database purposes. I sadly don't have anything dedicated for
workloads yet, so for now, both of these servers are intended to also run some
light workloads. I haven't made my mind up yet on how to solve the workload
issues. Now that I have a rack, I obviously want something rack-mountable, and
I probably want it to run a Kubernetes cluster too.
In this regard, I _could_ go for a set of [Raspberry Pi](https://www.raspberrypi.com/)
units, there's [3U mounts that can hold up to 12 Raspberry Pi machines](https://www.uctronics.com/uctronics-19-inch-3u-rack-mount-for-raspberry-pi-4-with-8-mounting-plates.html),
which would be a nice amount. However, I am not yet completely sold on full-ARM
workloads, and I'm not entirely convinced of the power of Raspberry Pi units in
general. I'd much rather standardize on another brand, [Odroid](https://www.hardkernel.com/),
as they have more types of units available, and are not limited to just ARM. But
since they're not the popular kid in class, there's very few off-the-shelf
rack mounting equipment for it. I'll be thinking about this for just a bit more
before making a decision.
For now, though, I wanted to talk about the setup of the first server, Mieshu,
who will be used as a storage server. Mieshu currently runs 8 HDDs, and 2 NVMe
drives. One of the NVMe drives is used for the rootfs, and the other is used for
caching in certain applications. The HDDs themselves offer the data storage
capacity.
The HDDs are currently comprised of four 16TB drives, and four 8TB drives. The
smaller disks come from my desktop, Edephas, which used to serve as data storage
until Mieshu took over. All disks are configured into pairs, which themselves
make mirrors. This means I have four sets of mirror pools, two times 16TB, and
two times 8TB, for a total of 48TB of storage. I'm currently using about half of
this, and it should give me plenty of time before needing to increase the size
again.
I chose to use mirrors since it has a good chance of your data being recoverable
on disk failure, and it allows me to buy disks per two, rather than in larger
numbers. This hopefully keeps the cost of expansion within reasonable limits.
The mirrors themselves are currently [ZFS](https://openzfs.org/wiki/Main_Page)
pools, but I hope to be able to use [bcachefs](https://bcachefs.org/) very soon
as well.
Just a bunch of mirrors is rather inconvenient, however, so I'm also leveraging
[MergerFS](https://github.com/trapexit/mergerfs) to combine all the mirrors into
a single usable pool. This slightly odd setup was chosen over RAID-0 or RAID-Z*
to lower the impact of disk failure. Even if two disks in the same mirror were
the die at the same time, I wouldn't lose _all_ data, just the bits on that
particular mirror. It would be very annoying, but it wouldn't be disastrous.
Apart from generic mass storage, I also host S3 buckets for personal use. This
is where I upload CI artifacts to, and [my MissKey instance](https://fedi.tyil.nl/@tyil)
uses it for storing objects as well. Future services such as [Mimir](https://grafana.com/oss/mimir/)
will probably leverage S3 for storage as well. This is achieved through
[Garage](https://garagehq.deuxfleurs.fr/). I've also tried [SeaweedFS](https://seaweedfs.github.io/),
which is a very neat project on its own, but Garage is just simpler to
configure, and allows a replicated setup with only two servers, whereas SeaweedFS
demands an odd number of master servers.
And lastly, Mieshu runs [K3s](https://k3s.io/) for its Kubernetes component. It
is currently not serving anything yet, as the other server is supposed to become
the database server, which is needed for most workloads. Once that is up and
running, Mieshu will start hosting things such as
[Grafana](https://grafana.com/oss/grafana) and [Loki](https://grafana.com/oss/loki/),
monitoring stuff basically. Perhaps I'll move [Laminar](https://laminar.ohwg.net/)
to this server as well, but I'm unsure if I will run that as a Kubernetes service.
The server itself runs on Gentoo, as it still is the most stable experience I
can get out of any GNU+Linux distribution. I am, however, not using the default
of OpenRC as the init system and service manager. For the first time, I'm
running Gentoo with systemd. After several years, it appears to have become
stable enough to trust with serious workloads. With its increased use, however,
some things have become simpler by just using systemd. I hope to get a better
understanding of it, and learn to bend it to my will as needed, by simply using
it on my own systems.
I hope to have time to work on the other server sooner rather than later, so
I can finish up the base of my new setup. Be on the lookout for the next post,
where I'll go into detail on Nouki, the database server.
|
