summaryrefslogtreecommitdiff
diff options
context:
space:
mode:
authorPatrick Spek <p.spek@tyil.nl>2022-01-18 15:50:55 +0100
committerPatrick Spek <p.spek@tyil.nl>2022-01-18 15:50:55 +0100
commit0a19618ae4ba2ab1dbcc243e0b91d4792349a2cd (patch)
tree5d887f210398722e6fd2719827fd2a32f52bd7ec
parent5d10c30fc5150ced04ebfcb1a9e100c43bef4762 (diff)
Add missing articles from 2020
-rw-r--r--content/posts/2020/2020-01-08-running-cgit-on-gentoo.md301
-rw-r--r--content/posts/2020/2020-06-21-lately-in-raku.md155
-rw-r--r--content/posts/2020/2020-07-15-config-3.0.md176
-rw-r--r--content/posts/2020/2020-07-19-freebsd-mailserver-part-6-system-updates.md341
-rw-r--r--content/posts/2020/2020-12-14-raku-modules-in-gentoo-portage.md114
-rw-r--r--content/posts/2020/2020-12-15-merging-json-in-postgresql.md50
-rw-r--r--content/posts/2020/key.ascbin8872 -> 0 bytes
7 files changed, 1137 insertions, 0 deletions
diff --git a/content/posts/2020/2020-01-08-running-cgit-on-gentoo.md b/content/posts/2020/2020-01-08-running-cgit-on-gentoo.md
new file mode 100644
index 0000000..085da26
--- /dev/null
+++ b/content/posts/2020/2020-01-08-running-cgit-on-gentoo.md
@@ -0,0 +1,301 @@
+---
+title: Running cgit on Gentoo
+date: 2020-01-08
+tags:
+- git
+- cgit
+- Gentoo
+---
+
+[cgit](https://git.zx2c4.com/cgit/about/), a web interface for git
+repositories, allows you to easily share your projects' source code over a web
+interface. It's running on my desktop right now, so you can [see for
+yourself](https://home.tyil.nl/git) what it looks like. On
+[Gentoo](https://www.gentoo.org/), the ebuild for this software can be found as
+`www-apps/cgit`. However, after installation, a number of configuration steps
+should be performed to make it accessible on `$HOSTNAME/git`, and index your
+repositories. This post will guide you through the steps I took.
+
+## Filesystem layout
+
+In my setup, my (bare) git repositories reside in `$HOME/.local/git`. But, some
+of the repositories should not be public, such as the
+[`pass`](https://www.passwordstore.org/) store. So, a different directory
+for cgit to look in exists, at `$HOME/.local/srv/cgit`. This directory contains
+symlinks to the actual repositories I want publicly available.
+
+## Installing the required software
+
+For this to work, there is more than just cgit to install. There are a number
+of ways to set this up, but I chose for Nginx as web server, and `uwsgi` as the
+handler for the fastcgi requests.
+
+```txt
+emerge dev-python/pygments www-apps/cgit www-servers/nginx www-servers/uwsgi
+```
+
+## Configuring all elements
+
+After installation, each of these packages needs to be configured.
+
+### cgit
+
+The configuration file for cgit resides in `/etc/cgitrc`. After removing all
+the comments, the contents of my `/etc/cgitrc` can be found below.
+
+```txt
+# Fixes for running cgit in a subdirectory
+css=/git/cgit.css
+logo=/git/cgit.png
+virtual-root=/git
+remove-suffix=1
+
+# Customization
+root-desc=All public repos from tyil
+enable-index-owner=0
+cache-size=1000
+snapshots=tar.gz tar.bz2
+clone-prefix=https://home.tyil.nl/git
+robots=index, follow
+
+readme=master:README.md
+readme=master:README.pod6
+
+# Add filters before repos (or filters won't work)
+about-filter=/usr/lib64/cgit/filters/about-formatting.sh
+source-filter=/usr/lib64/cgit/filters/syntax-highlighting.py
+
+# Scan paths for repos
+scan-path=/home/tyil/.local/srv/cgit
+```
+
+You should probably update the values of `root-desc`, `clone-prefix` and
+`scan-path`. The first describes the small line of text at the top of the web
+interface. `clone-prefix` is the prefix URL used for `git clone` URLs. The
+`scan-path` is the directory `cgit` will look for repositories in.
+
+Additionally, the `readme=master:README.pod6` only positively affects
+your setup if you also use my [Raku](https://raku.org/) customizations,
+outlined in the next section.
+
+For more information on the available settings and their impact, consult `man
+cgitrc`.
+
+#### Raku customizations
+
+Since I love working with Raku, I made some changes and a couple modules to get
+`README.pod6` files rendered on the *about* tab on projects. You should ensure
+the `cgit` user can run `raku` and has the
+[`Pod::To::Anything`](https://home.tyil.nl/git/raku/Pod::To::Anything/) and
+[`Pod::To::HTML::Section`](https://home.tyil.nl/git/raku/Pod::To::HTML::Section/)
+modules installed (including any dependencies). How to achieve this depends on
+how you installed Raku. Feel free to send me an email if you need help on this
+part!
+
+Once this works, however, the remaining step is quite simple. The
+`about-filter` configuration item in `/etc/cgitrc` points to a small shell
+script that invokes the required program to convert a document to HTML. In my
+case, this file is at `/usr/lib64/cgit/filters/about-formatting.sh`. Open up
+this file in your favorite `$EDITOR` and add another entry to the `case` for
+[Pod6](https://docs.raku.org/language/pod) to call Raku.
+
+```sh
+case "$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]')" in
+ *.markdown|*.mdown|*.md|*.mkd) exec ./md2html; ;;
+ *.pod6) exec raku --doc=HTML::Section; ;;
+ *.rst) exec ./rst2html; ;;
+ *.[1-9]) exec ./man2html; ;;
+ *.htm|*.html) exec cat; ;;
+ *.txt|*) exec ./txt2html; ;;
+esac
+```
+
+#### Highlighting style
+
+The `syntax-highlighting.py` filter carries the responsibility to get your code
+highlighted. This uses the Python library [pygments](https://pygments.org/),
+which comes with a number of styles. cgit uses *Pastie* by default. To change
+this, open the Python script, and look for the `HtmlFormatter`, which contains
+a `style='Pastie'` bit. You can change `Pastie` for any other style name. These
+styles are available in my version (2.4.2):
+
+- default
+- emacs
+- friendly
+- colorful
+- autumn
+- murphy
+- manni
+- monokai
+- perldoc
+- pastie
+- borland
+- trac
+- native
+- fruity
+- bw
+- vim
+- vs
+- tango
+- rrt
+- xcode
+- igor
+- paraiso-light
+- paraiso-dark
+- lovelace
+- algol
+- algol_nu
+- arduino
+- rainbow_dash
+- abap
+- solarized-dark
+- solarized-light
+- sas
+- stata
+- stata-light
+- stata-dark
+
+For those interested, I use the `emacs` theme.
+
+### uwsgi
+
+Next up, `uwsgi`. This needs configuration, which on Gentoo exists in
+`/etc/conf.d/uwsgi`. However, this file itself shouldn't be altered. Instead,
+make a copy of it, and call it `/etc/conf.d/uwsgi.cgit`. The standard file
+exists solely as a base template. For brevity, I left out all the comments in
+the contents below.
+
+```sh
+UWSGI_SOCKET=
+UWSGI_THREADS=0
+UWSGI_PROGRAM=
+UWSGI_XML_CONFIG=
+UWSGI_PROCESSES=1
+UWSGI_LOG_FILE=
+UWSGI_CHROOT=
+UWSGI_DIR=/home/tyil
+UWSGI_PIDPATH_MODE=0750
+UWSGI_USER=
+UWSGI_GROUP=
+UWSGI_EMPEROR_PATH=
+UWSGI_EMPEROR_PIDPATH_MODE=0770
+UWSGI_EMPEROR_GROUP=
+UWSGI_EXTRA_OPTIONS="--ini /etc/uwsgi.d/cgit.ini"
+```
+
+That covers the service configuration file. When things don't work the way you
+expect, specify a path in `UWSGI_LOG_FILE` to see its logs. Additionally, you
+may want to alter the value of `UWSGI_DIR`. This specifies the working
+directory from which the process starts.
+
+Now comes the application configuration, which will be read from
+`/etc/uwsgi.d/cgit.ini`, according to `UWSGI_EXTRA_OPTIONS`. Create that file
+with the following contents.
+
+```ini
+[uwsgi]
+master = true
+plugins = cgi
+socket = 127.0.0.1:1234
+uid = cgit
+gid = cgit
+procname-master = uwsgi cgit
+processes = 1
+threads = 2
+cgi = /usr/share/webapps/cgit/1.2.1/hostroot/cgi-bin/cgit.cgi
+```
+
+Note that the `cgi` value contains the version number of `www-apps/cgit`. You
+may need to come back after an upgrade and update it accordingly.
+
+As last step for `uwsgi` configuration, a service script, to manage it with
+`rc-service`. These scripts all exist in `/etc/conf.d`, and the package
+installed a script called `uwsgi` in there. Just like with the `conf.d`
+variant, its just a template. This time, however, don't make a copy of it, but
+a symlink. It does not need to be edited, but the name must be the same as the
+`conf.d` entry name. That would be `uwsgi.cgit`.
+
+```txt
+cd /etc/conf.d
+ln -s uwsgi uwsgi.cgit
+```
+
+Now you can start the service with `rc-service uwsgi.cgit start`. If a
+consequent `status` notes the state as *Started*, you're all good. If the state
+says *Crashed*, you should go back and double-check all configuration files.
+When those are correct and you can't figure out why, feel free to reach out to
+me via email.
+
+```txt
+rc-service uwsgi.cgit start
+rc-service uwsgi.cgit service
+
+# Start this after boot
+rc-update add uwsgi.cgit
+```
+
+### nginx
+
+The final element to make it accessible, the web server, `nginx`. How you
+organize the configuration files here is largely up to you. Explaining how to
+set up nginx from scratch is beyond the scope of this post. Assuming you know
+how to configure this, add the following `location` blocks to the `server`
+definition for the vhost you want to make `cgit` available on.
+
+```nginx
+location "/git" {
+ alias /usr/share/webapps/cgit/1.2.1/htdocs;
+ try_files $uri @cgit;
+}
+
+location @cgit {
+ include uwsgi_params;
+
+ gzip off;
+
+ uwsgi_modifier1 9;
+ uwsgi_pass 127.0.0.1:1234;
+
+ fastcgi_split_path_info ^(/git/?)(.+)$;
+ uwsgi_param PATH_INFO $fastcgi_path_info;
+}
+```
+
+Once saved, you can reload `nginx`, and the `$HOSTNAME/git` endpoint can be
+reached, and should display an cgit page, detailing there are no repositories.
+That can be easily solved by making some available in `$HOME/.local/srv/cgit`,
+through the power of symlinks.
+
+## Symlinking the repositories
+
+Go nuts with making symlinks to the various repositories you have gathered over
+the years. You don't need to use bare repositories, `cgit` will also handle
+regular repositories that you actively work in. As with the `nginx`
+configuration, explaining how to make symlinks is out of scope. In dire
+situations, consult `man ln`.
+
+### `git-mkbare`
+
+While making the symlinks is easy, I found that it sheepishly boring to do. I go
+to `$HOME/.local/git`, make a directory, `cd` to it, and create a bare
+repository. Then off to `$HOME/.local/srv/cgit` to make a symlink back to the
+newly created bare repository. I think you can see this will get tedious very
+quickly.
+
+So, to combat this, I made a small shell script to do all of that for me. I
+called it `git-mkbare`, and put it somewhere in my `$PATH`. This allows me to
+call it as `git mkbare repo-name`. It will ask for a small description as well,
+so I that can also be skipped as a manual task. This script may be of use to
+you if you want to more quickly start a new project.
+
+You can find this script [in my dotfiles
+repository](https://git.tyil.nl/dotfiles/tree/.local/bin/git-mkbare).
+
+## Wrapping up
+
+Now you should have cgit available from your site, allowing you to share the
+sources of all your projects easily with the world. No need to make use of a
+(proprietary) third-party service!
+
+If you have questions or comments on my setup, or the post in general, please
+contact me through email or irc.
diff --git a/content/posts/2020/2020-06-21-lately-in-raku.md b/content/posts/2020/2020-06-21-lately-in-raku.md
new file mode 100644
index 0000000..3d54bdc
--- /dev/null
+++ b/content/posts/2020/2020-06-21-lately-in-raku.md
@@ -0,0 +1,155 @@
+---
+title: Lately in Raku
+date: 2020-06-21
+tags:
+- Raku
+---
+
+I've been working on some Raku projects, but each of them is *just* too small
+to make an individual blog post about. So, I decided to just pack them together
+in a slightly larger blog post instead.
+
+## Binary Rakudo Star builds for GNU+Linux and FreeBSD
+
+A friend on IRC asked if it was possible to get Rakudo Star precompiled for
+ARM, since compiling it on his machine took forever. I took a look around for
+potential build services, and settled for [Sourcehut](https://builds.sr.ht/).
+
+I added build instructions for amd64 FreeBSD, GNU+Linux, musl+Linux, and ARM
+GNU+Linux. Tarballs with precompiled binaries get build whenever I push to the
+Rakudo Star mirror on Sourcehut, and are uploaded to
+[dist.tyil.nl/tmp](https://dist.tyil.nl/tmp/). Currently, these are not
+considered to be an official part of Rakudo Star, but if interest increases and
+more people can test these packages, I can include them in official releases.
+
+## `IRC::Client` plugins
+
+IRC bots are great fun, and the
+[`IRC::Client`](https://github.com/raku-community-modules/perl6-IRC-Client)
+module allows for easy extension through *plugins*. For my own IRC bot,
+[musashi](https://git.sr.ht/~tyil/raku-local-musashi), I've created two new
+plugins, which are now available in the Raku ecosystem for anyone to use.
+
+### `IRC::Client::Plugin::Dicerolls`
+
+The first plugin I've created can do dice rolls, D&D style. You can roll any
+number of dice, with any number of sides, and add (or subtract) bonusses from
+these.
+
+```txt
+<@tyil> .roll 1d20
+<+musashi> 1d20 = 1
+<@tyil> .roll 5d20
+<+musashi> 5d20 = 3 + 19 + 8 + 6 + 11 = 47
+<@tyil> .roll 1d8+2d6+10
+<+musashi> 1d8+2d6+10 = 4 + 6 + 4 + 10 = 24
+```
+
+Since this is ripe for abuse, the plugin allows to set limits, and sets some
+defaults for the limits as well. This should help prevent your bot from getting
+killed for spam.
+
+### `IRC::Client::Plugin::Reminders`
+
+Everyone forgets things, and there's various tools helping people remember
+things in various situations. For IRC based situations, I created a reminder
+plugin for `IRC::Client`.
+
+```txt
+10:19 <@tyil> musashi: remind me to write a blog post in 10 minutes
+10:19 <+musashi> Reminding you to write a blog post on 2020-06-21T08:29:00Z (UTC)
+10:29 <+musashi> tyil: Reminder to write a blog post
+```
+
+It's not very sophisticated yet, working only with numbers and certain
+identifiers (minutes, hours, days, weeks), but I may add more useful
+identifiers later on such as "tomorrow", or "next Sunday". Contributions for
+such extended functionality are obviously also very welcome!
+
+There's [a small
+issue](https://git.sr.ht/~tyil/raku-irc-client-plugin-reminders/tree/master/lib/IRC/Client/Plugin/Reminders.rakumod#L69)
+with logging in a `start` block. It seems the dynamic variable `$*LOG` is no
+longer defined within it. If anyone has an idea why, and how I could fix this,
+please let me know!
+
+## Template program for D&D
+
+Another little utility I made for D&D purposes. My DM asked me how hard it'd be
+to create a program to fill out a number of templates he made, so he could use
+them in the game with another party. He was able to hand me a list of variables
+in the form of a CSV, so I set out to use that. With some help from `Text::CSV`
+and `Template::Mustache`, I had a working solution in a couple minutes, with
+all the required things nicely fit into a single file.
+
+I had not used `$=pod` before in Raku, and I'm quite happy with how easy it is
+to use, though I would like a cleaner way to refer to a Pod block by name.
+
+```raku
+#!/usr/bin/env raku
+
+use v6.d;
+
+use Template::Mustache;
+use Text::CSV;
+
+#| Read a CSV input file to render contracts with.
+sub MAIN () {
+ # Set the directory to write the contracts to.
+ my $output-dir = $*PROGRAM.parent(2).add('out');
+
+ # Make sure the output directory exists
+ $output-dir.mkdir;
+
+ # Load the template
+ my $template = $=pod
+ .grep({ $_.^can('name') && $_.name eq 'template' })
+ .first
+ .contents
+ .map(*.contents)
+ .join("\n\n")
+ ;
+
+ # Parse STDIN as CSV
+ my @records = Text::CSV
+ .new
+ .getline_all($*IN)
+ .skip
+ ;
+
+ # Create a contract out of each record
+ for @records -> @record {
+ $output-dir.add("contract-{@record[0]}.txt").spurt(
+ Template::Mustache.render($template, {
+ contractor => @record[2],
+ date => @record[1],
+ description => @record[6],
+ item => @record[3],
+ location => @record[5],
+ price => @record[4]
+ }) ~ "\n"
+ );
+ }
+}
+
+=begin template
+As per our verbal agreement this contract will detail the rewards, rights, and
+obligations of both parties involved.
+
+The contractor, to be known henceforth as {{ contractor }}.
+The contractee, to be known henceforth as the Association.
+
+{{ contractor }} requests the delivery of an object identified as the "{{ item }}"
+to be delivered by the Association at the location specified for the agreed
+upon compensation. The Association shall deliver the object within two weeks of
+the signing of this contract and receive compensation upon delivery.
+
+The location is to be known as "{{ location }}", described as "{{ description }}".
+The compensation agreed upon is {{ price }} pieces of Hondia standard
+gold-coin currency, or an equivalent in precious gemstones.
+
+Written and signed on the {{ date }}.
+
+For the association, Lan Torrez
+For the {{ contractor }}
+=end template
+```
diff --git a/content/posts/2020/2020-07-15-config-3.0.md b/content/posts/2020/2020-07-15-config-3.0.md
new file mode 100644
index 0000000..67a64c4
--- /dev/null
+++ b/content/posts/2020/2020-07-15-config-3.0.md
@@ -0,0 +1,176 @@
+---
+title: Config 3.0
+date: 2020-07-15
+tags:
+- Raku
+- Programming
+---
+
+For those who don't know, the
+[`Config`](https://modules.raku.org/dist/Config:cpan:TYIL) module for the Raku
+programming language is a generic class to hold... well... configuration data.
+It supports
+[`Config::Parser`](https://modules.raku.org/search/?q=Config%3A%3AParser)
+modules to handle different configuration file formats, such as `JSON`, `YAML`
+and `TOML`.
+
+Up until now, the module didn't do much for you other than provide an interface
+that's generally the same, so you won't need to learn differing methods to
+handle differing configuration file formats. It was my first Raku module, and
+as such, the code wasn't the cleanest. I've written many new modules since
+then, and learned about a good number of (hopefully better) practices.
+
+For version 3.0, I specifically wanted to remove effort from using the `Config`
+module on the developer's end. It should check default locations for
+configuration files, so I don't have to rewrite that code in every other module
+all the time. Additionally, configuration using environment variables is quite
+popular in the current day and age, especially for Dockerized applications. So,
+I set out to make an automated way to read those too.
+
+## The Old Way
+
+First, let's take a look at how it used to work. Generally, I'd create the
+default configuration structure and values first.
+
+```raku
+use Config;
+
+my $config = Config.new.read({
+ foo => "bar",
+ alpha => {
+ beta => "gamma",
+ },
+ version => 3,
+});
+```
+
+And after that, check for potential configuration file locations, and read any
+that exist.
+
+```raku
+$config.read($*HOME.add('config/project.toml').absolute);
+```
+
+The `.absolute` call was necessary because I wrote the initial `Config` version
+with the `.read` method not supporting `IO::Path` objects. A fix for this has
+existed for a while, but wasn't released, so couldn't be relied on outside of
+my general development machines.
+
+If you wanted to add additional environment variable lookups, you'd have to
+check for those as well, and perhaps also cast them as well, since environment
+variables are all strings by default.
+
+## Version 3.0
+
+So, how does the new version improve this? For starters, the `.new` method of
+`Config` now takes a `Hash` as positional argument, in order to create the
+structure, and optionally types *or* default values of your configuration
+object.
+
+```raku
+use Config;
+
+my $config = Config.new({
+ foo => Str,
+ alpha => {
+ beta => "gamma",
+ },
+ version => 3,
+}, :name<project>);
+```
+
+{{< admonition title="note" >}}
+`foo` has been made into the `Str` *type object*, rather than a `Str` *value*.
+This was technically allowed in previous `Config` versions, but it comes with
+actual significance in 3.0.
+{{< / admonition_md >}}
+
+Using `.new` instead of `.read` is a minor syntactic change, which saves 1 word
+per program. This isn't quite that big of a deal. However, the optional `name`
+argument will enable the new automagic features. The name you give to `.new` is
+arbitrary, but will be used to deduce which directories to check, and which
+environment variables to read.
+
+### Automatic Configuration File Handling
+
+By setting `name` to the value `project`, `Config` will consult the
+configuration directories from the [XDG Base Directory
+Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html).
+It uses one of my other modules,
+[`IO::Path::XDG`](https://modules.raku.org/dist/IO::Path::XDG:cpan:TYIL), for
+this, together with
+[`IO::Glob`](https://modules.raku.org/dist/IO::Glob:cpan:HANENKAMP).
+Specifically, it will check my `$XDG_CONFIG_DIRS` and `$XDG_CONFIG_HOME` (in
+that order) for any files that match the globs `project.*` or
+`project/config.*`.
+
+If any files are found to match, they will be read as well, and the
+configuration values contained therein, merged into `$config`. It will load the
+appropriate `Config::Parser` implementation based on the file's extension. I
+intend to add a number of these to future Rakudo Star releases, to ensure most
+default configuration file formats are supported out of the box.
+
+### Automatic Environment Variable Handling
+
+After this step, it will try out some environment variables for configuration
+values. Which variables are checked depends on the structure (and `name`) of
+the `Config` object. The entire structure is squashed into a 1-dimensional list
+of fields. Each level is replaced by an `_`. Additionally, each variable name
+is prefixed with the `name`. Lastly, all the variable names are uppercased.
+
+For the example `Config` given above, this would result in the following
+environment variables being checked.
+
+```sh
+$PROJECT_FOO
+$PROJECT_ALPHA_BETA
+$PROJECT_VERSION
+```
+
+If any are found, they're also cast to the appropriate type. Thus,
+`$PROJECT_FOO` would be cast to a `Str`, and so would `$PROJECT_ALPHA_BETA`. In
+this case that doesn't do much, since they're already strings. But
+`$PROJECT_VERSION` would be cast to an `Int`, since it's default value is also
+of the `Int` type. This should ensure that your variables are always in the
+type you expected them to be originally, no matter the user's configuration
+choices.
+
+## Debugging
+
+In addition to these new features, `Config` now also makes use of my
+[`Log`](https://modules.raku.org/dist/Log:cpan:TYIL) module. This module is
+made around the idea that logging should be simple if module developers are to
+use it, and the way logs are represented is up to the end-user. When running an
+application in your local terminal, you may want more human-friendly logs,
+whereas in production you may want `JSON` formatted logs to make it fit better
+into other tools.
+
+You can tune the amount of logging performed using the `$RAKU_LOG_LEVEL`
+environment variable, as per the `Log` module's interface. When set to `7` (for
+"debug"), it will print the configuration files that are being merged into your
+`Config` and which environment veriables are being used as well.
+
+{{< admonition_md title="note" >}}
+A downside is that the application using `Config` for its configuration must
+also support `Log` to actually make the new logging work. Luckily, this is
+quite easy to set up, and there's example code for this in `Log`'s README.
+{{< / admonition >}}
+
+## Too Fancy For Me
+
+It could very well be that you don't want these features, and you want to stick
+to the old ways as much as possible. No tricks, just plain and simple
+configuration handling. This can be done by simply ommitting the `name`
+argument to `.new`. The new features depend on this name to be set, and won't
+do anything without it.
+
+Alternatively, both the automatic configuration file handling and the
+environment variable handling can be turned off individually using `:!from-xdg`
+and `:!from-env` arguments respectively.
+
+## In Conclusion
+
+The new `Config` module should result in cleaner code in modules using it, and
+more convenience for the developer. If you find any bugs or have other ideas
+for improving the module, feel free to send an email to
+`https://lists.sr.ht/~tyil/raku-devel`.
diff --git a/content/posts/2020/2020-07-19-freebsd-mailserver-part-6-system-updates.md b/content/posts/2020/2020-07-19-freebsd-mailserver-part-6-system-updates.md
new file mode 100644
index 0000000..f3d1e89
--- /dev/null
+++ b/content/posts/2020/2020-07-19-freebsd-mailserver-part-6-system-updates.md
@@ -0,0 +1,341 @@
+---
+title: "FreeBSD Email Server - Part 6: System Updates"
+date: 2020-07-19
+tags:
+- Email
+- FreeBSD
+- Tutorial
+social:
+ email: mailto:~tyil/public-inbox@lists.sr.ht&subject=FreeBSD Email Server
+---
+
+Four years have past, and my FreeBSD email server has keps on running without
+any problems. However, some people on IRC have recently been nagging me to
+support TLSv1.3 on my mailserver. Since the installation was done 4 years ago,
+it didn't do 1.3 yet, just 1.2. I set out to do a relatively simple system
+update, which didn't go as smooth as I had hoped. This tutorial post should
+help you avoid the mistakes I made, so your updates *will* go smooth.
+
+{{< admonition title="info" >}}
+The rest of this tutorial assumes you're running as the `root` user.
+{{< / admonition >}}
+
+## Preparations
+
+Before we do anything wild, let's do the obvious first step: backups. Since
+this is a FreeBSD server, it uses glorious
+[ZFS](https://en.wikipedia.org/wiki/ZFS) as the filesystem, which allows us to
+make use of
+[snapshots](https://docs.oracle.com/cd/E23824_01/html/821-1448/gbciq.html).
+Which subvolumes to make snapshots off depends on your particular setup. In my
+case, my actual email data is stored on `zroot/srv`, and all the services and
+their configurations are in `zroot/usr/local`. My database's data is stored on
+`zroot/postgres/data96`. Additionally, I want to make a snapshot of
+`zroot/usr/ports`.
+
+```txt
+zfs snapshot -r zroot/srv@`date +%Y%m%d%H%M%S`-11.0-final
+zfs snapshot -r zroot/usr/local@`date +%Y%m%d%H%M%S`-11.0-final
+zfs snapshot -r zroot/postgres@`date +%Y%m%d%H%M%S`-11.0-final
+zfs snapshot -r zroot/usr/ports@`date +%Y%m%d%H%M%S`-11.0-final
+```
+
+This will make a snapshot of each of these locations, for easy restoration in
+case any problems arise. You can list all your snapshots with `zfs list -t
+snapshot`.
+
+Your server is most likely hosted at a provider, not in your home. This means
+you won't be able to just physically access it and retrieve the harddrive if
+things go really bad. You might not be able to boot single-user mode either.
+Because of this, you might not be able to restore the snapshots if things go
+*really* bad. In this case, you should also make a local copy of the important
+data.
+
+The services and their configuration can be recreated, just follow the earlier
+parts of this series again. The email data, however, cannot. This is the data
+in `/srv/mail`. You can make a local copy of all this data using `rsync`.
+
+```txt
+rsync -av example.org:/srv/mail/ ~/mail-backup
+```
+
+There's one more thing to do, which I learned the hard way. Set your login
+shell to a simple one, provided by the base system. The obvious choice is
+`/bin/sh`, but some people may wrongly prefer `/bin/tcsh` as well. During a
+major version update, the ABI changes, which will temporarily break most of
+the user-installed packages, including your shell.
+
+```txt
+chsh
+```
+
+{{< admonition title="warning" >}}
+Be sure to change the shell for whatever user you're using to SSH into this
+machine too, if any!
+{{< / admonition >}}
+
+## Updating the Base System
+
+With the preparations in place in case things get royally screwed up, the
+actual updates can begin. FreeBSD has a dedicated program to handle updating
+the base system, `freebsd-update`. First off, fetch any updates, and make sure
+all the updates for your current version are applied.
+
+```txt
+freebsd-update fetch install
+```
+
+Afterwards, set the new system version you want to update to. In my case, this
+is `12.1-RELEASE`, but if you're reading this in the future, you most certainly
+want a newer version.
+
+```txt
+freebsd-update -r 12.1-RELEASE upgrade
+```
+
+This command will ask you to review the changes and confirm them as well. It
+should generally be fine, but this is your last chance to make any backups or
+perform other actions to secure your data! If you're ready to continue, install
+the updates to the machine.
+
+```txt
+freebsd-update install
+```
+
+At this point, your kernel has been updated. Next you must reboot to start
+using the new kernel.
+
+```txt
+reboot
+```
+
+Once the system is back online, you can continue installing the rest of the
+updates.
+
+```txt
+freebsd-update install
+```
+
+When this command finishes, the base system has been updated and should be
+ready for use. Next up is updating all the software you installed manually.
+
+## Updating User-Installed Packages
+
+Unlike GNU+Linux distributions, FreeBSD has a clear distinction between the
+*base system* and *user installed software*. The base system has now been
+updated, but everything installed through `pkg` or ports is still at the old
+version. If you performed a major version upgrade (say, FreeBSD 11.x to 12.x),
+the ABI has changed and few, if any, of the user-installed packages still work.
+
+### Binary Packages using `pkg`
+
+Binary packages are the most common packages used. These are the packages
+installed through `pkg`. Currently, `pkg` itself doesn't even work. Luckily,
+FreeBSD has `pkg-static`, which is a statically compiled version of `pkg`
+intended to fix this very problem. Let's fix up `pkg` itself first.
+
+```txt
+pkg-static install -f pkg
+```
+
+That will make `pkg` itself work again. Now you can use `pkg` to update package
+information, and upgrade all packages to a version that works under this
+FreeBSD version.
+
+```txt
+pkg update
+pkg upgrade
+```
+
+#### PostgreSQL
+
+A particular package that was installed through `pkg`, PostgreSQL, just got
+updated to the latest version. On FreeBSD, the data directory used by
+PostgreSQL is dependent on the version you're running. If you try to list
+databases now, you'll notice that the `mail` database used throughout the
+tutorial is gone. The data directory is still there, so you *could* downgrade
+PostgreSQL again, restart the database, run a `pgdump`, upgrade, restart and
+import. However, I find it much cleaner to use FreeBSD jails to solve this
+issue.
+
+{{< admonition title="info" >}}
+My original installation used PostgreSQL 9.6, you may need to update some
+version numbers accordingly!
+{{< / admonition >}}
+
+I generally put my jails in a ZFS subvolume, so let's create one of those
+first.
+
+```txt
+zfs create -o mountpoint=/usr/jails zroot/jails
+zfs create zroot/jails/postgres96
+```
+
+This will create a new subvolume at `/usr/jails/postgres96`. Using
+`bsdinstall`, a clean FreeBSD installation usable by the jail can be set up
+here. This command will give you some popups you may remember from installing
+FreeBSD initially. This time, you can uncheck *all* boxes, to get the most
+minimal system.
+
+```txt
+bsdinstall jail /usr/jails/postgres96
+```
+
+When `bsdinstall` finishes, you can configure the jail. This is done in
+`/etc/jail.conf`. If this file doesn't exist, you can create it. Make sure the
+following configuration block is written to the file.
+
+```cfg
+postgres96 {
+ # Init information
+ exec.start = "/bin/sh /etc/rc";
+ exec.stop = "/bin/sh /etc/rc.shutdown";
+ exec.clean;
+
+ # Set the root path of the jail
+ path = "/usr/jails/$name";
+
+ # Mount /dev
+ mount.devfs;
+
+ # Set network information
+ host.hostname = $name;
+ ip4.addr = "lo0|127.1.1.1/32";
+ ip6.addr = "lo0|fd00:1:1:1::1/64";
+
+ # Required for PostgreSQL to function
+ allow.raw_sockets;
+ allow.sysvipc;
+}
+```
+
+Now you can start up the jail, so it can be used.
+
+```txt
+service jail onestart postgres96
+```
+
+Using the host system's `pkg`, you can install PostgreSQL into the jail.
+
+```txt
+pkg -c /usr/jails/postgres96 install postgresql96-server
+```
+
+Now you just need to make the data directory available to the jail, which you
+can most easily do using
+[`nullfs`](https://www.freebsd.org/cgi/man.cgi?query=nullfs&sektion=&n=1).
+
+```txt
+mount -t nullfs /var/db/postgres/data96 /usr/jails/postgres96/var/db/postgres/data96
+```
+
+Now everything should be ready for use inside the jail. Let's head on in using
+`jexec`.
+
+```txt
+jexec postgres96
+```
+
+Once inside the jail, you can start the PostgreSQL service, and dump the `mail`
+database.
+
+```txt
+service postgresql onestart
+su - postgres
+pg_dump mail > ~/mail.sql
+```
+
+This will write the dump to `/usr/jails/postgres96/var/db/postgres/mail.sql` on
+the host system. You can leave the jail and close it down again.
+
+```txt
+exit
+exit
+service jail onestop postgres96
+```
+
+This dump can be imported in your updated PostgreSQL on the host system.
+Connect to the database first.
+
+```txt
+su - postgres
+psql
+```
+
+Then, recreate the user, database and import the data from the dump.
+
+```sql
+CREATE USER postfix WITH PASSWORD 'incredibly-secret!';
+CREATE DATABASE mail WITH OWNER postfix;
+\c mail
+\i /usr/jails/postgres96/var/db/postgres/mail.sql
+\q
+```
+
+The `mail` database is now back, and ready for use!
+
+### Packages from Ports
+
+With all the binary packages out of the way, it's time to update packages from
+ports. While it is very possible to just go to each port's directory and
+manually update each one individually, I opted to use `portupgrade`. This will
+need manual installation, but afterwards, we can rely on `portupgrade` to do
+the rest. Before doing anything with the ports collection, it should be
+updated, which is done using `portsnap`.
+
+```txt
+portsnap fetch extract
+```
+
+Once this is done, you can go to the `portupgrade` directory and install it.
+
+```txt
+cd /usr/ports/ports-mgmt/portupgrade
+make install clean
+```
+
+Now, to upgrade all other ports.
+
+```txt
+portupgrade -a
+```
+
+Be sure to double-check the compilation options that you are prompted about! If
+you're missing a certain option, you may miss an important feature that is
+required for your mailserver to work appropriately. This can be easily fixed by
+recompiling, but a few seconds checking now can save you an hour figuring it
+out later!
+
+## Tidying Up
+
+Now that all user-installed software has been updated too, it's time to
+finalize the update by running `freebsd-update` for a final time.
+
+```txt
+freebsd-update install
+```
+
+You can return to your favourite shell again.
+
+```txt
+chsh
+```
+
+And you can clean up the ports directories to get some wasted space back.
+
+```txt
+portsclean -C
+```
+
+I would suggest making a new snapshot as well, now that you're on a relatively
+clean and stable state.
+
+```txt
+zfs snapshot -r zroot/srv@`date +%Y%m%d%H%M%S`-12.1-clean
+zfs snapshot -r zroot/usr/local@`date +%Y%m%d%H%M%S`-12.1-clean
+zfs snapshot -r zroot/postgres@`date +%Y%m%d%H%M%S`-12.1-clean
+zfs snapshot -r zroot/usr/ports@`date +%Y%m%d%H%M%S`-12.1-clean
+```
+
+And that concludes your system update. Your mailserver is ready to be neglected
+for years again!
diff --git a/content/posts/2020/2020-12-14-raku-modules-in-gentoo-portage.md b/content/posts/2020/2020-12-14-raku-modules-in-gentoo-portage.md
new file mode 100644
index 0000000..480d3ae
--- /dev/null
+++ b/content/posts/2020/2020-12-14-raku-modules-in-gentoo-portage.md
@@ -0,0 +1,114 @@
+---
+title: Raku Modules in Gentoo Portage
+date: 2020-12-14
+tags:
+- Raku
+- Gentoo
+social:
+ email: mailto:~tyil/public-inbox@lists.sr.ht&subject=Raku modules in Gentoo's Portage
+---
+
+The last couple of days I've been taking another look at getting modules for
+the Raku programming language into Gentoo's Portage tree. Making new packages
+available in Gentoo is incredibly easy with their overlay system.
+
+The more complex part was Raku's side of things. While most languages just have
+a certain directory to drop files in, Raku *should* use a
+`CompUnit::Repository` object, which exposes the `.install` method. This is
+obviously slower than just copying the files around, but there are merits to
+this method. For one, it allows installation of the same module with different
+versions, or from different authors. It also handles all the Unicode complexity
+for you.
+
+{{< admonition title="note" >}}
+There *is* a
+[CompUnit::Repository::FileSystem](https://docs.raku.org/type/CompUnit::Repository::FileSystem)
+which would allow me to just copy over files to the right directory, however, I
+quite like the ability to have multiple versions of the same module installed.
+This is actually something Portage is designed with, too!
+{{< admonition >}}
+
+After an email to the Raku users mailing list, I got some pointers over IRC. I
+let these sink in for a couple days, considering how to approach the problem
+properly. Then, one night, a solution came to mind, and I set out to test it.
+
+*It actually worked*. And a similar method should be usable for other
+distributions too, such as Debian, OpenSUSE or Archlinux, to create packages
+out of Raku modules. This should greatly improve the ability to ship Raku
+programs to end-users, without requiring them to learn how Raku's ecosystem is
+modeled, or which module manager it uses.
+
+The most important part of this approach is the
+[`module-installer.raku`](https://git.sr.ht/~tyil/raku-overlay/tree/7494c81524ec1845c77dabfbb3303a34eb4b89f4/item/dev-lang/raku/files/module-installer.raku)
+program, which is part of `dev-lang/raku::raku-overlay`. It accepts a path to
+the module to install. It does not depend on any one module manager, so it can
+be used to bootstrap a user-friendly module manager (such as
+[`zef`](https://github.com/ugexe/zef/)) for the user.
+
+```raku
+#| Install a Raku module.
+sub MAIN (
+ #| The path to the Raku module sources.
+ IO() $path,
+
+ #| The repository to install it in. Options are "site" (ment for
+ #| user-installed modules), "vendor" (ment for distributions that want
+ #| to include more modules) and "core" (ment for modules distributed
+ #| along with Raku itself).
+ Str:D :$repo = 'site',
+
+ #| Force installation of the module.
+ Bool:D :$force = True,
+) {
+ CATCH {
+ default { $_.say; exit 1; }
+ }
+
+ die "This script should be used by Portage only!" unless %*ENV<D>;
+
+ my $prefix = %*ENV<D>.IO.add('usr/share/perl6').add($repo);
+ my $repository = CompUnit::Repository::Installation.new(:$prefix);
+ my $meta-file = $path.add('META6.json');
+ my $dist = Distribution::Path.new($path, :$meta-file);
+
+ $repository.install($dist, :$force);
+}
+```
+
+It's a fairly straightforward program. It checks for `$D` to be set in the
+environment, which is a variable Portage sets as the destination directory to
+install new files in. This directory gets merged into the root filesystem to
+finalize installation of any package.
+
+If `$D` is set, I append the path used by Raku in Gentoo to it, followed by a
+repo name. Next I create a `CompUnit::Repository` using this path. This is a
+trick to get the files to appear in the right directory for Portage, to
+eventually merge them in the system-wide `site` module repo used by Raku.
+Additionally, I can use the `CompUnit::Repository`'s `install` method to handle
+all the Raku specific parts that I don't want to handle myself.
+
+This leaves one last issue. By creating this new repo, I also get a couple
+files that already exist in the system wide `site` repo. Portage will complain
+about possible file collisions and refuse to install the package if these
+remain. However, this can be solved rather easily by calling `rm` on these files.
+
+```txt
+rm -- "${D}/usr/share/perl6/site/version"
+rm -- "${D}/usr/share/perl6/site/repo.lock"
+rm -- "${D}/usr/share/perl6/site/precomp/.lock"
+```
+
+And with this, my test module, `IO::Path::XDG`, installs cleanly through the
+power of Portage, and is usable by all users using the system-wide Raku
+installation.
+
+To make this work for other distributions, the `module-installer.raku` program
+should be modified slightly. Most notably, the `$prefix` must be altered to
+point towards the right directory, so the files will be installed into whatever
+directory will end up being packaged. Other than that, the standard means of
+packaging can be followed.
+
+For the Gentoo users, this overlay is available at
+[SourceHut](https://git.sr.ht/~tyil/raku-overlay). It currently holds only
+`IO::Path::XDG` (`dev-raku/io-path-xdg`), but you are invited to try it out and
+report any issues you may encounter.
diff --git a/content/posts/2020/2020-12-15-merging-json-in-postgresql.md b/content/posts/2020/2020-12-15-merging-json-in-postgresql.md
new file mode 100644
index 0000000..8d97e50
--- /dev/null
+++ b/content/posts/2020/2020-12-15-merging-json-in-postgresql.md
@@ -0,0 +1,50 @@
+---
+title: Merging JSON in PostgreSQL
+date: 2020-12-15
+tags:
+- JSON
+- PostgreSQL
+- Programming
+social:
+ email: mailto:~tyil/public-inbox@lists.sr.ht&subject=Merging JSON objects in PostgreSQL
+---
+
+At my `$day-job` we have a lot of `jsonb` in our database. From time to time, I
+have to manually run a query to fix something in there. This week was one of
+those times.
+
+While you can pretty much do everything you need with regards to JSON editing
+with `jsonb_se`t, I thought it might be nice if I were able to *merge* a given
+JSON object into an existing object. This might be cleaner in some situations,
+but mostly it is fun to figure it out. And who doesn’t like spending time with
+`plpgsql`?
+
+The way I wanted to have it work is like this:
+
+```sql
+UPDATE user SET properties = jsonb_merge(properties, '{"notifications": {"new_case": false, "new_document": true}}');
+```
+
+And this is the eventual function I produced to do it:
+
+```sql
+CREATE OR REPLACE FUNCTION jsonb_merge(original jsonb, delta jsonb) RETURNS jsonb AS $$
+ DECLARE result jsonb;
+ BEGIN
+ SELECT
+ json_object_agg(
+ COALESCE(original_key, delta_key),
+ CASE
+ WHEN original_value IS NULL THEN delta_value
+ WHEN delta_value IS NULL THEN original_value
+ WHEN (jsonb_typeof(original_value) <> 'object' OR jsonb_typeof(delta_value) <> 'object') THEN delta_value
+ ELSE jsonb_merge(original_value, delta_value)
+ END
+ )
+ INTO result
+ FROM jsonb_each(original) e1(original_key, original_value)
+ FULL JOIN jsonb_each(delta) e2(delta_key, delta_value) ON original_key = delta_key;
+ RETURN result;
+END
+$$ LANGUAGE plpgsql;
+```
diff --git a/content/posts/2020/key.asc b/content/posts/2020/key.asc
deleted file mode 100644
index 98ddd3a..0000000
--- a/content/posts/2020/key.asc
+++ /dev/null
Binary files differ