From 0a19618ae4ba2ab1dbcc243e0b91d4792349a2cd Mon Sep 17 00:00:00 2001 From: Patrick Spek Date: Tue, 18 Jan 2022 15:50:55 +0100 Subject: Add missing articles from 2020 --- .../2020/2020-01-08-running-cgit-on-gentoo.md | 301 ++++++++++++++++++ content/posts/2020/2020-06-21-lately-in-raku.md | 155 ++++++++++ content/posts/2020/2020-07-15-config-3.0.md | 176 +++++++++++ ...-19-freebsd-mailserver-part-6-system-updates.md | 341 +++++++++++++++++++++ .../2020-12-14-raku-modules-in-gentoo-portage.md | 114 +++++++ .../2020/2020-12-15-merging-json-in-postgresql.md | 50 +++ content/posts/2020/key.asc | Bin 8872 -> 0 bytes 7 files changed, 1137 insertions(+) create mode 100644 content/posts/2020/2020-01-08-running-cgit-on-gentoo.md create mode 100644 content/posts/2020/2020-06-21-lately-in-raku.md create mode 100644 content/posts/2020/2020-07-15-config-3.0.md create mode 100644 content/posts/2020/2020-07-19-freebsd-mailserver-part-6-system-updates.md create mode 100644 content/posts/2020/2020-12-14-raku-modules-in-gentoo-portage.md create mode 100644 content/posts/2020/2020-12-15-merging-json-in-postgresql.md delete mode 100644 content/posts/2020/key.asc diff --git a/content/posts/2020/2020-01-08-running-cgit-on-gentoo.md b/content/posts/2020/2020-01-08-running-cgit-on-gentoo.md new file mode 100644 index 0000000..085da26 --- /dev/null +++ b/content/posts/2020/2020-01-08-running-cgit-on-gentoo.md @@ -0,0 +1,301 @@ +--- +title: Running cgit on Gentoo +date: 2020-01-08 +tags: +- git +- cgit +- Gentoo +--- + +[cgit](https://git.zx2c4.com/cgit/about/), a web interface for git +repositories, allows you to easily share your projects' source code over a web +interface. It's running on my desktop right now, so you can [see for +yourself](https://home.tyil.nl/git) what it looks like. On +[Gentoo](https://www.gentoo.org/), the ebuild for this software can be found as +`www-apps/cgit`. However, after installation, a number of configuration steps +should be performed to make it accessible on `$HOSTNAME/git`, and index your +repositories. This post will guide you through the steps I took. + +## Filesystem layout + +In my setup, my (bare) git repositories reside in `$HOME/.local/git`. But, some +of the repositories should not be public, such as the +[`pass`](https://www.passwordstore.org/) store. So, a different directory +for cgit to look in exists, at `$HOME/.local/srv/cgit`. This directory contains +symlinks to the actual repositories I want publicly available. + +## Installing the required software + +For this to work, there is more than just cgit to install. There are a number +of ways to set this up, but I chose for Nginx as web server, and `uwsgi` as the +handler for the fastcgi requests. + +```txt +emerge dev-python/pygments www-apps/cgit www-servers/nginx www-servers/uwsgi +``` + +## Configuring all elements + +After installation, each of these packages needs to be configured. + +### cgit + +The configuration file for cgit resides in `/etc/cgitrc`. After removing all +the comments, the contents of my `/etc/cgitrc` can be found below. + +```txt +# Fixes for running cgit in a subdirectory +css=/git/cgit.css +logo=/git/cgit.png +virtual-root=/git +remove-suffix=1 + +# Customization +root-desc=All public repos from tyil +enable-index-owner=0 +cache-size=1000 +snapshots=tar.gz tar.bz2 +clone-prefix=https://home.tyil.nl/git +robots=index, follow + +readme=master:README.md +readme=master:README.pod6 + +# Add filters before repos (or filters won't work) +about-filter=/usr/lib64/cgit/filters/about-formatting.sh +source-filter=/usr/lib64/cgit/filters/syntax-highlighting.py + +# Scan paths for repos +scan-path=/home/tyil/.local/srv/cgit +``` + +You should probably update the values of `root-desc`, `clone-prefix` and +`scan-path`. The first describes the small line of text at the top of the web +interface. `clone-prefix` is the prefix URL used for `git clone` URLs. The +`scan-path` is the directory `cgit` will look for repositories in. + +Additionally, the `readme=master:README.pod6` only positively affects +your setup if you also use my [Raku](https://raku.org/) customizations, +outlined in the next section. + +For more information on the available settings and their impact, consult `man +cgitrc`. + +#### Raku customizations + +Since I love working with Raku, I made some changes and a couple modules to get +`README.pod6` files rendered on the *about* tab on projects. You should ensure +the `cgit` user can run `raku` and has the +[`Pod::To::Anything`](https://home.tyil.nl/git/raku/Pod::To::Anything/) and +[`Pod::To::HTML::Section`](https://home.tyil.nl/git/raku/Pod::To::HTML::Section/) +modules installed (including any dependencies). How to achieve this depends on +how you installed Raku. Feel free to send me an email if you need help on this +part! + +Once this works, however, the remaining step is quite simple. The +`about-filter` configuration item in `/etc/cgitrc` points to a small shell +script that invokes the required program to convert a document to HTML. In my +case, this file is at `/usr/lib64/cgit/filters/about-formatting.sh`. Open up +this file in your favorite `$EDITOR` and add another entry to the `case` for +[Pod6](https://docs.raku.org/language/pod) to call Raku. + +```sh +case "$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]')" in + *.markdown|*.mdown|*.md|*.mkd) exec ./md2html; ;; + *.pod6) exec raku --doc=HTML::Section; ;; + *.rst) exec ./rst2html; ;; + *.[1-9]) exec ./man2html; ;; + *.htm|*.html) exec cat; ;; + *.txt|*) exec ./txt2html; ;; +esac +``` + +#### Highlighting style + +The `syntax-highlighting.py` filter carries the responsibility to get your code +highlighted. This uses the Python library [pygments](https://pygments.org/), +which comes with a number of styles. cgit uses *Pastie* by default. To change +this, open the Python script, and look for the `HtmlFormatter`, which contains +a `style='Pastie'` bit. You can change `Pastie` for any other style name. These +styles are available in my version (2.4.2): + +- default +- emacs +- friendly +- colorful +- autumn +- murphy +- manni +- monokai +- perldoc +- pastie +- borland +- trac +- native +- fruity +- bw +- vim +- vs +- tango +- rrt +- xcode +- igor +- paraiso-light +- paraiso-dark +- lovelace +- algol +- algol_nu +- arduino +- rainbow_dash +- abap +- solarized-dark +- solarized-light +- sas +- stata +- stata-light +- stata-dark + +For those interested, I use the `emacs` theme. + +### uwsgi + +Next up, `uwsgi`. This needs configuration, which on Gentoo exists in +`/etc/conf.d/uwsgi`. However, this file itself shouldn't be altered. Instead, +make a copy of it, and call it `/etc/conf.d/uwsgi.cgit`. The standard file +exists solely as a base template. For brevity, I left out all the comments in +the contents below. + +```sh +UWSGI_SOCKET= +UWSGI_THREADS=0 +UWSGI_PROGRAM= +UWSGI_XML_CONFIG= +UWSGI_PROCESSES=1 +UWSGI_LOG_FILE= +UWSGI_CHROOT= +UWSGI_DIR=/home/tyil +UWSGI_PIDPATH_MODE=0750 +UWSGI_USER= +UWSGI_GROUP= +UWSGI_EMPEROR_PATH= +UWSGI_EMPEROR_PIDPATH_MODE=0770 +UWSGI_EMPEROR_GROUP= +UWSGI_EXTRA_OPTIONS="--ini /etc/uwsgi.d/cgit.ini" +``` + +That covers the service configuration file. When things don't work the way you +expect, specify a path in `UWSGI_LOG_FILE` to see its logs. Additionally, you +may want to alter the value of `UWSGI_DIR`. This specifies the working +directory from which the process starts. + +Now comes the application configuration, which will be read from +`/etc/uwsgi.d/cgit.ini`, according to `UWSGI_EXTRA_OPTIONS`. Create that file +with the following contents. + +```ini +[uwsgi] +master = true +plugins = cgi +socket = 127.0.0.1:1234 +uid = cgit +gid = cgit +procname-master = uwsgi cgit +processes = 1 +threads = 2 +cgi = /usr/share/webapps/cgit/1.2.1/hostroot/cgi-bin/cgit.cgi +``` + +Note that the `cgi` value contains the version number of `www-apps/cgit`. You +may need to come back after an upgrade and update it accordingly. + +As last step for `uwsgi` configuration, a service script, to manage it with +`rc-service`. These scripts all exist in `/etc/conf.d`, and the package +installed a script called `uwsgi` in there. Just like with the `conf.d` +variant, its just a template. This time, however, don't make a copy of it, but +a symlink. It does not need to be edited, but the name must be the same as the +`conf.d` entry name. That would be `uwsgi.cgit`. + +```txt +cd /etc/conf.d +ln -s uwsgi uwsgi.cgit +``` + +Now you can start the service with `rc-service uwsgi.cgit start`. If a +consequent `status` notes the state as *Started*, you're all good. If the state +says *Crashed*, you should go back and double-check all configuration files. +When those are correct and you can't figure out why, feel free to reach out to +me via email. + +```txt +rc-service uwsgi.cgit start +rc-service uwsgi.cgit service + +# Start this after boot +rc-update add uwsgi.cgit +``` + +### nginx + +The final element to make it accessible, the web server, `nginx`. How you +organize the configuration files here is largely up to you. Explaining how to +set up nginx from scratch is beyond the scope of this post. Assuming you know +how to configure this, add the following `location` blocks to the `server` +definition for the vhost you want to make `cgit` available on. + +```nginx +location "/git" { + alias /usr/share/webapps/cgit/1.2.1/htdocs; + try_files $uri @cgit; +} + +location @cgit { + include uwsgi_params; + + gzip off; + + uwsgi_modifier1 9; + uwsgi_pass 127.0.0.1:1234; + + fastcgi_split_path_info ^(/git/?)(.+)$; + uwsgi_param PATH_INFO $fastcgi_path_info; +} +``` + +Once saved, you can reload `nginx`, and the `$HOSTNAME/git` endpoint can be +reached, and should display an cgit page, detailing there are no repositories. +That can be easily solved by making some available in `$HOME/.local/srv/cgit`, +through the power of symlinks. + +## Symlinking the repositories + +Go nuts with making symlinks to the various repositories you have gathered over +the years. You don't need to use bare repositories, `cgit` will also handle +regular repositories that you actively work in. As with the `nginx` +configuration, explaining how to make symlinks is out of scope. In dire +situations, consult `man ln`. + +### `git-mkbare` + +While making the symlinks is easy, I found that it sheepishly boring to do. I go +to `$HOME/.local/git`, make a directory, `cd` to it, and create a bare +repository. Then off to `$HOME/.local/srv/cgit` to make a symlink back to the +newly created bare repository. I think you can see this will get tedious very +quickly. + +So, to combat this, I made a small shell script to do all of that for me. I +called it `git-mkbare`, and put it somewhere in my `$PATH`. This allows me to +call it as `git mkbare repo-name`. It will ask for a small description as well, +so I that can also be skipped as a manual task. This script may be of use to +you if you want to more quickly start a new project. + +You can find this script [in my dotfiles +repository](https://git.tyil.nl/dotfiles/tree/.local/bin/git-mkbare). + +## Wrapping up + +Now you should have cgit available from your site, allowing you to share the +sources of all your projects easily with the world. No need to make use of a +(proprietary) third-party service! + +If you have questions or comments on my setup, or the post in general, please +contact me through email or irc. diff --git a/content/posts/2020/2020-06-21-lately-in-raku.md b/content/posts/2020/2020-06-21-lately-in-raku.md new file mode 100644 index 0000000..3d54bdc --- /dev/null +++ b/content/posts/2020/2020-06-21-lately-in-raku.md @@ -0,0 +1,155 @@ +--- +title: Lately in Raku +date: 2020-06-21 +tags: +- Raku +--- + +I've been working on some Raku projects, but each of them is *just* too small +to make an individual blog post about. So, I decided to just pack them together +in a slightly larger blog post instead. + +## Binary Rakudo Star builds for GNU+Linux and FreeBSD + +A friend on IRC asked if it was possible to get Rakudo Star precompiled for +ARM, since compiling it on his machine took forever. I took a look around for +potential build services, and settled for [Sourcehut](https://builds.sr.ht/). + +I added build instructions for amd64 FreeBSD, GNU+Linux, musl+Linux, and ARM +GNU+Linux. Tarballs with precompiled binaries get build whenever I push to the +Rakudo Star mirror on Sourcehut, and are uploaded to +[dist.tyil.nl/tmp](https://dist.tyil.nl/tmp/). Currently, these are not +considered to be an official part of Rakudo Star, but if interest increases and +more people can test these packages, I can include them in official releases. + +## `IRC::Client` plugins + +IRC bots are great fun, and the +[`IRC::Client`](https://github.com/raku-community-modules/perl6-IRC-Client) +module allows for easy extension through *plugins*. For my own IRC bot, +[musashi](https://git.sr.ht/~tyil/raku-local-musashi), I've created two new +plugins, which are now available in the Raku ecosystem for anyone to use. + +### `IRC::Client::Plugin::Dicerolls` + +The first plugin I've created can do dice rolls, D&D style. You can roll any +number of dice, with any number of sides, and add (or subtract) bonusses from +these. + +```txt +<@tyil> .roll 1d20 +<+musashi> 1d20 = 1 +<@tyil> .roll 5d20 +<+musashi> 5d20 = 3 + 19 + 8 + 6 + 11 = 47 +<@tyil> .roll 1d8+2d6+10 +<+musashi> 1d8+2d6+10 = 4 + 6 + 4 + 10 = 24 +``` + +Since this is ripe for abuse, the plugin allows to set limits, and sets some +defaults for the limits as well. This should help prevent your bot from getting +killed for spam. + +### `IRC::Client::Plugin::Reminders` + +Everyone forgets things, and there's various tools helping people remember +things in various situations. For IRC based situations, I created a reminder +plugin for `IRC::Client`. + +```txt +10:19 <@tyil> musashi: remind me to write a blog post in 10 minutes +10:19 <+musashi> Reminding you to write a blog post on 2020-06-21T08:29:00Z (UTC) +10:29 <+musashi> tyil: Reminder to write a blog post +``` + +It's not very sophisticated yet, working only with numbers and certain +identifiers (minutes, hours, days, weeks), but I may add more useful +identifiers later on such as "tomorrow", or "next Sunday". Contributions for +such extended functionality are obviously also very welcome! + +There's [a small +issue](https://git.sr.ht/~tyil/raku-irc-client-plugin-reminders/tree/master/lib/IRC/Client/Plugin/Reminders.rakumod#L69) +with logging in a `start` block. It seems the dynamic variable `$*LOG` is no +longer defined within it. If anyone has an idea why, and how I could fix this, +please let me know! + +## Template program for D&D + +Another little utility I made for D&D purposes. My DM asked me how hard it'd be +to create a program to fill out a number of templates he made, so he could use +them in the game with another party. He was able to hand me a list of variables +in the form of a CSV, so I set out to use that. With some help from `Text::CSV` +and `Template::Mustache`, I had a working solution in a couple minutes, with +all the required things nicely fit into a single file. + +I had not used `$=pod` before in Raku, and I'm quite happy with how easy it is +to use, though I would like a cleaner way to refer to a Pod block by name. + +```raku +#!/usr/bin/env raku + +use v6.d; + +use Template::Mustache; +use Text::CSV; + +#| Read a CSV input file to render contracts with. +sub MAIN () { + # Set the directory to write the contracts to. + my $output-dir = $*PROGRAM.parent(2).add('out'); + + # Make sure the output directory exists + $output-dir.mkdir; + + # Load the template + my $template = $=pod + .grep({ $_.^can('name') && $_.name eq 'template' }) + .first + .contents + .map(*.contents) + .join("\n\n") + ; + + # Parse STDIN as CSV + my @records = Text::CSV + .new + .getline_all($*IN) + .skip + ; + + # Create a contract out of each record + for @records -> @record { + $output-dir.add("contract-{@record[0]}.txt").spurt( + Template::Mustache.render($template, { + contractor => @record[2], + date => @record[1], + description => @record[6], + item => @record[3], + location => @record[5], + price => @record[4] + }) ~ "\n" + ); + } +} + +=begin template +As per our verbal agreement this contract will detail the rewards, rights, and +obligations of both parties involved. + +The contractor, to be known henceforth as {{ contractor }}. +The contractee, to be known henceforth as the Association. + +{{ contractor }} requests the delivery of an object identified as the "{{ item }}" +to be delivered by the Association at the location specified for the agreed +upon compensation. The Association shall deliver the object within two weeks of +the signing of this contract and receive compensation upon delivery. + +The location is to be known as "{{ location }}", described as "{{ description }}". +The compensation agreed upon is {{ price }} pieces of Hondia standard +gold-coin currency, or an equivalent in precious gemstones. + +Written and signed on the {{ date }}. + +For the association, Lan Torrez +For the {{ contractor }} +=end template +``` diff --git a/content/posts/2020/2020-07-15-config-3.0.md b/content/posts/2020/2020-07-15-config-3.0.md new file mode 100644 index 0000000..67a64c4 --- /dev/null +++ b/content/posts/2020/2020-07-15-config-3.0.md @@ -0,0 +1,176 @@ +--- +title: Config 3.0 +date: 2020-07-15 +tags: +- Raku +- Programming +--- + +For those who don't know, the +[`Config`](https://modules.raku.org/dist/Config:cpan:TYIL) module for the Raku +programming language is a generic class to hold... well... configuration data. +It supports +[`Config::Parser`](https://modules.raku.org/search/?q=Config%3A%3AParser) +modules to handle different configuration file formats, such as `JSON`, `YAML` +and `TOML`. + +Up until now, the module didn't do much for you other than provide an interface +that's generally the same, so you won't need to learn differing methods to +handle differing configuration file formats. It was my first Raku module, and +as such, the code wasn't the cleanest. I've written many new modules since +then, and learned about a good number of (hopefully better) practices. + +For version 3.0, I specifically wanted to remove effort from using the `Config` +module on the developer's end. It should check default locations for +configuration files, so I don't have to rewrite that code in every other module +all the time. Additionally, configuration using environment variables is quite +popular in the current day and age, especially for Dockerized applications. So, +I set out to make an automated way to read those too. + +## The Old Way + +First, let's take a look at how it used to work. Generally, I'd create the +default configuration structure and values first. + +```raku +use Config; + +my $config = Config.new.read({ + foo => "bar", + alpha => { + beta => "gamma", + }, + version => 3, +}); +``` + +And after that, check for potential configuration file locations, and read any +that exist. + +```raku +$config.read($*HOME.add('config/project.toml').absolute); +``` + +The `.absolute` call was necessary because I wrote the initial `Config` version +with the `.read` method not supporting `IO::Path` objects. A fix for this has +existed for a while, but wasn't released, so couldn't be relied on outside of +my general development machines. + +If you wanted to add additional environment variable lookups, you'd have to +check for those as well, and perhaps also cast them as well, since environment +variables are all strings by default. + +## Version 3.0 + +So, how does the new version improve this? For starters, the `.new` method of +`Config` now takes a `Hash` as positional argument, in order to create the +structure, and optionally types *or* default values of your configuration +object. + +```raku +use Config; + +my $config = Config.new({ + foo => Str, + alpha => { + beta => "gamma", + }, + version => 3, +}, :name); +``` + +{{< admonition title="note" >}} +`foo` has been made into the `Str` *type object*, rather than a `Str` *value*. +This was technically allowed in previous `Config` versions, but it comes with +actual significance in 3.0. +{{< / admonition_md >}} + +Using `.new` instead of `.read` is a minor syntactic change, which saves 1 word +per program. This isn't quite that big of a deal. However, the optional `name` +argument will enable the new automagic features. The name you give to `.new` is +arbitrary, but will be used to deduce which directories to check, and which +environment variables to read. + +### Automatic Configuration File Handling + +By setting `name` to the value `project`, `Config` will consult the +configuration directories from the [XDG Base Directory +Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html). +It uses one of my other modules, +[`IO::Path::XDG`](https://modules.raku.org/dist/IO::Path::XDG:cpan:TYIL), for +this, together with +[`IO::Glob`](https://modules.raku.org/dist/IO::Glob:cpan:HANENKAMP). +Specifically, it will check my `$XDG_CONFIG_DIRS` and `$XDG_CONFIG_HOME` (in +that order) for any files that match the globs `project.*` or +`project/config.*`. + +If any files are found to match, they will be read as well, and the +configuration values contained therein, merged into `$config`. It will load the +appropriate `Config::Parser` implementation based on the file's extension. I +intend to add a number of these to future Rakudo Star releases, to ensure most +default configuration file formats are supported out of the box. + +### Automatic Environment Variable Handling + +After this step, it will try out some environment variables for configuration +values. Which variables are checked depends on the structure (and `name`) of +the `Config` object. The entire structure is squashed into a 1-dimensional list +of fields. Each level is replaced by an `_`. Additionally, each variable name +is prefixed with the `name`. Lastly, all the variable names are uppercased. + +For the example `Config` given above, this would result in the following +environment variables being checked. + +```sh +$PROJECT_FOO +$PROJECT_ALPHA_BETA +$PROJECT_VERSION +``` + +If any are found, they're also cast to the appropriate type. Thus, +`$PROJECT_FOO` would be cast to a `Str`, and so would `$PROJECT_ALPHA_BETA`. In +this case that doesn't do much, since they're already strings. But +`$PROJECT_VERSION` would be cast to an `Int`, since it's default value is also +of the `Int` type. This should ensure that your variables are always in the +type you expected them to be originally, no matter the user's configuration +choices. + +## Debugging + +In addition to these new features, `Config` now also makes use of my +[`Log`](https://modules.raku.org/dist/Log:cpan:TYIL) module. This module is +made around the idea that logging should be simple if module developers are to +use it, and the way logs are represented is up to the end-user. When running an +application in your local terminal, you may want more human-friendly logs, +whereas in production you may want `JSON` formatted logs to make it fit better +into other tools. + +You can tune the amount of logging performed using the `$RAKU_LOG_LEVEL` +environment variable, as per the `Log` module's interface. When set to `7` (for +"debug"), it will print the configuration files that are being merged into your +`Config` and which environment veriables are being used as well. + +{{< admonition_md title="note" >}} +A downside is that the application using `Config` for its configuration must +also support `Log` to actually make the new logging work. Luckily, this is +quite easy to set up, and there's example code for this in `Log`'s README. +{{< / admonition >}} + +## Too Fancy For Me + +It could very well be that you don't want these features, and you want to stick +to the old ways as much as possible. No tricks, just plain and simple +configuration handling. This can be done by simply ommitting the `name` +argument to `.new`. The new features depend on this name to be set, and won't +do anything without it. + +Alternatively, both the automatic configuration file handling and the +environment variable handling can be turned off individually using `:!from-xdg` +and `:!from-env` arguments respectively. + +## In Conclusion + +The new `Config` module should result in cleaner code in modules using it, and +more convenience for the developer. If you find any bugs or have other ideas +for improving the module, feel free to send an email to +`https://lists.sr.ht/~tyil/raku-devel`. diff --git a/content/posts/2020/2020-07-19-freebsd-mailserver-part-6-system-updates.md b/content/posts/2020/2020-07-19-freebsd-mailserver-part-6-system-updates.md new file mode 100644 index 0000000..f3d1e89 --- /dev/null +++ b/content/posts/2020/2020-07-19-freebsd-mailserver-part-6-system-updates.md @@ -0,0 +1,341 @@ +--- +title: "FreeBSD Email Server - Part 6: System Updates" +date: 2020-07-19 +tags: +- Email +- FreeBSD +- Tutorial +social: + email: mailto:~tyil/public-inbox@lists.sr.ht&subject=FreeBSD Email Server +--- + +Four years have past, and my FreeBSD email server has keps on running without +any problems. However, some people on IRC have recently been nagging me to +support TLSv1.3 on my mailserver. Since the installation was done 4 years ago, +it didn't do 1.3 yet, just 1.2. I set out to do a relatively simple system +update, which didn't go as smooth as I had hoped. This tutorial post should +help you avoid the mistakes I made, so your updates *will* go smooth. + +{{< admonition title="info" >}} +The rest of this tutorial assumes you're running as the `root` user. +{{< / admonition >}} + +## Preparations + +Before we do anything wild, let's do the obvious first step: backups. Since +this is a FreeBSD server, it uses glorious +[ZFS](https://en.wikipedia.org/wiki/ZFS) as the filesystem, which allows us to +make use of +[snapshots](https://docs.oracle.com/cd/E23824_01/html/821-1448/gbciq.html). +Which subvolumes to make snapshots off depends on your particular setup. In my +case, my actual email data is stored on `zroot/srv`, and all the services and +their configurations are in `zroot/usr/local`. My database's data is stored on +`zroot/postgres/data96`. Additionally, I want to make a snapshot of +`zroot/usr/ports`. + +```txt +zfs snapshot -r zroot/srv@`date +%Y%m%d%H%M%S`-11.0-final +zfs snapshot -r zroot/usr/local@`date +%Y%m%d%H%M%S`-11.0-final +zfs snapshot -r zroot/postgres@`date +%Y%m%d%H%M%S`-11.0-final +zfs snapshot -r zroot/usr/ports@`date +%Y%m%d%H%M%S`-11.0-final +``` + +This will make a snapshot of each of these locations, for easy restoration in +case any problems arise. You can list all your snapshots with `zfs list -t +snapshot`. + +Your server is most likely hosted at a provider, not in your home. This means +you won't be able to just physically access it and retrieve the harddrive if +things go really bad. You might not be able to boot single-user mode either. +Because of this, you might not be able to restore the snapshots if things go +*really* bad. In this case, you should also make a local copy of the important +data. + +The services and their configuration can be recreated, just follow the earlier +parts of this series again. The email data, however, cannot. This is the data +in `/srv/mail`. You can make a local copy of all this data using `rsync`. + +```txt +rsync -av example.org:/srv/mail/ ~/mail-backup +``` + +There's one more thing to do, which I learned the hard way. Set your login +shell to a simple one, provided by the base system. The obvious choice is +`/bin/sh`, but some people may wrongly prefer `/bin/tcsh` as well. During a +major version update, the ABI changes, which will temporarily break most of +the user-installed packages, including your shell. + +```txt +chsh +``` + +{{< admonition title="warning" >}} +Be sure to change the shell for whatever user you're using to SSH into this +machine too, if any! +{{< / admonition >}} + +## Updating the Base System + +With the preparations in place in case things get royally screwed up, the +actual updates can begin. FreeBSD has a dedicated program to handle updating +the base system, `freebsd-update`. First off, fetch any updates, and make sure +all the updates for your current version are applied. + +```txt +freebsd-update fetch install +``` + +Afterwards, set the new system version you want to update to. In my case, this +is `12.1-RELEASE`, but if you're reading this in the future, you most certainly +want a newer version. + +```txt +freebsd-update -r 12.1-RELEASE upgrade +``` + +This command will ask you to review the changes and confirm them as well. It +should generally be fine, but this is your last chance to make any backups or +perform other actions to secure your data! If you're ready to continue, install +the updates to the machine. + +```txt +freebsd-update install +``` + +At this point, your kernel has been updated. Next you must reboot to start +using the new kernel. + +```txt +reboot +``` + +Once the system is back online, you can continue installing the rest of the +updates. + +```txt +freebsd-update install +``` + +When this command finishes, the base system has been updated and should be +ready for use. Next up is updating all the software you installed manually. + +## Updating User-Installed Packages + +Unlike GNU+Linux distributions, FreeBSD has a clear distinction between the +*base system* and *user installed software*. The base system has now been +updated, but everything installed through `pkg` or ports is still at the old +version. If you performed a major version upgrade (say, FreeBSD 11.x to 12.x), +the ABI has changed and few, if any, of the user-installed packages still work. + +### Binary Packages using `pkg` + +Binary packages are the most common packages used. These are the packages +installed through `pkg`. Currently, `pkg` itself doesn't even work. Luckily, +FreeBSD has `pkg-static`, which is a statically compiled version of `pkg` +intended to fix this very problem. Let's fix up `pkg` itself first. + +```txt +pkg-static install -f pkg +``` + +That will make `pkg` itself work again. Now you can use `pkg` to update package +information, and upgrade all packages to a version that works under this +FreeBSD version. + +```txt +pkg update +pkg upgrade +``` + +#### PostgreSQL + +A particular package that was installed through `pkg`, PostgreSQL, just got +updated to the latest version. On FreeBSD, the data directory used by +PostgreSQL is dependent on the version you're running. If you try to list +databases now, you'll notice that the `mail` database used throughout the +tutorial is gone. The data directory is still there, so you *could* downgrade +PostgreSQL again, restart the database, run a `pgdump`, upgrade, restart and +import. However, I find it much cleaner to use FreeBSD jails to solve this +issue. + +{{< admonition title="info" >}} +My original installation used PostgreSQL 9.6, you may need to update some +version numbers accordingly! +{{< / admonition >}} + +I generally put my jails in a ZFS subvolume, so let's create one of those +first. + +```txt +zfs create -o mountpoint=/usr/jails zroot/jails +zfs create zroot/jails/postgres96 +``` + +This will create a new subvolume at `/usr/jails/postgres96`. Using +`bsdinstall`, a clean FreeBSD installation usable by the jail can be set up +here. This command will give you some popups you may remember from installing +FreeBSD initially. This time, you can uncheck *all* boxes, to get the most +minimal system. + +```txt +bsdinstall jail /usr/jails/postgres96 +``` + +When `bsdinstall` finishes, you can configure the jail. This is done in +`/etc/jail.conf`. If this file doesn't exist, you can create it. Make sure the +following configuration block is written to the file. + +```cfg +postgres96 { + # Init information + exec.start = "/bin/sh /etc/rc"; + exec.stop = "/bin/sh /etc/rc.shutdown"; + exec.clean; + + # Set the root path of the jail + path = "/usr/jails/$name"; + + # Mount /dev + mount.devfs; + + # Set network information + host.hostname = $name; + ip4.addr = "lo0|127.1.1.1/32"; + ip6.addr = "lo0|fd00:1:1:1::1/64"; + + # Required for PostgreSQL to function + allow.raw_sockets; + allow.sysvipc; +} +``` + +Now you can start up the jail, so it can be used. + +```txt +service jail onestart postgres96 +``` + +Using the host system's `pkg`, you can install PostgreSQL into the jail. + +```txt +pkg -c /usr/jails/postgres96 install postgresql96-server +``` + +Now you just need to make the data directory available to the jail, which you +can most easily do using +[`nullfs`](https://www.freebsd.org/cgi/man.cgi?query=nullfs&sektion=&n=1). + +```txt +mount -t nullfs /var/db/postgres/data96 /usr/jails/postgres96/var/db/postgres/data96 +``` + +Now everything should be ready for use inside the jail. Let's head on in using +`jexec`. + +```txt +jexec postgres96 +``` + +Once inside the jail, you can start the PostgreSQL service, and dump the `mail` +database. + +```txt +service postgresql onestart +su - postgres +pg_dump mail > ~/mail.sql +``` + +This will write the dump to `/usr/jails/postgres96/var/db/postgres/mail.sql` on +the host system. You can leave the jail and close it down again. + +```txt +exit +exit +service jail onestop postgres96 +``` + +This dump can be imported in your updated PostgreSQL on the host system. +Connect to the database first. + +```txt +su - postgres +psql +``` + +Then, recreate the user, database and import the data from the dump. + +```sql +CREATE USER postfix WITH PASSWORD 'incredibly-secret!'; +CREATE DATABASE mail WITH OWNER postfix; +\c mail +\i /usr/jails/postgres96/var/db/postgres/mail.sql +\q +``` + +The `mail` database is now back, and ready for use! + +### Packages from Ports + +With all the binary packages out of the way, it's time to update packages from +ports. While it is very possible to just go to each port's directory and +manually update each one individually, I opted to use `portupgrade`. This will +need manual installation, but afterwards, we can rely on `portupgrade` to do +the rest. Before doing anything with the ports collection, it should be +updated, which is done using `portsnap`. + +```txt +portsnap fetch extract +``` + +Once this is done, you can go to the `portupgrade` directory and install it. + +```txt +cd /usr/ports/ports-mgmt/portupgrade +make install clean +``` + +Now, to upgrade all other ports. + +```txt +portupgrade -a +``` + +Be sure to double-check the compilation options that you are prompted about! If +you're missing a certain option, you may miss an important feature that is +required for your mailserver to work appropriately. This can be easily fixed by +recompiling, but a few seconds checking now can save you an hour figuring it +out later! + +## Tidying Up + +Now that all user-installed software has been updated too, it's time to +finalize the update by running `freebsd-update` for a final time. + +```txt +freebsd-update install +``` + +You can return to your favourite shell again. + +```txt +chsh +``` + +And you can clean up the ports directories to get some wasted space back. + +```txt +portsclean -C +``` + +I would suggest making a new snapshot as well, now that you're on a relatively +clean and stable state. + +```txt +zfs snapshot -r zroot/srv@`date +%Y%m%d%H%M%S`-12.1-clean +zfs snapshot -r zroot/usr/local@`date +%Y%m%d%H%M%S`-12.1-clean +zfs snapshot -r zroot/postgres@`date +%Y%m%d%H%M%S`-12.1-clean +zfs snapshot -r zroot/usr/ports@`date +%Y%m%d%H%M%S`-12.1-clean +``` + +And that concludes your system update. Your mailserver is ready to be neglected +for years again! diff --git a/content/posts/2020/2020-12-14-raku-modules-in-gentoo-portage.md b/content/posts/2020/2020-12-14-raku-modules-in-gentoo-portage.md new file mode 100644 index 0000000..480d3ae --- /dev/null +++ b/content/posts/2020/2020-12-14-raku-modules-in-gentoo-portage.md @@ -0,0 +1,114 @@ +--- +title: Raku Modules in Gentoo Portage +date: 2020-12-14 +tags: +- Raku +- Gentoo +social: + email: mailto:~tyil/public-inbox@lists.sr.ht&subject=Raku modules in Gentoo's Portage +--- + +The last couple of days I've been taking another look at getting modules for +the Raku programming language into Gentoo's Portage tree. Making new packages +available in Gentoo is incredibly easy with their overlay system. + +The more complex part was Raku's side of things. While most languages just have +a certain directory to drop files in, Raku *should* use a +`CompUnit::Repository` object, which exposes the `.install` method. This is +obviously slower than just copying the files around, but there are merits to +this method. For one, it allows installation of the same module with different +versions, or from different authors. It also handles all the Unicode complexity +for you. + +{{< admonition title="note" >}} +There *is* a +[CompUnit::Repository::FileSystem](https://docs.raku.org/type/CompUnit::Repository::FileSystem) +which would allow me to just copy over files to the right directory, however, I +quite like the ability to have multiple versions of the same module installed. +This is actually something Portage is designed with, too! +{{< admonition >}} + +After an email to the Raku users mailing list, I got some pointers over IRC. I +let these sink in for a couple days, considering how to approach the problem +properly. Then, one night, a solution came to mind, and I set out to test it. + +*It actually worked*. And a similar method should be usable for other +distributions too, such as Debian, OpenSUSE or Archlinux, to create packages +out of Raku modules. This should greatly improve the ability to ship Raku +programs to end-users, without requiring them to learn how Raku's ecosystem is +modeled, or which module manager it uses. + +The most important part of this approach is the +[`module-installer.raku`](https://git.sr.ht/~tyil/raku-overlay/tree/7494c81524ec1845c77dabfbb3303a34eb4b89f4/item/dev-lang/raku/files/module-installer.raku) +program, which is part of `dev-lang/raku::raku-overlay`. It accepts a path to +the module to install. It does not depend on any one module manager, so it can +be used to bootstrap a user-friendly module manager (such as +[`zef`](https://github.com/ugexe/zef/)) for the user. + +```raku +#| Install a Raku module. +sub MAIN ( + #| The path to the Raku module sources. + IO() $path, + + #| The repository to install it in. Options are "site" (ment for + #| user-installed modules), "vendor" (ment for distributions that want + #| to include more modules) and "core" (ment for modules distributed + #| along with Raku itself). + Str:D :$repo = 'site', + + #| Force installation of the module. + Bool:D :$force = True, +) { + CATCH { + default { $_.say; exit 1; } + } + + die "This script should be used by Portage only!" unless %*ENV; + + my $prefix = %*ENV.IO.add('usr/share/perl6').add($repo); + my $repository = CompUnit::Repository::Installation.new(:$prefix); + my $meta-file = $path.add('META6.json'); + my $dist = Distribution::Path.new($path, :$meta-file); + + $repository.install($dist, :$force); +} +``` + +It's a fairly straightforward program. It checks for `$D` to be set in the +environment, which is a variable Portage sets as the destination directory to +install new files in. This directory gets merged into the root filesystem to +finalize installation of any package. + +If `$D` is set, I append the path used by Raku in Gentoo to it, followed by a +repo name. Next I create a `CompUnit::Repository` using this path. This is a +trick to get the files to appear in the right directory for Portage, to +eventually merge them in the system-wide `site` module repo used by Raku. +Additionally, I can use the `CompUnit::Repository`'s `install` method to handle +all the Raku specific parts that I don't want to handle myself. + +This leaves one last issue. By creating this new repo, I also get a couple +files that already exist in the system wide `site` repo. Portage will complain +about possible file collisions and refuse to install the package if these +remain. However, this can be solved rather easily by calling `rm` on these files. + +```txt +rm -- "${D}/usr/share/perl6/site/version" +rm -- "${D}/usr/share/perl6/site/repo.lock" +rm -- "${D}/usr/share/perl6/site/precomp/.lock" +``` + +And with this, my test module, `IO::Path::XDG`, installs cleanly through the +power of Portage, and is usable by all users using the system-wide Raku +installation. + +To make this work for other distributions, the `module-installer.raku` program +should be modified slightly. Most notably, the `$prefix` must be altered to +point towards the right directory, so the files will be installed into whatever +directory will end up being packaged. Other than that, the standard means of +packaging can be followed. + +For the Gentoo users, this overlay is available at +[SourceHut](https://git.sr.ht/~tyil/raku-overlay). It currently holds only +`IO::Path::XDG` (`dev-raku/io-path-xdg`), but you are invited to try it out and +report any issues you may encounter. diff --git a/content/posts/2020/2020-12-15-merging-json-in-postgresql.md b/content/posts/2020/2020-12-15-merging-json-in-postgresql.md new file mode 100644 index 0000000..8d97e50 --- /dev/null +++ b/content/posts/2020/2020-12-15-merging-json-in-postgresql.md @@ -0,0 +1,50 @@ +--- +title: Merging JSON in PostgreSQL +date: 2020-12-15 +tags: +- JSON +- PostgreSQL +- Programming +social: + email: mailto:~tyil/public-inbox@lists.sr.ht&subject=Merging JSON objects in PostgreSQL +--- + +At my `$day-job` we have a lot of `jsonb` in our database. From time to time, I +have to manually run a query to fix something in there. This week was one of +those times. + +While you can pretty much do everything you need with regards to JSON editing +with `jsonb_se`t, I thought it might be nice if I were able to *merge* a given +JSON object into an existing object. This might be cleaner in some situations, +but mostly it is fun to figure it out. And who doesn’t like spending time with +`plpgsql`? + +The way I wanted to have it work is like this: + +```sql +UPDATE user SET properties = jsonb_merge(properties, '{"notifications": {"new_case": false, "new_document": true}}'); +``` + +And this is the eventual function I produced to do it: + +```sql +CREATE OR REPLACE FUNCTION jsonb_merge(original jsonb, delta jsonb) RETURNS jsonb AS $$ + DECLARE result jsonb; + BEGIN + SELECT + json_object_agg( + COALESCE(original_key, delta_key), + CASE + WHEN original_value IS NULL THEN delta_value + WHEN delta_value IS NULL THEN original_value + WHEN (jsonb_typeof(original_value) <> 'object' OR jsonb_typeof(delta_value) <> 'object') THEN delta_value + ELSE jsonb_merge(original_value, delta_value) + END + ) + INTO result + FROM jsonb_each(original) e1(original_key, original_value) + FULL JOIN jsonb_each(delta) e2(delta_key, delta_value) ON original_key = delta_key; + RETURN result; +END +$$ LANGUAGE plpgsql; +``` diff --git a/content/posts/2020/key.asc b/content/posts/2020/key.asc deleted file mode 100644 index 98ddd3a..0000000 Binary files a/content/posts/2020/key.asc and /dev/null differ -- cgit v1.1