From 5c086bc52fa3a226bcf706b1f420a5d98ea377dd Mon Sep 17 00:00:00 2001 From: Patrick Spek Date: Wed, 15 Dec 2021 10:45:58 +0100 Subject: Redo most of the blog in Hugo Missing posts will have to be added later --- content/posts/2016/2016-10-01-on-pastebin.md | 80 +++ content/posts/2016/2016-10-01-on-systemd.md | 286 +++++++++ .../2016/2016-10-25-setup-a-vpn-with-cjdns.md | 212 ++++++ ...2016-10-25-setup-nginx-with-lets-encrypt-ssl.md | 229 +++++++ ...10-31-freebsd-mailserver-part-1-preparations.md | 140 ++++ ...eebsd-mailserver-part-2-mailing-with-postfix.md | 316 +++++++++ ...-freebsd-mailserver-part-3-dovecot-imap-sasl.md | 228 +++++++ ...bsd-mailserver-part-4-message-authentication.md | 159 +++++ ...-31-freebsd-mailserver-part-5-filtering-mail.md | 132 ++++ ...24-freebsd-mailserver-calendars-and-contacts.md | 141 ++++ content/posts/2016/_index.md | 3 + content/posts/2017/2017-09-14-how-to-git.md | 182 ++++++ ...17-09-28-perl6-creating-a-background-service.md | 157 +++++ content/posts/2017/_index.md | 3 + ...kerrank-solutions-python3-and-perl6-part-2.html | 709 +++++++++++++++++++++ content/posts/2018/_index.md | 3 + content/posts/2021/2021-05-13-a-new-irc-client.md | 85 +++ .../posts/2021/2021-05-22-raku-on-libera-chat.md | 35 + ...managing-docker-compose-projects-with-openrc.md | 194 ++++++ content/posts/2021/_index.md | 3 + content/posts/_index.md | 24 + 21 files changed, 3321 insertions(+) create mode 100644 content/posts/2016/2016-10-01-on-pastebin.md create mode 100644 content/posts/2016/2016-10-01-on-systemd.md create mode 100644 content/posts/2016/2016-10-25-setup-a-vpn-with-cjdns.md create mode 100644 content/posts/2016/2016-10-25-setup-nginx-with-lets-encrypt-ssl.md create mode 100644 content/posts/2016/2016-10-31-freebsd-mailserver-part-1-preparations.md create mode 100644 content/posts/2016/2016-10-31-freebsd-mailserver-part-2-mailing-with-postfix.md create mode 100644 content/posts/2016/2016-10-31-freebsd-mailserver-part-3-dovecot-imap-sasl.md create mode 100644 content/posts/2016/2016-10-31-freebsd-mailserver-part-4-message-authentication.md create mode 100644 content/posts/2016/2016-10-31-freebsd-mailserver-part-5-filtering-mail.md create mode 100644 content/posts/2016/2016-11-24-freebsd-mailserver-calendars-and-contacts.md create mode 100644 content/posts/2016/_index.md create mode 100644 content/posts/2017/2017-09-14-how-to-git.md create mode 100644 content/posts/2017/2017-09-28-perl6-creating-a-background-service.md create mode 100644 content/posts/2017/_index.md create mode 100644 content/posts/2018/2018-10-11-hackerrank-solutions-python3-and-perl6-part-2.html create mode 100644 content/posts/2018/_index.md create mode 100644 content/posts/2021/2021-05-13-a-new-irc-client.md create mode 100644 content/posts/2021/2021-05-22-raku-on-libera-chat.md create mode 100644 content/posts/2021/2021-06-04-managing-docker-compose-projects-with-openrc.md create mode 100644 content/posts/2021/_index.md create mode 100644 content/posts/_index.md (limited to 'content/posts') diff --git a/content/posts/2016/2016-10-01-on-pastebin.md b/content/posts/2016/2016-10-01-on-pastebin.md new file mode 100644 index 0000000..cb54542 --- /dev/null +++ b/content/posts/2016/2016-10-01-on-pastebin.md @@ -0,0 +1,80 @@ +--- +date: 2016-10-01 +title: On Pastebin +tags: +- Pastebin +- Security +- Cloudflare +- Privacy +--- + +Pastebin offers itself as a gratis paste service. Although it is probably the +most well known option out there, it is certainly not the best. + +## The security issue +Pastebin has a couple of issues that harm the visitor's security. This on +itself should be considered such a bad practice that no-one should consider +their service at all. + +### Cloudflare +Cloudflare is a [MITM][mitm]. It completely breaks the secure chain of TLS on +the web, and should not be used. Any service still using Cloudflare should be +shunned. There is [another article][cloudflare] on this site which has more +information on this specific issue. In addition, Cloudflare can be considered a +privacy issue for the same reasons, as is detailed below. + +### Advertisements +Another issue with regards to security on pastebin are the advertisements. +While it can be argued that "they need to make money somehow", using ads always +seems like the worst possible solution. Especially given the way they're +serving it. The past couple years have shown that advertisements on the web are +easily abused to serve malware to good netizens who decided to not block all +ads. + +A rant on the state of ads might be appropriate, but this article is +specifically about Pastebin, so I will just keep it at "third party +advertisements are a security risk, avoid sites who use them" + +## The privacy issue +Apart from their security issues, Pastebin also offers some privacy issues. As +stated above, they make use of Cloudflare. This means that whenever you visit +them, Cloudflare takes note of this. They may even decide that you need to +perform some additional tasks in order to be allowed to the resource. This +doesn't happen to most users, but if you're using any anonymization practices, +this will happen almost every time you visit a site behind Cloudflare. + +In addition to telling Cloudflare, you will also tell another third party, +Google, in case this "additional step" is required. This is done via the new +reCaptcha system which will inform Google of almost every detail of your +browser and the behaviour used to solve the puzzle. Incredibly useful for +fingerprinting you accross multiple locations. + +### Then there is Tor +But, if you're using an anonymization proxy such as Tor, even if you do not +care about the Cloudflare issue, and you solve the "security check" presented +to you, Pastebin still refuses to offer you their service. If they are going to +refuse you service, they should tell you up front, not after you have already +informed two other harmful parties of your attempt of accessing the resource. + +Actually, they should not. They should simply not require you to give up your +privacy and serve you the content you were looking for. Blocking resources to a +certain group of users is simply censorship, and should not be the status quo +on the free internet. + +## Alternatives +Luckily, there are plenty of alternatives that do not treat their users with +such disrespect. I ask anyone who is still using Pastebin to stop doing this, +and use any of the alternatives. + +* [0bin.net](https://0bin.net/) +* [cry.nu][crynu] (works like termbin: `nc cry.nu 9999 < file`) +* [ix.io][ix] +- [p.tyil.nl][tyilnl] (works like termbin: `nc p.tyil.nl 9999 < file`) + +[cloudflare]: /articles/on-cloudflare/ +[crynu]: https://cry.nu +[hastebin]: http://hastebin.com +[ix]: http://ix.io/ +[mitm]: https://en.wikipedia.org/wiki/Man-in-the-middle_attack +[termbin]: http://termbin.com +[tyilnl]: / diff --git a/content/posts/2016/2016-10-01-on-systemd.md b/content/posts/2016/2016-10-01-on-systemd.md new file mode 100644 index 0000000..9bd46d8 --- /dev/null +++ b/content/posts/2016/2016-10-01-on-systemd.md @@ -0,0 +1,286 @@ +--- +date: 2016-10-01 +title: On Systemd +tags: +- Systemd +- Security +- GNU+Linux +--- + +Systemd once presented itself as being the next generation init system for +GNU+Linux. When the project started it seemed to be headed in a good direction. +Unfortunately, it quickly became clear that systemd's goal was not only to +bring you a quick, new init system. It planned to do so much more. This was +part of the plan, since init systems were generally considered to be in a bad +state overall it was quickly accepted by most mainstream GNU+Linux +distributions. What was at first only an init system became so much more: +systemd-logind was made to manage tty's, systemd-resolvd was added to act as a +caching DNS server. Even networking was added with systemd-networkd to manage +network interfaces. + +**DISCLAIMER**: Systemd is a fast moving project, this may result in +information here to becoming outdated. If you find any information that is no +longer correct, please contact me. You can find my contact details [on my +homepage][tyil]. + +## Technical issues +### Security +From experience, we have seen that systemd's creator, Lennart Poettering, will +try to assimilate any functionality he can find and add it into systemd. This +causes systemd to have a large surface area of attack, adding to and magnifying +security attack vectors. An init system should be exactly the opposite. To +compound this issue, we have bugs like [the user-level DoS][systemd-dos], +which seem to indicate that the software is hardly tested or written by +programmers who don't use best practices. + +### POSIX +POSIX compliance. Systemd developers seem to detest it. Their common argument +against retaining POSIX compliance is that "systemd must break POSIX compliance +in order to further the development of GNU+Linux userland utilities". While +this may be true in some sense, it is a very bad idea to ignore POSIX +altogether. + +POSIX is one of the reasons that most applications running on GNU+Linux and +other Unix like systems are very portable. It's a standard that most OS's and +distro's try to meet, making it easy to port software. + +[natermeer on Reddit][reddit-natermeer] said +> POSIX has almost no relevance anymore. +> +> [...] +> +> If you care about portability you care about it running on OS X and Windows +> as well as your favorite \*nix system. POSIX gains you nothing here. A lot +> of the APIs from many of these systems will resemble POSIX closely, but if +> you don't take system-specific differences into account you are not going +> to accomplish much. + +> I really doubt that any Init system from any Unix system uses only POSIX +> interfaces, except maybe NetBSD. All of them are going to use scripts and +> services that are going to be running commands that use kernel-specific +> features at some point. Maybe a init will compile and can be executed on +> pure POSIX api, but that is a FAR FAR cry from actually having a booted and +> running system. + +Which was replied to by [aidanjt][reddit-aidanjt] +> Wrong, both OS X and Windows have POSIX support, although Window's is emulated, +> OS X certainly is not, it's fully POSIX compliant. and b) POSIX doesn't have to +> work identically everywhere, it only has to be more or less the same in most +> places and downstream can easily patch around OS-specific quirks. Even +> GNU/Linux and a bunch of the BSDs are merely regarded as 'mostly' POSIX +> compliant, after all. But if you ignore POSIX entirely, there's ZERO hope of +> portability. +> +> Actually sysvinit is very portable, init.c only has 1 single Linux header which +> has been #ifdef'ed, to handle the three-finger-salute. You see, init really +> isn't that complicated a programme, you tell the kernel to load it after it's +> done it's thing, init starts, and loads distro scripts which starts userspace +> programmes to carry on booting. No special voodoo magic is really required. +> POSIX is to thank for that. POSIX doesn't need to be the only library eva, it +> only needs to handle most of the things you can't do without, without having to +> directly poke at kernel-specific interfaces. +> +> This is why with POSIX, we can take a piece of software written for a PPC AIX +> mainframe, and make it work on x86 Linux without a complete rewrite, usually +> with only trivial changes. + +### Dependencies and unportability +Another common issue with systemd is that applications have started to +needlessly depend on it, forcing systemd onto users that do not wish to use +systemd for obvious reasons outlined here, reasons outside of this article, or +simply being unable to use it. Because systemd complies to no cross-platform +standard and uses many features only available in recent Linux version, it's +either very hard or impossible to implement systemd in some circumstances. + +The list of features it requires is no small one either, as you can see in the +list [posted by ohset][reddit-ohet]: + +- `/dev/char` +- `/dev/disk/by-label` +- `/dev/disk/by-uuid` +- `/dev/random` +- `/dev/rtc` +- `/dev/tty0` +- `/proc/$PID/cgroup` +- `/proc/${PID}/cmdline` +- `/proc/${PID}/comm` +- `/proc/${PID}/fd` +- `/proc/${PID}/root` +- `/proc/${PID}/stat` +- `/proc/cmdline` +- `/sys/class/dmi/id` +- `/sys/class/tty/console/active` +- `BTRFS_IOC_DEFRAG` +- `CLONE_xxx` +- `F_SETPIPE_SZ` +- `IP_TRANSPORT` +- `KDSKBMODE` +- `O_CLOEXEC` +- `PR_CAPBSET_DROP` +- `PR_GET_SECUREBITS` +- `PR_SET_NAME` +- `PR_SET_PDEATHSIG` +- `RLIMIT_RTPRIO` +- `RLIMIT_RTTIME` +- `SCHED_RESET_ON_FORK` +- `SOCK_CLOEXEC` +- `TIOCLINUX` +- `TIOCNXCL` +- `TIOCVHANGUP` +- `VT_ACTIVATE` +- `\033[3J` +- `audit` +- `autofs4` +- `capabilities` +- `cgroups` +- `fanotify` +- `inotify` +- `ionice` +- `namespaces` +- `oom score adjust` +- `openat()` and friends +- `selinux` +- `settimeofday()` and its semantics +- `udev` +- `waitid()` +- numerous GNU APIs like `asprintf` + +This made [Gnome][gnome] unavailable for a long time to BSD users and GNU+Linux +users who wanted to remain with a sane and proven system. Utilities like +[Gummiboot][gummiboot] are now being absorbed by systemd too. It is only a +matter of time before you can no longer use this utility without a systemd init +behind it. There are too many examples of software to list, which are being +assimilated or made unavailable by lazy or bad developers who choose to depend +on systemd for whatever reason. + +### Speed +The main selling point many systemd users hail all the time, is speed. They +place an unusual high amount of value on being a couple seconds faster on boot. +Systemd gains this speed gain by using parallelization, and many think this is +unique to systemd. Luckily for those who want to stick to a more sane system, +this is false. Other init systems, such as [OpenRC][openrc], used by +[Funtoo][funtoo], and [runit][runit], used by [Voidlinux][voidlinux] both +support parallel startup of services. Both these systems use small and +effective shell scripts for this, and support startup dependencies and the +like. Systemd brings nothing new to the init world, it just advertises these +features more agressively. + +### Modularity +The UNIX principle, *make an application perform one task very well*, seems to +be very unpopular among systemd developers. This principle is one of the +reasons why UNIX based systems have gotten so popular. Yet, the systemd +developers seem to despise this principle, and even try to argue that systemd +actually is modular because **it compiles down to multiple binaries**. This +shows a lack of understanding, which would make most users uneasy when they +consider that these people are working on one of the most critical pieces of +their OS. + +The technical problem this brings is that it is very hard to use systemd with +existing tools. `journald` for instance doesn't just output plain text you can +easily filter through, save or apply to a pager. I decides for you how to +represent this information, even if this might be an ineffective way to go +about it. + +### Binary logs +Hailed by systemd users and developers as a more efficient, fast and secure way +to store your logs, it is yet another middle finger to the UNIX principles, +which state that documents intended for the user should be human readable. +Binary logs are exactly not that. This forces you to use the tools bundled with +systemd, instead of your preferred solution. This means you need a system with +systemd in order to read your logs, which you generally need the most when the +system that generated it crashed. Thanks to systemd, these logs are now useless +unless you have another systemd available for it. + +These logs are also very fragile. It is a common "issue" to have corrupted logs +when using systemd. Corrupted is here within quotes because the systemd +developers do not recognize this as a bug. Instead, you should just rotate your +logs and hope it does not happen again. + +The usual counter to this issue is that you *can* tell systemd to use another +logger. However, this does not stop `journald` from processing them first or +just not having `journald` at all. As systemd is not modular, you will always +have all the pieces installed. It should also be noted that this is a +*workaround*, not a fix to the underlying problem. + +## Political issues +### Aggressively forced upon users +A point that has made many systemd opponents very wary of this huge piece of +software is the way it was introduced. Unlike most free software packages, +systemd was forced into the lives of many users by getting hard dependencies on +them, or simply absorbing a critical piece of software by the use of political +power. The two most prominent pieces of software where this has happened are +[Gnome][gnome] and [`udev`][udev]. + +The Gnome developers made a hard dependency on systemd. This in effect made +every gnome user suddenly require systemd. As a result, FreeBSD had to actually +drop Gnome for a while, as systemd does not run outside of GNU+Linux. + +The other, `udev`, was a critical piece of software to manage devices in +GNU+Linux. Sadly, some political power was shown by Red Hat and `udev` got +absorbed into systemd. Luckily, the Gentoo guys saw this issue and tried to +resolve it. As the systemd developers dislike anything that's not systemd +itself, they stubbornly refused the patches from the Gentoo folks which would +keep `udev` a single component (and thus usable without systemd). In the end, +the Gentoo developers forked `udev` into [`eudev`][eudev]. + +### Unwillingness to cooperate +Whenever someone from outside the systemd fangroups steps up to actually +improve systemd in whatever way, the systemd devs seem to be rather +uncooperative. It is not uncommon for developers from other projects to make a +change in order for their projects (and usually others) to improve. This +removes a lot of the cost for the systemd maintainers to deal with all the +issues created they are creating. + +There are some references to the systemd developers being against changes that +might make systemd less of a problem, but these changes are usually denied with +petty excuses. + +- https://lists.freedesktop.org/archives/systemd-devel/2012-June/005466.html +- https://lists.freedesktop.org/archives/systemd-devel/2012-June/005507.html + +## How to avoid it +### Choosing a better OS or distribution +Nowadays, the only way to avoid it without too much trouble, is by simply +choosing a better OS or distro that does not depend on systemd at all. There +are a few choices for this: + +- \*BSD ([FreeBSD][freebsd], [OpenBSD][openbsd], and others) +- [Devuan][devuan] +- [Funtoo][funtoo] +- [Voidlinux][voidlinux] + +It is a shame that it renders a very large chunk of the GNU+Linux world +unavailable when choosing a distro, but they have chosen laziness over a +working system. The only way to tell them at this point that they have made a +wrong decision, is to simply stop using these distros. + +### More links + +- [Broken by design: systemd][broken-systemd] +- [Without systemd][without-systemd] +- [systemd is the best example of Suck][suckless-systemd] +- [Thoughts on the systemd root exploit][agwa-systemd-root-exploit] (In response to [CVE-2016-10156][cve-2016-10156]) +- ["systemd: Please, No, Not Like This"](https://fromthecodefront.blogspot.nl/2017/10/systemd-no.html) + +[agwa-systemd-root-exploit]: https://www.agwa.name/blog/post/thoughts_on_the_systemd_root_exploit +[broken-systemd]: http://ewontfix.com/14/ +[cve-2016-10156]: http://www.openwall.com/lists/oss-security/2017/01/24/4 +[devuan]: https://devuan.org/ +[eudev]: https://wiki.gentoo.org/wiki/Eudev +[freebsd]: https://www.freebsd.org/ +[funtoo]: http://www.funtoo.org/Welcome +[gentoo]: https://gentoo.org +[gnome]: http://www.gnome.org/ +[gummiboot]: https://en.wikipedia.org/wiki/Gummiboot_(software) +[openbsd]: https://www.openbsd.org/ +[openrc]: https://en.wikipedia.org/wiki/OpenRC +[reddit-aidanjt]: https://www.reddit.com/r/linux/comments/132gle/eli5_the_systemd_vs_initupstart_controversy/c72saay +[reddit-natermeer]: https://www.reddit.com/r/linux/comments/132gle/eli5_the_systemd_vs_initupstart_controversy/c70hrsq +[reddit-ohet]: https://www.reddit.com/r/linux/comments/132gle/eli5_the_systemd_vs_initupstart_controversy/c70cao2 +[runit]: http://smarden.org/runit/ +[suckless-systemd]: http://suckless.org/sucks/systemd +[systemd-dos]: https://github.com/systemd/systemd/blob/b8fafaf4a1cffd02389d61ed92ca7acb1b8c739c/src/core/manager.c#L1666 +[tyil]: http://tyil.work +[udev]: https://wiki.gentoo.org/wiki/Eudev +[voidlinux]: http://www.voidlinux.eu/ +[without-systemd]: http://without-systemd.org/wiki/index.php/Main_Page diff --git a/content/posts/2016/2016-10-25-setup-a-vpn-with-cjdns.md b/content/posts/2016/2016-10-25-setup-a-vpn-with-cjdns.md new file mode 100644 index 0000000..52d9237 --- /dev/null +++ b/content/posts/2016/2016-10-25-setup-a-vpn-with-cjdns.md @@ -0,0 +1,212 @@ +--- +date: 2016-10-25 +title: Setup a VPN with cjdns +tags: +- Tutorial +- VPN +- cjdns +- GNU+Linux +- FreeBSD +--- + +In this tutorial I will outline a simple setup for a [VPN][vpn] using +[`cjdns`][cjdns]. Cjdns will allow you to setup a secure mesh vpn which uses +IPv6 internally. + +## Requirements +For this tutorial, I have used two client machines, both running Funtoo. A +FreeBSD 11 server is used as a global connection point. + +You are ofcourse able to use any other OS or distro supported by cjdns, but you +may have to update some steps to work on your environment in that case. + +## Installation of the server +### Dependencies +Before you can begin, we need some dependencies. There's only two of those, and +they are available via `pkg` to make it even easier. Install them as follows: + +``` +pkg install gmake node +``` + +### Compiling +Next up is getting the cjdns sources and compile these, as cjdns is not +available as a prebuilt package: + +``` +mkdir -p ~/.local/src +cd $_ +git clone https://github.com/cjdelisle/cjdns.git cjdns +cd $_ +./do +``` + +To make the compiled binary available system-wide so we can use it with a +system service, copy it to `/usr/local/bin` and rehash to make it available as +a direct command: + +``` +cp cjdroute /usr/local/bin/. +hash -r +``` + +### Configuring +Cjdns provides a flag to generate the initial configuration. This will provide +you with some sane defaults where only a couple of small changes are needed to +make it work properly. Generate these defaults with `--genconf`: + +``` +(umask 177 && cjdroute --genconf > /usr/local/etc/cjdroute.conf) +``` + +The umask will make all following commands write files using `600` permissions. +This makes sure the config file is not readable by people who shouldn't be able +to read it. Be sure to check wether the owner of the file is `root`! + +Now you can start actually configuring the node to allow incoming connections. +You have to find the `authorizedPasswords` array in the `cjdroute.conf` file +and remove the contents of it. Then you can add your own machines in it. This +guide follows the assumption of two clients, so the config for two clients will +be shown here. You can add more clients if you wish, ofcourse. + +```json +"authorizedPasswords": +[ + {"password": "aeQu6pa4Vuecai3iebah7ogeiShaeDaepha6Mae1yooThoF0oa0Eetha9oox", "user": "client_1"}, + {"password": "aiweequuthohkahx4tahLohPiezee9OhweiShoNeephe0iekai2jo9Toorah", "user": "client_2"}, +] +``` + +If you need to generate a password, you can make use of the tool `pwgen`, +available at your local package manager. You can then generate new passwords by +running `pwgen 60 -1`. Change the `60` around if you want passwords of a +different size. + +### Adding a startup service +rcinit has deceptively easy scripts to make applications available as services. +This in turn allows you to enable a service at startup. This way you can make +sure cjdns starts whenever the server boots. You can copy the following +contents directly into `/usr/local/etc/rc.d/cjdroute`: + +```sh +#!/bin/sh + +# PROVIDE: cjdroute +# KEYWORD: shutdown + +# +# Add the following lines to /etc/rc.conf to enable cjdroute: +# +#cjdroute_enable="YES" + +. /etc/rc.subr + +name="cjdroute" +rcvar="cjdroute_enable" + +load_rc_config $name + +: ${cjdroute_config:=/usr/local/etc/cjdroute.conf} + +command="/usr/local/bin/cjdroute" +command_args=" < ${cjdroute_config}" + +run_rc_command "$1" +``` + +Afterwards, you must enable the service in `/etc/rc.conf.local` like follows: + +``` +echo 'cjdroute_enable="YES"' >> /etc/rc.conf.local +``` + +## Installation of the clients +### Dependencies +The dependencies are still on `gmake` and `node`, so simply install those on +your clients. This guide assumes using Funtoo for the clients, so installation +would go as follows: + +``` +emerge gmake nodejs +``` + +### Compiling +Compilation is the same as for the server, so check back there for more +information if you have already forgotten. + +### Configuring +Generating the base configuration is again done using `cjdroute --genconf`, +just like on the server. On Funtoo, config files generally reside in `/etc` +instead of `/usr/local/etc`, so you should set the filepath you write the +configuration to accordingly: + +``` +cjdroute --genconf > /etc/cjdroute.conf +``` + +Setting up the connections differs as well, as the clients are going to make an +outbound connection to the server, which is configured to accept inbound +connections. + +You should still clean the `authorizedPasswords` array, as it comes with a +default entry that is uncommented. + +Now you can setup outbound connections on the clients. You set these up in the +`connectTo` block of `cjdroute.conf`. For this example, the IP 192.168.1.1 is +used to denote the server IP. Unsurprisingly, you should change this to your +server's actual IP. You can find the `publicKey` value at the top of your +server's `cjdroute.conf` file. + +On client 1, put the following in your `cjdroute.conf`: + +```json +"connectTo": +{ + "192.168.1.1:9416": + { + "login": "client_1", + "password": "aeQu6pa4Vuecai3iebah7ogeiShaeDaepha6Mae1yooThoF0oa0Eetha9oox", + "publicKey": "thisIsJustForAnExampleDoNotUseThisInYourConfFile_1.k" + } +} +``` + +On client 2: + +```json +"connectTo": +{ + "192.168.1.1:9416": + { + "login": "client_2", + "password": "aiweequuthohkahx4tahLohPiezee9OhweiShoNeephe0iekai2jo9Toorah", + "publicKey": "thisIsJustForAnExampleDoNotUseThisInYourConfFile_1.k" + } +} +``` + +That is all for configuring the nodes. + +### Adding a startup service +You probably want cjdroute to run at system startup so you can immediatly use +your VPN. For openrc based systems, such as Funtoo, cjdns comes with a ready to +use service script. To make this available to your system, copy it over to the +right directory: + +``` +cp ~/.local/src/cjdns/contrib/openrc/cjdns /etc/init.d/cjdroute +``` + +Now add the service to system startup and start the service: + +``` +rc-update add cjdroute default +rc-service cjdroute start +``` + +That should be sufficient to get cjdns up and running for an encrypted VPN. You +can find the IPs of each of your systems at the top of your `cjdroute.conf` +files, in the `ipv6` attribute. + +[cjdns]: https://github.com/cjdelisle/cjdns +[vpn]: https://en.wikipedia.org/wiki/Virtual_private_network diff --git a/content/posts/2016/2016-10-25-setup-nginx-with-lets-encrypt-ssl.md b/content/posts/2016/2016-10-25-setup-nginx-with-lets-encrypt-ssl.md new file mode 100644 index 0000000..8c7caa0 --- /dev/null +++ b/content/posts/2016/2016-10-25-setup-nginx-with-lets-encrypt-ssl.md @@ -0,0 +1,229 @@ +--- +date: 2016-10-25 +title: Setup nginx with Let's Encrypt SSL +tags: +- Tutorial +- LetsEncrypt +- Nginx +- SSL +- Encryption +--- + +This is a small tutorial to setup nginx with Let's Encrypt on a FreeBSD server +to host a static site. + +## Install required software +First you have to install all the packages we need in order to get this server +going: + +```sh +pkg install nginx py27-certbot +``` + +## Configure nginx +Next is nginx. To make life easier, you should configure nginx to read all +configuration files from another directory. This allows you to store all your sites in +separate configurations in a separate directory. Such a setup is a regular site on +nginx installations on GNU+Linux distributions, but not default on FreeBSD. + +Open up `/usr/local/etc/nginx/nginx.conf` and make the contents of the `http` +block look a as follows: + +```nginx +http { + include mime.types; + default_type application/octet-stream; + + sendfile on; + #tcp_nopush on; + + keepalive_timeout 65; + + # default paths + index index.html; + + # disable gzip - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=773332 + gzip off; + + # default ssl settings + ssl_session_cache shared:SSL:1m; + ssl_session_timeout 5m; + ssl_ciphers HIGH:!aNULL:!MD5:!AES128:!CAMELLIA128; + ssl_protocols TLSv1.2; + ssl_prefer_server_ciphers on; + ssl_dhparam /usr/local/etc/ssl/dhparam.pem; + + # default logs + error_log /var/log/nginx/error.log; + access_log /var/log/nginx/acces.log; + + # default server + server { + listen 80; + server_name localhost; + + location / { + root /usr/local/www/nginx; + index index.html index.htm; + } + + error_page 404 /404.html; + error_page 500 502 503 504 /50x.html; + + location = /50x.html { + root /usr/local/www/nginx-dist; + } + } + + # include site-specific configs + include sites/*.conf; +} +``` + +This sets default ssl settings for all server blocks that enable ssl. Note that +these are settings I use, and are in no way guaranteed to be perfect. I did some +minor research on these settings to get an acceptable rating on +[SSL Labs][ssllabs]. However, security is not standing still, and there is a +decent chance that my settings will become outdated. If you have better settings +that result in a safer setup, please [contact me][contact]. + +### Setup HTTP +Due to the way `certbot` works, you need a functioning web server. Since there +is no usable cert yet, this means hosting a HTTP version first. The tutorial +assumes a static HTML website to be hosted, so the configuration is pretty +easy. + +Put the following in `/usr/local/etc/nginx/sites/domain.conf`: + +```nginx +# static HTTP +server { + # listeners + listen 80; + server_name domain.tld www.domain.tld; + + # site path + root /srv/www/domain/_site; + + # / handler + location / { + try_files $uri $uri/ =404; + } + + # logs + error_log /var/log/nginx/error.log; + access_log /var/log/nginx/access.log; +} +``` + +If your site's sources do not reside in `/srv/www/domain/_site`, change the +path accordingly. This guide will continue using this path for all examples, so +be sure to modify this where needed. In the same vein, the domain `domain.tld` +will be used. Modify this to your own domain. + +### Start nginx +Nginx is now configured to host a single site over HTTP. Now is the time to enable +the nginx service. Execute the following: + +```sh +echo 'nginx_enable="YES"' >> /etc/rc.conf.local +``` + +This will enable nginx as a system service. On reboots, it will be started +automatically. You can also start it up without rebooting by running the +following: + +```sh +service nginx start +``` + +## Configure Let's Encrypt +Nginx is now running as your web server on port 80. Now you can request Let's +Encrypt certificates using `certbot`. You can do so as follows: + +```sh +certbot certonly --webroot -w /srv/www/domain/_site -d domain.tld -d www.domain.tld +``` + +In case you want to add any sub domains, simply add more `-d sub.domain.tld` +arguments at the end. If the DNS entries for the domains resolve properly, and +no unexpected errors occur on the Let's Encrypt side, you should see a message +congratulating you with your new certs. + +If your domains do not resolve correctly, `certbot` will complain about this. +You will have to resolve your DNS issues before attempting again. + +If `certbot` complains about an unexpected error on their side, wait a couple +minutes and retry the command. It should work, eventually. + +Once `certbot` has ran without errors, the required files should be available +in `/usr/local/etc/letsencrypt/live/domain.tld`. + +## Configure nginx with SSL +The certificate has been issued and base nginx is running. Now is the time to +re-configure your site on nginx to host the HTTPS version of your site instead. +Open up `/usr/local/etc/nginx/sites/domain.conf` again, and make the contents +look like the following: + +```nginx +# redirect HTTPS +server { + # listeners + listen 80; + server_name domain.tld *.domain.tld; + + # redirects + return 301 https://$host$request_uri; +} + +# static HTTPS +server { + # listeners + listen 443 ssl; + server_name domain.tld www.domain.tld; + + # site path + root /srv/www/domain/_site; + + # / handler + location / { + try_files $uri $uri/ =404; + } + + # enable HSTS + add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload"; + + # keys + ssl_certificate /usr/local/etc/letsencrypt/live/domain.tld/fullchain.pem; + ssl_certificate_key /usr/local/etc/letsencrypt/live/domain.tld/privkey.pem; +} +``` + +Do not forget to update all the paths to match your setup! + +As a final step, you should generate the dhparam file. This is to avoid the +issues as described on [Weak DH][weakdh]. + +```sh +openssl gendh -out /usr/local/etc/ssl/dhparam.pem 4096 +``` + +Be aware that this step can take a **very** long time. On the test machine I +used to test this tutorial, with 1 core and 1 GB ram, it took nearly 1 hour to +generate this file. + +### Reload nginx +The final step is to reload the nginx configuration so it hosts the SSL version +of your site, and redirects the HTTP version to the HTTPS version. To do this, +simply run + +```sh +service nginx reload +``` + +That should be all to get your site working with HTTP redirecting to HTTPS, and +HTTPS running using a gratis Let's Encrypt certificate. + +[contact]: https://www.tyil.work/ +[ssllabs]: https://www.ssllabs.com/ssltest/analyze.html?d=tyil.work&latest +[weakdh]: https://weakdh.org/ diff --git a/content/posts/2016/2016-10-31-freebsd-mailserver-part-1-preparations.md b/content/posts/2016/2016-10-31-freebsd-mailserver-part-1-preparations.md new file mode 100644 index 0000000..9fc04e7 --- /dev/null +++ b/content/posts/2016/2016-10-31-freebsd-mailserver-part-1-preparations.md @@ -0,0 +1,140 @@ +--- +date: 2016-10-31 07:57:50 +title: "FreeBSD email server - Part 1: Preparations" +tags: +- Tutorial +- FreeBSD +- Email +--- + +This tutorial is devised into multiple chapters to make it more manageable, and +to be able to better explain why certain parts are needed. + +The tutorial is created out of experience setting up my own email server. I have +read through quite a lot of documentation so you do not have to. Nonetheless, I +would recommend doing so. Email business is a tricky one, with a lot of moving +parts that have to fit into each other. Knowing how exactly each part works will +greatly help understanding why they are needed in a proper email server. +Besides that, it will make your life a lot more enjoyable if you want to tweak +some things after this tutorial. + +To kick off, some preparations should be done before you start on setting up +your own email server. + +## DNS setup +Some DNS setup is required for mail. Most importantly, the MX records of a +domain. Be sure you have a domain available, otherwise, get one. There are +plenty of registrars and the price is pretty low for most domains. If you want +to look hip, get a `.email` TLD for your email server. + +For the DNS records themselves, make sure you have an `A` record pointing to +the server IP you're going to use. If you have an IPv6 address, set up an +`AAAA` record as well. Mail uses the `MX` DNS records. Make one with the value +`10 @`. If you have multiple servers, you can make MX records for these as +well, but replace the `10` with a higher value each time (`20`, `30`, etc). +These will be used as fallback, in case the server with pointed to by the `10` +record is unavailable. + +## PostgreSQL +Next up you will have to install and configure [PostgreSQL][postgres]. Although +using a database is not required, this tutorial will make use of one. Using a +database makes administration easier and allows you to add a pretty basic web +interface for this task. + +### Installation +Since the tutorial uses FreeBSD 11, you can install PostgreSQL easily by running + +``` +pkg install postgresql96-server +``` + +### Starting up +In order to start Postfix, you should enable the system service for it. This +way, `service` can be used to easily manage it. In addition, it will start +automatically on boot. + +``` +echo 'postgresql_enable="YES"' >> /etc/rc.conf.local +service postgresql start +``` + +### Database initialization +Since PostgreSQL is a little different than the more popular [MySQL][mysql], I +will guide you through setting up the database as well. To begin, switch user +to `postgres`, which is the default administrative user for PostgreSQL. Then +simply open up the PostgreSQL CLI. + +``` +su postgres +psql +``` + +Once you are logged in to PostgreSQL, create a new user which will hold +ownership of the database and make a database for this user. + +```sql +CREATE USER postfix WITH PASSWORD 'incredibly-secret!'; +CREATE DATABASE mail WITH OWNER postfix; +``` + +Once this is done, create the tables which will hold some of our configuration +data. + +#### domains +```sql +CREATE TABLE domains ( + name VARCHAR(255) NOT NULL, + PRIMARY KEY (name) +); +``` + +#### users +```sql +CREATE TABLE users ( + local VARCHAR(64) NOT NULL, + domain VARCHAR(255) NOT NULL, + password VARCHAR(128) NOT NULL, + PRIMARY KEY (local, domain), + FOREIGN KEY (domain) REFERENCES domains(name) ON DELETE CASCADE +); +``` + +#### aliases +```sql +CREATE TABLE aliases ( + domain VARCHAR(255), + origin VARCHAR(256), + destination VARCHAR(256), + PRIMARY KEY (origin, destination), + FOREIGN KEY (domain) REFERENCES domains(name) ON DELETE CASCADE +); +``` + +## Let's Encrypt +### Installation +Installing the [Let's Encrypt][letsencrypt] client is just as straightforward +as the PostgreSQL database, using `pkg`. + +``` +pkg install py27-certbot +``` + +### Getting a certificate +Requesting a certificate requires your DNS entries to properly resolve. If they +do not resolve yet, Let's Encrypt will bother you with errors. If they do +resolve correctly, use `certbot` to get your certificate. + +``` +certbot certonly --standalone -d domain.tld +``` + +## Conclusion +This should be everything required to get started on setting up your own email +server. Continue to [part 2][part-2] of this series to start setting up +Postfix. + +[freebsd]: https://www.freebsd.org/ +[letsencrypt]: https://letsencrypt.org/ +[mysql]: https://www.mysql.com/ +[part-2]: /post/2016/10/31/freebsd-mailserver-part-2-mailing-with-postfix/ +[postgres]: https://www.postgresql.org/ diff --git a/content/posts/2016/2016-10-31-freebsd-mailserver-part-2-mailing-with-postfix.md b/content/posts/2016/2016-10-31-freebsd-mailserver-part-2-mailing-with-postfix.md new file mode 100644 index 0000000..58d822f --- /dev/null +++ b/content/posts/2016/2016-10-31-freebsd-mailserver-part-2-mailing-with-postfix.md @@ -0,0 +1,316 @@ +--- +date: 2016-10-31 +title: "FreeBSD email server - Part 2: Mailing with Postfix" +tags: +- Tutorial +- FreeBSD +- Email +- Postfix +--- + +Welcome to the second part of my FreeBSD email server series. In this series, I +will guide you through setting up your own email service. Be sure to done the +preparations from [part 1][part-1] of this series. + +This part will guide you through setting up email service on your machine using +[Postfix][postfix]. Basic installation is pretty straightforward, but there is +a lot to configure. If you are not sure what some configuration options do, +please read up on them. There is a lot to do wrong with a mail server, and +doing things wrong will likely get you on a blacklist which will make other +servers stop processing the mail you are trying to send out. + +Setting up Postfix is one of the harder parts of configuring a mail server. If +you have questions after reading the full guide, please find me on IRC. You can +find details on how to do so on [my homepage][home]. + +## Installing Postfix +Installation procedures on FreeBSD are pretty straightforward. Unlike `certbot` +from the previous part, we will need to compile Postfix from source in order to +use PostgreSQL as a database back-end. Thanks to FreeBSD's +[ports][freebsd-ports], this is not difficult either. If this is your first +port to compile, you probably need to get the ports tree first. You can +download and extract this using the following command. + +{% highlight sh %} +portsnap fetch extract +{% endhighlight %} + +Once that has finished running, go into the directory containing the build +instructions for Postfix, and start the installation process. + +{% highlight sh %} +cd /usr/ports/mail/postfix +make configure install +{% endhighlight %} + +This will open a pop up with a number of options you can enable or disable. The +enabled defaults are fine, but you will have to enable the `PGSQL` option. This +will allow you to use the configuration tables created in part 1. + +## Enabling Postfix +Enable the Postfix service for rcinit. This allows you to use `service postfix +start` once configuration is done, and will auto start the service on system +boot. In addition, the default mailer on FreeBSD, [sendmail][sendmail] should +be disabled so nothing is in Postfix's way when trying to deal with processing +email traffic. + +{% highlight sh %} +# disable the default sendmail system +echo 'daily_clean_hoststat_enable="NO"' >> /etc/periodic.conf.local +echo 'daily_status_mail_rejects_enable="NO"' >> /etc/periodic.conf.local +echo 'daily_status_include_submit_mailq="NO"' >> /etc/periodic.conf.local +echo 'daily_submit_queuerun="NO"' >> /etc/periodic.conf.local +echo 'sendmail_enable="NONE"' >> /etc/rc.conf.local + +# enable postfix +echo 'postfix_enable="YES"' >> /etc/rc.conf.local +{% endhighlight %} + +## Configuring Postfix +There is a ton to configure for Postfix. This configuration happens in two +files, `main.cf` and `master.cf`. Additionally, as some data is in the +PostgreSQL database, three files with information on how to query for this +information are needed. All of these files are in `/usr/local/etc/postfix`. + +The guide has a comment line for most blocks. It is advised that **if** you +decide to just copy and paste the contents, you copy that along so you have +some sort of indication of what is where. This could help you out if you ever +need to change anything later on. + +### main.cf +#### Compatibility +The configuration file starts off by setting the compatibility level. If +postfix updates the configuration scheme and deprecates certain options, you +will be notified of this in the logs. + +{% highlight ini %} +# compatibility +compatibility_level = 2 +{% endhighlight %} + +#### Directory paths +These options indicate where Postfix will look and keep certain files required +for correct operation. + +{% highlight ini %} +# directory paths +queue_directory = /var/spool/postfix +command_directory = /usr/local/sbin +daemon_directory = /usr/local/libexec/postfix +data_directory = /var/db/postfix +{% endhighlight %} + +#### Domain configuration +The domain configuration instruct the server of the domain(s) it should serve +for. Use your FQDN without sub domains for `mydomain`. You can use a sub domain +for `myhostname`, but you are not required to. The most common setting is +using a `mail` sub domain for all mail related activities, which would +result in something like this. + +{% highlight ini %} +# domain configuration +myhostname = mail.domain.tld +mydomain = domain.tld +myorigin = $mydomain +{% endhighlight %} + +#### Listening directives +All internet devices it should listen on, and all domains this server should +consider itself the endpoint for, should be listed here. The defaults in the +example block are good enough, as we put some of our data in the PostgreSQL +database instead. + +{% highlight ini %} +# listening directives +inet_interfaces = all +mydestination = $myhostname, localhost.$mydomain, localhost +{% endhighlight %} + +#### Reject unknown recipients +How to deal with messages sent to an email address whose domain points to your +server's address, but have no actual mailbox. A code of `550` means to inform +the remote server that delivery is not possible and will not be possible. This +should stop the remote server from trying it again. + +{% highlight ini %} +# reject unknown recipients +unknown_local_recipient_reject_code = 550 +{% endhighlight %} + +#### Trust +{% highlight ini %} +# trust +mynetworks_style = host +{% endhighlight %} + +#### Address extensions +This block is optional. It allows you to use email address extensions. These +are addresses with an additional character in them that will drop the email in +the non extended address' mailbox, but allows you to quickly filter on them as +the sent-to address contains the extension. + +{% highlight ini %} +# address extensions +recipient_delimiter = + +{% endhighlight %} + +#### Virtual domain directives +This part is where things get important. Virtual domains allow you to handle +mail for a large number of domains that are different from the actual server's +domain. This is where the database configuration comes in to play. It is +important to note the `static:125` values. The `125` should map to the `UID` of +the `postfix` user account on your system. + +{% highlight ini %} +# virtual domain directives +virtual_mailbox_base = /srv/mail +virtual_mailbox_domains = pgsql:/usr/local/etc/postfix/pgsql-virtual-domains.cf +virtual_mailbox_maps = pgsql:/usr/local/etc/postfix/pgsql-virtual-users.cf +virtual_alias_maps = pgsql:/usr/local/etc/postfix/pgsql-virtual-aliases.cf +virtual_uid_maps = static:125 +virtual_gid_maps = static:125 +virtual_transport = lmtp:unix:private/dovecot-lmtp +{% endhighlight %} + +#### TLS setup +The TLS setup configures your server to use secure connections. The keys used +here have been generated in the previous part of this series. + +{% highlight ini %} +# TLS setup +smtpd_tls_cert_file = /usr/local/etc/letsencrypt/live/domain.tld/fullchain.pem +smtpd_tls_key_file = /usr/local/etc/letsencrypt/live/domain.tld/privkey.pem +smtpd_use_tls = yes +smtpd_tls_auth_only = yes +{% endhighlight %} + +#### SASL setup +SASL deals with the authentication of the users to your email server. + +{% highlight ini %} +# SASL setup +smtpd_sasl_type = dovecot +smtpd_sasl_path = private/auth +smtpd_sasl_auth_enable = yes +smtpd_recipient_restrictions = + permit_sasl_authenticated, + permit_mynetworks, + reject_unauth_destination +smtpd_relay_restrictions = + permit_sasl_authenticated, + permit_mynetworks, + reject_unauth_destination +{% endhighlight %} + +#### Debugging +The debugging options are generally useful in case things break. If you have +little traffic, you could leave them on forever in case you want to debug +something later on. Once your server is working as intended, you should turn +these options off. The postfix logs get pretty big in a short amount of time. + +{% highlight ini %} +# debugging +debug_peer_level = 2 +debugger_command = + PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/binary + ddd $daemon_directory/$process_name $process_id & sleep 5 +{% endhighlight %} + +#### Installation time defaults +These options should not be touched, but are very important to have for your +server. + +{% highlight ini %} +# install-time defaults +sendmail_path = /usr/local/sbin/sendmail +newaliases_path = /usr/local/bin/newaliases +mailq_path = /usr/local/bin/mailq +setgid_group = maildrop +html_directory = /usr/local/share/doc/postfix +manpage_directory = /usr/local/man +sample_directory = /usr/local/etc/postfix +readme_directory = /usr/local/share/doc/postfix +inet_protocols = ipv4 +meta_directory = /usr/local/libexec/postfix +shlib_directory = /usr/local/lib/postfix +{% endhighlight %} + +### master.cf +For the `master.cf` file, you can use the following configuration block. + +{% highlight cfg %} +submission inet n - n - - smtpd + -o syslog_name=postfix/submission + -o smtpd_tls_security_level=encrypt + -o smtpd_sasl_auth_enable=yes + -o smtpd_reject_unlisted_recipient=no + -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject + -o milter_macro_daemon_name=ORIGINATING +pickup unix n - n 60 1 pickup +cleanup unix n - n - 0 cleanup +qmgr unix n - n 300 1 qmgr +tlsmgr unix - - n 1000? 1 tlsmgr +rewrite unix - - n - - trivial-rewrite +bounce unix - - n - 0 bounce +defer unix - - n - 0 bounce +trace unix - - n - 0 bounce +verify unix - - n - 1 verify +flush unix n - n 1000? 0 flush +proxymap unix - - n - - proxymap +proxywrite unix - - n - 1 proxymap +smtp unix - - n - - smtp +relay unix - - n - - smtp +showq unix n - n - - showq +error unix - - n - - error +retry unix - - n - - error +discard unix - - n - - discard +local unix - n n - - local +virtual unix - n n - - virtual +lmtp unix - - n - - lmtp +anvil unix - - n - 1 anvil +scache unix - - n - 1 scache +{% endhighlight %} + +### SQL query files +The following three configuration files deal with the SQL query files to make +Postfix able of getting some of its configuration from a database. You +obviously have to change the first 4 directives to match your database +authentication credentials. + +#### pgsql-virtual-domains.cf +{% highlight ini %} +user = postgres +password = incredibly-secret! +hosts = 127.1 +dbname = mail +query = SELECT 1 FROM domains WHERE name='%s'; +{% endhighlight %} + +#### pgsql-virtual-users.cf +{% highlight ini %} +user = postgres +password = incredibly-secret! +hosts = 127.1 +dbname = mail +query = SELECT 1 FROM users WHERE local='%u' AND domain='%d'; +{% endhighlight %} + +#### pgsql-virtual-aliases.cf +{% highlight ini %} +user = postfix +password = nope +hosts = 127.1 +dbname = mail +query = SELECT destination FROM aliases WHERE origin='%s'; +{% endhighlight %} + +## Conclusion +This should be enough Postfix configuration, for now. Next part involves +Dovecot, which will enable IMAP. It will also provide the SASL mechanism +defined in this part. + +[freebsd-ports]: https://www.freebsd.org/ports/ +[home]: / +[part-1]: /post/2016/10/31/freebsd-mailserver-part-1-preparations/ +[postfix]: http://www.postfix.org/ +[sendmail]: http://www.sendmail.com/sm/open_source/ diff --git a/content/posts/2016/2016-10-31-freebsd-mailserver-part-3-dovecot-imap-sasl.md b/content/posts/2016/2016-10-31-freebsd-mailserver-part-3-dovecot-imap-sasl.md new file mode 100644 index 0000000..0938a5e --- /dev/null +++ b/content/posts/2016/2016-10-31-freebsd-mailserver-part-3-dovecot-imap-sasl.md @@ -0,0 +1,228 @@ +--- +date: 2016-10-31 07:57:50 +title: "FreeBSD email server - Part 3: Dovecot, IMAP and SASL" +tags: +- Tutorial +- FreeBSD +- Email +- Dovecot +- IMAP +- SASL +--- + +Welcome to the second part of my FreeBSD email server series. In this series, I +will guide you through setting up your own email service. Be sure to read the +previous parts before trying to continue on this part in case you have not done +so yet. + +This part will guide you through setting up [Dovecot][dovecot]. This service +will deal with the SASL authentication to your email server and making your email +boxes accessible via IMAP. While this guide does not cover POP3 functionality, +Dovecot can handle this as well. + +Just like the Postfix setup, Dovecot has quite a few configuration options to +set before it will work as expected in this setup. If you have questions after +reading the full guide, please find me on IRC. You can find details on how to +do so on [my homepage][home]. + +## Installing Dovecot +Dovecot will also be installed from the ports tree from FreeBSD. As this guide +assumes you are working through them in order, explanation of acquiring the +ports tree will be omitted here. + +You can start the installation procedure with the following commands. + +``` +cd /usr/ports/mail/dovecot2 +make configure install +``` + +Again, like with the Postfix installation, leave the default options on and add +the `PGSQL` option so Dovecot can use PostgreSQL as the database back-end. + +## Enabling Dovecot +Enable the Dovecot service for rcinit. + +``` +echo 'dovecot_enable="YES"' >> /etc/rc.conf.local +``` + +## Configuring Dovecot +To start of with Dovecot configuration, copy over the sample files first. + +``` +cp -r /usr/local/etc/dovecot/example-config/* /usr/local/etc/dovecot/. +``` + +Now you can start editing a number of pesky files. The file names of the +headings all appear relative to `/usr/local/etc/dovecot`. + +### dovecot.conf +Here you only have to set which protocols you want to enable. Set them as +follows. + +```ini +protocols = imap lmtp +``` + +### conf.d/10-master.cf +The `master.cf` configuration file indicates which sockets Dovecot should use +and provide and as which user its processes should be ran. Keep the defaults as +they are, with the exception of the following two blocks. + +#### service imap-login +This will enable imaps, IMAP over SSL, and disable plain IMAP. + +```ini +service-imap-login { + inet_listener imap { + port = 0 + } + + inet_listener imaps { + port = 993 + ssl = yes + } +} +``` + +#### services +This will instruct Dovecot to provide a service for authentication and `lmtp` +the **local mail transport protocol**. This is required to deliver the email +files into the correct email box location in the file system. + +```ini +service auth { + unix_listener auth-userdb { + mode = 0600 + user = postfix + group = postfix + } + + unix_listener /var/spool/postfix/private/auth { + mode = 0666 + user = postfix + group = postfix + } + + user = dovecot +} + +service lmtp { + unix_listener /var/spool/postfix/private/dovecot-lmtp { + mode = 0600 + user = postfix + group = postfix + } +} + +service auth-worker { + user = postfix +} +``` + +### conf.d/10-ssl.conf +Here you have to enable SSL and provide the correct paths to your SSL key in +order for Dovecot to work with them. + +```ini +ssl = required +ssl_cert = < /usr/local/etc/letsencrypt/live/domain.tld/fullchain.pem +ssl_key = < /usr/local/etc/letsencrypt/live/domain.tld/privkey.pem +``` + +### conf.d/10-mail.conf +The mail.conf location instructs Dovecot which location to appoint for storing +the email files. `%d` expands to the domain name, while `%n` expands to the +local part of the email address. + +```ini +mail_home = /srv/mail/%d/%n +mail_location = maildir:~/Maildir +``` + +Make sure the location set by `mail_home` exists and is owned by `postfix`! + +``` +mkdir -p /srv/mail +chown postfix:postfix /srv/mail +``` + +### conf.d/10-auth.conf +This file deals with the authentication provided by Dovecot. Mostly, which +mechanisms should be supported and what mechanism should be used to get the +actual credentials to check against. Make sure the following options are set +as given + +```ini +disable_plaintext_auth = yes +auth_mechanisms = plain +``` + +Also, make sure `!include auth-system.conf.ext` is commented **out**. It is not +commented out by default, so you will have to do this manually. In addition, +you have to uncomment `!include auth-sql.conf.ext`. + +### conf.d/auth-sql.conf.ext +This is the file included from `10-auth.conf`. It instructs Dovecot to use SQL as +the driver for the password and user back-ends. + +```ini +passdb { + driver = sql + args = /usr/local/etc/dovecot/dovecot-sql-conf.ext +} + +userdb { + driver = prefetch +} + +userdb { + driver = sql + args = /usr/local/etc/dovecot/dovecot-sql-conf.ext +} +``` + +### dovecot-sql.conf.ext +The final configuration file entails the queries which should be used to get the +required information about the users. Make sure to update the `password` and possibly +other parameters used to connect to the database. You may have to update the `125` as +well, as this has to be identical to the `UID` of `postfix`. + +As a side note, if you are following this tutorial on a machine that does +**not** support Blowfish in the default glib, which is nearly every GNU+Linux +setup, you **can not** use `BLF-CRYPT` as the `default_pass_scheme`. You will +have to settle for the `SHA-512` scheme instead. + +```ini +driver = pgsql +connect = host=127.1 dbname=mail user=postfix password=incredibly-secret! +default_pass_scheme = BLF-CRYPT +password_query = \ + SELECT \ + local AS user, \ + password, \ + '/srv/mail/%d/%n' AS userdb_home, \ + 125 AS userdb_uid, \ + 125 AS userdb_gid \ + FROM users \ + WHERE local='%n' AND domain='%d'; + +user_query = \ + SELECT \ + '/srv/mail/%d/%n' AS home \ + 125 AS uid, \ + 125 AS gid \ + FROM users \ + WHERE local='%n' AND domain='%d'; +``` + +## Conclusion +After this part, you should be left with a functioning email server that +provides IMAP over a secure connection. While this is great on itself, for +actual use in the wild, you should setup some additional services. Therefore, +in the next part, we will deal with practices that "authenticate" your emails +as legit messages. Be sure to read up on it! + +[dovecot]: http://dovecot.org/ +[home]: / diff --git a/content/posts/2016/2016-10-31-freebsd-mailserver-part-4-message-authentication.md b/content/posts/2016/2016-10-31-freebsd-mailserver-part-4-message-authentication.md new file mode 100644 index 0000000..62a2799 --- /dev/null +++ b/content/posts/2016/2016-10-31-freebsd-mailserver-part-4-message-authentication.md @@ -0,0 +1,159 @@ +--- +date: 2016-10-31 20:00:38 +title: "FreeBSD email server - Part 4: Message authentication" +tags: +- Tutorial +- FreeBSD +- Email +- DKIM +- SPF +--- + +Welcome to another part in the FreeBSD email server series. This time, we are +going to setup some mechanisms to deal with message authentication. This +practice will make other email providers accept your email messages and deliver +them properly in the inbox of the receiving user, instead of their spam box. + +We will do so using three of the most common practices: [SPF][spf], +[DKIM][dkim] and [DMARC][dmarc]. + +## DKIM +### Installation +The tools for DKIM are easily installed using `pkg`. + +``` +pkg install opendkim +``` + +### Configuration +Write the following configuration into `/usr/local/etc/mail/opendkim.conf`. + +```apache +# logging +Syslog yes + +# permissions +UserID postfix +UMask 007 + +# general settings +AutoRestart yes +Background yes +Canonicalization relaxed/relaxed +DNSTimeout 5 +Mode sv +SignatureAlgorithm rsa-sha256 +SubDomains no +X-Header yes +OversignHeaders From + +# tables +KeyTable /usr/local/etc/opendkim/key.table +SigningTable /usr/local/etc/opendkim/signing.table + +# socket +Socket inet:8891@localhost + +# domains +Domain domain.tld.privkey +KeyFile /usr/local/etc/opendkim/domain.tld +Selector mail +``` + +#### Postfix +Postfix needs to be instructed to sign the messages with a DKIM header using +the opendkim service. You can do so by inserting the following configuration +block somewhere around the end of `/usr/local/etc/postfix/main.cf`. + +```ini +# milters +milter_protocol = 2 +milter_default_action = reject +smtpd_milters = + inet:localhost:8891 +``` + +#### System service +OpenDKIM runs as a system service. As such, you will have to enable this +service in rcinit. This is a simple step, achieved with the given command. + +``` +echo 'milteropendkim_enable="YES"' >> /etc/rc.conf.local +``` + +Do not forget to actually start the service when you are done with the +tutorial! + +### Creating and using keys +In order to use DKIM, you will need to generate some keys to sign the messages +with. You cannot use your Let's Encrypt SSL keys for this. First, create a +directory to house your domain's keys. + +``` +mkdir -p /usr/local/etc/opendkim/keys/domain.tld +chown -R postfix:wheel $_ +``` + +Next up, generate your first key. + +``` +opendkim-genkey -D /usr/local/etc/opendkim/keys -b 4096 -r -s $(date +%Y%m%d) -d domain.tld +``` + +I tend to use the current date for the key names so I can easily sort them by +the most recent one. + +Afterwards, you will have to add a line to two separate files to instruct DKIM +to use this key for a certain domain when signing mail. These are fairly +straightforward and can be done using a simple `echo` as well. + +``` +echo '*@domain.tld domain.tld' >> /usr/local/etc/opendkim/signing.table +echo "domain.tld domain.tld:$(date +%Y%m%d):/usr/local/etc/opendkim/keys/domain.tld/$(date +%Y%m%d).private" \ + >> /usr/local/etc/opendkim/key.table +``` + +### Adding the DNS records +You may have already noticed that `opendkim-genkey` also creates a `.txt` file +in addition to the private key. This text file contains the DNS record value +you need to add for your domain's DNS. Add the record to your DNS server, and +simply wait for it to propagate. + +## SPF +SPF is simply a DNS record that shows which IPs are allowed to email for that +domain. + +### Adding the DNS records +A simple example for an SPF record is the following. It allows mail to be sent +in the domain's name from any IP listed in the MX records. + +``` +v=spf1 mx -all +``` + +## DMARC +DMARC is, like SPF, a DNS record. It tells how to deal with messages coming +from the server and where to report abuse of your server. Some of the larger +email providers send out reports to the address given in the DMARC record so +you can figure out whether someone is spamming from your servers, for example. + +### Adding the DNS records +A simple DMARC policy to get started with is to quarantine all emails that fail +authentication. This means the emails will go into the receiving user's spam +box. In addition, abuse reports will be sent to the address defined in the +`rua`. + +``` +v=DMARC1; p=quarantine; rua=mailto:abuse@domain.tld +``` + +## Conclusion +These few simple measures will make receiving servers trust the authenticity of +the mails you send. In effect, your messages will be much less likely to be +marked as spam. However, you are a target of spam as well. How you can deal +with that, will be available in the next part of this series. + +[dkim]: http://www.dkim.org/ +[dmarc]: http://dmarc.org/ +[spf]: https://en.wikipedia.org/wiki/Sender_Policy_Framework + diff --git a/content/posts/2016/2016-10-31-freebsd-mailserver-part-5-filtering-mail.md b/content/posts/2016/2016-10-31-freebsd-mailserver-part-5-filtering-mail.md new file mode 100644 index 0000000..07f8e21 --- /dev/null +++ b/content/posts/2016/2016-10-31-freebsd-mailserver-part-5-filtering-mail.md @@ -0,0 +1,132 @@ +--- +date: 2016-10-31 20:02:19 +title: "FreeBSD email server - Part 5: Filtering mail" +tags: +- Tutorial +- FreeBSD +- Email +- Postfix +- SpamAssassin +- Pigeonhole +--- + +Being able to send mail and not be flagged as spam is pretty awesome on itself. +But you also get hit by a lot of spam. The more you give out your email address +and domain name, the more spam you will receive over time. I welcome you to +another part of the FreeBSD email server series. In this part, we will set up +email filtering at the server side. + +We will accomplish this with a couple packages, [SpamAssassin][spamassassin] +and [Pigeonhole][pigeonhole]. The former deals with scanning the emails to +deduce whether it is spam or not. The latter filters messages. We will use this +filtering to drop emails marked as spam by SpamAssassin into the Junk folder, +instead of the inbox. + +## Installing the packages +Both packages are available through FreeBSD's `pkg` utility. Install them as +such. + +``` +pkg install dovecot-pigeonhole spamassassin +``` + +## SpamAssassin +### Enabling the service +Like most services, you have to enable them as well. Pigeonhole is an extension +to Dovecot, and Dovecot will handle this one. SpamAssassin requires you to +configure the service as well. You can enable it and set sane configuration to +it with the following two commands. + +``` +echo 'spamd_enable="YES"' >> /etc/rc.conf.local +echo 'spamd_flags="-u spamd -H /srv/mail"' >> /etc/rc.conf.local +``` + +### Acquiring default spam rules +SpamAssassin has to "learn" what counts as *spam* and what counts as *ham*. To +fetch these rules, you should execute the updates for SpamAssassin with the +following command. + +``` +sa-update +``` + +You most likely want to run this once every while, so it is advised to setup a +cronjob for this purpose. + +## Postfix +In order to have mails checked by SpamAssassin, Postfix must be instructed to +pass all email through to SpamAssassin, which will hand them back with a +`X-Spam-Flag` header attached to them. This header can be used by other +applications to treat it as spam. + +### master.cf +There's not much to include to the already existing Postfix configuration to +enable SpamAssassin to do its job. Just open `/usr/local/etc/postfix/master.cf` +and append the block given below. + +```ini +spamassassin unix - n n - - pipe + user=spamd argv=/usr/local/bin/spamc + -f -e /usr/sbin/sendmail -oi -f ${sender} ${recipient} +``` + +## Pigeonhole +Pigeonhole is an implementation of Sieve for Dovecot. It deals with filtering +messages on the server side using a set of rules, defined in a file usually +named `sieve`. This file is generally saved at +`/srv/mail/domain.tld/user/sieve`. A default file to filter spam out is the +following example. + +```sieve +require [ + "fileinto", + "mailbox" +]; + +if header :contains "X-Spam-Flag" "YES" { + fileinto :create "Junk"; + stop; +} +``` + +This looks for the `X-Spam-Flag` header, which is added by SpamAssassin. If it +is set to `YES`, this indicates SpamAssassin thinks the message is spam. As +such, sieve is instructed to filter this message into the folder `Junk`, and to +create this folder if it does not exist yet. The `stop;` makes sieve stop +trying to process this message further based on later rules. + +## Dovecot +Dovecot needs some additional configuration to work with Pigeonhole. Modify the +following files and add the contents described. + +### conf.d/20-lmtp.conf +This will enable Pigeonhole in Dovecot. + +```ini +protocol lmtp { + mail_plugins = $mail_plugins sieve +} +``` + +### conf.d/90-plugin.conf +This configures Pigeonhole to look for a file named `sieve` in the mailbox +homedir, and execute that when delivering mail. + +```ini +plugin { + sieve = /srv/mail/%d/%n/sieve +} +``` + +## Conclusion +Spam is a pain, especially if you get a lot of it. The configuration added in +this part of the FreeBSD email server series should get rid of most of it. This +also concludes the series. If you have any questions or suggestions, please +contact me via any of the methods detailed on [my home page][home]. + +Thanks for reading along, and enjoy your very own email server! + +[home]: / +[pigeonhole]: http://pigeonhole.dovecot.org/ +[spamassassin]: https://spamassassin.apache.org/ diff --git a/content/posts/2016/2016-11-24-freebsd-mailserver-calendars-and-contacts.md b/content/posts/2016/2016-11-24-freebsd-mailserver-calendars-and-contacts.md new file mode 100644 index 0000000..b39120f --- /dev/null +++ b/content/posts/2016/2016-11-24-freebsd-mailserver-calendars-and-contacts.md @@ -0,0 +1,141 @@ +--- +date: 2016-11-24 08:26:09 +title: "FreeBSD email server - Part +: Calendars and contacts" +tags: +- Tutorial +- FreeBSD +- Email +- CalDAV +- CardDAV +--- + +This guide is an addition to the [FreeBSD email server series][tutorial-email]. +It is not required for your email server to operate properly, but it is often +considered a very important feature for those who want to switch from a third +party email provider to their own solution. It does build upon the completed +series, so be sure to work through that before starting on this. + +## Install required packages +``` +pkg install py27-radicale +``` + +## Configure Radicale +### /usr/local/etc/radicale/config +Open up the `/usr/local/etc/radicale/config` file, and update each `[block]`. + +#### [server] +The server is binding to `localhost` only. This way it is not accessible on +`:5232` from outside the server. Outside access will be provided through an +nginx reverse proxy instead. + +```ini +hosts = 127.1:5232 +daemon = True + +dns_lookup = True + +base_prefix = / +can_skip_base_prefix = False + +realm = Radicale - Password required +``` + +#### [encoding] +```ini +request = utf-8 +stock = utf-8 +``` + +#### [auth] +```ini +type = IMAP + +imap_hostname = localhost +imap_port = 143 +imap_ssl = False +``` + +#### [storage] +```ini +type = filesystem +filesystem_folder = /usr/local/share/radicale +``` + +#### [logging] +```ini +config = /usr/local/etc/radicale/logging +``` + +### /usr/local/etc/radicale/logging +This file is fine on the defaults in FreeBSD 11. This saves you from +configuring a little bit. + +## Configure Dovecot +### Enable imap +This option was disabled in the [IMAP server tutorial][tutorial-email], +however, if we want to auth using the same credentials as the mailserver, this +option is needed again. Bind it to `localhost`, so it can only be used +internally. In `/usr/local/etc/dovecont/conf.d/10-master.conf`, enable the +`imap` port again: + +```ini +... +service imap-login { + inet_listener imap { + address = 127.1 + port = 143 + } + ... +} +... +``` + +## Configure nginx +To make using the service easier, you can setup [nginx][nginx] to act as a +reverse proxy. If you followed the [webserver tutorial][tutorial-webserver], +you already have the basics for this set up. I do recommend you check this out, +as I will only explain how to configure a virtual host to deal with the reverse +proxy here. + +### Setup a reverse proxy +Assuming you have taken the crash-course in setting up the nginx webserver, you +can attain a reverse proxy using the following config block. Note that this block +only does HTTPS, as I use HTTP only to redirect to HTTPS. + +```nginx +# static HTTPS +server { + # listeners + listen 443 ssl; + server_name radicale.domain.tld; + + # enable HSTS + add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload"; + + # keys + ssl_certificate /usr/local/etc/letsencrypt/live/domain.tld/fullchain.pem; + ssl_certificate_key /usr/local/etc/letsencrypt/live/domain.tld/privkey.pem; + + # / handler + location / { + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_pass http://127.1:5232; + } +} +``` + +## Enable the service at startup +``` +echo 'radicale_enable="YES"' >> /etc/rc.conf.local +``` + +## Start the server +``` +service radicale start +``` + +[nginx]: https://www.nginx.com/ +[tutorial-email]: /post/2016/10/31/freebsd-mailserver-part-1-preparations/ +[tutorial-webserver]: /post/2016/10/25/setup-nginx-with-lets-encrypt-ssl/ diff --git a/content/posts/2016/_index.md b/content/posts/2016/_index.md new file mode 100644 index 0000000..74b1787 --- /dev/null +++ b/content/posts/2016/_index.md @@ -0,0 +1,3 @@ +--- +title: 2016 +--- diff --git a/content/posts/2017/2017-09-14-how-to-git.md b/content/posts/2017/2017-09-14-how-to-git.md new file mode 100644 index 0000000..39b884e --- /dev/null +++ b/content/posts/2017/2017-09-14-how-to-git.md @@ -0,0 +1,182 @@ +--- +date: 2017-09-14 +title: "How to: git" +tags: +- Tutorial +- Git +--- + +This guide will explain how to use `git` more efficiently, and why you should +use it as such. + +## Forking +When working in a team, there's generally a remote server which is used to sync +your repositories. There are gratis services, such as [GitHub][github], +[Gitlab][gitlab], [GOGS][gogs], and others. These services also allow you to +*fork* a repository. This basically makes a copy of the entire repository for +your own use. In it, you have full control over the branches, tags, merge +process and everything else you want to do with it. + +One the main reasons to do this is so you do not have to clutter up the main +repository with a ton of branches (these are explained later in the post). If +there are two people working in the same branch, it can help reduce conflicts, +as each developer is working on the branch in his own fork. + +As such, **always** use a fork. If the service does not have a fancy button for +you to click, you can still fork manually. Simply clone their repository as +usual, set a new remote and push it there: + +``` +git clone git@domain.tld:them/repo.git +cd repo +git remote rename origin upstream +git remote add origin git@domain.tld:you/repo.git +git push origin master +``` + +The default naming convention uses `upstream` for the base of your fork, and +`origin` for your remote version of the repository. If a merge request is +accepted on the original repo, you can apply it to your fork using + +``` +git pull upstream master +``` + +## Branching +Branching is the art of using separate branches to introduce new code into your +`master` branch. Every git repository starts with a `master` branch by default. +This is the *main* branch of your repository. + +Every time you want to add new code to your project, make a branch for the +feature or issue you are trying to solve. This way, you can commit freely +without having to worry about having untested or possibly broken code in the +`master` branch. If something were to come up with a higher priority, such as a +critical bug, you can simply create a new branch off of `master`, fix it and +merge that back into `master`, without having to worry about that other feature +you were working on, which is not in a releasable state yet. Once the fix is +applied, you go back to your feature branch on continue working on the cool new +stuff you wanted to implement. Now, the bug is fixed, and no code has been +released that should not have been released. If that's not convincing enough, +try some of the [Stack Overflow posts][so-git-branch] on this very topic. + +Branches can be made at your leisure, with next to no overhead on your project. +Do not be scared to play around with your code in a new branch to test +something out. You can also delete branches as quickly as you made them if you +are not satisfied with the result. + +Creating branches is done using `git checkout -b new-branch`. If you need to +switch to another existing branch to change something, use +`git checkout other-branch`. Deleting a branch can be done using +`git branch -D old-branch`. You can get a list of all branches in the +repository with `git branch`. The current branch is marked with an \*. + +If you start a new branch to implement a feature, be sure to always branch off +of `master`, unless you have a very compelling reason not to do so. If you are +not sure what reasons would validate branching off of another branch, you +should just branch off of `master`. If you branch off of another branch, you +will have the commit history of the other branch. This often includes commits +not accepted into master yet, which might result into commits getting into +master which should not be there (yet), or annoying merge conflicts later on. + +### Merging +Using multiple branches brings along the concept of *merging* branches +together. When working in a group, this is generally done by maintainers of the +upstream repository, via a *merge request*. For some reason, certain services +have named this as a *pull request* instead. The base idea of the process is as +follows: + +- Pull the latest `upstream/master` +- Create a new branch +- Apply the change you want +- Issue a merge request via the service you are using + - Generally, you want your change to be merged into their `master` branch +- Add a title and a description of your change: What does it do, and why should it be accepted +- Optionally, discuss the changes with the upstream maintainers +- Optionally, make a couple of changes to your branch, and push it again +- Upstream maintainer accepts your change + +When everything worked out, the upstream repository now contains your changes. +If you pull their branch again, it will contain your code. Using the merge +request process, your code can be easily reviewed by others, and discussed if +needed. + +## Committing +Whenever you have changed anything in the repository and you wish to share +these changes, you have to commit the changes. Committing in general is not +something people tend to have issues with. Simple add the changes you want to +commit using `git add` (add the `-p` switch if you want to commit only parts of +a changed file), then `git commit` and enter a descriptive message. And that is +where most annoyances come from: the commit *message*. There are no hard rules +on this forced by git itself. There are, however, some de-facto standards and +best practices which you should always follow. Even if you never intend to +share the repository with other people, having good commit messages can help +you identify a certain change when you look back into the history. + +A git commit message should be short, no more than 79 characters, on the first +line. It should be readable as "this commit message will ...", where your +commit message will replace the "...". It is a de-facto standard to start your +commit message with a capital letter, and leave off a finishing period. You do +not *have* to adhere to if you hate this, but be sure that all your commits are +consistent in how they are formatted. + +If you need to explain anything beyond that, such as a rationale for the +change, or things the reviewer should pay attention to in this particular +commit, you can leave an empty line and publish this message in the commit +body. + +When you are using a bug tracking system, you might also want to have a footer +with additional information. On services such as [Gitlab][gitlab] and +[GitHub][github], you can close issues by adding "Closes: #1" in the commit +message footer. A full commit message with all these things might look as +follows: + +``` +Fix overflow issue in table rendering mechanism + +An overflow issue was found in the table rendering mechanism, as explained in +CVE-0123-45678. Regression tests have been included as well. + +Closes: #35 +``` + +In order to achieve these kind of messages, you need to be sure that your +commits can fit in to this structure. This means you need to make small +commits. Having many smaller commits makes it easier to review the changes, +keep short, descriptive messages to describe each change, and revert a single +change in case it breaks something. + +### Signing your commits +You can set up git to cryptographically sign each commit you make. This will +ensure that the commit you made is proven to be from you, and not someone +impersonating you. People impersonating you might try to get harmful code into +a repo where you are a trusted contributor. Having all commits signed in a +repository can contribute in verifying the integrity of the project. + +Recently, [Github][github] has added the **Verified** tag to commits if the +commit contains a correct signature. + +To enable signing of all commits, add the following configuration to your +`~/.gitconfig`: + +```ini +[commit] + gpgsign = true + +[user] + signingkey = 9ACFE193FFBC1F50 +``` + +Ofcourse, you will have to update the value of the `signingkey` to match +the key you want to sign your commits with. + +## Closing words +I hope this post will help you in your adventures with git. It is a great tool +or working on projects together, but it gets much better when you stick to some +best practices. If you have any suggestions for this post, or any questions +after finishing it, contact me via any method listed on [my home page][home]. + +[github]: https://github.com +[gitlab]: https://gitlab.com +[gogs]: https://gogs.io +[home]: https://tyil.work +[so-git-branch]: https://softwareengineering.stackexchange.com/questions/335654/git-what-issues-arise-from-working-directly-on-master diff --git a/content/posts/2017/2017-09-28-perl6-creating-a-background-service.md b/content/posts/2017/2017-09-28-perl6-creating-a-background-service.md new file mode 100644 index 0000000..4f94bb6 --- /dev/null +++ b/content/posts/2017/2017-09-28-perl6-creating-a-background-service.md @@ -0,0 +1,157 @@ +--- +date: 2017-09-28 +title: Perl 6 - Creating a background service +tags: +- Tutorial +- Perl6 +- Programming +- Raku +--- + +I've recently made some progress on +[Shinrin](https://github.com/scriptkitties/perl6-Shinrin) a centralized logging +system in Perl 6. This has to run as service, which means that for most service +managers it has to be able to run in the background. + +{{< admonition title="Note" >}} +If you just want to get to the solution and don't care for the details, just +head straight to [the full script](#the-final-solution). +{{< / admonition >}} + +## It's not possible! + +After a lot of trying and talking with the folks at +[#perl6](irc://chat.freenode.net:6697/#perl6) I was told that it is not possible +to do this in pure Perl 6, explained by people with more knowledge than I have +on the internals: + +{{< quote attribution="jnthn" >}} +(jnthn suspects fork + multi-threaded VM = pain) Since fork only clones one +thread - the one that called it. So suddenly you've got an instance of the VM +missing most of its threads. +{{< / quote >}} + +{{< quote attribution="geekosaur" >}} +The most common failure mode is that some thread is holding e.g. a mutex (or a +userspace lock) during the fork. The thread goes away but the lock is process +level and remains, with nothing around to know to unlock it. So then things +work until something else needs that lock and suddenly you deadlock. +{{< / quote >}} + +Not much later, `jnthn` [pushed a +commit](https://github.com/perl6/doc/commit/8f9443c3ac) to update the docs to +clarify that a `fork` call through `NativeCall` will probably not give the +result you were hoping for. + +## Or is it? + +Luckily, the same people were able to think up of a work-around, which can be +made in POSIX sh, so it's usable on any decent OS. The workaround is to let a +little shell script fork into the background, and let that run the Perl +application. + +### A first example +This is fairly simple to create, as in this example to launch `shinrind` in the +background: + +```sh +#! /usr/bin/env sh + +main() +{ + perl6 -Ilib bin/shinrind "$@" +} + +main "$@" & +``` + +This works just fine if the working directory is correct. This means you need +to be in the parent directory to `lib` and `bin` of the program to make it +work. + +## Improving the forking script + +While that short script works fine to show a proof of concept, in order to make +it viable for real-world scenarios, it can use some improvements. After all, it +would be annoying if you'd have to `cd` to a specific directory any time you +want to start your application. + +### Ensure you are in the directory you should be in + +So for starters, let's make sure that you can run it from anywhere on your +system. For this, you should set the working directory for the script, so you +don't have to do it manually. Because the script runs in its own subshell, the +shell you're working from remains unaffected. + +A POSIX compliant way to get the directory the script is stored in is as +follows: + +```sh +DIR=$(CDPATH="" cd -- "$(dirname -- "$0")" && pwd) +``` + +This will set `$DIR` to the path of the directory the shell script is stored +in. You can simply `cd` to that and be assured you're in the right directory. + +In Perl 6, it is expected for executable files to live in the `bin` directory +of your project repository. So you should actually be in the parent of the +directory holding your script. Furthermore, you should check the `cd` command +executed correctly, just to be safe. + +```sh +cd -- "${DIR}/.." || exit +``` + +### Disable `STDOUT` and `STDERR` + +A started service should not be polluting your interactive shell, so you should +disable (or otherwise redirect) `STDOUT` and `STDERR`. This is done in the +shell using a small bit of code behind whatever you want to redirect: + +```sh +> /dev/null 2>&1 +``` + +This will set `STDOUT` to `/dev/null`, and set `STDERR` to the same stream as +`STDOUT`, which in effect will make all output go to `/dev/null`. If you want +to log everything to a single file, you can replace `/dev/null` with another +file of your choice. If you don't want logs to be overwritten on each start, +use a `>>` instead of a single `>` at the start. + +If you want to log errors and output in different files, you can use the +following: + +```sh +> /var/log/service.log 2> /var/log/service.err +``` + +This will put standard output in `/var/log/service.log` and errors in +`/var/log/service.err`. + +### Fork just the Perl 6 program + +In the initial example, I put the `&` behind the `main` call, at the bottom of +the script. While this works just fine for most simple usage, if you want to do +additional chores, like creating a pidfile after starting the Perl 6 program, +you're out of luck. If you were to only fork the Perl 6 application, you could +handle some other cases in the shell script. + +### The final solution + +For those eager to just get going with this, here is the complete example +script to just fork your Perl program into the background: + +```sh +#! /usr/bin/env sh + +readonly DIR=$(CDPATH="" cd -- "$(dirname -- "$0")" && pwd) + +main() +{ + cd -- "${DIR}/.." || exit + + perl6 -Ilib bin/shinrind "$@" > /dev/null >2&1 & +} + +main "$@" +``` diff --git a/content/posts/2017/_index.md b/content/posts/2017/_index.md new file mode 100644 index 0000000..141c28d --- /dev/null +++ b/content/posts/2017/_index.md @@ -0,0 +1,3 @@ +--- +title: 2017 +--- diff --git a/content/posts/2018/2018-10-11-hackerrank-solutions-python3-and-perl6-part-2.html b/content/posts/2018/2018-10-11-hackerrank-solutions-python3-and-perl6-part-2.html new file mode 100644 index 0000000..860a856 --- /dev/null +++ b/content/posts/2018/2018-10-11-hackerrank-solutions-python3-and-perl6-part-2.html @@ -0,0 +1,709 @@ +--- +layout: language-war +date: 2018-10-11 +title: "Hackerrank solutions: Python 3 and Perl 6 (part 2)" +tags: +- Hackerrank +- Perl6 +- Python +- Python3 +- Programming +- Raku +--- + +{% markdown %} +# Hackerrank solutions: Python 3 and Perl 6 (part 2) +{% endmarkdown %} + +{% markdown %} +As a continuation of the [previous +part](/post/2018/09/13/hackerrank-solutions-python3-and-perl6-part-1/) of this +series, I will be continuing to work through some Hackerrank challenges for +Python 3, and compare the solutions to how I would solve them in a language I'm +more proficient in, Perl 6. In this post, I will work through some of the +Python 3 string challenges from Hackerrank. + +Raiph [posted a comment on +Reddit](https://www.reddit.com/r/perl6/comments/9ffc2p/hackerrank_solutions_python_3_and_perl_6_part_1/e5xml3m) +suggesting a slightly different layout, which I will be using for this post. +Additional comments are always welcome as I try to improve the format. +{% endmarkdown %} + +{% admonition_md Disclaimer %} +Once again I'd like to make clear I'm trying to stick to the original +Hackerrank challenges by not using any imports not specifically used in the +original challenge. If you have suggestions for Python 3 or Perl 6 modules to +make a given task easier, I still appreciate them, but I won't update my +solutions to use a module. +{% endadmonition_md %} + +{% markdown %} +## Challenges +{% endmarkdown %} + +
+
+ +{% markdown %} +### String Split and Join + +This challenge involves a string containing spaces, where the spaces are to be +replaced with dashes (`-`) instead. +{% endmarkdown %} + +
+
+
+
+ +{% highlight python3 tio=https://tio.run/##K6gsycjPM/7/PyU1TaG4ICezJD4xLyU@Kz8zTyMnMy9V04pLAQiKUktKi/IUlHSV9OBSemDlGkoKSpqa/wuKMvNKNNAMyMwrKC3R0ARKp@XnKyQlFgFxFQA %} +def split_and_join(line): + return "-".join(line.split(" ")) +{% endhighlight %} + +
+
+ +{% markdown %} +I personally am not too fond that `join` takes a list of words to join +together, whereas `split` takes a word to split with. It feels a little +inconsistent. It also doesn't allow me to read the code logically from left to +right. +{% endmarkdown %} + +
+
+
+
+ +{% highlight perl6 tio=https://tio.run/##K0gtyjH7/7@4NEmhuCAns0Q3MS9FNys/M09BQyUnMy9VU6GaSwEIwBw9sBINJQUlTT2QGg0lXSVNrtr/KlqefnrFOaVFBXpqqKboFSdW/k/Lz1dISiwC4ioA %} +sub split-and-join ($line) { + $line.split(" ").join("-") +} +{% endhighlight %} + +
+
+ +{% markdown %} +The Perl 6 solution to the challenge does the same as the Python variant. Even +the function names are the same! The biggest difference is that I can chain the +functions from left to right, leading to clearer code. +{% endmarkdown %} + +
+
+
+
+ +{% markdown %} +### What's Your Name? + +The next challenge is a simply string formatting task. You get two inputs, a +first name and a last name, and have to put it in a string which will be +printed to `STDOUT`. +{% endmarkdown %} + +
+
+
+
+ +{% highlight python3 tio=https://tio.run/##XYzBCsIwEETv/YqxUEigePHmD@jZm6cSyRYj22ww2UK/PobqyWEOAzPz0laeEk@1epqR3iGWaVbmKbqFjBvxsOcOTXtl@isxC4bcfMBdFC/NBZ54JY@2EKQdeOwx4Pu39R8bYtJi7IhfsPWiwUu3uoib5KzLBw %} +def print_full_name(a, b): + print("Hello %s %s! You just delved into python." % (a, b)) +{% endhighlight %} + +
+
+ +{% markdown %} +Before you begin, I know this can be done using `f""` strings, and that was my +first attempt to use as well. However, Hackerrank did not accept this, +complaining about invalid syntax, so I assume they're running an older Python 3 +than I do. + +That said, this is a simple `printf` formatted string, which then accepts a +tuple of arguments to put into the string. `printf` formatted string are very +powerful in their possibilities, and it's clear to read. +{% endmarkdown %} + +
+
+
+
+ +{% highlight perl6 tio=https://tio.run/##K0gtyjH7/7@4NEmhoCgzr0Q3rTQnRzcvMTdVQyVRR0ElSVOhmksBCIoTKxWUPFJzcvIVVBKB4ooKkfmlClmlxSUKKak5ZakpCkDd@QoBQPMUzPSUuGr/Y5in5emnl55aAjQVytL875NYVFTJFZ6YkwMA %} +sub print-full-name($a, $b) { + say "Hello $a $b! You just delved into Perl 6." +} +{% endhighlight %} + +
+
+ +{% markdown %} +Perl 6 has double-quote semantics that many people may be familiar with from +other languages. When you insert a variable in a double-quoted string, it's +`.Str` value will be used. That is to say, the value will be converted to a +`Str` if required, and then put into the string. + +If you need it or want it for clarity, you can also use `"Hello {$a}"` in Perl +6, allowing you to use it similarly to Python 3's `f""` strings. +{% endmarkdown %} + +
+
+
+
+ +{% markdown %} +### Mutations + +You are given a string _string_, an integer _position_ and a character +_character_. In _string_, replace the character at _position_ to the given +_character_. The position is counted from starting point 0, so I don't have to +think about differences between what a human or computer considers to be +position 1 in a string. +{% endmarkdown %} + +
+
+
+
+ +{% highlight python3 tio=https://tio.run/##VY5BC8IwDIXv/RVhpxaKIN4Ef4mIjNm5iCYlTQ/79bXrpmIuyeN975E468R0KOUWRnhl7TVckwrS3a7LQ@SEikwehqmXftAg7migzqITnOCJSTfc/YzzJ3ipyDdqGiBBsxB03e7BSLbxrsTaoPb/C6SY1ToPi7WJptbLlZHZ7I28AQ %} +def mutate_string(string, position, character): + chars = list(string) + chars[position] = character + + return "".join(chars) +{% endhighlight %} + +
+
+ +{% markdown %} +This is basically what the example showed as well that came with the challenge, +so wasn't too hard to solve. My only complaint was that I couldn't call my list +"list", because that's a reserved keyword. +{% endmarkdown %} + +
+
+
+
+ +{% highlight perl6 tio=https://tio.run/##K0gtyjH7/7@4NEkht7QksSRVt7ikKDMvXUFDBcLQUVApyC/OLMnMzwMykzMSixKTS1KLNBWquRSAILdSwSEns7hEwVYBqkEvOT83yRosCZaJhuuPBSmCm2DNhVCjl5WfmWfNVfsfxQ0aKlqefnrpqSVAizFYmnrFiZX/0/LzuQy5igA %} +sub mutate-string ($string, $position, $character) { + my @list = $string.comb; + @list[$position] = $character; + + @list.join; +} +{% endhighlight %} + +
+
+ +{% markdown %} +The Perl 6 variant does the same things as the Python variant. `comb` without +arguments converts a string to a list of characters, and `join` without +arguments joins a list together to a string. +{% endmarkdown %} + +
+
+
+
+ +{% markdown %} +### Find a String + +In the following challenge you are given a string _string_, and a substring +_sub\_string_. The challenge is to find how often a substring occurs in the +_string_. The substrings may overlap one another, so the string `"ABCDCDC"` +contains the substring `"CDC"` twice. +{% endmarkdown %} + +
+
+
+
+ +{% highlight python3 tio=https://tio.run/##XU67CsMwDNz9FTfaxENKt4CHPv4ilNCHkwqKbBx76Ne7bhxIqYSQdNzp5N/x6Xif88OOuLvEcZjTbY6BeJK1aRRkqLPqBEosRBi0YllHF0AgRrjyZGWr8bK8qtUq@QaNqGBPHaGprO32Bcb8eG26zbIx2FXPYGMKXOHsCz/K//eJfYpSaayDUvlwPJ1LilIf %} +def count_substring(string, sub_string): + count = 0 + + for i in range(0, len(string)): + if string[i:i + len(sub_string)] == sub_string: + count += 1 + + return count +{% endhighlight %} + +
+
+ +{% markdown %} +As solution to this challenge I loop through the entire _string_, and check +whether it contains the _sub\_string_ at that point. If it does, I increment +_count_ by 1. Now, I learned that Python also has the inline `if`, just like +Perl 6 does, however, it also *needs* an `else` block. That put me off from +using it in this situation. I think it puts me off from using it in most +situations, actually. With an `else` coming after it, it just becomes messy to +read, in my opinion. +{% endmarkdown %} + +
+
+
+
+ +{% highlight perl6 +tio=https://tio.run/##K0gtyjH7/7@4NEkhOb80r0QXyCouKcrMS1fQUIEwdBRUgIK6EI6mQjUXZ2pOam6xAlRaoa5OIdcqvwxoUGKBPpJSfa7a/2hmaqhoefrppaeWAM2EsjT1ihMr/zs6ObsAIRcQAwA %} +sub count-substring ($string, $sub-string) { + elems $string ~~ m:overlap/$sub-string/ +} +{% endhighlight %} + +
+
+ +{% markdown %} +The Perl 6 version makes use of some regex magic, and the `elems` subroutine. +`elems` returns the number of elements in a list, which in this case would be +the number of matches found by the regex. The `m:overlap//` makes a regex to +*m*atch, with *overlap*ping strings. +{% endmarkdown %} + +
+
+
+
+ +{% markdown %} +### String Validators + +In the following challenge, the program is given a string _s_, and have to +validate a number of properties on this string. These are, in order, whether +they contain + +- alphanumeric characters (a-z, A-Z or 0-9), +- alphabetic characters (a-z or A-Z), +- digits (0-9), +- lowercase characters (a-z), +- uppercase characters (A-Z). + +If any character in the given string passes a validation, it must print +`"True"`, otherwise it must print `"False"`. + +{% endmarkdown %} + +
+
+
+
+ +{% highlight python3 tio=https://tio.run/##hZLNioMwEIDvPsXQPdTAIrS9CR72sk@wt1JCVtMaqklIIqWUPrubn6p1MZib38x8zmQi76YW/ND37AwYc9JSjKEoYItxSxjHeJsnYI@GAhiXnUlR4kFZ0/Lq6MN/urMhDe/aTQ7fpNH0853Lmizwil2YWeCNuFG1wDspJ@7xMzRzFso2RJRtERqmTapRPpbZybgwr4aPryZPQHjlazKmPUrfSqYBp/wCflRHk7jWzfhPa1FcG/JXtOGKZlqPYtohf0Ubbnim9SimHfJXtGFBM61HMe2QH7QQvFd6d@/K7zEkZg6lCI3hTAs1vkS3fAvd7l1s@pNUjA@Kow2dUN9//ZbVbn/4@AM %} +if __name__ == '__main__': + s = input() + + checks = { + "alnum": False, + "alpha": False, + "digit": False, + "lower": False, + "upper": False + } + + for char in list(s): + if not checks["alnum"] and char.isalnum(): + checks["alnum"] = True + + if not checks["alpha"] and char.isalpha(): + checks["alpha"] = True + + if not checks["digit"] and char.isdigit(): + checks["digit"] = True + + if not checks["lower"] and char.islower(): + checks["lower"] = True + + if not checks["upper"] and char.isupper(): + checks["upper"] = True + + keys = list(checks.keys()) + keys.sort() + + for key in keys: + print(checks[key]) +{% endhighlight %} + +
+
+ +{% markdown %} +As stated in the disclaimer, I don't want to make use of any `import` +statements unless these are explicitly given in the original challenges. This +means I can't use regexes, as these are stuffed away in the `re` packages in +Python. Luckily, Python has the correct check available as a method on the +string object, so I can still check them in a single line. + +I first tried to call the methods on _s_ directly, but this seemed to require +the entire string to match the check, instead of just any character in the +string. So I had to loop through the string by character, which I did. If any +character is found to validate, the appropriate key in the _checks_ dict will +be set to `True`. Once I've walked through the entire string, I sort the _keys_ +from _checks_ so I can be sure they're printed in the right order. +{% endmarkdown %} + +
+
+
+
+ +{% highlight perl6 +tio=https://tio.run/##K0gtyjH7/7@4NEnB19HTT6GaSwEIcisVVIoVbBVUtDz99NJTS6y5wMLFiZUKxfkgqbo6BX2bxJy80lw7fWvscgUZiTjkUjLTM0twyOXkl6cW4ZArLSiAyNX@/@@YlJxiaGSsDAA %} +sub MAIN { + my $s = $*IN.get; + + say so $s ~~ //; + say so $s ~~ //; + say so $s ~~ //; + say so $s ~~ //; + say so $s ~~ //; +} +{% endhighlight %} + +
+
+ +{% markdown %} +Perl 6 does have regexes available in the main namespace by default, so that +made this challenge a lot easier to work with. `$*IN` in a special variable +that refers to `STDIN`, and the `.slurp` method reads all remaining data from +the buffer. + +The next 5 lines all do a `say`, which acts like `print` in Python 3. The `so` +function coerces a value to become a `Bool`. When a `Bool` is given to `say`, +it will be coerced to a string representation again, and become either `"True"` +or `"False"`. The smartmatch operator `~~` has already been covered in the +previous post, so I recommend you read that as well if you haven't yet. + +In Perl 6, regexes are (usually) delimited by the `/` character. The ``, +`` etcetera parts are [predefined character classes][classes] in Perl 6 +regexes. These check for exactly what we need in the challenges, so were a good +pick to solve them. + +[classes]: https://docs.perl6.org/language/regexes.html#Predefined_character_classes +{% endmarkdown %} + +
+
+
+
+ +{% markdown %} +### Text Wrap + +You are given a string _s_ and a width _w_. The string should be split over +multiple lines so it is never more wide than _w_. +{% endmarkdown %} + +
+
+
+
+ +{% highlight python3 tio=https://tio.run/##K6gsycjPM/7/PzO3IL@oRKEktaKkvCixgIsrJTVNAcTSKC4pysxL11HITayIL89MKcnQtOJSAIKi1JLSojwFpZg8Jb2s/Mw8DZhePRzaNP8XAIVKNMDSmXkFpSUamjoKIBEoBwj@Ozo5u7i6uXt4enn7@Pr5BwQGBYeEhoVHREZxmQIA %} +import textwrap + +def wrap(string, max_width): + return "\n".join(textwrap.wrap(string, max_width)) +{% endhighlight %} + +
+
+ +{% markdown %} +This challenge introduces the first Python module: `textwrap`. This makes the +challenge very easy to solve as well, using the `wrap` function exposed by the +module. This function makes a list of strings, each no longer than the given +width. I then join these together with newlines to get the desired output. +{% endmarkdown %} + +
+
+
+
+ +{% highlight perl6 tio=https://tio.run/##K0gtyjH7/7@4NEmhvCixQEFDpbikKDMvXUdBpTwzpSRDU6GaSwEIoMJ6yfm5SRpQKb2s/Mw8DaWYPCVNrtr/IO0aKlqefnrpqSVA7VCWnmdeiaZecWLlf0cnZxdXN3cPTy9vH18//4DAoOCQ0LDwiMgoLlMA %} +sub wrap ($string, $width) { + $string.comb($width).join("\n") +} +{% endhighlight %} + +
+
+ +{% markdown %} +For the Perl 6 solution, I have not used an additional module, as all the +functionality are in the core namespace. I actually made a module in Perl 6 for +a less primitive wrapping functionality, called [`String::Fold`][string::fold]. + +In this solution, I use `comb` with the `$width` argument. This returns a list +of strings, each no longer than the given width, just like Python's +`textwrap.wrap`. I can then join these together with newlines as well to get +the same result. + +[string::fold]: https://modules.perl6.org/dist/String::Fold:cpan:TYIL +{% endmarkdown %} + +
+
+
+
+ +{% markdown %} +### Designer Door Mat + +This challenge is more complex than previous challenges. The task at hand is to +"draw" a certain "design" as the output. For the input, you are given both a +height _y_ and a width _x_, however _x_ must always be _y_ × 3, so you can +ignore the second argument. + +This one is much simpler to explain using two examples. The first example is +the output if the input were `7 21`. +{% endmarkdown %} + +{% highlight text %} +---------.|.--------- +------.|..|..|.------ +---.|..|..|..|..|.--- +-------WELCOME------- +---.|..|..|..|..|.--- +------.|..|..|.------ +---------.|.--------- +{% endhighlight %} + +{% markdown %} +In the second example, the input is `11 33`. +{% endmarkdown %} + +{% highlight text %} +---------------.|.--------------- +------------.|..|..|.------------ +---------.|..|..|..|..|.--------- +------.|..|..|..|..|..|..|.------ +---.|..|..|..|..|..|..|..|..|.--- +-------------WELCOME------------- +---.|..|..|..|..|..|..|..|..|.--- +------.|..|..|..|..|..|..|.------ +---------.|..|..|..|..|.--------- +------------.|..|..|.------------ +---------------.|.--------------- +{% endhighlight %} + +
+
+
+
+ +{% highlight python3 tio=https://tio.run/##zZDNS8MwGMbv@SseMwbJlrbrehCEnWS3DS@CBxGpNFsCJQ1p5xD832s@Nh2I7GpyCHmf3/u8H/ZjUJ2pxnFyg@LQu@JNm0Kad9gkEKKk3qsBK2gzMKaNPQyM571ttX/58@KFk6NuBuWJEzpDRVTd7s45p3CGkqPAkhMywWNnERiy6xxabaRH4Wqzl6wUUcHc83cE/pjOvDZ1r2TvLRmL@MwbRctIfKuxYOonA7tI9E3xc/mQYV1A6bQPl2IKRjPqqYQL0PwzD/8fCx@7IHgcY6ubppVxAPLbMRZKSSxoqa8idV5xkfSn9eb@Ybum4gpOUslNd5Tur92FsMBCIPvfuxvHWyzLLw %} +#! /usr/bin/env python3 + +height = int((input().split())[0]) +width = height * 3 +half = int((height - 1) / 2) + +# Top half +for line in range(1, half + 1): + non_dashes = ((line * 2) - 1) + dashes = int((width - (non_dashes * 3)) / 2) + + print("%s%s%s" % ("-" * dashes, ".|." * non_dashes, "-" * dashes)) + +# Middle line +print("%s%s%s" % ( + "-" * (int(width / 2) - 3), + "WELCOME", + "-" * (int(width / 2) - 3) +)) + +# Lower half +for line in range(half, 0, -1): + non_dashes = ((line * 2) - 1) + dashes = int((width - (non_dashes * 3)) / 2) + + print("%s%s%s" % ("-" * dashes, ".|." * non_dashes, "-" * dashes)) +{% endhighlight %} + +
+
+ +{% markdown %} +I split the code up in a top half, middle line and lower half, to make it +easier to reason about. The `for` loops contain some logic to get the right +output on every line. I found out that `range` supports a third argument, +allowing me to count down with it as well, which was perfect for this +situation. +{% endmarkdown %} + +
+
+
+
+ +{% highlight perl6 tio=https://tio.run/##zZBBTsMwEEX3PsXgdpEgxVFSqSwiVqiLSi1skFiiIE@xpeBEdtpQhZyDo3CAHix0UpKGSuzxypp5/8/XL9Bm87adXEG4dTZ80SZEs4OCxoy97WGqUL@qEm5her28Fy7b2kJUuZVOKEylWJoy6bhKy1IR9iM4fMLstFFptgkGG68HAoh8OHxBnDA2gce8AALZJrcQCfFLVTM4PvIyuQlk6hS6zuqZzsQ@eSUDM9qfQgXH30hIyfzhNKlcugde84DDey9vai4@RDc4S5sLhiesofBrLWWGkGmDbOzl9QnoFqWc@c3TYnX3sF7U8DfCu0ZWeYX23Il3UYovLO7QOvzX7bTtDcTRNw %} +#! /usr/bin/env perl6 + +my $height = $*IN.slurp.words.head.Int; +my $width = $height × 3; +my $half-height = ($height - 1) ÷ 2; + +# Top half +for 1..$half-height { + my $non-dashes = ($_ × 2) - 1; + my $dashes = ($width - ($non-dashes × 3)) ÷ 2; + + say "{"-" x $dashes}{".|." x $non-dashes}{"-" x $dashes}"; +} + +# Middle line +say "{"-" x (($width ÷ 2) - 3)}WELCOME{ "-" x (($width ÷ 2) - 3)}"; + +# Lower half +for (1..$half-height).reverse { + my $non-dashes = ($_ × 2) - 1; + my $dashes = ($width - ($non-dashes × 3)) ÷ 2; + + say "{"-" x $dashes}{".|." x $non-dashes}{"-" x $dashes}"; +} +{% endhighlight %} + +
+
+ +{% markdown %} +As usual, the code is functionally the same. I must admit I like the functional +style to get an `Int` from the first argument much more than the way I do it in +Python, though. + +A thing I learned is that the `..` operator that generates a sequence does not +have a way to make a sequence that counts down, so I had to use `.reverse` on a +sequence that counts up. I had expected this to Just Work as I expected and +count down if the left hand side would be larger than the right hand side. + +You may notice some fancy Unicode characters in the source, namely `×` for +multiplication, and ÷ for division. Perl 6 allows Unicode characters in the +source files, which can oftentimes lead to prettier code. In this particular +instance, there's no big difference in code readability, though. And for those +who don't yet have a modern editor that can make Unicode characters, do not +worry, as the ASCII equivalents (`*` and `/` respectively) still work as well. +{% endmarkdown %} + +
+
+
+
+ +{% markdown %} +### String Formatting + +In this challenge, you are to produce a table with four columns. The columns +should contain the decimal, octal, hexadecimal and binary values of the row +numbers. The function receives an int _number_. The table should contain that +many rows, starting with row number 1. +{% endmarkdown %} + +
+
+
+
+ +{% highlight python3 tio=https://tio.run/##jZFNCoMwEIX3nmIQhIQG0XYn9CwSTawpJoYYqVJ6dht/W7uQDlll3vveTKJ7W9bqMgyMF6CNUDYtaiOptZwh1cqMG5x44ErSLn0IZku4QsUV8p9Rkr38cJavWuxNYncJAoQCQ9WNo5jA3IcTxAtvLMZzIWnliI72YQkcmnvbWLRl4s1S53Y1JPV/lpJ3dJeUdDtjqzU36ACQCUVNv3iz49DNND0m8oMG1uNDAGgZhMyLkO/hyBKE8fD7EyNKKN1ahF33HL0B %} +def print_formatted(number): + max_width = len("{0:b}".format(number)) + + for i in range(1, number + 1): + decimal = "{0}".format(i).rjust(max_width) + octal = "{0:o}".format(i).rjust(max_width) + hexadecimal = "{0:x}".format(i).upper().rjust(max_width) + binary = "{0:b}".format(i).rjust(max_width) + + print("%s %s %s %s" % (decimal, octal, hexadecimal, binary)) +{% endhighlight %} + +
+
+ +{% markdown %} +In the Python 3 solution I first calculate the max width I need to take into +account. Then I loop from 1 until _number_ to get the right amount of rows. +Each iteration, I format the number correctly, and then print it out using a +printf format string. + +The hardest part of this challenge was to get formatting right the way +Hackerrank wanted it. But I guess that was the entire point of the challenge. +{% endmarkdown %} + +
+
+
+
+ +{% highlight perl6 tio=https://tio.run/##XY9NCsIwEIX3PcUQIiRiB1tEBPEA3bhxK0iq0VZMWpIUK1KvXmNb/HurGWbeN29KaS7ztrVVCqXJtQuPhVHCOXkARnWlUmk43APwUjegStThNT@4DFYwjDEVVrKY48YZ3GfC2OV7vYeF1nn0yVsYGd0/jMYSqGuYcTwXuWYECIcHkK0my6BDeDtEiMOhIcZLv2Dskh8Z3U2A7vo8C/6po/lXE3Pe52uCpv17mdFxssaTdJhox9t4@gQ %} +sub print-formatted ($number) { + my $max-width = $number.base(2).Str.chars; + my $format-string = ("%{$max-width}s" xx 4).join(" ") ~ "\n"; + + for 1..$number { + $format-string.printf($_, $_.base(8), $_.base(16), $_.base(2)); + } +} +{% endhighlight %} + +
+
+ +{% markdown %} +The Perl 6 solution starts of the same, in that it first calculates the max +width I need to take into account. Next, however, I generate the format string +using the `$max-width` to make the `printf` subroutine pad it for me. The `xx` +operator makes a total of 4 such strings, and puts them into a list, which I +can then `join` together with a space character, and add a `\n` at the end of +it (the `~` operator is for string concatenation). + +I'm assuming something similar is possible in Python 3 as well, and I would +like to have an example so I can compare it more fairly. + +In the Perl 6 solution I am also able to make use of the `base` method to +convert the numbers into the right base, something I could not find for Python +3. +{% endmarkdown %} + +
+
+
+
+ +{% markdown %} +## Wrap-up + +This time I did not do all of the challenges, as the post would probably get +too long. I still did 8 of them, and might do the rest of the string challenges +in a later part anyway. + +I still find Perl 6 to produce much cleaner code, which is shown best with the +first challenge. In Perl 6 (`$line.split(" ").join("-")`), I can read from left +to right to see what I'm doing: I have a `$line`, which I split, and then join. +In the Python variant (`"-".join(line.split(" "))`), it is much less clear what +the actual item I'm working on is, as it's hidden inbetween the `join` and +`split` calls. + +Of course, I'm still not an expert on Python 3 code, so I'm sure that there are +many parts that could be written in a cleaner fashion. I'm still open for +feedback to improve my Python 3 skills (hence I'm publishing these posts), so +please let me know if you know better ways to solve some challenges. +{% endmarkdown %} diff --git a/content/posts/2018/_index.md b/content/posts/2018/_index.md new file mode 100644 index 0000000..e1bb4e6 --- /dev/null +++ b/content/posts/2018/_index.md @@ -0,0 +1,3 @@ +--- +title: 2018 +--- diff --git a/content/posts/2021/2021-05-13-a-new-irc-client.md b/content/posts/2021/2021-05-13-a-new-irc-client.md new file mode 100644 index 0000000..7003cb5 --- /dev/null +++ b/content/posts/2021/2021-05-13-a-new-irc-client.md @@ -0,0 +1,85 @@ +--- +date: 2021-05-13 +title: A New IRC::Client +tags: +- Raku +- IRC +- Programming +social: + email: mailto:~tyil/public-inbox@lists.sr.ht&subject=A New IRC::Client +--- + +The Raku programming language has a popular module for creating IRC bots, +[`IRC::Client`](https://github.com/raku-community-modules/IRC-Client). However, +it's been stale for quite a while, and one of the bots I host, +[Geth](https://github.com/Raku/geth), is having troubles on a regular basis. + +I've looked at the source code, and found a lot of neat tricks, but when +maintaining a library, I generally want clean and straightforward code instead. +To that end, I decided to just write my own from scratch. Given that [the IRC +protocol is rather simple](https://tools.ietf.org/html/rfc2812), this was the +easiest way to go about it. + +Sure enough, after a couple hours of playing around, I had something that +worked reasonably well. A few more hours a day afterwards brought me to an +`IRC::Client` that is usable in mostly the same way as the current +`IRC::Client`, to save me effort in getting my current bots to make use of it +for a few test runs. + +Geth was my main target, as I wanted it to stop from getting timed out so +often. For the past week, Geth has been running stable, without any time +out, so I think I've succeeded in the main goal. + +However, how to continue next? Since it is mostly compatible, but not +*completely* compatible, if I were to adopt `IRC::Client` from the Raku +ecosystem and push my version, many people's IRC bots would break when people +update their dependencies. There is a solution for this built into the entire +ecosystem, which is using the correct `:ver` and `:auth` (and in some cases, +`:api`) so you can ensure your project is always getting the "correct" +dependency. However, from personal experience, I know these additional +dependency restrictions are rarely used in practice. + +I hope that with this blog post, I can inform the community in time about the +changes that are coming to `IRC::Client`, so people have ample time to set +their dependencies just right to keep their projects working. Of course, the +real solution for the long term would be to make the small changes required to +use the latest `IRC::Client` again. + +For convenience sake, I've added a small number of methods for backwards +compatibility as well, though these will generate [a deprecation +warning](https://docs.raku.org/routine/is%20DEPRECATED), and will be removed in +a later `IRC::Client` release. + +There's two major changes that are not backwards compatible, however. The first +one is the removal of the ability to have a single `IRC::Client` connect to +multiple servers. This is also the easiest to remedy, by simply creating +multiple instances of `IRC::Client`. + +The second major incompatible change is how events are dispatched to plugins. +This used to be handled by going through all the plugins sequentially, allowing +one plugin to stop the dispatch to another plugin. In my new version, events +are dispatched to all plugins in parallel. This allows for faster execution, +and for multiple plugins to handle an event without having to use `$*NEXT` +(which has been removed). My main motivation is that a buggy plugin will no +longer interfere with the interactions provided by other plugins. The ordering +of your plugins will also stop being an issue to worry about. + +Geth's updates to actually use my updated `IRC::Client` module was introduced +in +[`edc6b08`](https://github.com/Raku/geth/commit/edc6b08036c8ede08afdea39fd1768655a2766aa), +and most if it was updates to the way it handled logging. The actual changes +needed to make Geth play nice were + +- [Adding the `IRC::Client::Plugin` role to `Geth::Plugin::Info`](https://github.com/Raku/geth/commit/edc6b08036c8ede08afdea39fd1768655a2766aa#diff-104b5cdb61f2aa423eb941ce32a4412b4cb814014728cae968e5aeff7dc587d2R14); +- [And to `Geth::Plugin::Uptime`](https://github.com/Raku/geth/commit/edc6b08036c8ede08afdea39fd1768655a2766aa#diff-104b5cdb61f2aa423eb941ce32a4412b4cb814014728cae968e5aeff7dc587d2R24); +- [Using `nicks` instead of `nick`](https://github.com/Raku/geth/commit/edc6b08036c8ede08afdea39fd1768655a2766aa#diff-104b5cdb61f2aa423eb941ce32a4412b4cb814014728cae968e5aeff7dc587d2R34); +- [Using `start` instead of `run`](https://github.com/Raku/geth/commit/edc6b08036c8ede08afdea39fd1768655a2766aa#diff-104b5cdb61f2aa423eb941ce32a4412b4cb814014728cae968e5aeff7dc587d2R46); +- [Using `privmsg` instead of `send`](https://github.com/Raku/geth/commit/edc6b08036c8ede08afdea39fd1768655a2766aa#diff-c388af832470886cdd4304aeb17c4c2f406ac184d33afb956b0ef8a92b69855bR57); + +The last two changes aren't strictly necessary, as there are backwards +compatibility methods made for these, but it's a rather small change and +reduces the amount of noise in the logs. + +With this, I hope everyone using `IRC::Client` is prepared for the coming +changes. If you have any comments or questions, do not hesitate to reach out to +me and share your thoughts! diff --git a/content/posts/2021/2021-05-22-raku-on-libera-chat.md b/content/posts/2021/2021-05-22-raku-on-libera-chat.md new file mode 100644 index 0000000..87ce7ff --- /dev/null +++ b/content/posts/2021/2021-05-22-raku-on-libera-chat.md @@ -0,0 +1,35 @@ +--- +date: 2021-05-22 +title: Raku is moving to Libera.chat +tags: +- Raku +- LiberaChat +- IRC +social: + email: mailto:~tyil/public-inbox@lists.sr.ht&subject=Raku is moving to Libera.chat +--- + +Earlier this week, the staff at the Freenode IRC network have resigned en +masse, and created a new network, Libera. This was sparked by [new ownership of +Freenode](https://kline.sh/). Due to concerns with the new ownership, the Raku +Steering Council has decided to also migrate our IRC channels to Libera. + +This requires us to take a couple steps. First and foremost, we need to +register the Raku project, to allow us a claim to the `#raku` and related +channels. Approval for this happened within 24 hours, and as such, we can +continue on the more noticable steps. + +The IRC bots we're using for various tasks will be moved next, and the Raku +documentation has to be updated to refer to Libera instead of Freenode. The +coming week we'll be working on that, together with the people who provide +those bots. + +Once this is done, the last step involves the Matrix bridge. Libera and +Matrix.org staff are working on this, but there's no definite timeline +available just yet. This may mean that Matrix users will temporarily not be +able to join the discussions happening at Libera. We will keep an eye on the +progress of this, and set up the bridge as soon as it has been made available. + +If you have any questions regarding the migration, feel free to reach out to us +via email (`rsc@raku.org`) or on IRC (`#raku-steering-council` on +irc.libera.chat). diff --git a/content/posts/2021/2021-06-04-managing-docker-compose-projects-with-openrc.md b/content/posts/2021/2021-06-04-managing-docker-compose-projects-with-openrc.md new file mode 100644 index 0000000..c182654 --- /dev/null +++ b/content/posts/2021/2021-06-04-managing-docker-compose-projects-with-openrc.md @@ -0,0 +1,194 @@ +--- +date: 2021-06-04 +title: Managing Docker Compose with OpenRC +tags: +- Gentoo +- OpenRC +- Docker +- DockerCompose +--- + +On one of my machines, I host a couple services using `docker-compose`. I +wanted to start/restart/stop these using the default init/service manager used +on the machine, `openrc`. This would allow them to start/stop automatically +with Docker (which coincides with the machine powering on or off, +respectively). + +I've set this up through a single `docker-compose` meta-service. To add new +`docker-compose` projects to be managed, all I need to do for `openrc` +configuration is creating a symlink, and configure the path to the +`docker-compose.yaml` file. + +The meta-service lives at `/etc/init.d/docker-compose`, just like all other +services managed by `openrc`. This file is quite straightforward. To start off, +a number of variables are set and exported. + +```sh +name="$RC_SVCNAME" +description="OpenRC script for managing the $name docker-compose project" + +# Set default values +DOCKER_COMPOSE="${DOCKER_COMPOSE:-docker-compose} $DOCKER_COMPOSE_ARGS" + +COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-$name}" + +# Export all variables used by docker-compose CLI +export COMPOSE_PROJECT_NAME +export COMPOSE_FILE +export COMPOSE_PROFILES +export COMPOSE_API_VERSION +export DOCKER_HOST +export DOCKER_TLS_VERIFY +export DOCKER_CERT_PATH +export COMPOSE_HTTP_TIMEOUT +export COMPOSE_TLS_VERSION +export COMPOSE_CONVERT_WINDOWS_PATHS +export COMPOSE_PATH_SEPARATOR +export COMPOSE_FORCE_WINDOWS_HOST +export COMPOSE_IGNORE_ORPHANS +export COMPOSE_PARALLEL_LIMIT +export COMPOSE_INTERACTIVE_NO_CLI +export COMPOSE_DOCKER_CLI_BUILD +``` + +One of the services I use is also configured with its own `external` network. I +want it to be created if it doesn't exist, to ensure that the service can start +up properly. I do *not* want it to be removed, so I left that out. + +```sh +# Set up (external) networks +for name in "${DOCKER_NETWORKS[@]}" +do + # Create the network if needed + if ! docker network ls | awk '{ print $2 }' | grep -q "$name" + then + einfo "Creating docker network '$name'" + docker network create "$name" > /dev/null + fi + + # Expose some variables for the networks + network_id="DOCKER_NETWORK_${name}_ID" + + declare -gx DOCKER_NETWORK_${name}_ID="$(docker network ls | awk '$2 == "'"$name"'" { print $1 }')" + declare -gx DOCKER_NETWORK_${name}_GATEWAY="$(docker network inspect "${!network_id}" | jq -r '.[0].IPAM.Config[0].Gateway')" + + unset network_id +done +``` + +And lastly, there's the four simple functions to declare dependencies, +configure how to `start` or `stop`, and how to get the `status` of the service. + +```sh +depend() { + need docker +} + +start() { + $DOCKER_COMPOSE --project-directory "$COMPOSE_PROJECT_DIRECTORY" up -d +} + +status() { + $DOCKER_COMPOSE --project-directory "$COMPOSE_PROJECT_DIRECTORY" ps +} + +stop() { + $DOCKER_COMPOSE --project-directory "$COMPOSE_PROJECT_DIRECTORY" down +} +``` + +Now, to actually create a service file to manage a `docker-compose` project, a +symlink must be made. I'll take my +[`botamusique`](https://github.com/azlux/botamusique) service as an example. + +``` +ln -s /etc/init.d/docker-compose /etc/init.d/botamusique +``` + +This service can't start just yet, as there's no `$COMPOSE_PROJECT_DIRECTORY` +configured for it yet. For this, a similarly named file should be made in +`/etc/conf.d`. In here, any variable used by the service can be configured. + +``` +$EDITOR /etc/conf.d/botamusique +``` + +In my case, it only pertains the `$COMPOSE_PROJECT_DIRECTORY` variable. + +``` +COMPOSE_PROJECT_DIRECTORY="/var/docker-compose/botamusique" +``` + +And that's it. For additional `docker-compose` projects I need to make only a +symlink and a configuration file. If I discover a bug or nuisance, only a +single file needs to be altered to get the benefit on all the `docker-compose` +services. + +For reference, here's the full `/etc/init.d/docker-compose` file. + +```sh +#!/sbin/openrc-run +# Copyright 2021 Gentoo Authors +# Distributed under the terms of the GNU General Public License v2 + +name="$RC_SVCNAME" +description="OpenRC script for managing the $name docker-compose project" + +# Set default values +DOCKER_COMPOSE="${DOCKER_COMPOSE:-docker-compose} $DOCKER_COMPOSE_ARGS" + +COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-$name}" + +# Export all variables used by docker-compose CLI +export COMPOSE_PROJECT_NAME +export COMPOSE_FILE +export COMPOSE_PROFILES +export COMPOSE_API_VERSION +export DOCKER_HOST +export DOCKER_TLS_VERIFY +export DOCKER_CERT_PATH +export COMPOSE_HTTP_TIMEOUT +export COMPOSE_TLS_VERSION +export COMPOSE_CONVERT_WINDOWS_PATHS +export COMPOSE_PATH_SEPARATOR +export COMPOSE_FORCE_WINDOWS_HOST +export COMPOSE_IGNORE_ORPHANS +export COMPOSE_PARALLEL_LIMIT +export COMPOSE_INTERACTIVE_NO_CLI +export COMPOSE_DOCKER_CLI_BUILD + +# Set up (external) networks +for name in "${DOCKER_NETWORKS[@]}" +do + # Create the network if needed + if ! docker network ls | awk '{ print $2 }' | grep -q "$name" + then + einfo "Creating docker network '$name'" + docker network create "$name" > /dev/null + fi + + # Expose some variables for the networks + network_id="DOCKER_NETWORK_${name}_ID" + + declare -gx DOCKER_NETWORK_${name}_ID="$(docker network ls | awk '$2 == "'"$name"'" { print $1 }')" + declare -gx DOCKER_NETWORK_${name}_GATEWAY="$(docker network inspect "${!network_id}" | jq -r '.[0].IPAM.Config[0].Gateway')" + + unset network_id +done + +depend() { + need docker +} + +start() { + $DOCKER_COMPOSE --project-directory "$COMPOSE_PROJECT_DIRECTORY" up -d +} + +status() { + $DOCKER_COMPOSE --project-directory "$COMPOSE_PROJECT_DIRECTORY" ps +} + +stop() { + $DOCKER_COMPOSE --project-directory "$COMPOSE_PROJECT_DIRECTORY" down +} +``` diff --git a/content/posts/2021/_index.md b/content/posts/2021/_index.md new file mode 100644 index 0000000..1287d56 --- /dev/null +++ b/content/posts/2021/_index.md @@ -0,0 +1,3 @@ +--- +title: 2021 +--- diff --git a/content/posts/_index.md b/content/posts/_index.md new file mode 100644 index 0000000..b80ef2e --- /dev/null +++ b/content/posts/_index.md @@ -0,0 +1,24 @@ +--- +title: Blog +--- + +Over time, I've written a number of articles. Some to voice my opinion, some to +help people out with a tutorial. These articles are listed below, sorted by +publication date. If you have any comments on any of my articles, feel free to +reach out to me through any of the contact details found [on the +homepage][home]. Alternatively, I have a public inbox on Sourcehut that you +can send any comments to: +[`~tyil/public-inbox@lists.sr.ht`](mailto:~tyil/public-inbox@lists.sr.ht). + +All my blog posts are available under the [Creative Commons (CC BY-SA +3.0)][cc-by-sa] license, which means you are free to use it for any purpose so +long as you keep attribution to me (and preferably also just link to the +original article) and do not relicense the article. + +I'd also like to note that **these articles reflect my opinion, and only +mine**. Please refrain from accusing other people of holding my opinion for +simply being referenced in my articles. + +[cc-by-sa]: https://creativecommons.org/licenses/by-sa/3.0/ +[home]: / +[pgp]: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x7A6AC285E2D98827 -- cgit v1.1