diff options
| author | Patrick Spek <p.spek@tyil.nl> | 2021-02-05 09:55:25 +0100 |
|---|---|---|
| committer | Patrick Spek <p.spek@tyil.nl> | 2021-04-01 07:26:50 +0200 |
| commit | c455896ae9e69e2498742ff795e7886dee1ffa23 (patch) | |
| tree | 62a7353cbd1b14b4dd93b453f1f38cfe533c66ec /_posts | |
| parent | 32acfbf6bf21d52e987d1b7599128f75b759ab28 (diff) | |
Move source files into src
Diffstat (limited to '_posts')
33 files changed, 0 insertions, 8119 deletions
diff --git a/_posts/2016-10-01-on-pastebin.md b/_posts/2016-10-01-on-pastebin.md deleted file mode 100644 index b5cab12..0000000 --- a/_posts/2016-10-01-on-pastebin.md +++ /dev/null @@ -1,78 +0,0 @@ ---- -layout: post -date: 2016-10-01 10:20:27 +0200 -tags: Pastebin Security Cloudflare Privacy -authors: - - ["Patrick Spek", "http://tyil.work"] ---- - -Pastebin offers itself as a gratis paste service. Although it is probably the -most well known option out there, it is certainly not the best. - -## The security issue -Pastebin has a couple of issues that harm the visitor's security. This on -itself should be considered such a bad practice that no-one should consider -their service at all. - -### Cloudflare -Cloudflare is a [MITM][mitm]. It completely breaks the secure chain of TLS on -the web, and should not be used. Any service still using Cloudflare should be -shunned. There is [another article][cloudflare] on this site which has more -information on this specific issue. In addition, Cloudflare can be considered a -privacy issue for the same reasons, as is detailed below. - -### Advertisements -Another issue with regards to security on pastebin are the advertisements. -While it can be argued that "they need to make money somehow", using ads always -seems like the worst possible solution. Especially given the way they're -serving it. The past couple years have shown that advertisements on the web are -easily abused to serve malware to good netizens who decided to not block all -ads. - -A rant on the state of ads might be appropriate, but this article is -specifically about Pastebin, so I will just keep it at "third party -advertisements are a security risk, avoid sites who use them" - -## The privacy issue -Apart from their security issues, Pastebin also offers some privacy issues. As -stated above, they make use of Cloudflare. This means that whenever you visit -them, Cloudflare takes note of this. They may even decide that you need to -perform some additional tasks in order to be allowed to the resource. This -doesn't happen to most users, but if you're using any anonymization practices, -this will happen almost every time you visit a site behind Cloudflare. - -In addition to telling Cloudflare, you will also tell another third party, -Google, in case this "additional step" is required. This is done via the new -reCaptcha system which will inform Google of almost every detail of your -browser and the behaviour used to solve the puzzle. Incredibly useful for -fingerprinting you accross multiple locations. - -### Then there is Tor -But, if you're using an anonymization proxy such as Tor, even if you do not -care about the Cloudflare issue, and you solve the "security check" presented -to you, Pastebin still refuses to offer you their service. If they are going to -refuse you service, they should tell you up front, not after you have already -informed two other harmful parties of your attempt of accessing the resource. - -Actually, they should not. They should simply not require you to give up your -privacy and serve you the content you were looking for. Blocking resources to a -certain group of users is simply censorship, and should not be the status quo -on the free internet. - -## Alternatives -Luckily, there are plenty of alternatives that do not treat their users with -such disrespect. I ask anyone who is still using Pastebin to stop doing this, -and use any of the alternatives. - -* [0bin.net](https://0bin.net/) -* [cry.nu][crynu] (works like termbin: `nc cry.nu 9999 < file`) -* [ix.io][ix] -- [p.tyil.nl][tyilnl] (works like termbin: `nc p.tyil.nl 9999 < file`) - -[cloudflare]: /articles/on-cloudflare/ -[crynu]: https://cry.nu -[hastebin]: http://hastebin.com -[ix]: http://ix.io/ -[mitm]: https://en.wikipedia.org/wiki/Man-in-the-middle_attack -[termbin]: http://termbin.com -[tyilnl]: / diff --git a/_posts/2016-10-01-on-systemd.md b/_posts/2016-10-01-on-systemd.md deleted file mode 100644 index 9dc410a..0000000 --- a/_posts/2016-10-01-on-systemd.md +++ /dev/null @@ -1,286 +0,0 @@ ---- -layout: post -date: 2016-10-01 10:20:27 +0200 -tags: Systemd Security GNU+Linux -authors: - - ["Patrick Spek", "http://tyil.work"] - - ["Samantha McVey", "https://cry.nu"] ---- - -Systemd once presented itself as being the next generation init system for -GNU+Linux. When the project started it seemed to be headed in a good direction. -Unfortunately, it quickly became clear that systemd's goal was not only to -bring you a quick, new init system. It planned to do so much more. This was -part of the plan, since init systems were generally considered to be in a bad -state overall it was quickly accepted by most mainstream GNU+Linux -distributions. What was at first only an init system became so much more: -systemd-logind was made to manage tty's, systemd-resolvd was added to act as a -caching DNS server. Even networking was added with systemd-networkd to manage -network interfaces. - -**DISCLAIMER**: Systemd is a fast moving project, this may result in -information here to becoming outdated. If you find any information that is no -longer correct, please contact me. You can find my contact details [on my -homepage][tyil]. - -## Technical issues -### Security -From experience, we have seen that systemd's creator, Lennart Poettering, will -try to assimilate any functionality he can find and add it into systemd. This -causes systemd to have a large surface area of attack, adding to and magnifying -security attack vectors. An init system should be exactly the opposite. To -compound this issue, we have bugs like [the user-level DoS][systemd-dos], -which seem to indicate that the software is hardly tested or written by -programmers who don't use best practices. - -### POSIX -POSIX compliance. Systemd developers seem to detest it. Their common argument -against retaining POSIX compliance is that "systemd must break POSIX compliance -in order to further the development of GNU+Linux userland utilities". While -this may be true in some sense, it is a very bad idea to ignore POSIX -altogether. - -POSIX is one of the reasons that most applications running on GNU+Linux and -other Unix like systems are very portable. It's a standard that most OS's and -distro's try to meet, making it easy to port software. - -[natermeer on Reddit][reddit-natermeer] said -> POSIX has almost no relevance anymore. -> -> [...] -> -> If you care about portability you care about it running on OS X and Windows -> as well as your favorite \*nix system. POSIX gains you nothing here. A lot -> of the APIs from many of these systems will resemble POSIX closely, but if -> you don't take system-specific differences into account you are not going -> to accomplish much. - -> I really doubt that any Init system from any Unix system uses only POSIX -> interfaces, except maybe NetBSD. All of them are going to use scripts and -> services that are going to be running commands that use kernel-specific -> features at some point. Maybe a init will compile and can be executed on -> pure POSIX api, but that is a FAR FAR cry from actually having a booted and -> running system. - -Which was replied to by [aidanjt][reddit-aidanjt] -> Wrong, both OS X and Windows have POSIX support, although Window's is emulated, -> OS X certainly is not, it's fully POSIX compliant. and b) POSIX doesn't have to -> work identically everywhere, it only has to be more or less the same in most -> places and downstream can easily patch around OS-specific quirks. Even -> GNU/Linux and a bunch of the BSDs are merely regarded as 'mostly' POSIX -> compliant, after all. But if you ignore POSIX entirely, there's ZERO hope of -> portability. -> -> Actually sysvinit is very portable, init.c only has 1 single Linux header which -> has been #ifdef'ed, to handle the three-finger-salute. You see, init really -> isn't that complicated a programme, you tell the kernel to load it after it's -> done it's thing, init starts, and loads distro scripts which starts userspace -> programmes to carry on booting. No special voodoo magic is really required. -> POSIX is to thank for that. POSIX doesn't need to be the only library eva, it -> only needs to handle most of the things you can't do without, without having to -> directly poke at kernel-specific interfaces. -> -> This is why with POSIX, we can take a piece of software written for a PPC AIX -> mainframe, and make it work on x86 Linux without a complete rewrite, usually -> with only trivial changes. - -### Dependencies and unportability -Another common issue with systemd is that applications have started to -needlessly depend on it, forcing systemd onto users that do not wish to use -systemd for obvious reasons outlined here, reasons outside of this article, or -simply being unable to use it. Because systemd complies to no cross-platform -standard and uses many features only available in recent Linux version, it's -either very hard or impossible to implement systemd in some circumstances. - -The list of features it requires is no small one either, as you can see in the -list [posted by ohset][reddit-ohet]: - -- `/dev/char` -- `/dev/disk/by-label` -- `/dev/disk/by-uuid` -- `/dev/random` -- `/dev/rtc` -- `/dev/tty0` -- `/proc/$PID/cgroup` -- `/proc/${PID}/cmdline` -- `/proc/${PID}/comm` -- `/proc/${PID}/fd` -- `/proc/${PID}/root` -- `/proc/${PID}/stat` -- `/proc/cmdline` -- `/sys/class/dmi/id` -- `/sys/class/tty/console/active` -- `BTRFS_IOC_DEFRAG` -- `CLONE_xxx` -- `F_SETPIPE_SZ` -- `IP_TRANSPORT` -- `KDSKBMODE` -- `O_CLOEXEC` -- `PR_CAPBSET_DROP` -- `PR_GET_SECUREBITS` -- `PR_SET_NAME` -- `PR_SET_PDEATHSIG` -- `RLIMIT_RTPRIO` -- `RLIMIT_RTTIME` -- `SCHED_RESET_ON_FORK` -- `SOCK_CLOEXEC` -- `TIOCLINUX` -- `TIOCNXCL` -- `TIOCVHANGUP` -- `VT_ACTIVATE` -- `\033[3J` -- `audit` -- `autofs4` -- `capabilities` -- `cgroups` -- `fanotify` -- `inotify` -- `ionice` -- `namespaces` -- `oom score adjust` -- `openat()` and friends -- `selinux` -- `settimeofday()` and its semantics -- `udev` -- `waitid()` -- numerous GNU APIs like `asprintf` - -This made [Gnome][gnome] unavailable for a long time to BSD users and GNU+Linux -users who wanted to remain with a sane and proven system. Utilities like -[Gummiboot][gummiboot] are now being absorbed by systemd too. It is only a -matter of time before you can no longer use this utility without a systemd init -behind it. There are too many examples of software to list, which are being -assimilated or made unavailable by lazy or bad developers who choose to depend -on systemd for whatever reason. - -### Speed -The main selling point many systemd users hail all the time, is speed. They -place an unusual high amount of value on being a couple seconds faster on boot. -Systemd gains this speed gain by using parallelization, and many think this is -unique to systemd. Luckily for those who want to stick to a more sane system, -this is false. Other init systems, such as [OpenRC][openrc], used by -[Funtoo][funtoo], and [runit][runit], used by [Voidlinux][voidlinux] both -support parallel startup of services. Both these systems use small and -effective shell scripts for this, and support startup dependencies and the -like. Systemd brings nothing new to the init world, it just advertises these -features more agressively. - -### Modularity -The UNIX principle, *make an application perform one task very well*, seems to -be very unpopular among systemd developers. This principle is one of the -reasons why UNIX based systems have gotten so popular. Yet, the systemd -developers seem to despise this principle, and even try to argue that systemd -actually is modular because **it compiles down to multiple binaries**. This -shows a lack of understanding, which would make most users uneasy when they -consider that these people are working on one of the most critical pieces of -their OS. - -The technical problem this brings is that it is very hard to use systemd with -existing tools. `journald` for instance doesn't just output plain text you can -easily filter through, save or apply to a pager. I decides for you how to -represent this information, even if this might be an ineffective way to go -about it. - -### Binary logs -Hailed by systemd users and developers as a more efficient, fast and secure way -to store your logs, it is yet another middle finger to the UNIX principles, -which state that documents intended for the user should be human readable. -Binary logs are exactly not that. This forces you to use the tools bundled with -systemd, instead of your preferred solution. This means you need a system with -systemd in order to read your logs, which you generally need the most when the -system that generated it crashed. Thanks to systemd, these logs are now useless -unless you have another systemd available for it. - -These logs are also very fragile. It is a common "issue" to have corrupted logs -when using systemd. Corrupted is here within quotes because the systemd -developers do not recognize this as a bug. Instead, you should just rotate your -logs and hope it does not happen again. - -The usual counter to this issue is that you *can* tell systemd to use another -logger. However, this does not stop `journald` from processing them first or -just not having `journald` at all. As systemd is not modular, you will always -have all the pieces installed. It should also be noted that this is a -*workaround*, not a fix to the underlying problem. - -## Political issues -### Aggressively forced upon users -A point that has made many systemd opponents very wary of this huge piece of -software is the way it was introduced. Unlike most free software packages, -systemd was forced into the lives of many users by getting hard dependencies on -them, or simply absorbing a critical piece of software by the use of political -power. The two most prominent pieces of software where this has happened are -[Gnome][gnome] and [`udev`][udev]. - -The Gnome developers made a hard dependency on systemd. This in effect made -every gnome user suddenly require systemd. As a result, FreeBSD had to actually -drop Gnome for a while, as systemd does not run outside of GNU+Linux. - -The other, `udev`, was a critical piece of software to manage devices in -GNU+Linux. Sadly, some political power was shown by Red Hat and `udev` got -absorbed into systemd. Luckily, the Gentoo guys saw this issue and tried to -resolve it. As the systemd developers dislike anything that's not systemd -itself, they stubbornly refused the patches from the Gentoo folks which would -keep `udev` a single component (and thus usable without systemd). In the end, -the Gentoo developers forked `udev` into [`eudev`][eudev]. - -### Unwillingness to cooperate -Whenever someone from outside the systemd fangroups steps up to actually -improve systemd in whatever way, the systemd devs seem to be rather -uncooperative. It is not uncommon for developers from other projects to make a -change in order for their projects (and usually others) to improve. This -removes a lot of the cost for the systemd maintainers to deal with all the -issues created they are creating. - -There are some references to the systemd developers being against changes that -might make systemd less of a problem, but these changes are usually denied with -petty excuses. - -- https://lists.freedesktop.org/archives/systemd-devel/2012-June/005466.html -- https://lists.freedesktop.org/archives/systemd-devel/2012-June/005507.html - -## How to avoid it -### Choosing a better OS or distribution -Nowadays, the only way to avoid it without too much trouble, is by simply -choosing a better OS or distro that does not depend on systemd at all. There -are a few choices for this: - -- \*BSD ([FreeBSD][freebsd], [OpenBSD][openbsd], and others) -- [Devuan][devuan] -- [Funtoo][funtoo] -- [Voidlinux][voidlinux] - -It is a shame that it renders a very large chunk of the GNU+Linux world -unavailable when choosing a distro, but they have chosen laziness over a -working system. The only way to tell them at this point that they have made a -wrong decision, is to simply stop using these distros. - -### More links - -- [Broken by design: systemd][broken-systemd] -- [Without systemd][without-systemd] -- [systemd is the best example of Suck][suckless-systemd] -- [Thoughts on the systemd root exploit][agwa-systemd-root-exploit] (In response to [CVE-2016-10156][cve-2016-10156]) -- ["systemd: Please, No, Not Like This"](https://fromthecodefront.blogspot.nl/2017/10/systemd-no.html) - -[agwa-systemd-root-exploit]: https://www.agwa.name/blog/post/thoughts_on_the_systemd_root_exploit -[broken-systemd]: http://ewontfix.com/14/ -[cve-2016-10156]: http://www.openwall.com/lists/oss-security/2017/01/24/4 -[devuan]: https://devuan.org/ -[eudev]: https://wiki.gentoo.org/wiki/Eudev -[freebsd]: https://www.freebsd.org/ -[funtoo]: http://www.funtoo.org/Welcome -[gentoo]: https://gentoo.org -[gnome]: http://www.gnome.org/ -[gummiboot]: https://en.wikipedia.org/wiki/Gummiboot_(software) -[openbsd]: https://www.openbsd.org/ -[openrc]: https://en.wikipedia.org/wiki/OpenRC -[reddit-aidanjt]: https://www.reddit.com/r/linux/comments/132gle/eli5_the_systemd_vs_initupstart_controversy/c72saay -[reddit-natermeer]: https://www.reddit.com/r/linux/comments/132gle/eli5_the_systemd_vs_initupstart_controversy/c70hrsq -[reddit-ohet]: https://www.reddit.com/r/linux/comments/132gle/eli5_the_systemd_vs_initupstart_controversy/c70cao2 -[runit]: http://smarden.org/runit/ -[suckless-systemd]: http://suckless.org/sucks/systemd -[systemd-dos]: https://github.com/systemd/systemd/blob/b8fafaf4a1cffd02389d61ed92ca7acb1b8c739c/src/core/manager.c#L1666 -[tyil]: http://tyil.work -[udev]: https://wiki.gentoo.org/wiki/Eudev -[voidlinux]: http://www.voidlinux.eu/ -[without-systemd]: http://without-systemd.org/wiki/index.php/Main_Page diff --git a/_posts/2016-10-25-setup-a-vpn-with-cjdns.md b/_posts/2016-10-25-setup-a-vpn-with-cjdns.md deleted file mode 100644 index 74446b0..0000000 --- a/_posts/2016-10-25-setup-a-vpn-with-cjdns.md +++ /dev/null @@ -1,211 +0,0 @@ ---- -title: Setup a VPN with cjdns -date: 2016-10-25 08:00:34 -tags: Tutorial VPN cjdns GNU+Linux FreeBSD -layout: post -authors: - - ["Patrick Spek", "http://tyil.work"] ---- - -In this tutorial I will outline a simple setup for a [VPN][vpn] using -[`cjdns`][cjdns]. Cjdns will allow you to setup a secure mesh vpn which uses -IPv6 internally. - -## Requirements -For this tutorial, I have used two client machines, both running Funtoo. A -FreeBSD 11 server is used as a global connection point. - -You are ofcourse able to use any other OS or distro supported by cjdns, but you -may have to update some steps to work on your environment in that case. - -## Installation of the server -### Dependencies -Before you can begin, we need some dependencies. There's only two of those, and -they are available via `pkg` to make it even easier. Install them as follows: - -{% highlight sh %} -pkg install gmake node -{% endhighlight %} - -### Compiling -Next up is getting the cjdns sources and compile these, as cjdns is not -available as a prebuilt package: - -{% highlight sh %} -mkdir -p ~/.local/src -cd $_ -git clone https://github.com/cjdelisle/cjdns.git cjdns -cd $_ -./do -{% endhighlight %} - -To make the compiled binary available system-wide so we can use it with a -system service, copy it to `/usr/local/bin` and rehash to make it available as -a direct command: - -{% highlight sh %} -cp cjdroute /usr/local/bin/. -hash -r -{% endhighlight %} - -### Configuring -Cjdns provides a flag to generate the initial configuration. This will provide -you with some sane defaults where only a couple of small changes are needed to -make it work properly. Generate these defaults with `--genconf`: - -{% highlight sh %} -(umask 177 && cjdroute --genconf > /usr/local/etc/cjdroute.conf) -{% endhighlight %} - -The umask will make all following commands write files using `600` permissions. -This makes sure the config file is not readable by people who shouldn't be able -to read it. Be sure to check wether the owner of the file is `root`! - -Now you can start actually configuring the node to allow incoming connections. -You have to find the `authorizedPasswords` array in the `cjdroute.conf` file -and remove the contents of it. Then you can add your own machines in it. This -guide follows the assumption of two clients, so the config for two clients will -be shown here. You can add more clients if you wish, ofcourse. - -{% highlight json %} -"authorizedPasswords": -[ - {"password": "aeQu6pa4Vuecai3iebah7ogeiShaeDaepha6Mae1yooThoF0oa0Eetha9oox", "user": "client_1"}, - {"password": "aiweequuthohkahx4tahLohPiezee9OhweiShoNeephe0iekai2jo9Toorah", "user": "client_2"}, -] -{% endhighlight %} - -If you need to generate a password, you can make use of the tool `pwgen`, -available at your local package manager. You can then generate new passwords by -running `pwgen 60 -1`. Change the `60` around if you want passwords of a -different size. - -### Adding a startup service -rcinit has deceptively easy scripts to make applications available as services. -This in turn allows you to enable a service at startup. This way you can make -sure cjdns starts whenever the server boots. You can copy the following -contents directly into `/usr/local/etc/rc.d/cjdroute`: - -{% highlight sh %} -#! /bin/sh - -# PROVIDE: cjdroute -# KEYWORD: shutdown - -# -# Add the following lines to /etc/rc.conf to enable cjdroute: -# -#cjdroute_enable="YES" - -. /etc/rc.subr - -name="cjdroute" -rcvar="cjdroute_enable" - -load_rc_config $name - -: ${cjdroute_config:=/usr/local/etc/cjdroute.conf} - -command="/usr/local/bin/cjdroute" -command_args=" < ${cjdroute_config}" - -run_rc_command "$1" -{% endhighlight %} - -Afterwards, you must enable the service in `/etc/rc.conf.local` like follows: - -{% highlight sh %} -echo 'cjdroute_enable="YES"' >> /etc/rc.conf.local -{% endhighlight %} - -## Installation of the clients -### Dependencies -The dependencies are still on `gmake` and `node`, so simply install those on -your clients. This guide assumes using Funtoo for the clients, so installation -would go as follows: - -{% highlight sh %} -emerge gmake nodejs -{% endhighlight %} - -### Compiling -Compilation is the same as for the server, so check back there for more -information if you have already forgotten. - -### Configuring -Generating the base configuration is again done using `cjdroute --genconf`, -just like on the server. On Funtoo, config files generally reside in `/etc` -instead of `/usr/local/etc`, so you should set the filepath you write the -configuration to accordingly: - -{% highlight sh %} -cjdroute --genconf > /etc/cjdroute.conf -{% endhighlight %} - -Setting up the connections differs as well, as the clients are going to make an -outbound connection to the server, which is configured to accept inbound -connections. - -You should still clean the `authorizedPasswords` array, as it comes with a -default entry that is uncommented. - -Now you can setup outbound connections on the clients. You set these up in the -`connectTo` block of `cjdroute.conf`. For this example, the IP 192.168.1.1 is -used to denote the server IP. Unsurprisingly, you should change this to your -server's actual IP. You can find the `publicKey` value at the top of your -server's `cjdroute.conf` file. - -On client 1, put the following in your `cjdroute.conf`: - -{% highlight json %} -"connectTo": -{ - "192.168.1.1:9416": - { - "login": "client_1", - "password": "aeQu6pa4Vuecai3iebah7ogeiShaeDaepha6Mae1yooThoF0oa0Eetha9oox", - "publicKey": "thisIsJustForAnExampleDoNotUseThisInYourConfFile_1.k" - } -} -{% endhighlight %} - -On client 2: - -{% highlight json %} -"connectTo": -{ - "192.168.1.1:9416": - { - "login": "client_2", - "password": "aiweequuthohkahx4tahLohPiezee9OhweiShoNeephe0iekai2jo9Toorah", - "publicKey": "thisIsJustForAnExampleDoNotUseThisInYourConfFile_1.k" - } -} -{% endhighlight %} - -That is all for configuring the nodes. - -### Adding a startup service -You probably want cjdroute to run at system startup so you can immediatly use -your VPN. For openrc based systems, such as Funtoo, cjdns comes with a ready to -use service script. To make this available to your system, copy it over to the -right directory: - -{% highlight sh %} -cp ~/.local/src/cjdns/contrib/openrc/cjdns /etc/init.d/cjdroute -{% endhighlight %} - -Now add the service to system startup and start the service: - -{% highlight sh %} -rc-update add cjdroute default -rc-service cjdroute start -{% endhighlight %} - -That should be sufficient to get cjdns up and running for an encrypted VPN. You -can find the IPs of each of your systems at the top of your `cjdroute.conf` -files, in the `ipv6` attribute. - -[cjdns]: https://github.com/cjdelisle/cjdns -[vpn]: https://en.wikipedia.org/wiki/Virtual_private_network - diff --git a/_posts/2016-10-25-setup-nginx-with-lets-encrypt-ssl.md b/_posts/2016-10-25-setup-nginx-with-lets-encrypt-ssl.md deleted file mode 100644 index a2802f8..0000000 --- a/_posts/2016-10-25-setup-nginx-with-lets-encrypt-ssl.md +++ /dev/null @@ -1,228 +0,0 @@ ---- -title: Setup nginx with Let's Encrypt SSL -date: 2016-10-25 08:00:34 -tags: Tutorial LetsEncrypt Nginx SSL Encryption -layout: post -authors: - - ["Patrick Spek", "http://tyil.work"] ---- - -This is a small tutorial to setup nginx with Let's Encrypt on a FreeBSD server -to host a static site. - -## Install required software -First you have to install all the packages we need in order to get this server -going: - -{% highlight sh %} -pkg install nginx py27-certbot -{% endhighlight %} - -## Configure nginx -Next is nginx. To make life easier, you should configure nginx to read all -configuration files from another directory. This allows you to store all your sites in -separate configurations in a separate directory. Such a setup is a regular site on -nginx installations on GNU+Linux distributions, but not default on FreeBSD. - -Open up `/usr/local/etc/nginx/nginx.conf` and make the contents of the `http` -block look a as follows: - -{% highlight nginx %} -http { - include mime.types; - default_type application/octet-stream; - - sendfile on; - #tcp_nopush on; - - keepalive_timeout 65; - - # default paths - index index.html; - - # disable gzip - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=773332 - gzip off; - - # default ssl settings - ssl_session_cache shared:SSL:1m; - ssl_session_timeout 5m; - ssl_ciphers HIGH:!aNULL:!MD5:!AES128:!CAMELLIA128; - ssl_protocols TLSv1.2; - ssl_prefer_server_ciphers on; - ssl_dhparam /usr/local/etc/ssl/dhparam.pem; - - # default logs - error_log /var/log/nginx/error.log; - access_log /var/log/nginx/acces.log; - - # default server - server { - listen 80; - server_name localhost; - - location / { - root /usr/local/www/nginx; - index index.html index.htm; - } - - error_page 404 /404.html; - error_page 500 502 503 504 /50x.html; - - location = /50x.html { - root /usr/local/www/nginx-dist; - } - } - - # include site-specific configs - include sites/*.conf; -} -{% endhighlight %} - -This sets default ssl settings for all server blocks that enable ssl. Note that -these are settings I use, and are in no way guaranteed to be perfect. I did some -minor research on these settings to get an acceptable rating on -[SSL Labs][ssllabs]. However, security is not standing still, and there is a -decent chance that my settings will become outdated. If you have better settings -that result in a safer setup, please [contact me][contact]. - -### Setup HTTP -Due to the way `certbot` works, you need a functioning web server. Since there -is no usable cert yet, this means hosting a HTTP version first. The tutorial -assumes a static HTML website to be hosted, so the configuration is pretty -easy. - -Put the following in `/usr/local/etc/nginx/sites/domain.conf`: - -{% highlight nginx %} -# static HTTP -server { - # listeners - listen 80; - server_name domain.tld www.domain.tld; - - # site path - root /srv/www/domain/_site; - - # / handler - location / { - try_files $uri $uri/ =404; - } - - # logs - error_log /var/log/nginx/error.log; - access_log /var/log/nginx/access.log; -} -{% endhighlight %} - -If your site's sources do not reside in `/srv/www/domain/_site`, change the -path accordingly. This guide will continue using this path for all examples, so -be sure to modify this where needed. In the same vein, the domain `domain.tld` -will be used. Modify this to your own domain. - -### Start nginx -Nginx is now configured to host a single site over HTTP. Now is the time to enable -the nginx service. Execute the following: - -{% highlight sh %} -echo 'nginx_enable="YES"' >> /etc/rc.conf.local -{% endhighlight %} - -This will enable nginx as a system service. On reboots, it will be started -automatically. You can also start it up without rebooting by running the -following: - -{% highlight sh %} -service nginx start -{% endhighlight %} - -## Configure Let's Encrypt -Nginx is now running as your web server on port 80. Now you can request Let's -Encrypt certificates using `certbot`. You can do so as follows: - -{% highlight sh %} -certbot certonly --webroot -w /srv/www/domain/_site -d domain.tld -d www.domain.tld -{% endhighlight %} - -In case you want to add any sub domains, simply add more `-d sub.domain.tld` -arguments at the end. If the DNS entries for the domains resolve properly, and -no unexpected errors occur on the Let's Encrypt side, you should see a message -congratulating you with your new certs. - -If your domains do not resolve correctly, `certbot` will complain about this. -You will have to resolve your DNS issues before attempting again. - -If `certbot` complains about an unexpected error on their side, wait a couple -minutes and retry the command. It should work, eventually. - -Once `certbot` has ran without errors, the required files should be available -in `/usr/local/etc/letsencrypt/live/domain.tld`. - -## Configure nginx with SSL -The certificate has been issued and base nginx is running. Now is the time to -re-configure your site on nginx to host the HTTPS version of your site instead. -Open up `/usr/local/etc/nginx/sites/domain.conf` again, and make the contents -look like the following: - -{% highlight nginx %} -# redirect HTTPS -server { - # listeners - listen 80; - server_name domain.tld *.domain.tld; - - # redirects - return 301 https://$host$request_uri; -} - -# static HTTPS -server { - # listeners - listen 443 ssl; - server_name domain.tld www.domain.tld; - - # site path - root /srv/www/domain/_site; - - # / handler - location / { - try_files $uri $uri/ =404; - } - - # enable HSTS - add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload"; - - # keys - ssl_certificate /usr/local/etc/letsencrypt/live/domain.tld/fullchain.pem; - ssl_certificate_key /usr/local/etc/letsencrypt/live/domain.tld/privkey.pem; -} -{% endhighlight %} - -Do not forget to update all the paths to match your setup! - -As a final step, you should generate the dhparam file. This is to avoid the -issues as described on [Weak DH][weakdh]. - -{% highlight sh %} -openssl gendh -out /usr/local/etc/ssl/dhparam.pem 4096 -{% endhighlight %} - -Be aware that this step can take a **very** long time. On the test machine I -used to test this tutorial, with 1 core and 1 GB ram, it took nearly 1 hour to -generate this file. - -### Reload nginx -The final step is to reload the nginx configuration so it hosts the SSL version -of your site, and redirects the HTTP version to the HTTPS version. To do this, -simply run - -{% highlight sh %} -service nginx reload -{% endhighlight %} - -That should be all to get your site working with HTTP redirecting to HTTPS, and -HTTPS running using a gratis Let's Encrypt certificate. - -[contact]: https://www.tyil.work/ -[ssllabs]: https://www.ssllabs.com/ssltest/analyze.html?d=tyil.work&latest -[weakdh]: https://weakdh.org/ - diff --git a/_posts/2016-10-31-freebsd-mailserver-part-1-preparations.md b/_posts/2016-10-31-freebsd-mailserver-part-1-preparations.md deleted file mode 100644 index 2439349..0000000 --- a/_posts/2016-10-31-freebsd-mailserver-part-1-preparations.md +++ /dev/null @@ -1,138 +0,0 @@ ---- -title: "FreeBSD email server - Part 1: Preparations" -date: 2016-10-31 07:57:50 -tags: Tutorial FreeBSD Email -layout: post ---- - -This tutorial is devised into multiple chapters to make it more manageable, and -to be able to better explain why certain parts are needed. - -The tutorial is created out of experience setting up my own email server. I have -read through quite a lot of documentation so you do not have to. Nonetheless, I -would recommend doing so. Email business is a tricky one, with a lot of moving -parts that have to fit into each other. Knowing how exactly each part works will -greatly help understanding why they are needed in a proper email server. -Besides that, it will make your life a lot more enjoyable if you want to tweak -some things after this tutorial. - -To kick off, some preparations should be done before you start on setting up -your own email server. - -## DNS setup -Some DNS setup is required for mail. Most importantly, the MX records of a -domain. Be sure you have a domain available, otherwise, get one. There are -plenty of registrars and the price is pretty low for most domains. If you want -to look hip, get a `.email` TLD for your email server. - -For the DNS records themselves, make sure you have an `A` record pointing to -the server IP you're going to use. If you have an IPv6 address, set up an -`AAAA` record as well. Mail uses the `MX` DNS records. Make one with the value -`10 @`. If you have multiple servers, you can make MX records for these as -well, but replace the `10` with a higher value each time (`20`, `30`, etc). -These will be used as fallback, in case the server with pointed to by the `10` -record is unavailable. - -## PostgreSQL -Next up you will have to install and configure [PostgreSQL][postgres]. Although -using a database is not required, this tutorial will make use of one. Using a -database makes administration easier and allows you to add a pretty basic web -interface for this task. - -### Installation -Since the tutorial uses FreeBSD 11, you can install PostgreSQL easily by running - -{% highlight sh %} -pkg install postgresql96-server -{% endhighlight %} - -### Starting up -In order to start Postfix, you should enable the system service for it. This -way, `service` can be used to easily manage it. In addition, it will start -automatically on boot. - -{% highlight sh %} -echo 'postgresql_enable="YES"' >> /etc/rc.conf.local -service postgresql start -{% endhighlight %} - -### Database initialization -Since PostgreSQL is a little different than the more popular [MySQL][mysql], I -will guide you through setting up the database as well. To begin, switch user -to `postgres`, which is the default administrative user for PostgreSQL. Then -simply open up the PostgreSQL CLI. - -{% highlight sh %} -su postgres -psql -{% endhighlight %} - -Once you are logged in to PostgreSQL, create a new user which will hold -ownership of the database and make a database for this user. - -{% highlight sql %} -CREATE USER postfix WITH PASSWORD 'incredibly-secret!'; -CREATE DATABASE mail WITH OWNER postfix; -{% endhighlight %} - -Once this is done, create the tables which will hold some of our configuration -data. - -#### domains -{% highlight sql %} -CREATE TABLE domains ( - name VARCHAR(255) NOT NULL, - PRIMARY KEY (name) -); -{% endhighlight %} - -#### users -{% highlight sql %} -CREATE TABLE users ( - local VARCHAR(64) NOT NULL, - domain VARCHAR(255) NOT NULL, - password VARCHAR(128) NOT NULL, - PRIMARY KEY (local, domain), - FOREIGN KEY (domain) REFERENCES domains(name) ON DELETE CASCADE -); -{% endhighlight %} - -#### aliases -{% highlight sql %} -CREATE TABLE aliases ( - domain VARCHAR(255), - origin VARCHAR(256), - destination VARCHAR(256), - PRIMARY KEY (origin, destination), - FOREIGN KEY (domain) REFERENCES domains(name) ON DELETE CASCADE -); -{% endhighlight %} - -## Let's Encrypt -### Installation -Installing the [Let's Encrypt][letsencrypt] client is just as straightforward -as the PostgreSQL database, using `pkg`. - -{% highlight sh %} -pkg install py27-certbot -{% endhighlight %} - -### Getting a certificate -Requesting a certificate requires your DNS entries to properly resolve. If they -do not resolve yet, Let's Encrypt will bother you with errors. If they do -resolve correctly, use `certbot` to get your certificate. - -{% highlight sh %} -certbot certonly --standalone -d domain.tld -{% endhighlight %} - -## Conclusion -This should be everything required to get started on setting up your own email -server. Continue to [part 2][part-2] of this series to start setting up -Postfix. - -[freebsd]: https://www.freebsd.org/ -[letsencrypt]: https://letsencrypt.org/ -[mysql]: https://www.mysql.com/ -[part-2]: {{ "/post/2016/10/31/freebsd-mailserver-part-2-mailing-with-postfix/" | prepend: site.baseurl }} -[postgres]: https://www.postgresql.org/ diff --git a/_posts/2016-10-31-freebsd-mailserver-part-2-mailing-with-postfix.md b/_posts/2016-10-31-freebsd-mailserver-part-2-mailing-with-postfix.md deleted file mode 100644 index 1818c9b..0000000 --- a/_posts/2016-10-31-freebsd-mailserver-part-2-mailing-with-postfix.md +++ /dev/null @@ -1,313 +0,0 @@ ---- -title: "FreeBSD email server - Part 2: Mailing with Postfix" -date: 2016-10-31 07:57:50 -tags: Tutorial FreeBSD Email Postfix -layout: post ---- - -Welcome to the second part of my FreeBSD email server series. In this series, I -will guide you through setting up your own email service. Be sure to done the -preparations from [part 1][part-1] of this series. - -This part will guide you through setting up email service on your machine using -[Postfix][postfix]. Basic installation is pretty straightforward, but there is -a lot to configure. If you are not sure what some configuration options do, -please read up on them. There is a lot to do wrong with a mail server, and -doing things wrong will likely get you on a blacklist which will make other -servers stop processing the mail you are trying to send out. - -Setting up Postfix is one of the harder parts of configuring a mail server. If -you have questions after reading the full guide, please find me on IRC. You can -find details on how to do so on [my homepage][home]. - -## Installing Postfix -Installation procedures on FreeBSD are pretty straightforward. Unlike `certbot` -from the previous part, we will need to compile Postfix from source in order to -use PostgreSQL as a database back-end. Thanks to FreeBSD's -[ports][freebsd-ports], this is not difficult either. If this is your first -port to compile, you probably need to get the ports tree first. You can -download and extract this using the following command. - -{% highlight sh %} -portsnap fetch extract -{% endhighlight %} - -Once that has finished running, go into the directory containing the build -instructions for Postfix, and start the installation process. - -{% highlight sh %} -cd /usr/ports/mail/postfix -make configure install -{% endhighlight %} - -This will open a pop up with a number of options you can enable or disable. The -enabled defaults are fine, but you will have to enable the `PGSQL` option. This -will allow you to use the configuration tables created in part 1. - -## Enabling Postfix -Enable the Postfix service for rcinit. This allows you to use `service postfix -start` once configuration is done, and will auto start the service on system -boot. In addition, the default mailer on FreeBSD, [sendmail][sendmail] should -be disabled so nothing is in Postfix's way when trying to deal with processing -email traffic. - -{% highlight sh %} -# disable the default sendmail system -echo 'daily_clean_hoststat_enable="NO"' >> /etc/periodic.conf.local -echo 'daily_status_mail_rejects_enable="NO"' >> /etc/periodic.conf.local -echo 'daily_status_include_submit_mailq="NO"' >> /etc/periodic.conf.local -echo 'daily_submit_queuerun="NO"' >> /etc/periodic.conf.local -echo 'sendmail_enable="NONE"' >> /etc/rc.conf.local - -# enable postfix -echo 'postfix_enable="YES"' >> /etc/rc.conf.local -{% endhighlight %} - -## Configuring Postfix -There is a ton to configure for Postfix. This configuration happens in two -files, `main.cf` and `master.cf`. Additionally, as some data is in the -PostgreSQL database, three files with information on how to query for this -information are needed. All of these files are in `/usr/local/etc/postfix`. - -The guide has a comment line for most blocks. It is advised that **if** you -decide to just copy and paste the contents, you copy that along so you have -some sort of indication of what is where. This could help you out if you ever -need to change anything later on. - -### main.cf -#### Compatibility -The configuration file starts off by setting the compatibility level. If -postfix updates the configuration scheme and deprecates certain options, you -will be notified of this in the logs. - -{% highlight ini %} -# compatibility -compatibility_level = 2 -{% endhighlight %} - -#### Directory paths -These options indicate where Postfix will look and keep certain files required -for correct operation. - -{% highlight ini %} -# directory paths -queue_directory = /var/spool/postfix -command_directory = /usr/local/sbin -daemon_directory = /usr/local/libexec/postfix -data_directory = /var/db/postfix -{% endhighlight %} - -#### Domain configuration -The domain configuration instruct the server of the domain(s) it should serve -for. Use your FQDN without sub domains for `mydomain`. You can use a sub domain -for `myhostname`, but you are not required to. The most common setting is -using a `mail` sub domain for all mail related activities, which would -result in something like this. - -{% highlight ini %} -# domain configuration -myhostname = mail.domain.tld -mydomain = domain.tld -myorigin = $mydomain -{% endhighlight %} - -#### Listening directives -All internet devices it should listen on, and all domains this server should -consider itself the endpoint for, should be listed here. The defaults in the -example block are good enough, as we put some of our data in the PostgreSQL -database instead. - -{% highlight ini %} -# listening directives -inet_interfaces = all -mydestination = $myhostname, localhost.$mydomain, localhost -{% endhighlight %} - -#### Reject unknown recipients -How to deal with messages sent to an email address whose domain points to your -server's address, but have no actual mailbox. A code of `550` means to inform -the remote server that delivery is not possible and will not be possible. This -should stop the remote server from trying it again. - -{% highlight ini %} -# reject unknown recipients -unknown_local_recipient_reject_code = 550 -{% endhighlight %} - -#### Trust -{% highlight ini %} -# trust -mynetworks_style = host -{% endhighlight %} - -#### Address extensions -This block is optional. It allows you to use email address extensions. These -are addresses with an additional character in them that will drop the email in -the non extended address' mailbox, but allows you to quickly filter on them as -the sent-to address contains the extension. - -{% highlight ini %} -# address extensions -recipient_delimiter = + -{% endhighlight %} - -#### Virtual domain directives -This part is where things get important. Virtual domains allow you to handle -mail for a large number of domains that are different from the actual server's -domain. This is where the database configuration comes in to play. It is -important to note the `static:125` values. The `125` should map to the `UID` of -the `postfix` user account on your system. - -{% highlight ini %} -# virtual domain directives -virtual_mailbox_base = /srv/mail -virtual_mailbox_domains = pgsql:/usr/local/etc/postfix/pgsql-virtual-domains.cf -virtual_mailbox_maps = pgsql:/usr/local/etc/postfix/pgsql-virtual-users.cf -virtual_alias_maps = pgsql:/usr/local/etc/postfix/pgsql-virtual-aliases.cf -virtual_uid_maps = static:125 -virtual_gid_maps = static:125 -virtual_transport = lmtp:unix:private/dovecot-lmtp -{% endhighlight %} - -#### TLS setup -The TLS setup configures your server to use secure connections. The keys used -here have been generated in the previous part of this series. - -{% highlight ini %} -# TLS setup -smtpd_tls_cert_file = /usr/local/etc/letsencrypt/live/domain.tld/fullchain.pem -smtpd_tls_key_file = /usr/local/etc/letsencrypt/live/domain.tld/privkey.pem -smtpd_use_tls = yes -smtpd_tls_auth_only = yes -{% endhighlight %} - -#### SASL setup -SASL deals with the authentication of the users to your email server. - -{% highlight ini %} -# SASL setup -smtpd_sasl_type = dovecot -smtpd_sasl_path = private/auth -smtpd_sasl_auth_enable = yes -smtpd_recipient_restrictions = - permit_sasl_authenticated, - permit_mynetworks, - reject_unauth_destination -smtpd_relay_restrictions = - permit_sasl_authenticated, - permit_mynetworks, - reject_unauth_destination -{% endhighlight %} - -#### Debugging -The debugging options are generally useful in case things break. If you have -little traffic, you could leave them on forever in case you want to debug -something later on. Once your server is working as intended, you should turn -these options off. The postfix logs get pretty big in a short amount of time. - -{% highlight ini %} -# debugging -debug_peer_level = 2 -debugger_command = - PATH=/bin:/usr/bin:/usr/local/bin:/usr/X11R6/binary - ddd $daemon_directory/$process_name $process_id & sleep 5 -{% endhighlight %} - -#### Installation time defaults -These options should not be touched, but are very important to have for your -server. - -{% highlight ini %} -# install-time defaults -sendmail_path = /usr/local/sbin/sendmail -newaliases_path = /usr/local/bin/newaliases -mailq_path = /usr/local/bin/mailq -setgid_group = maildrop -html_directory = /usr/local/share/doc/postfix -manpage_directory = /usr/local/man -sample_directory = /usr/local/etc/postfix -readme_directory = /usr/local/share/doc/postfix -inet_protocols = ipv4 -meta_directory = /usr/local/libexec/postfix -shlib_directory = /usr/local/lib/postfix -{% endhighlight %} - -### master.cf -For the `master.cf` file, you can use the following configuration block. - -{% highlight cfg %} -submission inet n - n - - smtpd - -o syslog_name=postfix/submission - -o smtpd_tls_security_level=encrypt - -o smtpd_sasl_auth_enable=yes - -o smtpd_reject_unlisted_recipient=no - -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject - -o milter_macro_daemon_name=ORIGINATING -pickup unix n - n 60 1 pickup -cleanup unix n - n - 0 cleanup -qmgr unix n - n 300 1 qmgr -tlsmgr unix - - n 1000? 1 tlsmgr -rewrite unix - - n - - trivial-rewrite -bounce unix - - n - 0 bounce -defer unix - - n - 0 bounce -trace unix - - n - 0 bounce -verify unix - - n - 1 verify -flush unix n - n 1000? 0 flush -proxymap unix - - n - - proxymap -proxywrite unix - - n - 1 proxymap -smtp unix - - n - - smtp -relay unix - - n - - smtp -showq unix n - n - - showq -error unix - - n - - error -retry unix - - n - - error -discard unix - - n - - discard -local unix - n n - - local -virtual unix - n n - - virtual -lmtp unix - - n - - lmtp -anvil unix - - n - 1 anvil -scache unix - - n - 1 scache -{% endhighlight %} - -### SQL query files -The following three configuration files deal with the SQL query files to make -Postfix able of getting some of its configuration from a database. You -obviously have to change the first 4 directives to match your database -authentication credentials. - -#### pgsql-virtual-domains.cf -{% highlight ini %} -user = postgres -password = incredibly-secret! -hosts = 127.1 -dbname = mail -query = SELECT 1 FROM domains WHERE name='%s'; -{% endhighlight %} - -#### pgsql-virtual-users.cf -{% highlight ini %} -user = postgres -password = incredibly-secret! -hosts = 127.1 -dbname = mail -query = SELECT 1 FROM users WHERE local='%u' AND domain='%d'; -{% endhighlight %} - -#### pgsql-virtual-aliases.cf -{% highlight ini %} -user = postfix -password = nope -hosts = 127.1 -dbname = mail -query = SELECT destination FROM aliases WHERE origin='%s'; -{% endhighlight %} - -## Conclusion -This should be enough Postfix configuration, for now. Next part involves -Dovecot, which will enable IMAP. It will also provide the SASL mechanism -defined in this part. - -[freebsd-ports]: https://www.freebsd.org/ports/ -[home]: {{ "/" | prepend: site.baseurl }} -[part-1]: {{ "/post/2016/10/31/freebsd-mailserver-part-1-preparations/" | prepend: site.baseurl }} -[postfix]: http://www.postfix.org/ -[sendmail]: http://www.sendmail.com/sm/open_source/ diff --git a/_posts/2016-10-31-freebsd-mailserver-part-3-dovecot-imap-sasl.md b/_posts/2016-10-31-freebsd-mailserver-part-3-dovecot-imap-sasl.md deleted file mode 100644 index cbb476b..0000000 --- a/_posts/2016-10-31-freebsd-mailserver-part-3-dovecot-imap-sasl.md +++ /dev/null @@ -1,223 +0,0 @@ ---- -title: "FreeBSD email server - Part 3: Dovecot, IMAP and SASL" -date: 2016-10-31 07:57:50 -tags: Tutorial FreeBSD Email Dovecot IMAP SASL -layout: post ---- - -Welcome to the second part of my FreeBSD email server series. In this series, I -will guide you through setting up your own email service. Be sure to read the -previous parts before trying to continue on this part in case you have not done -so yet. - -This part will guide you through setting up [Dovecot][dovecot]. This service -will deal with the SASL authentication to your email server and making your email -boxes accessible via IMAP. While this guide does not cover POP3 functionality, -Dovecot can handle this as well. - -Just like the Postfix setup, Dovecot has quite a few configuration options to -set before it will work as expected in this setup. If you have questions after -reading the full guide, please find me on IRC. You can find details on how to -do so on [my homepage][home]. - -## Installing Dovecot -Dovecot will also be installed from the ports tree from FreeBSD. As this guide -assumes you are working through them in order, explanation of acquiring the -ports tree will be omitted here. - -You can start the installation procedure with the following commands. - -{% highlight sh %} -cd /usr/ports/mail/dovecot2 -make configure install -{% endhighlight %} - -Again, like with the Postfix installation, leave the default options on and add -the `PGSQL` option so Dovecot can use PostgreSQL as the database back-end. - -## Enabling Dovecot -Enable the Dovecot service for rcinit. - -{% highlight sh %} -echo 'dovecot_enable="YES"' >> /etc/rc.conf.local -{% endhighlight %} - -## Configuring Dovecot -To start of with Dovecot configuration, copy over the sample files first. - -{% highlight sh %} -cp -r /usr/local/etc/dovecot/example-config/* /usr/local/etc/dovecot/. -{% endhighlight %} - -Now you can start editing a number of pesky files. The file names of the -headings all appear relative to `/usr/local/etc/dovecot`. - -### dovecot.conf -Here you only have to set which protocols you want to enable. Set them as -follows. - -{% highlight ini %} -protocols = imap lmtp -{% endhighlight %} - -### conf.d/10-master.cf -The `master.cf` configuration file indicates which sockets Dovecot should use -and provide and as which user its processes should be ran. Keep the defaults as -they are, with the exception of the following two blocks. - -#### service imap-login -This will enable imaps, IMAP over SSL, and disable plain IMAP. - -{% highlight ini %} -service-imap-login { - inet_listener imap { - port = 0 - } - - inet_listener imaps { - port = 993 - ssl = yes - } -} -{% endhighlight %} - -#### services -This will instruct Dovecot to provide a service for authentication and `lmtp` -the **local mail transport protocol**. This is required to deliver the email -files into the correct email box location in the file system. - -{% highlight ini %} -service auth { - unix_listener auth-userdb { - mode = 0600 - user = postfix - group = postfix - } - - unix_listener /var/spool/postfix/private/auth { - mode = 0666 - user = postfix - group = postfix - } - - user = dovecot -} - -service lmtp { - unix_listener /var/spool/postfix/private/dovecot-lmtp { - mode = 0600 - user = postfix - group = postfix - } -} - -service auth-worker { - user = postfix -} -{% endhighlight %} - -### conf.d/10-ssl.conf -Here you have to enable SSL and provide the correct paths to your SSL key in -order for Dovecot to work with them. - -{% highlight ini %} -ssl = required -ssl_cert = < /usr/local/etc/letsencrypt/live/domain.tld/fullchain.pem -ssl_key = < /usr/local/etc/letsencrypt/live/domain.tld/privkey.pem -{% endhighlight %} - -### conf.d/10-mail.conf -The mail.conf location instructs Dovecot which location to appoint for storing -the email files. `%d` expands to the domain name, while `%n` expands to the -local part of the email address. - -{% highlight ini %} -mail_home = /srv/mail/%d/%n -mail_location = maildir:~/Maildir -{% endhighlight %} - -Make sure the location set by `mail_home` exists and is owned by `postfix`! - -{% highlight sh %} -mkdir -p /srv/mail -chown postfix:postfix /srv/mail -{% endhighlight %} - -### conf.d/10-auth.conf -This file deals with the authentication provided by Dovecot. Mostly, which -mechanisms should be supported and what mechanism should be used to get the -actual credentials to check against. Make sure the following options are set -as given - -{% highlight ini %} -disable_plaintext_auth = yes -auth_mechanisms = plain -{% endhighlight %} - -Also, make sure `!include auth-system.conf.ext` is commented **out**. It is not -commented out by default, so you will have to do this manually. In addition, -you have to uncomment `!include auth-sql.conf.ext`. - -### conf.d/auth-sql.conf.ext -This is the file included from `10-auth.conf`. It instructs Dovecot to use SQL as -the driver for the password and user back-ends. - -{% highlight ini %} -passdb { - driver = sql - args = /usr/local/etc/dovecot/dovecot-sql-conf.ext -} - -userdb { - driver = prefetch -} - -userdb { - driver = sql - args = /usr/local/etc/dovecot/dovecot-sql-conf.ext -} -{% endhighlight %} - -### dovecot-sql.conf.ext -The final configuration file entails the queries which should be used to get the -required information about the users. Make sure to update the `password` and possibly -other parameters used to connect to the database. You may have to update the `125` as -well, as this has to be identical to the `UID` of `postfix`. - -As a side note, if you are following this tutorial on a machine that does -**not** support Blowfish in the default glib, which is nearly every GNU+Linux -setup, you **can not** use `BLF-CRYPT` as the `default_pass_scheme`. You will -have to settle for the `SHA-512` scheme instead. - -{% highlight ini %} -driver = pgsql -connect = host=127.1 dbname=mail user=postfix password=incredibly-secret! -default_pass_scheme = BLF-CRYPT -password_query = \ - SELECT \ - local AS user, \ - password, \ - '/srv/mail/%d/%n' AS userdb_home, \ - 125 AS userdb_uid, \ - 125 AS userdb_gid \ - FROM users \ - WHERE local='%n' AND domain='%d'; - -user_query = \ - SELECT \ - '/srv/mail/%d/%n' AS home \ - 125 AS uid, \ - 125 AS gid \ - FROM users \ - WHERE local='%n' AND domain='%d'; -{% endhighlight %} - -## Conclusion -After this part, you should be left with a functioning email server that -provides IMAP over a secure connection. While this is great on itself, for -actual use in the wild, you should setup some additional services. Therefore, -in the next part, we will deal with practices that "authenticate" your emails -as legit messages. Be sure to read up on it! - -[dovecot]: http://dovecot.org/ -[home]: {{ "/" | prepend: site.baseurl }} diff --git a/_posts/2016-10-31-freebsd-mailserver-part-4-message-authentication.md b/_posts/2016-10-31-freebsd-mailserver-part-4-message-authentication.md deleted file mode 100644 index e5cee0e..0000000 --- a/_posts/2016-10-31-freebsd-mailserver-part-4-message-authentication.md +++ /dev/null @@ -1,155 +0,0 @@ ---- -title: "FreeBSD email server - Part 4: Message authentication" -date: 2016-10-31 20:00:38 -tags: Tutorial FreeBSD Email DKIM SPF -layout: post ---- - -Welcome to another part in the FreeBSD email server series. This time, we are -going to setup some mechanisms to deal with message authentication. This -practice will make other email providers accept your email messages and deliver -them properly in the inbox of the receiving user, instead of their spam box. - -We will do so using three of the most common practices: [SPF][spf], -[DKIM][dkim] and [DMARC][dmarc]. - -## DKIM -### Installation -The tools for DKIM are easily installed using `pkg`. - -{% highlight sh %} -pkg install opendkim -{% endhighlight %} - -### Configuration -Write the following configuration into `/usr/local/etc/mail/opendkim.conf`. - -{% highlight apache %} -# logging -Syslog yes - -# permissions -UserID postfix -UMask 007 - -# general settings -AutoRestart yes -Background yes -Canonicalization relaxed/relaxed -DNSTimeout 5 -Mode sv -SignatureAlgorithm rsa-sha256 -SubDomains no -X-Header yes -OversignHeaders From - -# tables -KeyTable /usr/local/etc/opendkim/key.table -SigningTable /usr/local/etc/opendkim/signing.table - -# socket -Socket inet:8891@localhost - -# domains -Domain domain.tld.privkey -KeyFile /usr/local/etc/opendkim/domain.tld -Selector mail -{% endhighlight %} - -#### Postfix -Postfix needs to be instructed to sign the messages with a DKIM header using -the opendkim service. You can do so by inserting the following configuration -block somewhere around the end of `/usr/local/etc/postfix/main.cf`. - -{% highlight ini %} -# milters -milter_protocol = 2 -milter_default_action = reject -smtpd_milters = - inet:localhost:8891 -{% endhighlight %} - -#### System service -OpenDKIM runs as a system service. As such, you will have to enable this -service in rcinit. This is a simple step, achieved with the given command. - -{% highlight sh %} -echo 'milteropendkim_enable="YES"' >> /etc/rc.conf.local -{% endhighlight %} - -Do not forget to actually start the service when you are done with the -tutorial! - -### Creating and using keys -In order to use DKIM, you will need to generate some keys to sign the messages -with. You cannot use your Let's Encrypt SSL keys for this. First, create a -directory to house your domain's keys. - -{% highlight sh %} -mkdir -p /usr/local/etc/opendkim/keys/domain.tld -chown -R postfix:wheel $_ -{% endhighlight %} - -Next up, generate your first key. - -{% highlight sh %} -opendkim-genkey -D /usr/local/etc/opendkim/keys -b 4096 -r -s $(date +%Y%m%d) -d domain.tld -{% endhighlight %} - -I tend to use the current date for the key names so I can easily sort them by -the most recent one. - -Afterwards, you will have to add a line to two separate files to instruct DKIM -to use this key for a certain domain when signing mail. These are fairly -straightforward and can be done using a simple `echo` as well. - -{% highlight sh %} -echo '*@domain.tld domain.tld' >> /usr/local/etc/opendkim/signing.table -echo "domain.tld domain.tld:$(date +%Y%m%d):/usr/local/etc/opendkim/keys/domain.tld/$(date +%Y%m%d).private" \ - >> /usr/local/etc/opendkim/key.table -{% endhighlight %} - -### Adding the DNS records -You may have already noticed that `opendkim-genkey` also creates a `.txt` file -in addition to the private key. This text file contains the DNS record value -you need to add for your domain's DNS. Add the record to your DNS server, and -simply wait for it to propagate. - -## SPF -SPF is simply a DNS record that shows which IPs are allowed to email for that -domain. - -### Adding the DNS records -A simple example for an SPF record is the following. It allows mail to be sent -in the domain's name from any IP listed in the MX records. - -``` -v=spf1 mx -all -``` - -## DMARC -DMARC is, like SPF, a DNS record. It tells how to deal with messages coming -from the server and where to report abuse of your server. Some of the larger -email providers send out reports to the address given in the DMARC record so -you can figure out whether someone is spamming from your servers, for example. - -### Adding the DNS records -A simple DMARC policy to get started with is to quarantine all emails that fail -authentication. This means the emails will go into the receiving user's spam -box. In addition, abuse reports will be sent to the address defined in the -`rua`. - -``` -v=DMARC1; p=quarantine; rua=mailto:abuse@domain.tld -``` - -## Conclusion -These few simple measures will make receiving servers trust the authenticity of -the mails you send. In effect, your messages will be much less likely to be -marked as spam. However, you are a target of spam as well. How you can deal -with that, will be available in the next part of this series. - -[dkim]: http://www.dkim.org/ -[dmarc]: http://dmarc.org/ -[spf]: https://en.wikipedia.org/wiki/Sender_Policy_Framework - diff --git a/_posts/2016-10-31-freebsd-mailserver-part-5-filtering-mail.md b/_posts/2016-10-31-freebsd-mailserver-part-5-filtering-mail.md deleted file mode 100644 index d1797eb..0000000 --- a/_posts/2016-10-31-freebsd-mailserver-part-5-filtering-mail.md +++ /dev/null @@ -1,127 +0,0 @@ ---- -title: "FreeBSD email server - Part 5: Filtering mail" -date: 2016-10-31 20:02:19 -tags: Tutorial FreeBSD Email Postfix SpamAssassin Pigeonhole -layout: post ---- - -Being able to send mail and not be flagged as spam is pretty awesome on itself. -But you also get hit by a lot of spam. The more you give out your email address -and domain name, the more spam you will receive over time. I welcome you to -another part of the FreeBSD email server series. In this part, we will set up -email filtering at the server side. - -We will accomplish this with a couple packages, [SpamAssassin][spamassassin] -and [Pigeonhole][pigeonhole]. The former deals with scanning the emails to -deduce whether it is spam or not. The latter filters messages. We will use this -filtering to drop emails marked as spam by SpamAssassin into the Junk folder, -instead of the inbox. - -## Installing the packages -Both packages are available through FreeBSD's `pkg` utility. Install them as -such. - -{% highlight sh %} -pkg install dovecot-pigeonhole spamassassin -{% endhighlight %} - -## SpamAssassin -### Enabling the service -Like most services, you have to enable them as well. Pigeonhole is an extension -to Dovecot, and Dovecot will handle this one. SpamAssassin requires you to -configure the service as well. You can enable it and set sane configuration to -it with the following two commands. - -{% highlight sh %} -echo 'spamd_enable="YES"' >> /etc/rc.conf.local -echo 'spamd_flags="-u spamd -H /srv/mail"' >> /etc/rc.conf.local -{% endhighlight %} - -### Acquiring default spam rules -SpamAssassin has to "learn" what counts as *spam* and what counts as *ham*. To -fetch these rules, you should execute the updates for SpamAssassin with the -following command. - -{% highlight sh %} -sa-update -{% endhighlight %} - -You most likely want to run this once every while, so it is advised to setup a -cronjob for this purpose. - -## Postfix -In order to have mails checked by SpamAssassin, Postfix must be instructed to -pass all email through to SpamAssassin, which will hand them back with a -`X-Spam-Flag` header attached to them. This header can be used by other -applications to treat it as spam. - -### master.cf -There's not much to include to the already existing Postfix configuration to -enable SpamAssassin to do its job. Just open `/usr/local/etc/postfix/master.cf` -and append the block given below. - -{% highlight ini %} -spamassassin unix - n n - - pipe - user=spamd argv=/usr/local/bin/spamc - -f -e /usr/sbin/sendmail -oi -f ${sender} ${recipient} -{% endhighlight %} - -## Pigeonhole -Pigeonhole is an implementation of Sieve for Dovecot. It deals with filtering -messages on the server side using a set of rules, defined in a file usually -named `sieve`. This file is generally saved at -`/srv/mail/domain.tld/user/sieve`. A default file to filter spam out is the -following example. - -``` -require [ - "fileinto", - "mailbox" -]; - -if header :contains "X-Spam-Flag" "YES" { - fileinto :create "Junk"; - stop; -} -``` - -This looks for the `X-Spam-Flag` header, which is added by SpamAssassin. If it -is set to `YES`, this indicates SpamAssassin thinks the message is spam. As -such, sieve is instructed to filter this message into the folder `Junk`, and to -create this folder if it does not exist yet. The `stop;` makes sieve stop -trying to process this message further based on later rules. - -## Dovecot -Dovecot needs some additional configuration to work with Pigeonhole. Modify the -following files and add the contents described. - -### conf.d/20-lmtp.conf -This will enable Pigeonhole in Dovecot. - -{% highlight ini %} -protocol lmtp { - mail_plugins = $mail_plugins sieve -} -{% endhighlight %} - -### conf.d/90-plugin.conf -This configures Pigeonhole to look for a file named `sieve` in the mailbox -homedir, and execute that when delivering mail. - -{% highlight ini %} -plugin { - sieve = /srv/mail/%d/%n/sieve -} -{% endhighlight %} - -## Conclusion -Spam is a pain, especially if you get a lot of it. The configuration added in -this part of the FreeBSD email server series should get rid of most of it. This -also concludes the series. If you have any questions or suggestions, please -contact me via any of the methods detailed on [my home page][home]. - -Thanks for reading along, and enjoy your very own email server! - -[home]: {{ "/" | prepend: site.baseurl }} -[pigeonhole]: http://pigeonhole.dovecot.org/ -[spamassassin]: https://spamassassin.apache.org/ diff --git a/_posts/2016-11-24-freebsd-mailserver-calendars-and-contacts.md b/_posts/2016-11-24-freebsd-mailserver-calendars-and-contacts.md deleted file mode 100644 index 0e7d953..0000000 --- a/_posts/2016-11-24-freebsd-mailserver-calendars-and-contacts.md +++ /dev/null @@ -1,137 +0,0 @@ ---- -title: "FreeBSD email server - Part +: Calendars and contacts" -date: 2016-11-24 08:26:09 -tags: Tutorial FreeBSD Email CalDAV CardDAV -layout: post ---- - -This guide is an addition to the [FreeBSD email server series][tutorial-email]. -It is not required for your email server to operate properly, but it is often -considered a very important feature for those who want to switch from a third -party email provider to their own solution. It does build upon the completed -series, so be sure to work through that before starting on this. - -## Install required packages -{% highlight sh %} -pkg install py27-radicale -{% endhighlight %} - -## Configure Radicale -### /usr/local/etc/radicale/config -Open up the `/usr/local/etc/radicale/config` file, and update each `[block]`. - -#### [server] -The server is binding to `localhost` only. This way it is not accessible on -`:5232` from outside the server. Outside access will be provided through an -nginx reverse proxy instead. - -{% highlight ini %} -hosts = 127.1:5232 -daemon = True - -dns_lookup = True - -base_prefix = / -can_skip_base_prefix = False - -realm = Radicale - Password required -{% endhighlight %} - -#### [encoding] -{% highlight ini %} -request = utf-8 -stock = utf-8 -{% endhighlight %} - -#### [auth] -{% highlight ini %} -type = IMAP - -imap_hostname = localhost -imap_port = 143 -imap_ssl = False -{% endhighlight %} - -#### [storage] -{% highlight ini %} -type = filesystem -filesystem_folder = /usr/local/share/radicale -{% endhighlight %} - -#### [logging] -{% highlight ini %} -config = /usr/local/etc/radicale/logging -{% endhighlight %} - -### /usr/local/etc/radicale/logging -This file is fine on the defaults in FreeBSD 11. This saves you from -configuring a little bit. - -## Configure Dovecot -### Enable imap -This option was disabled in the [IMAP server tutorial][tutorial-email], -however, if we want to auth using the same credentials as the mailserver, this -option is needed again. Bind it to `localhost`, so it can only be used -internally. In `/usr/local/etc/dovecont/conf.d/10-master.conf`, enable the -`imap` port again: - -``` -... -service imap-login { - inet_listener imap { - address = 127.1 - port = 143 - } - ... -} -... -``` - -## Configure nginx -To make using the service easier, you can setup [nginx][nginx] to act as a -reverse proxy. If you followed the [webserver tutorial][tutorial-webserver], -you already have the basics for this set up. I do recommend you check this out, -as I will only explain how to configure a virtual host to deal with the reverse -proxy here. - -### Setup a reverse proxy -Assuming you have taken the crash-course in setting up the nginx webserver, you -can attain a reverse proxy using the following config block. Note that this block -only does HTTPS, as I use HTTP only to redirect to HTTPS. - -{% highlight nginx %} -# static HTTPS -server { - # listeners - listen 443 ssl; - server_name radicale.domain.tld; - - # enable HSTS - add_header Strict-Transport-Security "max-age=31536000; includeSubdomains; preload"; - - # keys - ssl_certificate /usr/local/etc/letsencrypt/live/domain.tld/fullchain.pem; - ssl_certificate_key /usr/local/etc/letsencrypt/live/domain.tld/privkey.pem; - - # / handler - location / { - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_pass http://127.1:5232; - } -} -{% endhighlight %} - -## Enable the service at startup -{% highlight sh %} -echo 'radicale_enable="YES"' >> /etc/rc.conf.local -{% endhighlight %} - -## Start the server -{% highlight sh %} -service radicale start -{% endhighlight %} - -[nginx]: https://www.nginx.com/ -[tutorial-email]: {{ "/post/2016/10/31/freebsd-mailserver-part-1-preparations/" | prepend: site.baseurl }} -[tutorial-webserver]: {{ "/post/2016/10/25/setup-nginx-with-lets-encrypt-ssl/" | prepend: site.baseurl }} diff --git a/_posts/2017-09-14-how-to-git.md b/_posts/2017-09-14-how-to-git.md deleted file mode 100644 index b16a6ae..0000000 --- a/_posts/2017-09-14-how-to-git.md +++ /dev/null @@ -1,184 +0,0 @@ ---- -title: "How to: git" -date: 2017-09-14 19:46:34 -tags: Tutorial Git -layout: post -authors: - - ["Patrick Spek", "http://tyil.work"] ---- - -This guide will explain how to use `git` more efficiently, and why you should -use it as such. - -## Forking -When working in a team, there's generally a remote server which is used to sync -your repositories. There are gratis services, such as [GitHub][github], -[Gitlab][gitlab], [GOGS][gogs], and others. These services also allow you to -*fork* a repository. This basically makes a copy of the entire repository for -your own use. In it, you have full control over the branches, tags, merge -process and everything else you want to do with it. - -One the main reasons to do this is so you do not have to clutter up the main -repository with a ton of branches (these are explained later in the post). If -there are two people working in the same branch, it can help reduce conflicts, -as each developer is working on the branch in his own fork. - -As such, **always** use a fork. If the service does not have a fancy button for -you to click, you can still fork manually. Simply clone their repository as -usual, set a new remote and push it there: - -{% highlight sh %} -git clone git@domain.tld:them/repo.git -cd repo -git remote rename origin upstream -git remote add origin git@domain.tld:you/repo.git -git push origin master -{% endhighlight %} - -The default naming convention uses `upstream` for the base of your fork, and -`origin` for your remote version of the repository. If a merge request is -accepted on the original repo, you can apply it to your fork using - -{% highlight sh %} -git pull upstream master -{% endhighlight %} - -## Branching -Branching is the art of using separate branches to introduce new code into your -`master` branch. Every git repository starts with a `master` branch by default. -This is the *main* branch of your repository. - -Every time you want to add new code to your project, make a branch for the -feature or issue you are trying to solve. This way, you can commit freely -without having to worry about having untested or possibly broken code in the -`master` branch. If something were to come up with a higher priority, such as a -critical bug, you can simply create a new branch off of `master`, fix it and -merge that back into `master`, without having to worry about that other feature -you were working on, which is not in a releasable state yet. Once the fix is -applied, you go back to your feature branch on continue working on the cool new -stuff you wanted to implement. Now, the bug is fixed, and no code has been -released that should not have been released. If that's not convincing enough, -try some of the [Stack Overflow posts][so-git-branch] on this very topic. - -Branches can be made at your leisure, with next to no overhead on your project. -Do not be scared to play around with your code in a new branch to test -something out. You can also delete branches as quickly as you made them if you -are not satisfied with the result. - -Creating branches is done using `git checkout -b new-branch`. If you need to -switch to another existing branch to change something, use -`git checkout other-branch`. Deleting a branch can be done using -`git branch -D old-branch`. You can get a list of all branches in the -repository with `git branch`. The current branch is marked with an \*. - -If you start a new branch to implement a feature, be sure to always branch off -of `master`, unless you have a very compelling reason not to do so. If you are -not sure what reasons would validate branching off of another branch, you -should just branch off of `master`. If you branch off of another branch, you -will have the commit history of the other branch. This often includes commits -not accepted into master yet, which might result into commits getting into -master which should not be there (yet), or annoying merge conflicts later on. - -### Merging -Using multiple branches brings along the concept of *merging* branches -together. When working in a group, this is generally done by maintainers of the -upstream repository, via a *merge request*. For some reason, certain services -have named this as a *pull request* instead. The base idea of the process is as -follows: - -- Pull the latest `upstream/master` -- Create a new branch -- Apply the change you want -- Issue a merge request via the service you are using - - Generally, you want your change to be merged into their `master` branch -- Add a title and a description of your change: What does it do, and why should it be accepted -- Optionally, discuss the changes with the upstream maintainers -- Optionally, make a couple of changes to your branch, and push it again -- Upstream maintainer accepts your change - -When everything worked out, the upstream repository now contains your changes. -If you pull their branch again, it will contain your code. Using the merge -request process, your code can be easily reviewed by others, and discussed if -needed. - -## Committing -Whenever you have changed anything in the repository and you wish to share -these changes, you have to commit the changes. Committing in general is not -something people tend to have issues with. Simple add the changes you want to -commit using `git add` (add the `-p` switch if you want to commit only parts of -a changed file), then `git commit` and enter a descriptive message. And that is -where most annoyances come from: the commit *message*. There are no hard rules -on this forced by git itself. There are, however, some de-facto standards and -best practices which you should always follow. Even if you never intend to -share the repository with other people, having good commit messages can help -you identify a certain change when you look back into the history. - -A git commit message should be short, no more than 79 characters, on the first -line. It should be readable as "this commit message will ...", where your -commit message will replace the "...". It is a de-facto standard to start your -commit message with a capital letter, and leave off a finishing period. You do -not *have* to adhere to if you hate this, but be sure that all your commits are -consistent in how they are formatted. - -If you need to explain anything beyond that, such as a rationale for the -change, or things the reviewer should pay attention to in this particular -commit, you can leave an empty line and publish this message in the commit -body. - -When you are using a bug tracking system, you might also want to have a footer -with additional information. On services such as [Gitlab][gitlab] and -[GitHub][github], you can close issues by adding "Closes: #1" in the commit -message footer. A full commit message with all these things might look as -follows: - -``` -Fix overflow issue in table rendering mechanism - -An overflow issue was found in the table rendering mechanism, as explained in -CVE-0123-45678. Regression tests have been included as well. - -Closes: #35 -``` - -In order to achieve these kind of messages, you need to be sure that your -commits can fit in to this structure. This means you need to make small -commits. Having many smaller commits makes it easier to review the changes, -keep short, descriptive messages to describe each change, and revert a single -change in case it breaks something. - -### Signing your commits -You can set up git to cryptographically sign each commit you make. This will -ensure that the commit you made is proven to be from you, and not someone -impersonating you. People impersonating you might try to get harmful code into -a repo where you are a trusted contributor. Having all commits signed in a -repository can contribute in verifying the integrity of the project. - -Recently, [Github][github] has added the **Verified** tag to commits if the -commit contains a correct signature. - -To enable signing of all commits, add the following configuration to your -`~/.gitconfig`: - -{% highlight ini %} -[commit] - gpgsign = true - -[user] - signingkey = 9ACFE193FFBC1F50 -{% endhighlight %} - -Ofcourse, you will have to update the value of the `signingkey` to match -the key you want to sign your commits with. - -## Closing words -I hope this post will help you in your adventures with git. It is a great tool -or working on projects together, but it gets much better when you stick to some -best practices. If you have any suggestions for this post, or any questions -after finishing it, contact me via any method listed on [my home page][home]. - -[github]: https://github.com -[gitlab]: https://gitlab.com -[gogs]: https://gogs.io -[home]: https://tyil.work -[so-git-branch]: https://softwareengineering.stackexchange.com/questions/335654/git-what-issues-arise-from-working-directly-on-master - diff --git a/_posts/2017-09-28-perl6-creating-a-background-service.adoc b/_posts/2017-09-28-perl6-creating-a-background-service.adoc deleted file mode 100644 index 4cc759d..0000000 --- a/_posts/2017-09-28-perl6-creating-a-background-service.adoc +++ /dev/null @@ -1,159 +0,0 @@ ---- -date: 2017-09-28 15:11:43 -tags: Tutorial Perl6 Programming Raku -description: > - I've recently made some progress on Shinrin, a centralized logging system in - Perl 6. This has to run as service, which means that for most service - managers it has to be able to run in the background. ---- -= Perl 6 - Creating a background service -:toc: preamble - -I've recently made some progress on -https://github.com/scriptkitties/perl6-Shinrin[Shinrin], a centralized logging -system in Perl 6. This has to run as service, which means that for most service -managers it has to be able to run in the background. - -[NOTE] -==== -If you just want to get to the solution and don't care for the details, just -head straight to link:#the-final-solution[the full script]. -==== - -== It's not possible! -After a lot of trying and talking with the folks at -irc://chat.freenode.net:6697/#perl6[#perl6] I was told that it is not possible -to do this in pure Perl 6, explained by people with more knowledge than I have -on the internals: - -[quote, jnthn] -____ -(jnthn suspects fork + multi-threaded VM = pain) Since fork only clones one -thread - the one that called it. So suddenly you've got an instance of the VM -missing most of its threads. -____ - -[quote, geekosaur] -____ -The most common failure mode is that some thread is holding e.g. a mutex (or a -userspace lock) during the fork. The thread goes away but the lock is process -level and remains, with nothing around to know to unlock it. So then things -work until something else needs that lock and suddenly you deadlock. -____ - -Not much later, `jnthn` https://github.com/perl6/doc/commit/8f9443c3ac[pushed a -commit] to update the docs to clarify that a `fork` call through `NativeCall` -will probably not give the result you were hoping for. - -== Or is it? -Luckily, the same people were able to think up of a work-around, which can be -made in POSIX sh, so it's usable on any decent OS. The workaround is to let a -little shell script fork into the background, and let that run the Perl -application. - -=== A first example -This is fairly simple to create, as in this example to launch `shinrind` in the -background: - -[source,sh] ----- -#! /usr/bin/env sh - -main() -{ - perl6 -Ilib bin/shinrind "$@" -} - -main "$@" & ----- - -This works just fine if the working directory is correct. This means you need -to be in the parent directory to `lib` and `bin` of the program to make it -work. - -== Improving the forking script -While that short script works fine to show a proof of concept, in order to make -it viable for real-world scenarios, it can use some improvements. After all, it -would be annoying if you'd have to `cd` to a specific directory any time you -want to start your application. - -=== Ensure you are in the directory you should be in -So for starters, let's make sure that you can run it from anywhere on your -system. For this, you should set the working directory for the script, so you -don't have to do it manually. Because the script runs in its own subshell, the -shell you're working from remains unaffected. - -A POSIX compliant way to get the directory the script is stored in is as -follows: - -[source,sh] ----- -DIR=$(CDPATH="" cd -- "$(dirname -- "$0")" && pwd) ----- - -This will set `$DIR` to the path of the directory the shell script is stored -in. You can simply `cd` to that and be assured you're in the right directory. - -In Perl 6, it is expected for executable files to live in the `bin` directory -of your project repository. So you should actually be in the parent of the -directory holding your script. Furthermore, you should check the `cd` command -executed correctly, just to be safe. - -[source,sh] ----- -cd -- "${DIR}/.." || exit ----- - -=== Disable `STDOUT` and `STDERR` -A started service should not be polluting your interactive shell, so you should -disable (or otherwise redirect) `STDOUT` and `STDERR`. This is done in the -shell using a small bit of code behind whatever you want to redirect: - -[source,sh] ----- -> /dev/null 2>&1 ----- - -This will set `STDOUT` to `/dev/null`, and set `STDERR` to the same stream as -`STDOUT`, which in effect will make all output go to `/dev/null`. If you want -to log everything to a single file, you can replace `/dev/null` with another -file of your choice. If you don't want logs to be overwritten on each start, -use a `>>` instead of a single `>` at the start. - -If you want to log errors and output in different files, you can use the -following: - -[source,sh] ----- -> /var/log/service.log 2> /var/log/service.err ----- - -This will put standard output in `/var/log/service.log` and errors in -`/var/log/service.err`. - -=== Fork just the Perl 6 program -In the initial example, I put the `&` behind the `main` call, at the bottom of -the script. While this works just fine for most simple usage, if you want to do -additional chores, like creating a pidfile after starting the Perl 6 program, -you're out of luck. If you were to only fork the Perl 6 application, you could -handle some other cases in the shell script. - -== The final solution -For those eager to just get going with this, here is the complete example -script to just fork your Perl program into the background: - -[source,sh] ----- -#! /usr/bin/env sh - -readonly DIR=$(CDPATH="" cd -- "$(dirname -- "$0")" && pwd) - -main() -{ - cd -- "${DIR}/.." || exit - - perl6 -Ilib bin/shinrind "$@" > /dev/null >2&1 & -} - -main "$@" ----- diff --git a/_posts/2017-11-01-hacktoberfest-2017.adoc b/_posts/2017-11-01-hacktoberfest-2017.adoc deleted file mode 100644 index 67c7746..0000000 --- a/_posts/2017-11-01-hacktoberfest-2017.adoc +++ /dev/null @@ -1,197 +0,0 @@ ---- -date: 2017-11-01 12:37:53 -tags: FreeSoftware Github Hacktoberfest Contributions -description: > - This year I actively participated in the Hacktoberfest event, which is 'a - month-long celebration of open source software'. Ironic, given that the - companies organising it don't have their own software stack open source. - Nevertheless, I did contribute to free software, and this article lists these - contributions. ---- -= Hacktoberfest 2017 -:toc: preamble - -This year I actively participated in the Hacktoberfest event, which is "a -month-long celebration of open source software". Ironic, given that the -companies organising it don't have their own software stack open source. - -I've found some issues to solve in https://perl6.org/[Perl 6] projects, and -that lead to trying to solve issues in some other projects, and eventually I -got more PRs out than there are days in the month. It did go at the cost of -some sleep, but in the end it seems worth it. In this article, I'll give a -small overview of all those PRs, in no particular order. - -== Projects contributed to -=== Funtoo -==== funtoo/boot-update -- https://github.com/funtoo/boot-update/pull/14 - -When reinstalling my server to try out https://docker.com[Docker], I noticed an -error in the output of the `boot-update` utility, a tool from -https://www.funtoo.org/Welcome[Funtoo] to make installing and configuring the -bootloader easier. The error itself was a small type of a `-` which had to be a -`_`. - -==== scriptkitties/overlay -- https://github.com/scriptkitties/overlay/pull/14 -- https://github.com/scriptkitties/overlay/pull/15 -- https://github.com/scriptkitties/overlay/pull/16 - -This is the overlay of the https://scriptkitties.church[Scriptkitties] -community. It's got some additional software released under a free license that -is not available in the main portage repository. Most of the packages in here -are of software made by the Scriptkitties community. - -This month I updated the readme to be in asciidoc, my new favourite format for -documentation. The Travis builds should also no longer throw errors, so those -can be used again to ensure the overlay is meeting quality standards. One -package has also been updated to be at it's latest version again. - -=== Perl 6 -==== moznion/p6-HTML-Escape -- https://github.com/moznion/p6-HTML-Escape/pull/1 - -On this repository, I added a subroutine to also handle unescaping HTML special -characters. Sadly, the owner of this repository has shown no sign of life, and -the PR remains open. - -==== rakudo/rakudo -- https://github.com/rakudo/rakudo/pull/1180 - -This is a rather small issue, but I noticed it when compiling Perl 6 with -https://github.com/tadzik/rakudobrew[Rakudobrew] and it annoyed me. -http://zoffix.com/[Zoffix] was a great help in getting me started on this one, -and in general with many other Perl related contributions as well. - -==== scriptkitties/perl6-IRC-Client-Plugin-Github -- https://github.com/scriptkitties/perl6-IRC-Client-Plugin-Github/pull/2 - -A neat feature for the Github notification system, HMAC adds a header that can -be used to verify the body of the request, and can be used to verify the other -end of the connection knows the right "secret". Inspired by a Perl 6 bot that -already did this, I made a PR to make this a proper -https://github.com/zoffixznet/perl6-IRC-Client[IRC::Client] plugin. It is still -being tested in https://github.com/scriptkitties/musashi[musashi]. - -==== perl6/roast -- https://github.com/perl6/roast/pull/342 - -Roast is the test suite for Perl 6. There was an open issue for the IO::File -tests, which needed expansion. As my first contribution during a Perl 6 -squashaton, I expanded these tests to fix the issue that was open for it. - -==== vim-perl/vim-perl6 -- https://github.com/vim-perl/vim-perl6/pull/9 -- https://github.com/vim-perl/vim-perl6/pull/10 - -This first PR has become a bit of a drag, with the maintainers not responding -for two weeks, but suddenly very eager to respond when I mention I'm going to -fork off and update the reference on the Perl documentation to my fork. -Nonetheless, it's sorted out, and the abbreviations for unicode operators -have been merged in! - -==== timo/json_fast -- https://github.com/timo/json_fast/pull/32 - -`JSON::Fast` is the de-facto standard for dealing with JSON data in Perl 6 it -seems. For my work with `App::Cpan6` I wanted the JSON data to be ordered, so I -added that as an option when calling `to-json`. Having the JSON data ordered -makes it easier to compare diffs of two different versions of the data, making -git diffs a lot cleaner. - -Sadly, timo has not merged the PR yet, so I can't properly depend on it in -`App::Cpan6`. - -==== scriptkitties/perl6-SemVer -- https://github.com/scriptkitties/perl6-SemVer/pull/1 - -This is one of the new projects I started. It is intended to be used in -`App::Cpan6`, since that uses https://semver.org[Semantic Versioning] for all -modules it works with. This module defines a class that can interpret a SemVer -notation, and exposes methods to bump any part of the version. - -==== perl6/doc -- https://github.com/perl6/doc/pull/1614 - -This has been one of the more annoying PRs to work on, as the current `zef` -maintainer insists everything but his module is wrong, and seemed very -uninterested to improve the situation for users. After some discussion on IRC, -some more discussion on IRC, and then some discussion on the PR itself, I -decided to just word the paragraph differently. - -I am still interested in improving the documentation here and the ecosystem -itself, mainly the `META6.json` specification, and getting `zef` to play nice -with this spec. If anyone else is interested in helping me out on this, do -message me on IRC! - -==== perl6/perl6.org -- https://github.com/perl6/perl6.org/pull/86 -- https://github.com/perl6/perl6.org/pull/87 - -There were some open issues for the https://perl6.org[perl6.org] website, and I -decided to take a look at some and try to fix them. This resulted in NeoVim -being added to the list of recommended editors for Perl 6, and the list of IRC -bots being updated to include all bots in use right now. - -==== scriptkitties/p6-MPD-Client -- https://github.com/scriptkitties/p6-MPD-Client/pull/1 -- https://github.com/scriptkitties/p6-MPD-Client/pull/2 - -As I was making `App::MPD::AutoQueue` and `App::MPD::Notify`, I found some -issues in `MPD::Client`. I fixed those to get my two new projects working -nicely. - -==== melezhik/sparrowdo -- https://github.com/melezhik/sparrowdo/pull/15 -- https://github.com/melezhik/sparrowdo/pull/18 - -Sparrowdo is a configuration management system, written in Perl 6. I learned -about it after a reference from the Perl 6 Weekly, and set out to try it. I ran -into some issues, which I reported and eventually fixed. - -In addition, I also rewrote the testing script for Travis, which enables -paralel builds of the tests. This has nearly halved the time required for -running the full test suite. - -==== perl6/ecosystem -- https://github.com/perl6/ecosystem/pull/371 -- https://github.com/perl6/ecosystem/pull/372 -- https://github.com/perl6/ecosystem/pull/374 - -These PRs added a module, and removed that one and more later on, since I got a -PAUSE ID and uploaded my modules to CPAN. - -==== scriptkitties/perl6-App-Cpan6 -- https://github.com/scriptkitties/perl6-App-Cpan6/pull/1 -- https://github.com/scriptkitties/perl6-App-Cpan6/pull/2 -- https://github.com/scriptkitties/perl6-App-Cpan6/pull/3 -- https://github.com/scriptkitties/perl6-App-Cpan6/pull/4 -- https://github.com/scriptkitties/perl6-App-Cpan6/pull/12 -- https://github.com/scriptkitties/perl6-App-Cpan6/pull/13 -- https://github.com/scriptkitties/perl6-App-Cpan6/pull/14 -- https://github.com/scriptkitties/perl6-App-Cpan6/pull/15 - -`App::Cpan6` is a tool I've started working on to assist me in creating new -Perl 6 modules. There's been a couple of tasks that I do often in the process -of creating a module, and those tasks should become easier and faster using -this module. - -If everything works out and I learn enough of the module installation process, -I might consider letting this deal with the installation and updating of -modules as well. - -== In retrospect -The Hacktoberfest has been an interesting month for me. I've gotten to -contribute to a project I have come to love a lot, Perl 6. I've also made some -new friends with similar goals. Sadly I can't put in this much time every month -of the year, but I would if I could! - -I learned many interesting things for Perl 6, new operators, new functions, all -kinds of cool stuff to improve my Perl scripts with. I also got to learn about -parallelizing Travis builds with the Sparrowdo project, of which I will write -another tutorial post later. - -I've greatly enjoyed contributing to all the various projects, and would -recommend other people to check it out too. The people on the respective -project's IRC channels have been a great help to me to get started, and I can -help out getting you started as well now. diff --git a/_posts/2017-11-16-perl6-setting-up-a-raspberry-perl.adoc b/_posts/2017-11-16-perl6-setting-up-a-raspberry-perl.adoc deleted file mode 100644 index 7ea72fa..0000000 --- a/_posts/2017-11-16-perl6-setting-up-a-raspberry-perl.adoc +++ /dev/null @@ -1,236 +0,0 @@ ---- -date: 2017-11-16 11:58:40 -tags: Tutorial Perl6 RaspberryPi Raku -description: > - In this tutorial I'll get you through setting up a Raspberry Pi with Perl 6. - I am using a Raspberry Pi 3 myself, but other versions should work fine too. - However, older versions are slower, so it might take a bit longer to install - completely. ---- -= Perl 6 - Setting up a Raspberry Perl -:toc: preamble - -In this tutorial I'll get you through setting up a Raspberry Pi with -https://perl6.org/[Perl 6]. I am using a Raspberry Pi 3 myself, but other -versions should work fine too. However, older versions are slower, so it might -take a bit longer to install completely. - -[NOTE] -==== -For those who have never had a Raspberry Pi before, you will need -the following: - -- Raspberry Pi board -- Power supply (5v 2A, micro USB) -- SD card of at least 4gb, but I would advise at least 8gb -- Monitor with HDMI cable -- Keyboard -==== - -Perl 6 will be installed using -https://github.com/tadzik/rakudobrew[Rakudobrew], which I'll also be using to -get https://github.com/ugexe/zef[zef] installed. Zef is the recommended module -manager for Perl 6. - -== Setting up Raspbian - -The first step is getting the OS set up. To keep this tutorial simple, I will -stick to https://www.raspbian.org/[Raspbian], but if you feel confident in your -skills you can use any other distribution or OS. Perl 6 installs the same on -all UNIX(-like) operating systems. - -=== Get the image - -First, https://www.Raspberrypi.org/downloads/raspbian/[download the Raspbian -image from the Raspberry Pi download page]. I chose the `LITE` version, but if -you prefer having a graphical desktop you can go for the `DESKTOP` version -instead. - -At the time of writing, this means I got the -`2017-09-07-raspbian-stretch-lite.zip`. If you want to verify you got the -correct download and nothing went wrong saving it to your disk, you can verify -the checksum. The checksum for your download is noted below the download links. -To get the checksum of the file you downloaded, use `sha256sum` as follows: - -NOTE: Lines prepended with a `$` are to be ran as your normal user, whereas -lines with a `#` are ment to be ran as "super user". This can be done by using -a privilege escalation program, such as -https://www.linux.com/blog/how-use-sudo-and-su-commands-linux-introduction[`sudo`]. - -[source] ----- -$ sha256sum 2017-09-07-raspbian-stretch-lite.zip ----- - -If the checksum matches the one noted below the download button you used, it -should be fine, and you can continue with extracting the image from the zip -using `unzip`: - -[source] ----- -$ unzip 2017-09-07-raspbian-stretch-lite.zip ----- - -This will result in a similarly named file, but with a `.img` extension instead -of `.zip`. This is the image that you can write to the SD card. - -=== Write the image to the SD card - -This step is pretty easy, but typos here can be disastrous for the system -you're using to write to the SD card. - -Open a terminal and run `dmesg -w` as super user (usually doable using `sudo -dmesg -w`). This will give immediate feedback when you insert your SD card, and -shows which device it is being assigned to. In my case, this was `sdb`, which -means the device file resides at `/dev/sdb`. - -Now, to actually write the image, I'll use `dd` since this is everyone's -favourite tool, it seems. If you feel adventurous enough to try out something -different, feel free to read up on -https://www.vidarholen.net/contents/blog/?p=479[Useless Use of dd]. - -Make sure to make the `if` argument point to the correct path with your -extracted raspbian image, and `of` to point to the correct device as identified -earlier. In order to be allowed to run this command, you must be root, which -can be achieved by using `sudo` or `doas` again. - -[source] ----- -# dd bs=4M status=progress if=/path/to/2017-09-07-raspbian-stretch-lite.img of=/dev/sdb -$ sync ----- - -Afterwards, plug it into your Raspberry Pi and attach all cables you might -need. Think of stuff like a keyboard, mouse, monitor, internet, power. Do power -last, as the Raspberry Pi will start immediatly once it receives power. - -=== First boot - -The Raspberry Pi should start booting the moment you supply it with power. If -you attach the HDMI after the power, it's possible you won't have display -working, so make sure HDMI is attached before powering up. - -You'll see some text scrolling by, up to a point where it asks you for a -`login`, and accepts keyboard input. The default username is `pi`, and the -default password is `Raspberry`. You are strongly advised to change the -password upon login, which can be done in the next step. - -=== Configuration - -The Raspberry Pi comes with its own configuration tool, `raspi-config`. Run -this with `sudo` prepended in front of it so you gain the right privileges. I -would advise you to at least change the user password from here. After this you -should go to `Advanced Options` and expand the filesystem. This will grow the -filesystem to the entire SD card's size. - -TIP: To get to the buttons on the bottom (`Select`, `Finish` and `Back`), use -the arrow keys to go left or right. - -You can look around the tool for other interesting things to modify. Once you -are satisfied, go back to the main menu and choose `Finish`. It will ask to -reboot, which you should accept. This will apply all the new configurations you -just made. - -=== Updating and installing additional packages - -It's rare for the system to be completely up to date after installing the image -on the SD card. Additionally, you also need some extra packages in order to get -rakudobrew, and to install Perl 6 itself. For this, we use the package manager -bundled with raspbian, `apt`: - -[source] ----- -# apt update -# apt upgrade ----- - -This will update the package lists, and then upgrade all outdated packages to -their newest versions. You should do this at least once a week to make sure -your system stays up to date. - -Once the upgrades are finished, you can install some new packages which are -needed later on in this tutorial: - -[source] ----- -# apt install git build-essential ----- - -`git` is required to get the rakudobrew repository and is also used by -rakudobrew itself to get the sources needed to build Perl 6 and to install zef. -The `build-essential` package comes with all sorts of tools to build software, -which is required to build Perl 6. - -== Installing Perl 6 - -Now, we've got a working Raspberry Pi installation. We can start doing things -with it, such as playing around with Perl 6. - -=== Setting up Rakudobrew - -Rakudobrew is a nice tool to manage Perl 6 installations on your system. It can -also install `zef` for you, so you don't have to deal with this manually. This -is all documented on the repository's `README.md` file as well, but I'll -explain it here too. I do make a few small tweaks here and there to match my -preferred setup more closely. - -Clone the repository to your system, and add it to your `$PATH` to be able to -use the scripts bundled with it: - -[source] ----- -$ mkdir -p ~/.local/var -$ git clone https://github.com/tadzik/rakudobrew.git ~/.local/var/rakudobrew -$ export PATH=${HOME}/.local/var/rakudobrew/bin:$PATH -$ hash -r ----- - -The `hash -r` call will rehash your PATH, so you can tab-complete `rakudobrew`. -Next, initialize rakudobrew: - -[source] ----- -$ rakudobrew init ----- - -This will give you a notification to automatically load rakudobrew next time. -It is advised you follow that message, so you won't have to do it manually each -time you log in to the system. - -=== Installing Perl 6 with MoarVM backend - -Now that rakudobrew is installed and available to use, it's time to make use of -it to install Perl 6. - -[source] ----- -$ rakudobrew build moar ----- - -=== Installing zef, the module manager - -Getting zef to work isn't much harder than installing Perl 6, but its a lot -faster. You can have rakudobrew take care of this too: - -[source] ----- -$ rakudobrew build zef ----- - -== Final words - -And that should be it, you now have a working Perl 6 installation with the zef -module manager to take care of installing and upgrading modules. Now you just -need to come up with a nice project to work on to start using and learning the -wonders of Perl 6. - -If you need any help on getting started, try the `#perl6` IRC channel on -Freenode, or check out some of the Perl 6 documentation and introduction sites: - -- https://docs.perl6.org/ -- http://perl6intro.com/ - -For starting projects that are easy to start with and can bring quick results, -consider making an IRC bot using -https://github.com/zoffixznet/perl6-IRC-Client[`IRC::Client`], or a small web -application using https://github.com/Bailador/Bailador[`Bailador`]. diff --git a/_posts/2017-12-17-on-cloudflare.adoc b/_posts/2017-12-17-on-cloudflare.adoc deleted file mode 100644 index 9b831a8..0000000 --- a/_posts/2017-12-17-on-cloudflare.adoc +++ /dev/null @@ -1,129 +0,0 @@ ---- -date: 2017-12-17 10:13:26 -tags: Cloudflare Security Privacy -description: > - Cloudflare is a threat to online security and privacy. I am not the first on - to address this issue, and I probably will not be the last either. Sadly, - people still seem to be very uninformed as to what issues Cloudflare actually - solves, or introduces. ---- -= On Cloudflare -:toc: - -== Foreword -Cloudflare is a threat to online security and privacy. I am not the first on to -address this issue, and I probably will not be the last either. Sadly, people -still seem to be very uninformed as to what issues Cloudflare actually poses. -There also seems to be a big misconception about the benefits provided by using -Cloudflare. I would suggest reading the -http://cryto.net/~joepie91/blog/2016/07/14/cloudflare-we-have-a-problem/[article -on Cloudflare by joepie91] for a more thorough look at Cloudflare. - -If anyone is using Cloudflare, please tell them to stop doing it. Link them to -this page or any of the articles referenced here. Cloudflare is harmful to your -visitors, and if you do not care about them, they will stop caring about you -too. - -== A literal MITM attack -Cloudflare poses a huge risk by completely breaking the TLS/SSL chain used by -browsers by setting itself up as a -https://en.wikipedia.org/wiki/Man-in-the-middle_attack[man in the middle]. -Cloudflare doesn't do actual DDoS protection, they just make the request to the -origin server for you. Once they have received the data, they decrypt it and -re-encrypts it with their own certificate. This means that Cloudflare has -access to all requests in plain text and can optionally modify the data you -see. TLS/SSL is meant to prevent this very issue, but Cloudflare seems to care -very little. - -If we would consider Cloudflare to be a benevolent entity and surely never -modify any data ever, this is still an issue. Much data can be mined from the -plain text communications between you and the origin server. This data can be -used for all kinds of purposes. It is not uncommon for the USA government to -request a massive amount of surveillance information from companies without the -companies being able to speak up about it due to a gag order. This has become -clear once more by the -https://whispersystems.org/bigbrother/eastern-virginia-grand-jury/[subpoena on -Signal]. It should be clear to anyone that end-to-end encryption has to be a -standard and implemented properly. Cloudflare goes out of its way to break this -implementation. - -=== Cloudbleed -The danger of their MITM style of operation was shown be the -https://en.wikipedia.org/wiki/Cloudbleed[Cloudbleed] vulnerability. It also -shows that they make use of their MITM position to scan the data your site and -a visitor are exchanging. This includes private data, such as passwords. - -Even if you have an SSL connection to Cloudflare, they still decrypt it on -their end. They then serve the content under their own certificate. This makes -it look to the visitor like everything is secure, the browser says so after -all. But in reality, they don't have a secure connection to your server. They -only have one up to Cloudflare, and when it reaches Cloudflare, they decrypt it -and re-encrypt it using your certificate again. If you use one, of course, -otherwise they'll pass it on in plaintext back to your server, which is even -more dangerous. Whether or not you do, the content exists in plaintext on -Cloudflare's servers, which is not what you want, if you truly care about -security. - -== Eliminating your privacy -If Cloudflare were to fix their MITM behavior, the privacy problem would not -be solved all of a sudden. There are more questionable practices in use by -Cloudflare. - -People who are using a VPN or an anonimization service such as Tor are usually -greeted by a warning from Cloudflare. Let's not talk about this warning being -incorrect about the reason behind the user receiving the warning, but instead -about the methodology used to "pass" this "warning". Cloudflare presents you -with a page that requires you to solve a reCaptcha puzzle, which is hosted by a -well known third party that tries to harm your privacy as much as possible, -Google. If you do not wish to have Google tracking you all the time, you will -not be able to solve these puzzles, and in effect, unable to access the site -you were visiting. It is also interesting to note that this reCaptcha system is -sometimes broken if your browser does not identify itself as one of the regular -mainstream browsers such as Firefox or Chrome. - -Some site administrators disable this specific check. However, this still means -all your requests are logged by another third party, namely Cloudflare itself. -As noted in _A literal MITM attack_, this data is still very interesting to -some parties. And do not fool yourself: meta data is still very worthwhile and -can tell a huge amount of information about a person. - -=== Forcing JavaScript -This issue generally does not concern many people, as most people online -nowadays use a big mainstream browser with JavaScript enabled. However, there -are still people, services and applications that do not use JavaScript. This -makes sites unavailable when they are in the "under attack" mode by Cloudflare. -This will run a check sending Cloudflare your browser information before -deciding whether you are allowed to access the website. This is yet another -privacy issue, but at the same time, a usability issue. It makes your site -unavailable to people who simply do not wish to use JavaScript or people who -are currently limited to a browser with no JavaScript support. - -It is also common for Cloudflare to -http://www.tedunangst.com/flak/post/cloudflare-and-rss[Break RSS readers] by -presenting them with this check. This check is often presented to common user -agents used by services and programs. Since these do not include a big -JavaScript engine, there is no way for them to pass the test. - -== False advertising -=== DDoS protection -Cloudflare is hailed by many as a gratis DDoS protection service, and they -advertise themselves as such. However, Cloudflare does not offer DDoS -protection, they simply act as a pin cushion to soak the hit. Real DDoS -protection works by analyzing traffic, spotting unusual patterns and blocking -these requests. If they were to offer real DDoS protection like this, they -would be able to tunnel TLS/SSL traffic straight to the origin server, thereby -not breaking the TLS/SSL chain as they do right now. - -It should also be noted that this gratis "protection" truly gratis either. If -your site gets attacked for long enough, or for enough times in a short enough -time frame, you will be kicked off of the gratis plan and be moved onto the -"business" plan. This requires you to pay $200 per month for a service that does -not do what it is advertised to do. If you do not go to the business plan, you will -have about the same protection as you would have without it, but with the -addition of ruining the privacy and security of your visitors. - -=== Faster page loads -This is very well explained on -http://cryto.net/~joepie91/blog/2016/07/14/cloudflare-we-have-a-problem/[joepie91's -article] under the heading _But The Speed! The Speed!_. As such, I will refer -to his article instead of repeating him here. diff --git a/_posts/2017-12-21-funding-yourself-as-free-software-developer.adoc b/_posts/2017-12-21-funding-yourself-as-free-software-developer.adoc deleted file mode 100644 index de687ed..0000000 --- a/_posts/2017-12-21-funding-yourself-as-free-software-developer.adoc +++ /dev/null @@ -1,231 +0,0 @@ ---- -title: Funding Yourself As A Free Software Developer -date: 2017-12-21 05:29:26 -tags: FreeSoftware Programming Funding -description: > - An overview of multiple funding platforms, with pros and cons, from the - perspective of a free software developer. It's intent is to help others find - a possible form of income while doing what they love, and what I consider is - immensely important for society. ---- -= Funding Yourself As A Free Software Developer -:toc: preamble - -I've been meaning to spend more time on developing free software, helping out -new users on IRC and writing more tutorials to get others started. All of these -cost time, and time is money - so I've set out to set up donation accounts. -In the hopes of helping other developers who struggle to fund their work, I've -written up this article to talk about my experience. This is a living -document! As you explore this yourself, please send me your thoughts on each -platform and turn me on to interesting platforms I missed. - -I'll be focussing on platforms allowing for recurring donations, as these are -more useful for procuring a stable income. - -== Platforms -=== BountySource -[WARNING] -==== -- Requires 3rd-party link:/articles/on-cloudflare/[Cloudflare]-hosted - JavaScript sources to function. -==== - -BountySource lets people donate money towards an issue on Github your projects. -Once an issue gets fixed, you can claim the "bounty" that was on this issue. -This can also help in making clear which issue you should aim for next, and -can increase interest in contributors for your project. - -There's also BountySource Salt, which is a recurring donation platform. -Projects or teams can use this to gain monthly income to sustain the -development of their project(s). - -Support for this platform is offered through the IRC channel -https://kiwiirc.com/client/chat.freenode.net:+6697/#bountysource[`#bountysource` -on Freenode]. - -The BountySource platform itself is also free software, and the source code -for it can be found https://github.com/bountysource/core[on github]. - -You can find BountySource at https://www.bountysource.com/. - -=== LiberaPay -This service seems to be completely free as in freedom. They even -https://github.com/liberapay/liberapay.com[publish their source on GitHub]. -Their own funding comes through donations on their own platform, instead of -taking a cut of each donation like most other services. - -It's possible to connect other accounts to your LiberaPay account. While this -feature in general is pretty common, they allow you to link to sites which are -interesting to show as developer, such as GitHub, GitLab, and BitBucket. They -also let you link to a Mastodon account, if you have one. - -To let people know you're accepting donations through LiberaPay, you can use -one of the widgets they make available for you. This will show a donate button -which will link to you profile. Do note, this is not a regular HTML button or -cleverly implemented anchor tag, but a JavaScript-based button. - -Another thing LiberaPay lacks is a rewards system. Most other platforms allow -you to set reward tiers, which allow you to give certain benefits to donors. - -You can find Liberapay at https://liberapay.com/. - -=== MakerSupport -[WARNING] -==== -- The site requires a 3rd-party hosted jQuery. -- You have to solve a Google reCaptcha in order to register a new account. -==== - -MakerSupport seems to be another option, aimed at content creators who might -need freedom of speech more than others. It seems to be less focused on -software development, as you cannot link to any of the major git hosting -platforms. - -There are options here to set up "tiers" for your donors; which is a convenient -way to provide them with perks for their support. For a free software -developer, this might be something like access to more direct support from the -developer. - -Sadly, registration wasn't as smooth as most other platforms. My preferred -username, "tyil" is too short. There's no indication of the requirements of any -of the fields, you just get a popup on submission of the form saying a field is -wrong. - -Additionally, the registration form requires some 3rd-party JavaScript to work, -and a Google reCaptcha to be solved in order to get the submit button to show -up. As I have set up uMatrix in my browser, this cost me some extra time to -finish registration. - -Setting a profile image proved to be a little harder. First off, I'm still -using uMatrix so I had to allow a 3rd-party (Amazon, in this case) XHR -requests. Secondly, their error when uploading a "wrong" format is also not -very user friendly, as it won't give you any details on why it's disallowed, -nor what images are allowed instead. - -[NOTE] -==== -It seems they check the extension of the uploaded image's filename. As far as I -can tell, you're allowed to upload files that end with `.jpg` and `.png`. -==== - -You can find MakerSupport at https://www.makersupport.com/. - -=== Patreon -[WARNING] -==== -- Requires 3rd-party link:/articles/on-cloudflare[Cloudflare]-hosted - JavaScript sources to function. -- You have to solve a Google reCaptcha in order to register a new account. -==== - -Patreon is possibly the most famous donation-based funding platform available -right now. Its popularity is a good thing, since this means there's probably -many donors already using this platform. - -At Patreon, you can set up so-called goals. Goals are the thing I haven't found -with other funding platforms. It allows you to set a goal for an amount of -money, and add a reward to this. This way, you can inform your donors you will -be creating a certain kind of content once a one-time goal has been reached. -Basically, you can show your donors what you're going to do with the money -they're donating to you. - -Another interesting thing that I haven't seen on other platforms is the option -to charge donors per creation, instead of per month. While this may seem less -fitting for software developers (unless you want to get paid per commit, I -guess), it's an interesting feature that's pretty unique. If you publish many -tutorials, guides or other posts, this might fit you very well. - -You can link your account to other services, similarly to other platforms, but -it seems to only allow you to be linked with proprietary social media -platforms. - -You can find Patreon at https://www.patreon.com/home. - -=== (Dis)honorable mentions -==== Hatreon -I've included this because I found people talking about it on IRC. However, it -seems to be nothing more than a joke that's gone too far. Its main reason for -existing seems to be to get away from the political correctness found with -earlier crowdfunding platforms, yet their site is invite-only, so those who are -actually interested can't even use it. It seems that pledging is currently -disabled as well, and has been for at least 10 days. - -== But that's not all -Just setting up an account on a funding platform isn't enough. There's more to -keeping a healthy and happy supporter base. - -=== Spread awareness of your work -Whether you're writing articles or publishing new releases of projects, tell -the world you're doing whatever it is you're doing. If nobody knows about your -project, they won't be able to give any kind of appreciation for it. Use social -media outlets, public forums, mailing lists, anything! Tell them what you made, -why it's useful and how they could use it to improve their digital life. - -[WARNING] -==== -Ofcourse, don't spam it to unrelated communication channels. This will only -backfire. -==== - -=== Using the rewards system -On the platforms that support a rewards system, make use of it. There's some -little things you can do that go a long way with your supporters. For instance, -you can offer things like stickers to donors that donate a certain amount of -money to you. These are reasonably cheap to produce and ship, and many people -like these. - -Another idea that seems to strike well with donors is having a way to talk with -the person they're supporting directly. This can be done by giving them access -to an IRC channel for you and your donors. You can use another platform for -this, but most free software enthousiasts are already on IRC, and there's few -real-time communication alternatives that they're already using. - -=== Don't stick to a single platform -There's multiple platforms out there, use them! Not all of them have the same -userbase, and you can reach more people by giving them more options to work -with. - -=== Let people know you're accepting donations -If people don't know you're even accepting donations, chances are pretty high -you won't get any. Or if it's too hard to figure out how to donate to you, -people will simply not take the effort. Make sure people can easily find out -that you're accepting donations, and how to donate to you. - -=== Show what you're doing with donation money -Have a page with information about what you're using with the money. This can -be as simple as just saying you pay the rent and buy food with it. Most donors -don't mind too much what you're doing with the money they donate to you, but a -few do appreciate having this information available to them. - -It can be as simple as adding a `/donate` link to your site where you explain -how to donate to you, and what you do with the donation money. - -[WARNING] -==== -Don't let it turn into an annoying advertisement though, this will surely have -an opposite effect. -==== - -== Further reading -There's more places to go for tips and tricks in getting funds to sustain your -free software development work. I've listed a couple of these here for those -interested. - -- https://wiki.snowdrift.coop/market-research/other-crowdfunding[snowdrift.coop wiki on crowdfunding/fundraising services] -- https://github.com/nayafia/lemonade-stand[A handy guide to financial support for open source] - -== RFC -I'd love to receive feedback on this, as I think being able to get donations -easily for the work free software developers put in to their projects is -important. - -Getting to know more platforms and giving them a small write-up should help out -other developers like me looking for the best platform for their use case. I'd -also like to hear from developers already using a platform, to extend this -article with more useful information on how to successfully get donors for -their work. - -If you want to contact me, do take a look at the link:/[Contact] section, and -let me know about your experiences with funding. - -And if you liked this post, consider link:/support/[donating] ;) diff --git a/_posts/2018-02-05-why-perl6.adoc b/_posts/2018-02-05-why-perl6.adoc deleted file mode 100644 index 8c9a299..0000000 --- a/_posts/2018-02-05-why-perl6.adoc +++ /dev/null @@ -1,280 +0,0 @@ ---- -title: Why Perl 6? -date: 2018-02-05 18:22:20 -tags: Perl6 Raku -description: > - I've grown quite fond of Perl 6 the more I learn about it, yet the general - developer community still seems to think Perl is a dirty word. In this - article, I will detail some of the features that make me like Perl 6, and why - I try to use it wherever possible. ---- -= Why Perl 6? -:toc: preamble - -For about a year now, I've been working in Perl 6. Telling this to other people -often brings about some confused faces. I've grown quite fond of Perl 6 the -more I learn about it, yet the general developer community still seems to think -Perl is a dirty word. In this article, I will detail some of the features that -make me like Perl 6, and why I try to use it wherever possible. - -== Hassle-free command line arguments -Whet creating an application, you usually want to be able to specify some -arguments at runtime. Most times this happens using command line arguments or -options. Perl 6 allows you to specify these in the -https://docs.perl6.org/language/functions#index-entry-MAIN[`MAIN`] subroutine -signature. - -For instance, if I want the application to accept two string arguments, I can -do it as easy as this: - -[source,perl6] ----- -sub MAIN ( - Str $arg-one, - Str $arg-two, -) { - ... -} ----- - -Now, if you wanted to add an option like `--output=/path/to/file`, you can do -it just like this: - -[source,perl6] ----- -sub MAIN ( - Str $arg-one, - Str $arg-two, - Str :$output, -) { - ... -} ----- - -By default, if there's a `MAIN` available in your Perl 6 program, but the -arguments or options supplied by the user are incorrect, it will display the -right way to invoke the command, called the -https://docs.perl6.org/language/functions#index-entry-USAGE[`USAGE`]. Ofcourse, -this message can be changed if you wish, but the default is quite good for most -use-cases. - -However, sometimes you want to add a little explanation to what the argument or -option is intended for. Just for a liitle bit of additional user friendliness. - -Fear not, for this is also already covered by the defaults. In Perl, there was -POD to document your code. In Perl 6, we have -https://docs.perl6.org/language/glossary#index-entry-POD[POD] as well. And -these comments can be inspected at runtime to provide the user some -information. And that's exactly what the default `USAGE` also does. So if you -want to add some helpful comments to the arguments or the program itself, -simply add the comments where you want them: - -[source,perl6] ----- -#| This is a sample program, just to showcase the awesome stuff available in -#| Perl 6. -sub MAIN ( - Str $arg-one, #= Just a random argument - Str $arg-two, #= Yet another argument used for showcasing - Str :$output, #= Last but not least, an option which allows for a value -) { - ... -} ----- - -== Unicode -What if you could support all languages with a single implementation? That's -where unicode comes in. And Perl 6 currently has the best support for Unicode -out of all programming languages available. Its only real competitor seems to -be Swift (at the time of writing this). - -But not just for handling strings, Perl 6 uses unicode as a core language -feature. This means you can use them in your source code as well. And that -opens up some nice possibilities. Using the right unicode characters allows you -to write cleaner and more concise code, reducing the cognitive load while -trying to understand the program. - -For instance, if you're trying to do any kind of math, you can just use the -π character as a regular character. Or use the ² to get the square of a certain -number. This little piece is completely valid in Perl 6: - -[source,perl6] ----- -my $a = $r² ÷ π; ----- - -Now, if you're thinking "that looks neat, but how am I ever going to write -these?", do not worry. Most operating systems and many editors have tools to -let you input these. For instance, using `vim` with -https://github.com/vim-perl/vim-perl6[`vim-perl6`], you can just write "pi" and -hit space (or type any non-alphabetical character). - -But not everyone is using an OS or an editor that makes it easy. And for those -people, Perl 6 simply supports using -https://docs.perl6.org/language/unicode_ascii[ascii based operators]. The -previous block could also be written as follows: - -[source,perl6] ----- -my $a = $r ^ 2 / pi; ----- - -As unicode becomes more accepted, input methods will hopefully improve to make -input easier for everyone in the long run. Those who can already input it -easily don't have to wait for this future, Perl 6 already supports it. - -== Multithreading -Multi-core processors are virtually everywhere these days. Yet many programming -languages still don't support multithreaded application development natively, -if at all. In Perl 6, running something in a different thread is as easy as -wrapping it in a https://docs.perl6.org/routine/start[`start`] block: - -[source,perl6] ----- -start { - do-something(); -} ----- - -`start` returns a https://docs.perl6.org/type/Promise[`Promise`], which you can -store in a scalar variable just like any other object. You can check on whether -the `Promise` has completed already and check whether it died, for instance. - -Other aspects which can often be spread over multiple threads are loops or -maps. For instance, consider the following -https://docs.perl6.org/routine/map[map] function: - -[source,perl6] ----- -@cats.map: { - $^cat.pat; -} ----- - -This will pat each cat in turn, in the order they appear in the list. But you -can speed up the patting process by patting multiple cats at the same time. And -to get there, all you need to do is add a -https://docs.perl6.org/routine/race[`race`]: - -[source,perl6] ----- -@cats.race.map: { - $^cat.pat; -} ----- - -This will attempt to pat the cats over multiple threads, speeding up the -process to pat all the cats. If the result of the pattings needs to be in the -same order as the patting order, you use -https://docs.perl6.org/routine/hyper[`hyper`] instead of `race`: - -[source,perl6] ----- -@cats.hyper.map: { - $^cat.pat; -} ----- - -== Object orientation -Object oriented programming seems to be getting out of fashion with the new -generation of developers. But it's still in wide use, being taught at most -universities, and is often easy to explain to new developers as well. - -And Perl 6 has https://docs.perl6.org/language/classtut#index-entry-OOP[OO] -support built into its core: - -[source,perl6] ----- -class Foo -{ - has Str $some-field; - - method bar ( - Str $some-arg, - ) { - ... - } -} ----- - -You can also have -https://docs.perl6.org/language/glossary#index-entry-Multi-Dispatch[multi-dispatch] -methods on your classes, which are methods with the same names, but accepting -different arguments or argument types. For instance: - -[source,perl6] ----- -class Foo -{ - multi method bar ( - Str $some-arg, - ) { - ... - } - - multi method bar ( - Int $some-arg, - ) { - ... - } -} ----- - -Which method is being used will be decided by the type of argument is being -passed in, in this case either a https://docs.perl6.org/type/Str[`Str`] or an -https://docs.perl6.org/type/Int[`Int`]. - -== Functional programming -Whilst OO is considered being old more and more, functional programming is -gaining ground. And this paradigm is fully supported in the core of Perl 6 as -well. You've seen the `map` example already while patting cats earlier, for -instance. - -But there's much more on the functional playing field, such as the -https://docs.perl6.org/routine/==%3E[`=\=>`] operator, known as the -https://docs.perl6.org/language/operators#infix_==%3E[feed operator]. It simply -passed the output of a statement as the last argument to the next statement: - -[source,perl6] ----- -@grumpy-cats - ==> feed() - ==> pat() - ==> snuggle() - ==> my @happy-cats; ----- - -This will take the `@grumpy-cats`, feed them, pat them, snuggle them and put -the result into `@happy-cats`. You could've chained the calls using a `.` -instead, and Perl 6 allows you to do this too. But the `=\=>` looks much more -readable to me, which is why I prefer using this instead. - -I'm still exploring the functional programming field myself, but these few -things have made me happy exploring it. - -== Community -(Almost) last, but certainly not least, the Perl 6 community is amazing. It's -been the friendliest bunch I've been with, both on IRC, their mailing lists and -in real life. Everyone is welcoming, and they try to help you whenever they -can. - -Community is important to help you out whenever you get stuck for whatever -reason. A friendly community is the best you can get here to keep you a happy -developer yourself as well. - -== Other little aspects -There's a few neat things I can do in Perl 6 that I can't do in (most) other -languages, but aren't important enough to warrant a large section to show them -off. - -=== Dashes in names -You can use dashes in names: Things like `my $foo-bar` is valid, just like -`method foo-bar`. It's nothing big on itself, but I've found it makes reading -code much more enjoyable than pascalCase, CamelCase or snake_case. - -=== Gradual typing -You don't *need* to use types in Perl 6. But when you want to use them (for -making use of multi-dispatch, for example), you can just start using them. If -types are added, the compiler will make sure the types are correct. If not, you -can always do them yourself (but why would you, when the compiler can do a -better job for free). diff --git a/_posts/2018-03-20-perl6-introduction-to-application-programming.adoc b/_posts/2018-03-20-perl6-introduction-to-application-programming.adoc deleted file mode 100644 index fc00bd3..0000000 --- a/_posts/2018-03-20-perl6-introduction-to-application-programming.adoc +++ /dev/null @@ -1,784 +0,0 @@ ---- -title: "Perl 6 - Introduction to application programming" -date: 2018-03-20 11:08:00 -tags: Tutorial Perl6 Assixt GTK Programming Raku -layout: post -authors: - - ["Patrick Spek", "https://tyil.nl"] ---- -= Perl 6 - Introduction to application programming -:toc: preamble - -In this tutorial, I'll be guiding you through creating a simple application in -Perl 6. If you don't have Perl 6 installed yet, get the -http://rakudo.org/how-to-get-rakudo/[Rakudo Star] distribution for your OS. -Alternatively, you can use the https://hub.docker.com/_/rakudo-star/[Docker -image]. - -The application itself will be a simple dice-roller. You give it a number of -dice to roll, and the number of sides the die has. We'll start off by creating -it as a console application, then work to make it a GUI as well with the -`GTK::Simple` module. - -== Preparation -First, you'll want to install the libgtk headers. How to get these depends on -your distro of choice. For Debian-based systems, which includes Ubuntu and -derivatives, this command would be the following `apt` invocation: - -[source] ----- -$ apt install libgtk-3-dev ----- - -For other distros, please consult your documentation. - -To ease up module/application building, I'll use `App::Assixt`. This module -eases up on common tasks required for building other modules or applications. -So we'll start by installing this module through `zef`. - -[source] ----- -$ zef install App::Assixt ----- - -[NOTE] -==== -You may need to rehash your `$PATH` as well, which can be done using `hash -r` -on `bash`, or `rehash` for `zsh`. For other shells, consult your manual. -==== - -Next up, we can use `assixt` to create the new skeleton of our application, -with the `new` subcommand. This will ask for some user input, which will be -recorded in the `META6.json`, a json-formatted file to keep track of meta -information about the module. `assixt` should take care of this file for you, -so you never need to actually deal with it. - -[source] ----- -$ assixt new ----- - -=== assixt input -Since the `assixt new` command requires some input, I'll walk through these -options and explain how these would affect your eventual application. - -==== Name of the module -This is the name given to the module. This will be used for the directory name, -which by default in `assixt` will be `perl6-` prepended to a lower-case version -of the module name. If you ever wish to make a module that is to be shared in -the Perl 6 ecosystem, this should be unique across the entire ecosystem. If -you're interested in some guidelines, the -https://pause.perl.org/pause/query?ACTION=pause_namingmodules[PAUSE guidelines] -seem to apply pretty well to Perl 6 as well. - -For this application, we'll use `Local::App::Dicer`, but you can use whatever -name you'd prefer here. - -==== Your name -Your name. This will be used as the author's name in the `META6.json`. It is -used to find out who made it, in order to report issues (or words of praise, -of course). - -==== Your email address -Your email address. Like your name, it will be used in case someone has to -contact you in regards off the module. - -==== Perl 6 version -This defaults to `c` right now, and you can just hit enter to accept it. In the -future, there will be a Perl 6.d available as well, in which case you can use -this to indicate you want to use the newer features introduced in 6.d. This is -not the case yet, so you just want to go with the default `c` value here. - -==== Module description -A short description of your module, preferably a single sentence. This is -useful to people wondering what the module is for, and module managers can show -to the user. - -==== License key -This indicates the license under which your module is distributed. This -defaults to `GPL-3.0`, which I strongly recommend to use. The de-facto -default seems to be `Artistic-2.0`, which is also used for Perl 6 itself. - -This identifier is based on the https://spdx.org/licenses/[SPDX license list]. -Anything not mentioned in this list is not acceptable. #TODO Clarify why - -== Writing your first test -With the creation of the directory structure and metadata being taken care of -by `assixt`, we can now start on writing things. Tests are not mandatory, but -are a great tool for quickly checking if everything works. If you make larger -applications, it really helps not having to manually test anything. Another -benefit is that you can quickly see if your changes, or those of someone else, -break anything. - -Creating the base template for tests, `assixt` can help you out again: `assixt -touch` can create templates in the right location, so you don't have to deal -with it. In this case we want to create a test, which we'll call "basic". - -[source] ----- -$ assixt touch test basic ----- - -This will create the file `t/basic.t` in your module directory. Its contents -will look as follows: - -[source,perl6] ----- -#! /usr/bin/env perl6 - -use v6.c; - -use Test; - -ok True; - -done-testing; - -# vim: ft=perl6 ----- - -The only test it has right now is `ok True`, which will always pass testing. We -will change that line into something more usable for this application: - -[source,perl6] ----- -use Local::App::Dicer; - -plan 2; - -subtest "Legal rolls", { - plan 50; - - for 1..50 { - ok 1 ≤ roll($_) ≤ $_, "Rolls between 1 and $_"; - } -} - -subtest "Illegal rolls", { - plan 3; - - throws-like { roll(0) }, X::TypeCheck::Binding::Parameter, "Zero is not accepted"; - throws-like { roll(-1) }, X::TypeCheck::Binding::Parameter, "Negative rolls are not accepted"; - throws-like { roll(1.5) }, X::TypeCheck::Binding::Parameter, "Can't roll half sides"; -} ----- - -[NOTE] -==== -Perl 6 allows mathematical characters to make your code more concise, as with -the ≤ in the above block. If you use http://www.vim.org/[vim], you can make use -of the https://github.com/vim-perl/vim-perl6[vim-perl6] plugin, which has an -option to change the longer, ascii-based ops (in this case `\<=`) into the -shorter unicode based ops (in this case `≤`). This specific feature requires -`let g:perl6_unicode_abbrevs = 1` in your `vimrc` to be enabled with -`vim-perl6`. - -If that's not an option, you can use a -https://en.wikipedia.org/wiki/Compose_key[compose key]. If that is not viable -either, you can also stick to using the ascii-based ops. Perl 6 supports both -of them. -==== - -This will run 53 tests, split up in two -https://docs.perl6.org/language/testing#Grouping_tests[subtests]. Subtests are -used to logically group your tests. In this case, the calls that are correct -are in one subtest, the calls that should be rejected are in another. - -The `plan` keywords indicate how many tests should be run. This will help spot -errors in case your expectations were not matched. For more information on -testing, check out https://docs.perl6.org/language/testing[the Perl 6 docs on -testing]. - -We're making use of two test routines, `ok` and `throws-like`. `ok` is a -simple test: if the given statement is truthy, the test succeeds. The other -one, `throws-like`, might require some more explanation. The first argument it -expects is a code block, hence the `{ }`. Inside this block, you can run any -code you want. In this case, we run code that we know shouldn't work. The -second argument is the exception it should throw. The test succeeds if the -right exception is thrown. Both `ok` and `throws-like` accept a descriptive -string as optional last argument. - -=== Running the tests -A test is useless if you can't easily run it. For this, the `prove` utility -exists. You can use `assixt test` to run these tests properly as well, saving -you from having to manually type out the full `prove` command with options. - -[source] ----- -$ assixt test ----- - -You might notice the tests are currently failing, which is correct. The -`Local::App::Dicer` module doesn't exist yet to test against. We'll be working -on that next. - -[NOTE] -==== -For those interested, the command run by `assixt test` is `prove -e "perl6 --Ilib" t`. This will include the `lib` directory into the `PERL6PATH` to be -able to access the libraries we'll be making. The `t` argument specifies the -directory containing the tests. -==== - -== Creating the library -Again, let's start with a `assixt` command to create the base template. This -time, instead of `touch test`, we'll use `touch lib`. - -[source] ----- -$ assixt touch unit Local::App::Dicer ----- - -This will generate a template file at `lib/Local/App/Dicer.pm6` which some -defaults set. The file will look like this. - -[source,perl6] ----- -#! /usr/bin/env false - -use v6.c; - -unit module Local::App::Dicer; ----- - -The first line is a https://en.wikipedia.org/wiki/Shebang_(Unix)[shebang]. It -informs the shell what to do when you try to run the file as an executable -program. In this case, it will run `false`, which immediately exits with a -non-success code. This file needs to be run as a Perl 6 module file, and -running it as a standalone file is an error. - -The `use v6.c` line indicates what version of Perl 6 should be used, and is -taken from the `META6.json`, which was generated with `assixt new`. The last -line informs the name of this module, which is `Local::App::Dicer`. Beneath -this, we can add subroutines, which can be exported. These can then be accessed -from other Perl 6 files that `use` this module. - -=== Creating the `roll` subroutine -Since we want to be able to `roll` a die, we'll create a subroutine to do -exactly that. Let's start with the signature, which tells the compiler the name -of the subroutine, which arguments it accepts, their types and what type the -subroutine will return. - -[TIP] -==== -Perl 6 is gradually typed, so all type information is optional. The subroutine -arguments are optional too, but you will rarely want a subroutine that doesn't -have an argument list. -==== - -[source,perl6] ----- -sub roll($sides) is export -{ - $sides -} ----- - -Let's break this down. - -- `sub` informs the compiler we're going to create a subroutine. -- `roll` is the name of the subroutine we're going to create. -- `$sides` defines an argument used by the subroutine. -- `is export` tells the compiler that this subroutine is to be exported. This - allows access to the subroutine to another program that imports this module - through a `use`. -- `{ $sides }` is the subroutine body. In Perl 6, the last statement is also - the return value in a code block, thus this returns the value of $sides. A - closing `;` is also not required for the last statement in a block. - -If you run `assixt test` now, you can see it only fails 1/2 subtests: - -[source] ----- -# TODO: Add output of failing tests ----- - -Something is going right, but not all of it yet. The 3 tests to check for -illegal rolls are still failing, because there's no constraints on the input of -the subroutine. - -=== Adding constraints -The first constraint we'll add is to limit the value of `$sides` to an `Int:D`. -The first part of this constraint is common in many languages, the `Int` part. -The `:D` requires the argument to be **defined**. This forces an actual -existing instance of `Int`, not a `Nil` or undefined value. - -[source,perl6] ----- -sub roll(Int:D $sides) is export ----- - -Fractional input is no longer allowed, since an `Int` is always a round number. -But an `Int` is still allowed to be 0 or negative, which isn't possible in a -dice roll. Nearly every language will make you solve these two cases in the -subroutine body. But in Perl 6, you can add another constraint in the signature -that checks for exactly that: - -[source,perl6] ----- -sub roll(Int:D $sides where $sides > 0) is export ----- - -The `where` part specifies additional constraints, in this case `$sides > 0`. -So now, only round numbers larger than 0 are allowed. If you run `assixt test` -again, you should see all tests passing, indicating that all illegal rolls are -now correctly disallowed. - -=== Returning a random number -So now that we can be sure that the input is always correct, we can start on -making the output more random. In Perl 6, you can take a number and call -`.rand` on it, to get a random number between 0 and the value of the number you -called it on. This in turn can be rounded up to get a number ranging from 1 to -the value of the number you called `.rand` on. These two method calls can also -be changed to yield concise code: - -[source,perl6] ----- -sub roll(Int:D $sides where $sides > 0) is export -{ - $sides.rand.ceiling -} ----- - -That's all we need from the library itself. Now we can start on making a usable -program out of it. - -== Adding a console interface -First off, a console interface. `assixt` can `touch` a starting point for an -executable script as well, using `assixt touch bin`: - -[source] ----- -$ assixt touch bin dicer ----- - -This will create the file `bin/dicer` in your repository, with the following -template: - -[source,perl6] ----- -#! /usr/bin/env perl6 - -use v6.c; - -sub MAIN -{ - … -} ----- - -The program will run the `MAIN` sub by default. We want to slightly change this -`MAIN` signature though, since we want to accept user input. And it just so -happens that you can specify the command line parameters in the `MAIN` -signature in Perl 6. This lets us add constraints to the parameters and give -them better names with next to no effort. We want to accept two numbers, one -for the number of dice, and one for the number of sides per die: - -[source,perl6] ----- -sub MAIN(Int:D $dice, Int:D $sides where { $dice > 0 && $sides > 0 }) ----- - -Here we see the `where` applying constraints again. If you try running this -program in its current state, you'll have to run the following: - -[source] ----- -$ perl6 -Ilib bin/dicer -Usage: - bin/dicer <dice> <sides> ----- - -This will return a list of all possible ways to invoke the program. There's one -slight problem right now. The usage description does not inform the user that -both arguments need to be larger than 0. We'll take care of that in a moment. -First we'll make this part work the way we want. - -To do that, let's add a `use` statement to our `lib` directory, and call the -`roll` function we created earlier. The `bin/dicer` file will come to look as -follows: - -[source,perl6] ----- -#! /usr/bin/env perl6 - -use v6.c; - -use Local::App::Dicer; - -sub MAIN(Int:D $dice, Int:D $sides where { $dice > 0 && $sides > 0 }) -{ - say $dice × roll($sides) -} ----- - -[NOTE] -==== -Just like the `≤` character, Perl 6 allows to use the proper multiplication -character `×` (this is not the letter `x`!). You can use the more widely known -`*` for multiplication as well. -==== - -If you run the program with the arguments `2` and `20` now, you'll get a random -number between 2 and 40, just like we expect: - -[source] ----- -$ perl6 -Ilib bin/dicer 2 20 -18 ----- - -=== The usage output -Now, we still have the trouble of illegal number input not clearly telling -what's wrong. We can do a neat trick with -https://docs.perl6.org/language/functions#index-entry-USAGE[the USAGE sub] to -achieve this. Perl 6 allows a subroutine with the name `USAGE` to be defined, -overriding the default behaviour. - -Using this, we can generate a friendlier message informing the user what they -need to supply more clearly. The `USAGE` sub would look like this: - -[source,perl6] ----- -sub USAGE -{ - say "Dicer requires two positive, round numbers as arguments." -} ----- - -If you run the program with incorrect parameters now, it will show the text -from the `USAGE` subroutine. If the parameters are correct, it will run the -`MAIN` subroutine. - -You now have a working console application in Perl 6! - -== Making a simple GUI -But that's not all. Perl 6 has a module to create GUIs with the -https://www.gtk.org/[GTK library] as well. For this, we'll use the -https://github.com/perl6/gtk-simple[`GTK::Simple`] module. - -You can add this module as a dependency to the `Local::App::Dicer` repository -with `assixt` as well, using the `depend` command. By default, this will also -install the dependency locally so you can use it immediately. - -[source] ----- -$ assixt depend GTK::Simple ----- - -=== Multi subs -Next, we could create another executable file and call it `dicer-gtk`. However, -I can also use this moment to introduce -https://docs.perl6.org/language/glossary#index-entry-multi-method[multi -methods]. These are subs with the same name, but differing signatures. If a -call to such a sub could potentially match multiple signatures, the most -specific one will be used. We will add another `MAIN` sub, which will be called -when `bin/dicer` is called with the `--gtk` parameter. - -We should also update the `USAGE` sub accordingly, of course. And while we're -at it, let's also include the `GTK::Simple` and `GTK::Simple::App` modules. The -first pulls in all the different GTK elements we will use later on, while the -latter pulls in the class for the base GTK application window. The updated -`MAIN`, `USAGE` and `use` parts will now look like this: - -[source,perl6] ----- -use Local::App::Dicer; -use GTK::Simple; -use GTK::Simple::App; - -multi sub MAIN(Int:D $dice, Int:D $sides where { $dice > 0 && $sides > 0 }) -{ - say $dice × roll($sides) -} - -multi sub MAIN(Bool:D :$gtk where $gtk == True) -{ - # TODO: Create the GTK version -} - -sub USAGE -{ - say "Launch Dicer as a GUI with --gtk, or supply two positive, round numbers as arguments."; -} ----- - -There's a new thing in a signature header here as well, `:$gtk`. The `:` in -front of it makes it a named argument, instead of a positional one. When used -in a `MAIN`, this will allow it to be used like a long-opt, thus as `--gtk`. -Its use in general subroutine signatures is explained in the next chapter. - -Running the application with `--gtk` gives no output now, because the body only -contains a comment. Let's fix that. - -=== Creating the window -First off, we require a `GTK::Simple::App` instance. This is the main window, -in which we'll be able to put elements such as buttons, labels, and input -fields. We can create the `GTK::Simple::App` as follows: - -[source,perl6] ----- -my GTK::Simple::App $app .= new(title => "Dicer"); ----- - -This one line brings in some new Perl 6 syntax, namely the `.=` operator. -There's also the use of a named argument in a regular subroutine. - -The `.=` operator performs a method on the variable on the left. In our case, -it will call the `new` subroutine, which creates a new instance of the -`GTK::Simple::App` class. This is commonly referred to as the **constructor**. - -The named argument list (`title \=> "Dicer"`) is another commonly used feature -in Perl 6. Any method can be given a non-positional, named parameter. This is -done by appending a `:` in front of the variable name in the sub signature. -This has already been used in our code, in `multi sub MAIN(Bool :$gtk where -$gtk == True)`. This has a couple of benefits, which are explained in the -https://docs.perl6.org/type/Signature#index-entry-positional_argument_%28Signature%29_named_argument_%28Signature%29[Perl -6 docs on signatures]. - -=== Creating the elements -Next up, we can create the elements we'd like to have visible in our -application window. We needed two inputs for the console version, so we'll -probably need two for the GUI version as well. Since we have two inputs, we -want labels for them. The roll itself will be performed on a button press. -Lastly, we will want another label to display the outcome. This brings us to 6 -elements in total: - -- 3 labels -- 2 entries -- 1 button - -[source,perl6] ----- -my GTK::Simple::Label $label-dice .= new(text => "Amount of dice"); -my GTK::Simple::Label $label-sides .= new(text => "Dice value"); -my GTK::Simple::Label $label-result .= new(text => ""); -my GTK::Simple::Entry $entry-dice .= new(text => 0); -my GTK::Simple::Entry $entry-sides .= new(text => 0); -my GTK::Simple::Button $button-roll .= new(label => "Roll!"); ----- - -This creates all elements we want to show to the user. - -=== Show the elements in the application window -Now that we have our elements, let's put them into the application window. -We'll need to put them into a layout as well. For this, we'll use a grid. The -`GTK::Simple::Grid` constructor takes pairs, with the key being a tuple -containing 4 elements, and the value containing the element you want to show. -The tuple's elements are the `x`, `y`, `w` and `h`, which are the x -coordinates, y coordinates, width and height respectively. - -This in turn takes us to the following statement: - -[source,perl6] ----- -$app.set-content( - GTK::Simple::Grid.new( - [0, 0, 1, 1] => $label-dice, - [1, 0, 1, 1] => $entry-dice, - [0, 1, 1, 1] => $label-sides, - [1, 1, 1, 1] => $entry-sides, - [0, 2, 2, 1] => $button-roll, - [0, 3, 2, 1] => $label-result, - ) -); ----- - -Put a `$app.run` beneath that, and try running `perl6 -Ilib bin/dicer --gtk`. -That should provide you with a GTK window with all the elements visible in the -position we want. To make it a little more appealing, we can add a -`border-width` to the `$app`, which adds a margin between the border of the -application window, and the grid inside the window. - -[source,perl6] ----- -$app.border-width = 20; -$app.run; ----- - -You may notice that there's no `()` after the `run` method call. In Perl 6, -these are optional if you're not supplying any arguments any way. - -=== Binding an action to the button -Now that we have a visible window, it's time to make the button perform an -action. The action we want to execute is to take the values from the two -inputs, roll the correct number of dice with the correct number of sides, and -present it to the user. - -The base code for binding an action to a button is to call `.clicked.tap` on it, -and provide it with a code block. This code will be executed whenever the -button is clicked. - -[source,perl6] ----- -$button-roll.clicked.tap: { -}; ----- - -You see we can also invoke a method using `:`, and then supplying its -arguments. This saves you the trouble of having to add additional `( )` around -the call, and in this case it would be annoying to have to deal with yet -another set of parens. - -Next, we give the code block something to actually perform: - -[source,perl6] ----- -$button-roll.clicked.tap: { - CATCH { - $label-result.text = "Can't roll with those numbers"; - } - - X::TypeCheck::Binding::Parameter.new.throw if $entry-dice.text.Int < 1; - - $label-result.text = ($entry-dice.text.Int × roll($entry-sides.text.Int)).Str; -}; ----- - -There's some new things in this block of code, so let's go over these. - -- `CATCH` is the block in which we'll end up if an exception is thrown in this - scope. `roll` will throw an exception if the parameters are wrong, and this - allows us to cleanly deal with that. -- `X::TypeCheck::Binding::Parameter.new.throw` throws a new exception of type - `X::TypeCheck::Binding::Parameter`. This is the same exception type as thrown - by `roll` if something is wrong. We need to check the number of dice manually - here, since `roll` doesn't take care of it, nor does any signature impose any - restrictions on the value of the entry box. -- `if` behind another statement. This is something Perl 6 allows, and in some - circumstances can result in cleaner code. It's used here because it improves - the readability of the code, and to show that it's possible. - -== The completed product -And with that, you should have a dice roller in Perl 6, with both a console and -GTK interface. Below you can find the complete, finished sourcefiles which you -should have by now. - -=== t/basic.t -[source,perl6] ----- -#! /usr/bin/env perl6 - -use v6.c; - -use Test; -use Local::App::Dicer; - -plan 2; - -subtest "Legal rolls", { - plan 50; - - for 1..50 { - ok 1 ≤ roll($_) ≤ $_, "Rolls between 1 and $_"; - } -} - -subtest "Illegal rolls", { - plan 3; - - throws-like { roll(0) }, X::TypeCheck::Binding::Parameter, "Zero is not accepted"; - throws-like { roll(-1) }, X::TypeCheck::Binding::Parameter, "Negative rolls are not accepted"; - throws-like { roll(1.5) }, X::TypeCheck::Binding::Parameter, "Can't roll half sides"; -} - -done-testing; - -# vim: ft=perl6 ----- - -=== lib/Local/App/Dicer.pm6 -[source,perl6] ----- -#! /usr/bin/env false - -use v6.c; - -unit module Local::App::Dicer; - -sub roll(Int:D $sides where $sides > 0) is export -{ - $sides.rand.ceiling; -} ----- - -=== bin/dicer -[source,perl6] ----- -#! /usr/bin/env perl6 - -use v6.c; - -use Local::App::Dicer; -use GTK::Simple; -use GTK::Simple::App; - -multi sub MAIN(Int:D $dice, Int:D $sides where { $dice > 0 && $sides > 0 }) -{ - say $dice × roll($sides) -} - -multi sub MAIN(Bool:D :$gtk where $gtk == True) -{ - my GTK::Simple::App $app .= new(title => "Dicer"); - my GTK::Simple::Label $label-dice .= new(text => "Number of dice"); - my GTK::Simple::Label $label-sides .= new(text => "Number of sides per die"); - my GTK::Simple::Label $label-result .= new(text => ""); - my GTK::Simple::Entry $entry-dice .= new(text => 0); - my GTK::Simple::Entry $entry-sides .= new(text => 0); - my GTK::Simple::Button $button-roll .= new(label => "Roll!"); - - $app.set-content( - GTK::Simple::Grid.new( - [0, 0, 1, 1] => $label-dice, - [1, 0, 1, 1] => $entry-dice, - [0, 1, 1, 1] => $label-sides, - [1, 1, 1, 1] => $entry-sides, - [0, 2, 2, 1] => $button-roll, - [0, 3, 2, 1] => $label-result, - ) - ); - - $button-roll.clicked.tap: { - CATCH { - $label-result.text = "Can't roll with those numbers"; - } - - X::TypeCheck::Binding::Parameter.new.throw if $entry-dice.text.Int < 1; - - $label-result.text = ($entry-dice.text.Int × roll($entry-sides.text.Int)).Str; - }; - - $app.border-width = 20; - - $app.run; -} - -sub USAGE -{ - say "Launch Dicer as a GUI with --gtk, or supply two positive, round numbers as arguments."; -} ----- - -== Installing your module -Now that you have a finished application, you probably want to install it as -well, so you can run it by calling `dicer` in your shell. For this, we'll be -using `zef`. - -To install a local module, tell `zef` to try and install the local directory -you're in: - -[source] ----- -$ zef install . ----- - -This will resolve the dependencies of the local module, and then install it. -You should now be able to run `dicer` from anywhere. - -[WARNING] -==== -With most shells, you have to "rehash" your `$PATH` as well. On `bash`, this is -done with `hash -r`, on `zsh` it's `rehash`. If you're using any other shell, -please consult the manual. -==== diff --git a/_posts/2018-05-07-sparrowdo-getting-started.adoc b/_posts/2018-05-07-sparrowdo-getting-started.adoc deleted file mode 100644 index cbe2201..0000000 --- a/_posts/2018-05-07-sparrowdo-getting-started.adoc +++ /dev/null @@ -1,239 +0,0 @@ ---- -date: 2018-05-07 14:04:43 -tags: Tutorial Perl6 Sparrowdo Raku LoneStar -description: > - Nondescript ---- -= Sparrowdo - Getting started -:toc: preamble - -https://github.com/melezhik/sparrowdo[Sparrowdo] is a Perl 6 project to -facilitate automatic configuration of systems. There's a -https://sparrowhub.org/[repository of useful modules] to make specific cases -easier to work with, but the -https://github.com/melezhik/sparrowdo/blob/master/core-dsl.md[Core DSL] can -already take care of many tasks. In this tutorial, I'll guide you through -setting up Sparrowdo, bootstrapping it onto your local system, writing a task -and running it. - -== Install Sparrowdo -Sparrowdo is a http://perl6.org/[Perl 6] project, so you'll need to have Perl 6 -installed. We'll also use the Perl 6 package manager -https://github.com/ugexe/zef/[zef] to install Sparrowdo itself. Luckily for us, -there's a stable distribution of Perl 6 with everything we need added to it, -called https://rakudo.org/files[Rakudo Star]. And to make it easier for -GNU+Linux users, I wrote a tool to fetch the latest Rakudo Star release, compile -it and install it, called https://github.com/Tyil/lonestar[LoneStar]. Since this -tutorial will aim at GNU+Linux users, I'll use that to install Perl 6. - -=== Installing Perl 6 with LoneStar -LoneStar is a Bash application to download, compile and set up Perl 6. It's a -standalone application, meaning you don't have to install it to your system. You -can just run it from the source directory. First, we'll have to get the source -directory, which we'll do using `git`. - -[source,sh] ----- -mkdir -p ~/.local/src -git clone https://github.com/tyil/lonestar.git ~/.local/src/lonestar -cd !$ ----- - -Now you have the LoneStar sources available in `~/.local/src/lonestar`. You can -run the application using `./bin/lonestar`. Running it, you'll get some help -output: - -[source] ----- -$ ./bin/lonestar -lonestar - Installation manager for Rakudo Star - -Usage: lonestar <action> [arguments...] - -Actions: - help [action] - init [version=latest] - install [version=latest] - path [version=latest] - reinstall [version=latest] - upgrade ----- - -We'll be needing the `install` action to get Perl 6 installed, and the `init` -action to configure the `$PATH` environment variable. Depending on your -hardware, `install` may take a couple minutes as it will compile Rakudo Perl 6 -and install some base modules. You might want to grab a drink during this -period. - -[source] ----- -$ ./bin/lonestar install -$ eval $(./bin/lonestar init) -$ perl6 -v -This is Rakudo Star version 2018.04.1 built on MoarVM version 2018.04.1 -implementing Perl 6.c. ----- - -[NOTE] -==== -If there's a newer version available of Rakudo Star, the version numbers given -by `perl6 -v` will differ for you. -==== - -=== Installing Sparrowdo with zef -Now that you have Perl 6 available and installed, you can continue on using -`zef` to install Sparrowdo. `zef` is bundled with Rakudo Star, so you don't have -to do anything to get it working. - -[source,sh] ----- -zef install Sparrowdo ----- - -This will instruct `zef` to install Sparrowdo and all its dependencies. This can -take a couple minutes, again depending on the hardware of your machine. - -== Bootstrapping your system -The first step to working with Sparrowdo is bootstrapping the system you wish to -use it with. In this case, that'll be the local system. There's a `--bootstrap` -option to do this automatically. - -[source,sh] ----- -sparrowdo --bootstrap ----- - -[TIP] -==== -If you wish to bootstrap a remote system, you can use the `--host` option to -specify the system. For example: `sparrowdo --host=192.168.1.2 --bootstrap`. -==== - -Now your system is ready to be configured automatically using Sparrowdo! - -== Sparrowfiles -Sparrowfiles are the files that describe the tasks Sparrow should execute to -get you the configuration you want. They are valid Perl 6 code, and call the -subroutines (or _sparrowtasks_) that will handle the actual actions. By default, -when running `sparrowdo`, it will look for a file named `sparrowfile` in the -current directory. - -To make our sample, we'll create a new directory to work in, so we have clean -directory that can be shared easily. You can also keep this directory under -version control, so you can distribute the `sparrowfile` with all its templates. - -[TIP] -==== -If you just want to create an empty directory to test things in, without -"polluting" the rest of your system, just call `cd -- "$(mktemp -d)"`. This will -create a temporary directory and change the working directory to there. -==== - -I'll be using `~/.local/sparrowdo/local-dns` to work in, as I'll be setting up a -local dns cache with http://www.thekelleys.org.uk/dnsmasq/doc.html[dnsmasq] for -the sample code. - -=== Writing a `sparrowfile` -As noted in the previous paragraph, for the sake of a demo I'll guide you -through creating a `sparrowfile` to install and configure `dnsmasq` as a local -DNS cache. Using your favourite `$EDITOR`, write the following to `sparrowfile`: - -[source,perl6] ----- -package-install "dnsmasq"; -directory "/etc/dnsmasq.d"; -file-create "/etc/dnsmasq.conf", %(content => slurp "dnsmasq.conf"); -file-create "/etc/dnsmasq.d/resolv.conf", %(content => slurp "resolv.conf"); -service-start "dnsmasq"; ----- - -This `sparrowfile` will set up the following configuration for `dnsmasq`: - -- Install the `dnsmasq` package -- Create the `/etc/dnsmasq.d` directory in which we'll store configuration files - for `dnsmasq` -- Create the configuration files `dnsmasq.conf` at `/etc/dnsmasq.conf` -- Create the `resolv.conf` in the `dnsmasq.d` directory -- Start the `dnsmasq` service - -The configuration files will be created based on the configuration files in the -current directory. So for this to work, you'll need to also create the -appropriate configuration files. Let's start off with the main `dnsmasq` -configuration in `dnsmasq.conf`: - -[source,conf] ----- -listen-address=127.0.0.1 - -no-dhcp-interface= -resolv-file=/etc/dnsmasq.d/resolv.conf ----- - -This will make `dnsmasq` listen on the loopback interface, so it'll only be able -to be used by the local machine. Furthermore, DHCP functionality will be -disabled, and the upstream resolvers are read from `/etc/dnsmasq.d/resolv.conf`. -The contents of that file are as follows: - -[source,conf] ----- -nameserver 37.235.1.174 -nameserver 37.235.1.177 ----- - -These nameservers are part of the https://freedns.zone/en/[FreeDNS] project. You -can of course use whatever other DNS provider you want to use as your upstream -servers. Now, for `dnsmasq` to be used, you will also need to set your machine's -DNS resolvers to point to the `dnsmasq` service. This is defined in -`/etc/resolv.conf`, so lets append the following to our `sparrowfile` to set -that up. - -[source,conf] ----- -bash "chattr -i /etc/resolv.conf"; -file-delete "/etc/resolv.conf"; -file-create "/etc/resolv.conf", %(content => "nameserver 127.0.0.1"); -bash "chattr +i /etc/resolv.conf"; ----- - -This will remove the "immutable" attribute from `/etc/resolv.conf` if it's set. -Next it will remove the current `/etc/resolv.conf` and write out a new one which -only refers to the local machine as DNS resolver. This is to ensure an existing -`/etc/resolv.conf` gets recreated with the configuration we want. Finally, it -adds back the immutable attribute to the file, so other processes won't -overwrite it. - -=== Running the `sparrowfile` -To run the `sparrowfile` and get the setup you desire, run the `sparrowdo` -command with `--local_mode` and wait. - -[source] ----- -sparrowdo --local_mode ----- - -[NOTE] -==== -If you want to run this on a remote machine to configure that one instead, you -can use `--host=<ip>` instead of `--local_mode`. -==== - -You can check whether it actually worked by inspecting the files in -`/etc/dnsmasq.d` and your `/etc/resolv.conf`. The easiest way to check their -contents would be by using `cat`: - -[source] ----- -cat /etc/dnsmasq.d/dnsmasq.conf -cat /etc/dnsmasq.d/resolv.conf -cat /etc/resolv.conf ----- - -== Closing words - -You should now have a working local DNS setup, configured programmatically -through Sparrowdo. This allows you easily get it working on other machines as -well, and updates can be done in a much simpler fashion for all of them -together. - -If you have more interest in automating configuration with Sparrowdo, go check -their website, https://sparrowdo.wordpress.com/. diff --git a/_posts/2018-08-15-the-perl-conference-in-glasgow.adoc b/_posts/2018-08-15-the-perl-conference-in-glasgow.adoc deleted file mode 100644 index 6fd8f90..0000000 --- a/_posts/2018-08-15-the-perl-conference-in-glasgow.adoc +++ /dev/null @@ -1,240 +0,0 @@ ---- -date: 2018-08-23 -tags: Perl Conference -description: > - My feedback and comments on some of the talks I attended during The Perl - Conference in Glasgow, in 2018. ---- -= The Perl Conference in Glasgow -:toc: preamble - -This year the European Perl Conference was hosted in Glasgow, and of course -I've attended a number of presentations there. On some of these, I have some -feedback or comments. These talks, and the feedback I have for them, are -detailed in this blog post. - -== Discourse Without Drama - -[NOTE] -==== -There used to be comments here to show a different perspective to the speaker. -However, someone's lightning talk has been removed because -https://act.perlconference.org/tpc-2018-glasgow/news/1568[some people played -the "offended" card], and no specifics are being published about it. Therefore, -I cannot make any assertions on what is and what is not allowed, and have to -assume that criticism of harmful beliefs will be shut down. As such, I have -removed the comments which used to be here, in an attempt to keep it from -detracting from the rest of my post. -==== - -== European Perl Mongers Organiser's Forum 2018 - -The Perl community isn't big nowadays, however, the Perl 6 language also offers -a lot of concepts which are very well suited for modern programming. Sadly, if -no new users try out the language, it will be all for nothing. As such, we need -to bring new blood in to the community. - -One of the ways of doing this is by extending our promoting efforts outside of -the Perl community. Most people who like Perl are in a social bubble with other -people that are also familiar with the Perl programming language, be it 5 or 6. -But we need to reach new people as well, who will most likely be outside of -this social bubble. These people don't have to be techies either, they might -just as well be marketeers or designers. - -I myself am part of the "techies", so I'll stick to this particular group for -now. And I know people like me can be found at meetups, so it would be -worthwhile to promote Perl at meetups which are not dedicated to Perl. Think of -more generic programming meetups, or GNU+Linux User Groups. We have to be -mindful not to be too pushy, though. Listen to other people, and try to -understand the problem they're facing. Most of them will not be open to using a -different language immediately, especially not Perl (which sadly has a -particularly bad standing amongst people unfamiliar with it). Try to assist -them with their issues, and slowly introduce them to Perl (6) if it helps to -showcase what you mean. It might also be interesting to show people examples on -how to solve certain issues before telling them the language's name, so they -don't have a negative preconception solely from the name. - -Another thing to note is that Perl is more than just a programming language. -It's a community, and a large library of modules, known as CPAN. And CPAN -offers some nifty tools, such as the CPAN testers, which help ensure module -developers that their code runs on a massive set of platforms and Perl -versions. - -This has led me to consider the creation of a new Perl 6 module: -`CPAN::Tester`, to make it easy for people to contribute to a large-scale -testing environment for Perl 6. The idea is that one can run `CPAN::Tester` on -their machine, which will keep track of new Perl 6 modules being uploaded to -CPAN. The results are to be sent to another server (or multiple servers), which -can aggregate the data and show a matrix of test results. This aggregating -server could also be built as a Perl 6 module, possibly named -`CPAN::Tester::ResultsServer`. This would make setting up an environment -similar to CPAN testers for Perl 5 quite easy for Perl 6. - -== Perl 6 in Real Life $Work - -The speaker shows the perfect use case for -https://docs.perl6.org/language/grammars[Perl 6 grammars], advanced yet -readable parsing of text and performing actions with the results. It's an -interesting talk, showcasing some nifty grammar constructs. The best part of -this is that it actually runs in production, where it parses over 700 files, -consisting over 100,000 lines of code, in about 22 seconds (on his laptop). -This goes to show that Perl 6 is no longer "too slow to use in production". - -It might be interesting to run this application of grammars on every Perl 6 -release to gather more information on the speed improvements of Perl 6, much -like Tux's `Text::CSV` runs. - -== Releasing a Perl 6 Module - -The speaker starts off with detailing the platform which most Perl 6 modules -use to host their code repository, GitHub. He also touched upon automated -testing using Travis and AppVeyor. It was good to show how to make use of -these, as automated testing oftentimes stops unintended bugs from reaching end -users. But, I personally prefer GitLab over GitHub, as they have much better -testing functionality, and they actually release their own platform as an open -source package. I'd like more GitLab love from the community and speakers as -well if possible. This would also make the speaker's CI configuration simpler, -for which he currently uses a `.travis.yml` file. This requires him to build -Perl 6 from source every test run, wasting quite a lot of time. - -It was also noted that there's a module to help you set up this module -skeleton, `mi6`. The speaker also noted that it doesn't seem to add much once -you know how a Perl 6 module is organized, and I tend to agree with this. -Actually, I made a module precisely because I agree with him here, -`App::Assixt`. This module intends to smoothen the entire course of module -development, not just the creation of a skeleton file. It will take care of -keeping your `META6.json` up to date, and ease uploading your module to CPAN as -well. - -Lastly, the speaker says the `META6.json` documentation can be found in S22. -While this is technically correct, S22 is *not* the implementation's -documentation, this lives in the official Perl 6 documentation instead. S22 -offers many additional information to be stored in the `META6.json`, but using -these fields will actually break installation of your module through `zef`, -rendering it unusable by others. I would strongly recommend people not to use -S22 when trying to figure out what they can or cannot do with their -`META6.json`. - -== How to become CPAN contributor? - -Submitting a pull request (or more correctly named, merge request) to a -repository is possibly the most straightforward way to help out other projects. -However, sometimes it will take a long time to get a response. The speaker -notes this can actually be on the scale of years. I have authored a number of -modules myself, and have been in the situation where I had not realized I got a -merge request from another person (same goes for issue reports). I would -recommend people who are not getting timely responses to their contributions to -contact the maintainer via other channels which are more suited for -communications. Think of email or IRC, for instance. You'll generally have a -much better chance of getting a timely response from the author, and then you -can work out your contribution and see if you can get it merged into the main -project. - -The speaker also lists a couple of ways to get started with contributing to -modules. One thing I missed in particular was the Squashathons -footnote:[A Squashathon is like a hackathon, except everyone in the world is -invited, and you can help out over the Internet, staying in your own home. Of -course, you can still meet up with other developers and make it a social -gathering in the real world as well!] for Perl 6. -These generally offer a good entry point to help out with the language's -development and the ecosystem's maintainance. - -Near the end, it was pointed out that it is a good idea to have a thick skin. -Even when it's not intended, people can come accross as rude. This is in -opposition to the talking point of the speaker yesterday (_Discourse Without -Drama_), but he does raise a good point here. People oftentimes don't mean to -insult you, but context is easily lost in written communications. Try to stay -mature and professional, you can simply ask for clarification. If you feel the -person remains hostile towards you, walk away. There's plenty of other projects -that would love your contributions! - -== Conference Organizers & European Perl Mongers Organiser's Forum 2018 BoF - -Well, that's certainly a mouthful for a heading, and it even contains an -abbreviation! This event was not a presentation, but a platform to exchange -ideas together. - -One of the items that were up for discussion was _A Conference Toolkit_, or ACT -for short. This is the platform used to organize Perl events, such as this -conference and Perl workshops throughout the world. However, ACT is dated. -They enabled HTTPS a short while ago, but it's still not the default because -people don't want to risk breaking the platform. I think this is enough of -an indication that it might be time to make something new to replace it. - -And I'm not alone in that sentiment, it seems. However, ACT is big and contains -a lot of data we don't want to lose. It's a massive undertaking to make a new -tool that works at least as well, and allows us to make use of the old data as -well. There is a Trello board available that lists all the features that would -be required to implement, so that's a good start already. I think now it needs -a dedicated product owner with people contributing code, so a start can be -made. This does seem like a touchy subject, since I'm far from the first person -to want this. Many before me have tried and failed already. - -As such, I'd propose not making it a Perl centric tool. Make it a modular, -generic event organizing tool. Get a good database design that we can import -our old data into, so nothing is lost, but things can be converted to be more -useful for our current needs. This way, we can work in small steps, and maybe -even reach contributors from outside the regular Perl circles. This might even -bring in new partnerships (or sponsors) towards the Perl community. - -Personally, I'd like to see something like this to be written in Perl 6. This -way, it could also be used as a showcase project for the Perl 6 programming -language. - -== Writing a Perl 6 Module - -Perl 6 has this very neat feature called -https://docs.perl6.org/language/typesystem#index-entry-subset-subset[subsets]. -These can be used to make your own types with very little effort, which can -help tremendously to keep your code clean and concise. There are two arguments -I have in favour of subsets that the speaker did not touch upon. - -First off, using a subset instead of a `where` clause in a sub or method -signature will bring much better error messages. If you use a `where` in your -signature, and the check fails, you'll get an error that there was no signature -that matched `where { ... }`. - -Secondly, if you want to use abstract methods, you can't really use a `where`. -https://stackoverflow.com/questions/51570655/how-to-use-abstract-multi-methods-containing-a-where[I've -asked a question about this on Stack Overflow], which has the details as to why -this doesn't work the way you might expect. - -Next, there's some cool things about operators in Perl 6. There are many of -these available by default, and it's _very_ easy to add new ones yourself as -well. In fact, the `Math::Matrix` module used throughout the presentation makes -some available as well. Thanks to the ease of adding operators in Perl 6, if -you have a `Math::Matrix $m` in Perl 6, you can get the norm by writing `|| $m -||`. This is the mathematically correct way to write this, making it easy to -understand for everyone using matrixes in their daily lives. If you're a -mathematician, small things like these are great to have. - -I have some comments on the `Math::Matrix` module itself as well, based on -slides shown in the presentiation. The first thing I noticed is that there's a -`norm` method using a `where` clause when it's not needed: - -[source,perl6] ----- -method norm (Str $which where * eq 'row-sum') ----- - -This can be written instead as: - -[source,perl6] ----- -method norm ('row-sum') ----- - -This is shorter and clearer, and you'll get better feedback from the compiler -as well. I https://github.com/pierre-vigier/Perl6-Math-Matrix/pull/49[submitted -a pull request on the GitHub repository] in an attempt to improve this, which -got merged! The speaker was not aware it could be done in this manner, so I'm -proud I got to teach him something right after he did his presentation. - -== Winding down - -I've had a great time at the Perl conference, spoke to many people with whom -I've had some great discussions. I got to meet and personally thank a number of -people who've helped me out over the past year as well. - -A big thank you to all the people who made this conference possible, and I hope -to see you all again in Riga! diff --git a/_posts/2018-09-04-setting-up-pgp-with-a-yubikey.adoc b/_posts/2018-09-04-setting-up-pgp-with-a-yubikey.adoc deleted file mode 100644 index 4b7cee0..0000000 --- a/_posts/2018-09-04-setting-up-pgp-with-a-yubikey.adoc +++ /dev/null @@ -1,466 +0,0 @@ ---- -date: 2018-09-04 -tags: Security YubiKey PGP GPG -description: > - An introduction to decent security using the Yubikey as your physical - security card. ---- -= Setting up PGP with a Yubikey -:toc: preamble - -I've recently started a job where I am required to have above-average security -practices in place on my machine. I already had some standard security in -place, such as full disk encryption and PGP encrypted email, but I thought that -this would be a good time to up my game. To accomplish this, I purchased a -Yubikey to act as my physical security token. Additionally, I have a USB device -which is also encrypted to hold backups of the keys. - -In this blogpost, I will detail how I set up my security policies in the hopes -it will be able to help out other people looking to improve their security, and -to get feedback to improve my set up as well. - -[NOTE] -==== -I am using the Yubikey 4. If you're using another version, some steps may -differ. -==== - -== Installing required software - -You'll need some software to set all of this up. Depending on your -distribution, some of it might already be installed. Everything not installed -yet should be installed with your distribution's package manager. - -For encrypting the disk and the USB key, you will need `cryptsetup`. To -generate and use the PGP keys, you will need `gpg`, at least version 2.0.12. To -interface with the Yubikey itself, you'll need `pcsc-lite`, and start the -service as well. It may be necessary to restart the `gpg-agent` after -installing `pcsc-lite`, which you can do by simply killing the existing -`gpg-agent` process. It restarts itself when needed. - -To securely remove the temporary data we need, you should make sure you have -`secure-delete` available on your system as well. - -== Personalizing the Yubikey - -The Yubikey can be personalized. Some of this personalization is completely -optional, such as setting personal information. However, setting new PIN codes -is strongly advised, as the default values are publicly known. - -=== PIN codes - -The PIN codes are short combinations of numbers, letters and symbols to grant -permission to write to or retrieve data from the Yubikey. The default value for -the user PIN is `123456`. The admin PIN is `12345678` by default. These should -be changed, as they're publicly known and allow the usage of your private keys. -To change these, use the `gpg` program and enter admin mode: - -[source] ----- -gpg --card-edit - -gpg/card> admin -Admin commands are allowed ----- - -You'll notice it immediately says that admin commands are now allowed to be -used. The admin PIN (`12345678`) will be asked whenever an admin command is -executed. It will then be stored for this session, so you won't have to enter -it right away. To update the PIN values, run the following commands: - -[source] ----- -gpg/card> passwd -gpg/card> 3 ----- - -This will change the admin PIN first. This PIN is required for managing the -keys and user PIN on the Yubikey. To set the user PIN, pick `1` instead of `3`: - -[source] ----- -gpg/card> 1 ----- - -Once this is done, you can quit the `passwd` submenu using `q`: - -[source] ----- -gpg/card> q ----- - -You may have noticed we skipped the reset code. Resetting the device will wipe -existing keys, so it's not a serious risk to keep this at the default. The -private keys will be backed up to an encrypted USB drive, so we can always -retrieve them and put them back on the Yubikey if ever needed. - -=== Personal information - -The personal information is optional, but could be used by a friendly person to -find out who a found Yubikey belongs to. They can contact the owner, and send -the key back. You can set as many of the personally identifying fields as you -want. If you're interested in setting this information, plug in your Yubikey -and edit the card information with `gpg`: - -[source] ----- -gpg --card-edit ----- - -Once you're back in the GPG shell, you can update your personal information. -There are 5 attributes that you can set in this way: - -- `name`, which is your real name; -- `lang`, which is your preferred contact language; -- `sex`, which is your real sex; -- `url`, which indicates a location to retrieve your public key from; -- `login`, which indicates your email address. - -Each of these attributes can be updated by running the command in the GPG -shell. For instance, to update your real name, run the following: - -[source] ----- -gpg/card> name ----- - -You do not need to explicitly save once you're done. You can run `quit` to quit -the GPG shell and return to your regular shell. - -== Creating PGP keys - -To create the PGP keys, we'll create a temporary directory which will function -as our working directory to store the keys in. This way you can't accidentally -break existing keys if you have them, and ensure that the private keys don't -accidentally linger on in your filesystem. - -=== Preparing a clean environment - -To create such a temporary directory, we'll use `mktemp`, and store the result -in an environment variable so we can easily re-use it: - -[source,sh] ----- -export GNUPGHOME="$(mktemp -d)" ----- - -Now you can switch to that directory using `cd "$GNUPGHOME"`. Additionally, -`$GNUPGHOME` is also the directory `gpg` uses as its working directory, if it -is set. This means you can use a temporary custom configuration for `gpg` as -well, without it affecting your normal setup. The following configuration is -recommended to set in `$GNUPGHOME/gpg.conf` before starting: - -[source,conf] ----- -use-agent -charset utf-8 -no-comments -keyid-format 0xlong -list-options show-uid-validity -verify-options show-uid-validity -with-fingerprint ----- - -If you have a `gpg-agent` running, it is recommended to stop it before -continuing with `killall gpg-agent`. - -=== Creating the master key - -For our master key, we'll go for a 4096 bytes RSA key. 2048 would be plenty as -well, if you want the generation to be a tad quicker. `gpg` will ask you a -couple questions to establish your identity, which is required for a PGP key. -You can add more identities later, in case you're using multiple email -addresses, for instance. - -Start the key generation process with `gpg`: - -[source] ----- -gpg --full-generate-key ----- - -When asked what kind of key you want, choose `4` (RSA (sign only)). Next is the -key size, which should be `4096`. - -The key's expiration is optional, though highly recommended. It will be more -effort to maintain the keys, as you'll occasionally need the private master -keys to extend the validity, but you can also guarantee that your keys won't -stay valid in case you ever lose them. If you don't want to bother with -refreshing your keys from time to time, just press enter here to continue. - -When prompted on whether the data is correct, doublecheck whether the data is -really correct, and then enter `y` and press enter to accept the current -values. `gpg` will continue with your identity information, which you should -fill out with your real information. The comment field can be left empty, this -is an optional field to add a comment to your identity, such as "School", or -"Work keys". `gpg` will ask your confirmation one final time. Enter an `o` -(it's not case sensitive) and press enter again. The final step before it will -generate a key is to enter a passphrase. This is technically optional, but -highly recommended. If anyone ever gets their hands on your private master key, -they will need the passphrase in order to use it. Adding one is yet another -layer against malicious use of your key. - -Once you've chosen a passphrase, it will generate they key and output some -information about the key. Verify whether this information is correct one more -time, and if it is, you can continue to the next step. If it is not, redo the -whole PGP section of this post. - -Take note of the line starting with `pub`. It shows that the key is an -`rsa4096` key, followed by a `/`, and then the key ID. You'll need this key ID -throughout the rest of this post. For convenience, you can store this ID in -a variable, and just refer to the variable when you need it's value again: - -[source,sh] ----- -export KEYID=0x27F53A16486878C7 ----- - -This post will use the `$KEYID` variable from now on, to make it easier to -follow. - -=== Creating a revocation certificate - -The revocation certificate can be used to invalidate your newly created key. -You should store it seperately from the private master key, preferably printed -on a sheet of paper. If you want to be able to easily read it back in, consider -printing it as a QR code. - -To create the certificate, run the following: - -[source] ----- -gpg --gen-revoke $KEYID > $GNUPGHOME/revoke.txt ----- - -This will prompt you to specify a reason, for which you'll want to use `1`. -This way you can easily revoke the key's validity if you ever lose it. If you -want to revoke your keys in the future for any other reason, you can always -generate a new revocation certificate for that specific purpose. You don't have -to supply an additional description, so just hit enter. A revocation -certificate will be written to `$GNUPGHOME/revoke.txt`. - -=== Creating the subkeys - -Now that you have your master key and the ability to revoke it in case anything -goes wrong in the future, it's time to create a couple of subkeys which can be -stored on the Yubikey, and used in your daily life. We'll create seperate keys -for _encryption_, _signing_ and _authentication_, and store each of them in -their own dedicated slot on the Yubikey. - -To add subkeys to your master key, enter a GPG shell to edit your existing -key with `gpg --expert --edit-key $KEYID`. The `--expert` is required to show -all the options we're going to need. Once the GPG shell has started, run -`addkey` to add a new key. - -Just like with the master key, a number of questions will be asked. Expiration -for subkeys is generally not advised, as the subkeys will be considered invalid -whenever the master key has expired. The key sizes for the subkeys can be left -at 2048 as well, which is also the maximum size for keys for the older Yubikey -models. The key type is different for all 3 subkeys. - -You will want to select type `4` (RSA (sign only)) for your signing key, type -`6` (RSA (encrypt only)) for the encryption key, and type `8` (RSA (set your -own capabilities)) for the authentication key. With the final key, it will ask -you what capabilities you want to enable. The only capability you want it to -have is *Authentication*. - -Once you've created the subkeys, you can check `gpg --list-secret-keys` to look -at your newly created keys. You should have 1 `sec` key, which is the master -key, and 3 `ssb` keys, which are the subkeys. One line should end with `[S]`, -one with `[E]` and one with `[A]`. These denote the capabilities of the -subkeys, _Sign_, _Encrypt_ and _Authenticate_, respectively. - -=== Export the keys - -Now that you have your keys generated, you should export them, allowing you to -easily import them in another environment in case you ever need to generate -more keys, invalidate some keys, or extend the validity of the keys in case you -set an expiry date. This can be done with the following commands: - -[source,sh] ----- -gpg --armor --export-secret-keys $KEYID > masterkey.asc -gpg --armor --export-secret-subkeys $KEYID > subkeys.asc ----- - -== Creating a backup USB - -For the backup of the private keys, I'm using an encrypted USB device. You can -also opt to print the keys to paper, and retype them if you ever need them. Or -print a QR code that you can scan. But for convenience sake, I went with a USB -device. I encrypted it, and stored it in a safe and sealed location, so it's -easy to detect unwanted attempted access. - -=== Encrypting the USB - -For the encryption, I went with full device encryption using LUKS. You will -need the `cryptsetup` utility to apply the encryption, and to unlock the drive. -You can find out the device name from `dmesg` or `lsblk`. Once you know it, -encrypt the drive with the `luksFormat` subcommand. - -[WARNING] -==== -Using the wrong name for the device can irrecoverably destroy data from another -drive! -==== - -[source,sh] ----- -cryptsetup luksFormat /dev/sdb ----- - -It will prompt you whether you want to continue, and ask twice for a passphrase -to ensure it is correct. Make sure you don't forget the passphrase, or you'll -lose access to your backup keys. - -Once it has been encrypted, unlock the device. - -[source,sh] ----- -cryptsetup luksOpen /dev/sdb crypt ----- - -This will open the device as `/dev/mapper/crypt`. Format it with your favourite -filesystem. I used `ext4`. - -[source,sh] ----- -mkfs.ext4 /dev/mapper/crypt ----- - -Once it has been formatted, you can mount it as a regular device. - -[source,sh] ----- -mount /dev/mapper/crypt /mnt/usb ----- - -=== Copying the keys - -Copying the keys is as straightforward as copying other files. You can use -`$GNUPGHOME` to target the source directory. - -[source,sh] ----- -cp -arv "$GNUPGHOME"/* /mnt/usb/. ----- - -Once the files are copied, you can unmount the drive, lock it and unplug the -USB. - -[source,sh] ----- -sync -umount /mnt/usb -cryptsetup luksClose crypt ----- - -Store the USB in a safe location, because these private keys can give someone -full control of your identity. - -== Storing the private keys on the Yubikey - -The Yubikey has key slots for encryption, signing and authentication. These -need to be set individually, which can be done using `gpg`. First, you need to -select a key using the `key` command, then store it on the card using -`keytocard` and select a slot to store it in, then finally deselect the key by -using the `key` command again. - -[source] ----- -gpg --edit-key $KEYID - -gpg> key 1 -gpg> keytocard -Your selection? 1 -gpg> key 1 - -gpg> key 2 -gpg> keytocard -Your selection? 2 -gpg> key 2 - -gpg> key 3 -gpg> keytocard -Your selection? 3 - -gpg> save ----- - -You can verify whether the keys are available on the Yubikey now using `gpg ---card-status`. It will show the key fingerprints for the `Signature key`, -`Encryption key` and `Authentication key`. - -=== Sharing your public key - -You can share your public keys in many ways. Mine is hosted link:/pubkey.txt[on -my own site], for instance. There are also https://sks-keyservers.net/[public -keyservers] on which you can upload your keys. `gpg` has the `--send-keys` and -`--recv-keys` switches to interact with these public keyservers. For ease of -use, I would recommend uploading them to a public keyserver, so that other -people can easily import it. For instance, my key can be imported using `gpg`: - -[source] ----- -gpg --recv-keys 0x7A6AC285E2D98827 ----- - -== Clean up - -The keys are on the Yubikey, and you probably do not want to leave traces on -your local system of these new keys, so you should clean up the `$GNUPGHOME` -directory. There's a utility for securely removing a directory with all its -contents, called `secure-delete`, which provides the `srm` program. You can use -it just like the regular `rm` on the temporary directory. - -[source,sh] ----- -srm -r "$GNUPGHOME" ----- - -You can also `unset` the `$GNUPGHOME` variable at this point, so `gpg` will use -it's default configuration again. - -[source,sh] ----- -unset GNUPGHOME ----- - -== Configure GPG - -Finally, you have your keys on the Yubikey and the traces that might have been -left on your device are wiped clean. Now you should configure `gpg` for regular -use as well, however, this is completely optional. All this configuration does -is ensure you have good defaults for the current day and age. - -[source] ----- -auto-key-locate keyserver -keyserver hkps://hkps.pool.sks-keyservers.net -keyserver-options no-honor-keyserver-url -personal-cipher-preferences AES256 AES192 AES CAST5 -personal-digest-preferences SHA512 SHA384 SHA256 SHA224 -default-preference-list SHA512 SHA384 SHA256 SHA224 AES256 AES192 AES CAST5 -ZLIB BZIP2 ZIP Uncompressed -cert-digest-algo SHA512 -s2k-cipher-algo AES256 -s2k-digest-algo SHA512 -charset utf-8 -fixed-list-mode -no-comments -no-emit-version -keyid-format 0xlong -list-options show-uid-validity -verify-options show-uid-validity -with-fingerprint -use-agent -require-cross-certification ----- - -== Conclusion - -You now have PGP keys available on your Yubikey. These keys are only available -to your system if the Yubikey is inserted, and the user PIN is given. You can -use these keys for authentication, signing and encrypting/decrypting messages. -In a future post, I'll detail how to set up a number of services to use these -keys as well. diff --git a/_posts/2018-09-13-hackerrank-solutions-python3-and-perl6-part-1.adoc b/_posts/2018-09-13-hackerrank-solutions-python3-and-perl6-part-1.adoc deleted file mode 100644 index 6dd01a6..0000000 --- a/_posts/2018-09-13-hackerrank-solutions-python3-and-perl6-part-1.adoc +++ /dev/null @@ -1,465 +0,0 @@ ---- -tags: Hackerrank Perl6 Python Python3 Programming Raku -description: > - A number of solutions to Hackerrank challenges in both the Python 3 and the - Perl 6 programming languages. Compare the results and see which language - works best for you! ---- -= Hackerrank solutions: Python 3 and Perl 6 (part 1) -:toc: preamble - -I recently started at a new company, for which I will have to write Python 3 -code. To make sure I still know how to do basic stuff in Python, I started to -work on some https://www.hackerrank.com/[Hackerrank challenges]. In this post, -I will show solutions to some challenges to show the differences. I hope that I -can show that Perl doesn't have to be the "write only" language that many -people make it out to be. - -[NOTE] -==== -I am _much_ more proficient in the Perl 6 programming language than in Python -(2 or 3), so I might not always use the most optimal solutions in the Python -variants. Suggestions are welcome via email, though I most likely won't update -this post with better solutions. I ofcourse also welcome feedback on the Perl 6 -solutions! -==== - -== Challenges - -The challenges covered in this post are the -https://www.hackerrank.com/domains/algorithms?filters%5Bsubdomains%5D%5B%5D=warmup[warmup -challenges] you are recommended to solve when you make a new account. The code -around the function I'm expected to solve won't be included, as this should be -irrelevant (for now). Additionally, I may rename the sub to conform to -https://en.wikipedia.org/wiki/Letter_case#Special_case_styles[kebab-case], as -this is more readable (in my opinion), and allowed in Perl 6. - -=== Solve Me First - -This challenge is just a very simple example to introduce how the site works. -It required me to make a simple `a + b` function. - -[source,py3] ----- -def solveMeFirst(a,b): - return a+b ----- - -The Perl 6 variant isn't going to very different here. - -[source,perl6] ----- -sub solve-me-first ($a, $b) { - $a + $b -} ----- - -For those not familiar with Perl 6, the `$` in front of the variable names is -called a https://docs.perl6.org/language/glossary#index-entry-Sigil[Sigil], and -it signals that the variable contains only a single value. - -You may have noticed that there's also no `return` in the Perl 6 variant of -this example. In Perl 6, the last statement in a block is also the implicit -return value (just like in Perl 5 or Ruby). - -=== Simple Array Sum - -For this challenge I had to write a function that would return the sum of a -list of values. Naturally, I wanted to use a `reduce` function, but Python 3 -does not support these. So I wrote it with a `for` loop instead. - -[source,py3] ----- -def simpleArraySum(ar): - sum = 0 - - for i in ar: - sum += i - - return sum ----- - -Perl 6 does have a `reduce` function, so I would use that to solve the problem -here. - -[source,perl6] ----- -sub simple-array-sum (@ar) { - @ar.reduce(sub ($a, $b) { $a + $b }) -} ----- - -Here you can see a different sigil for `@ar`. The `@` sigil denotes a list of -scalars in Perl 6. In most other languages this would simply be an array. - -This code can be written even shorter, however. Perl 6 has -https://docs.perl6.org/language/operators#index-entry-%5B%2B%5D_%28reduction_metaoperators%29[reduction -meta-operators]. This allows you to put an operator between brackets, like -`[+]`, to apply a certain operator as a reduce function. - -[source,perl6] ----- -sub simple-array-sum (@ar) { - [+] @ar -} ----- - -[NOTE] -==== -After publishing this post I have learned that both Python 3 and Perl 6 have a -`.sum` function that can also be called on the array, simplifying the code in -both languages. -==== - -=== Compare the Triplets - -This challenge provides you with 2 lists of 3 elements each. The lists should -be compared to one another, and a "score" should be kept. For each index, if -the first list contains a larger number, the first list's score must be -incremented. Similarly, if the second list contains a larger number on that -index, the second list's score must be incremented. If the values are equal, do -nothing. - -[source,py3] ----- -def compareTriplets(a, b): - scores = [0, 0] - - for i in range(3): - if a[i] > b[i]: - scores[0] += 1 - - if a[i] < b[i]: - scores[1] += 1 - - return scores ----- - -I learned that Python 3 has no `++` operator to increment a value by 1, so I -had to use `+= 1` instead. - -[source,perl6] ----- -sub compare-triplets (@a, @b) { - my @scores = [0, 0]; - - for ^3 { - @scores[0]++ if @a[$_] > @b[$_]; - @scores[1]++ if @a[$_] < @b[$_]; - } -} ----- - -In Perl 6, the `^3` notation simply means a range from 0 to 3, non-inclusive, -so `0`, `1`, `2`, meaning it will loop 3 times. The `$_` is called the -__topic__, and in a `for` loop it is the current element of the iteration. - -Both of these loops could use a `continue` (or `next` in Perl 6) to skip the -second `if` in case the first `if` was true, but for readability I chose not -to. - -[NOTE] -==== -After publishing this post I learned that Python 3 also supports the inline if -syntax, just like Perl 6, so I could've used this in Python 3 as well. -==== - -=== A Very Big Sum - -In this challenge, you need to write the function body for `aVeryBigSum`, which -gets an array of integers, and has to return the sum of this array. Both Python -3 and Perl 6 handle the large integers transparently for you, so I was able to -use the same code as I used for the simple array sum challenge. - -[source,py3] ----- -def aVeryBigSum(ar): - sum = 0 - - for i in ar: - sum += i - - return sum ----- - -And for Perl 6 using the `[+]` reduce meta-operation. - -[source,perl6] ----- -sub a-very-big-sum (@ar) { - [+] @ar -} ----- - -=== Plus Minus - -The next challenge gives a list of numbers, and wants you to return the -fractions of its elements which are positive, negative or zero. The fractions -should be rounded down to 6 decimals. I made a counter just like in the -*Compare the Triplets* challenge, and calculated the fractions and rounded them -at the end. - -[source,py3] ----- -def plusMinus(arr): - counters = [0, 0, 0] - - for i in arr: - if (i > 0): - counters[0] += 1 - continue - - if (i < 0): - counters[1] += 1 - continue - - counters[2] += 1 - - for i in counters: - print("%.6f" % (i / len(arr))) ----- - -For the Perl 6 solution, I went for a `given/when`, `map` and the `fmt` -function to format the fractions. - -[source,perl6] ----- -sub plus-minus (@arr) { - my @counters = [0, 0, 0]; - - for @arr -> $i { - given $i { - when * > 0 { @counters[0]++ } - when * < 0 { @counters[1]++ } - default { @counters[2]++ } - } - } - - @counters.map({ $_.fmt("%.6f").say }); -} ----- - -You may notice a number of statements do not have a terminating `;` at the end. -In Perl 6, this is not needed if it's the last statement in a block (any code -surrounded by a `{` and `}`. - -The `given/when` construct is similar to a `switch/case` found in other -languages (but not Python, sadly), but uses the -https://docs.perl6.org/language/operators#index-entry-smartmatch_operator[smartmatch -operator] implicitly to check if the statements given to `when` are `True`. The -`*` is the https://docs.perl6.org/type/Whatever[Whatever operator], which in -this case will get the value of `$i`. - -Lastly, he `$_` in the `map` function is similar to inside a `for` loop, -it's the current element. Since the code given to `map` is inside a block, -there's no need for a `;` after `say` either. - -=== Staircase - -This challenge gives you an integer 𝓃, and you're tasked with "drawing" a -staircase that is 𝓃 high, and 𝓃 wide at the base. The staircase must be made -using `#` characters, and for the spacing you must use regular spaces. - -It seems that in Python, you _must_ specify the `i in` part oft the `for i in -range`. Since I don't really care for the value, I assigned it to `_`. - -[source,py3] ----- -def staircase(n): - for i in range(1, n + 1): - for _ in range(n - i): - print(" ", end="") - - for _ in range(i): - print("#", end="") - - print("") ----- - -In Perl 6, there's also a `print` function, which is like `say`, but does not -append a `\n` at the end of the string. The `for` loop in Perl 6 allows for -just a range to operate as expected. The `..` operator creates a range from the -left-hand side up to the right hand side, inclusive. - -[source,perl6] ----- -sub staircase ($n) { - for 1..$n -> $i { - print(" ") for 0..($n - $i); - print("#") for ^$i; - print("\n"); - } -} ----- - -=== Mini-Maxi Sum - -Here you will be given 5 integers, and have to calculate the minimum and -maximum values that can be calculated using only 4 of them. - -I sort the array, and iterate over the first 4 values to calculate the sum and -print it. I then do the same but sort it in reverse for the sum of the 4 -highest values. - -[source,py3] ----- -def miniMaxSum(arr): - arr.sort() - sum = 0 - - for i in range(4): - sum += arr[i] - - print(str(sum) + " ", end="") - - arr.sort(reverse=True) - sum = 0 - - for i in range(4): - sum += arr[i] - - print(str(sum)) ----- - -Perl 6 has immutable lists, so calling `sort` on them will return a new list -which has been sorted. I can call `reverse` on that list to get the highest -number at the top instead. `head` allows me to get the first 4 elements in a -functional way. You've already seen the meta-reduce operator `[+]`, which will -get me the sum of the 4 elements I got from `head`. I wrap the calculation in -parenthesis so I can call `print` on the result immediately. - -[source,perl6] ----- -sub mini-maxi-sum (@arr) { - ([+] @arr.sort.head(4)).print; - print(" "); - ([+] @arr.sort.reverse.head(4)).print; -} ----- - -=== Birthday Cake Candles - -In this challenge, you're given a list of numbers. You must find the highest -number in the list, and return how often that number occurs in the list. - -It's fairly straightforward, I keep track of the current largest value as -`size`, and a `count` that I reset whenever I find a larger value than I -currently have. - -[source,py3] ----- -def birthdayCakeCandles(ar): - size = 0 - count = 0 - - for i in ar: - if i > size: - size = i - count = 0 - - if i == size: - count += 1 - - return count ----- - -The Perl 6 variant does not differ in how it solves the problem, apart from -having a very different syntax of course. - -[source,perl6] ----- -sub birthday-cake-candles (@ar) { - my ($size, $count) = (0, 0); - - for @ar { - if ($_ > $size) { - $size = $_; - $count = 0; - } - - $count++ if $size == $_; - } - - $count; -} ----- - -[NOTE] -==== -On IRC, someone showed me a clean solution in Python 3: `return -ar.count(max(ar))`. This feels like a much cleaner solution than what I had -created. -==== - -=== Time Conversion - -This is the final challenge of this section on Hackerrank, and also this post. -You're given a timestamp in 12-hour AM/PM format, and have to convert it to a -24-hour format. - -I split the AM/PM identifier from the actual time by treating the string as a -list of characters and taking two slices, one of the last two characters, and -one of everything _but_ the last two characters. Then I split the time into -parts, and convert the first part (hours) to integers for calculations. Next I -set the hours to 0 if it's set to 12, and add 12 hours if the timestamp was -post meridiem. Finally, I convert the hours back to a string with leading -zeroes, and join all the parts together to form a timestamp again. - -[source,py3] ----- -def timeConversion(s): - meridiem = s[-2:] - hours = int(s[:2]) - rest = s[2:-2] - - if (hours > 11): - hours = 0 - - if (meridiem.lower() == "pm"): - hours += 12 - - return ("%02d:%s" % (hours, rest)) ----- - -The Perl 6 solution again doesn't differ much from the Python solution in terms -of the logic it's using to get the result. The biggest difference is that in -Perl 6, strings can't be accessed as lists, so I use the `substr` method to -extract the parts that I want. The first one starts at `*-2`, which means 2 -places before the end. The others get a -https://docs.perl6.org/type/Range[`Range`] as argument, and will get the -characters that exist in that range. - -[source,perl6] ----- -sub time-conversion ($s) { - my $meridiem = $s.substr(*-2); - my $hours = $s.substr(0..2).Int; - my $rest = $s.substr(2..*-2); - - $hours = 0 if $hours > 11; - $hours += 12 if $meridiem.lc eq "pm"; - - sprintf("%02d:%s", $hours, $rest); -} ----- - -The `.Int` method converts the `Str` object into an `Int` object, so we can -perform calculations on it. The `eq` operator checks specifically for -https://docs.perl6.org/routine/eq[__string equality__]. Since Perl 6 is a -https://en.wikipedia.org/wiki/Gradual_typing[gradually typed programming -language], there's a dedicated operator to ensure that you're checking string -equality correctly. - -== Wrap-up - -These challenges were just the warm-up challenges I was given after creating a -new account and choosing Python as a language to use. I intend to write up more -posts like this, for the near future I'll stick to Python 3 challenges since I -want to get better at that specific language for work. - -This is also the first post in which I have tried this format to show off two -languages side-by-side, and to highlight differences in how you can accomplish -certain (relatively simple) tasks with them. If you have suggestions to improve -this format, do not hesitate to contact me. I am always open for feedback, -preferably via email. You can find my contact details on the link:/[homepage]. - diff --git a/_posts/2018-10-11-hackerrank-solutions-python3-and-perl6-part-2.html b/_posts/2018-10-11-hackerrank-solutions-python3-and-perl6-part-2.html deleted file mode 100644 index 60a9a75..0000000 --- a/_posts/2018-10-11-hackerrank-solutions-python3-and-perl6-part-2.html +++ /dev/null @@ -1,706 +0,0 @@ ---- -title: "Hackerrank solutions: Python 3 and Perl 6 (part 2)" -layout: language-war -tags: Hackerrank Perl6 Python Python3 Programming Raku -description: > - A number of solutions to Hackerrank challenges in both the Python 3 and the - Perl 6 programming languages. This is the second part of the series, and will - work through the subdomain of Strings. ---- - -{% markdown %} -# Hackerrank solutions: Python 3 and Perl 6 (part 2) -{% endmarkdown %} - -{% markdown %} -As a continuation of the [previous -part](/post/2018/09/13/hackerrank-solutions-python3-and-perl6-part-1/) of this -series, I will be continuing to work through some Hackerrank challenges for -Python 3, and compare the solutions to how I would solve them in a language I'm -more proficient in, Perl 6. In this post, I will work through some of the -Python 3 string challenges from Hackerrank. - -Raiph [posted a comment on -Reddit](https://www.reddit.com/r/perl6/comments/9ffc2p/hackerrank_solutions_python_3_and_perl_6_part_1/e5xml3m) -suggesting a slightly different layout, which I will be using for this post. -Additional comments are always welcome as I try to improve the format. -{% endmarkdown %} - -{% admonition_md Disclaimer %} -Once again I'd like to make clear I'm trying to stick to the original -Hackerrank challenges by not using any imports not specifically used in the -original challenge. If you have suggestions for Python 3 or Perl 6 modules to -make a given task easier, I still appreciate them, but I won't update my -solutions to use a module. -{% endadmonition_md %} - -{% markdown %} -## Challenges -{% endmarkdown %} - -<div class="language-war"> - <div class="language-announcer"> - -{% markdown %} -### String Split and Join - -This challenge involves a string containing spaces, where the spaces are to be -replaced with dashes (`-`) instead. -{% endmarkdown %} - - </div> - <div class="language-arena"> - <div class="language-challenger"> - <div class="language-code"> - -{% highlight python3 tio=https://tio.run/##K6gsycjPM/7/PyU1TaG4ICezJD4xLyU@Kz8zTyMnMy9V04pLAQiKUktKi/IUlHSV9OBSemDlGkoKSpqa/wuKMvNKNNAMyMwrKC3R0ARKp@XnKyQlFgFxFQA %} -def split_and_join(line): - return "-".join(line.split(" ")) -{% endhighlight %} - - </div> - <div class="language-commentary"> - -{% markdown %} -I personally am not too fond that `join` takes a list of words to join -together, whereas `split` takes a word to split with. It feels a little -inconsistent. It also doesn't allow me to read the code logically from left to -right. -{% endmarkdown %} - - </div> - </div> - <div class="language-defender"> - <div class="language-code"> - -{% highlight perl6 tio=https://tio.run/##K0gtyjH7/7@4NEmhuCAns0Q3MS9FNys/M09BQyUnMy9VU6GaSwEIwBw9sBINJQUlTT2QGg0lXSVNrtr/KlqefnrFOaVFBXpqqKboFSdW/k/Lz1dISiwC4ioA %} -sub split-and-join ($line) { - $line.split(" ").join("-") -} -{% endhighlight %} - - </div> - <div class="language-commentary"> - -{% markdown %} -The Perl 6 solution to the challenge does the same as the Python variant. Even -the function names are the same! The biggest difference is that I can chain the -functions from left to right, leading to clearer code. -{% endmarkdown %} - - </div> - </div> - </div> - <div class="language-announcer"> - -{% markdown %} -### What's Your Name? - -The next challenge is a simply string formatting task. You get two inputs, a -first name and a last name, and have to put it in a string which will be -printed to `STDOUT`. -{% endmarkdown %} - - </div> - <div class="language-arena"> - <div class="language-challenger"> - <div class="language-code"> - -{% highlight python3 tio=https://tio.run/##XYzBCsIwEETv/YqxUEigePHmD@jZm6cSyRYj22ww2UK/PobqyWEOAzPz0laeEk@1epqR3iGWaVbmKbqFjBvxsOcOTXtl@isxC4bcfMBdFC/NBZ54JY@2EKQdeOwx4Pu39R8bYtJi7IhfsPWiwUu3uoib5KzLBw %} -def print_full_name(a, b): - print("Hello %s %s! You just delved into python." % (a, b)) -{% endhighlight %} - - </div> - <div class="language-commentary"> - -{% markdown %} -Before you begin, I know this can be done using `f""` strings, and that was my -first attempt to use as well. However, Hackerrank did not accept this, -complaining about invalid syntax, so I assume they're running an older Python 3 -than I do. - -That said, this is a simple `printf` formatted string, which then accepts a -tuple of arguments to put into the string. `printf` formatted string are very -powerful in their possibilities, and it's clear to read. -{% endmarkdown %} - - </div> - </div> - <div class="language-defender"> - <div class="language-code"> - -{% highlight perl6 tio=https://tio.run/##K0gtyjH7/7@4NEmhoCgzr0Q3rTQnRzcvMTdVQyVRR0ElSVOhmksBCIoTKxWUPFJzcvIVVBKB4ooKkfmlClmlxSUKKak5ZakpCkDd@QoBQPMUzPSUuGr/Y5in5emnl55aAjQVytL875NYVFTJFZ6YkwMA %} -sub print-full-name($a, $b) { - say "Hello $a $b! You just delved into Perl 6." -} -{% endhighlight %} - - </div> - <div class="language-commentary"> - -{% markdown %} -Perl 6 has double-quote semantics that many people may be familiar with from -other languages. When you insert a variable in a double-quoted string, it's -`.Str` value will be used. That is to say, the value will be converted to a -`Str` if required, and then put into the string. - -If you need it or want it for clarity, you can also use `"Hello {$a}"` in Perl -6, allowing you to use it similarly to Python 3's `f""` strings. -{% endmarkdown %} - - </div> - </div> - </div> - <div class="language-announcer"> - -{% markdown %} -### Mutations - -You are given a string _string_, an integer _position_ and a character -_character_. In _string_, replace the character at _position_ to the given -_character_. The position is counted from starting point 0, so I don't have to -think about differences between what a human or computer considers to be -position 1 in a string. -{% endmarkdown %} - - </div> - <div class="language-arena"> - <div class="language-challenger"> - <div class="language-code"> - -{% highlight python3 tio=https://tio.run/##VY5BC8IwDIXv/RVhpxaKIN4Ef4mIjNm5iCYlTQ/79bXrpmIuyeN975E468R0KOUWRnhl7TVckwrS3a7LQ@SEikwehqmXftAg7migzqITnOCJSTfc/YzzJ3ipyDdqGiBBsxB03e7BSLbxrsTaoPb/C6SY1ToPi7WJptbLlZHZ7I28AQ %} -def mutate_string(string, position, character): - chars = list(string) - chars[position] = character - - return "".join(chars) -{% endhighlight %} - - </div> - <div class="language-commentary"> - -{% markdown %} -This is basically what the example showed as well that came with the challenge, -so wasn't too hard to solve. My only complaint was that I couldn't call my list -"list", because that's a reserved keyword. -{% endmarkdown %} - - </div> - </div> - <div class="language-defender"> - <div class="language-code"> - -{% highlight perl6 tio=https://tio.run/##K0gtyjH7/7@4NEkht7QksSRVt7ikKDMvXUFDBcLQUVApyC/OLMnMzwMykzMSixKTS1KLNBWquRSAILdSwSEns7hEwVYBqkEvOT83yRosCZaJhuuPBSmCm2DNhVCjl5WfmWfNVfsfxQ0aKlqefnrpqSVAizFYmnrFiZX/0/LzuQy5igA %} -sub mutate-string ($string, $position, $character) { - my @list = $string.comb; - @list[$position] = $character; - - @list.join; -} -{% endhighlight %} - - </div> - <div class="language-commentary"> - -{% markdown %} -The Perl 6 variant does the same things as the Python variant. `comb` without -arguments converts a string to a list of characters, and `join` without -arguments joins a list together to a string. -{% endmarkdown %} - - </div> - </div> - </div> - <div class="language-announcer"> - -{% markdown %} -### Find a String - -In the following challenge you are given a string _string_, and a substring -_sub\_string_. The challenge is to find how often a substring occurs in the -_string_. The substrings may overlap one another, so the string `"ABCDCDC"` -contains the substring `"CDC"` twice. -{% endmarkdown %} - - </div> - <div class="language-arena"> - <div class="language-challenger"> - <div class="language-code"> - -{% highlight python3 tio=https://tio.run/##XU67CsMwDNz9FTfaxENKt4CHPv4ilNCHkwqKbBx76Ne7bhxIqYSQdNzp5N/x6Xif88OOuLvEcZjTbY6BeJK1aRRkqLPqBEosRBi0YllHF0AgRrjyZGWr8bK8qtUq@QaNqGBPHaGprO32Bcb8eG26zbIx2FXPYGMKXOHsCz/K//eJfYpSaayDUvlwPJ1LilIf %} -def count_substring(string, sub_string): - count = 0 - - for i in range(0, len(string)): - if string[i:i + len(sub_string)] == sub_string: - count += 1 - - return count -{% endhighlight %} - - </div> - <div class="language-commentary"> - -{% markdown %} -As solution to this challenge I loop through the entire _string_, and check -whether it contains the _sub\_string_ at that point. If it does, I increment -_count_ by 1. Now, I learned that Python also has the inline `if`, just like -Perl 6 does, however, it also *needs* an `else` block. That put me off from -using it in this situation. I think it puts me off from using it in most -situations, actually. With an `else` coming after it, it just becomes messy to -read, in my opinion. -{% endmarkdown %} - - </div> - </div> - <div class="language-defender"> - <div class="language-code"> - -{% highlight perl6 -tio=https://tio.run/##K0gtyjH7/7@4NEkhOb80r0QXyCouKcrMS1fQUIEwdBRUgIK6EI6mQjUXZ2pOam6xAlRaoa5OIdcqvwxoUGKBPpJSfa7a/2hmaqhoefrppaeWAM2EsjT1ihMr/zs6ObsAIRcQAwA %} -sub count-substring ($string, $sub-string) { - elems $string ~~ m:overlap/$sub-string/ -} -{% endhighlight %} - - </div> - <div class="language-commentary"> - -{% markdown %} -The Perl 6 version makes use of some regex magic, and the `elems` subroutine. -`elems` returns the number of elements in a list, which in this case would be -the number of matches found by the regex. The `m:overlap//` makes a regex to -*m*atch, with *overlap*ping strings. -{% endmarkdown %} - - </div> - </div> - </div> - <div class="language-announcer"> - -{% markdown %} -### String Validators - -In the following challenge, the program is given a string _s_, and have to -validate a number of properties on this string. These are, in order, whether -they contain - -- alphanumeric characters (a-z, A-Z or 0-9), -- alphabetic characters (a-z or A-Z), -- digits (0-9), -- lowercase characters (a-z), -- uppercase characters (A-Z). - -If any character in the given string passes a validation, it must print -`"True"`, otherwise it must print `"False"`. - -{% endmarkdown %} - - </div> - <div class="language-arena"> - <div class="language-challenger"> - <div class="language-code"> - -{% highlight python3 tio=https://tio.run/##hZLNioMwEIDvPsXQPdTAIrS9CR72sk@wt1JCVtMaqklIIqWUPrubn6p1MZib38x8zmQi76YW/ND37AwYc9JSjKEoYItxSxjHeJsnYI@GAhiXnUlR4kFZ0/Lq6MN/urMhDe/aTQ7fpNH0853Lmizwil2YWeCNuFG1wDspJ@7xMzRzFso2RJRtERqmTapRPpbZybgwr4aPryZPQHjlazKmPUrfSqYBp/wCflRHk7jWzfhPa1FcG/JXtOGKZlqPYtohf0Ubbnim9SimHfJXtGFBM61HMe2QH7QQvFd6d@/K7zEkZg6lCI3hTAs1vkS3fAvd7l1s@pNUjA@Kow2dUN9//ZbVbn/4@AM %} -if __name__ == '__main__': - s = input() - - checks = { - "alnum": False, - "alpha": False, - "digit": False, - "lower": False, - "upper": False - } - - for char in list(s): - if not checks["alnum"] and char.isalnum(): - checks["alnum"] = True - - if not checks["alpha"] and char.isalpha(): - checks["alpha"] = True - - if not checks["digit"] and char.isdigit(): - checks["digit"] = True - - if not checks["lower"] and char.islower(): - checks["lower"] = True - - if not checks["upper"] and char.isupper(): - checks["upper"] = True - - keys = list(checks.keys()) - keys.sort() - - for key in keys: - print(checks[key]) -{% endhighlight %} - - </div> - <div class="language-commentary"> - -{% markdown %} -As stated in the disclaimer, I don't want to make use of any `import` -statements unless these are explicitly given in the original challenges. This -means I can't use regexes, as these are stuffed away in the `re` packages in -Python. Luckily, Python has the correct check available as a method on the -string object, so I can still check them in a single line. - -I first tried to call the methods on _s_ directly, but this seemed to require -the entire string to match the check, instead of just any character in the -string. So I had to loop through the string by character, which I did. If any -character is found to validate, the appropriate key in the _checks_ dict will -be set to `True`. Once I've walked through the entire string, I sort the _keys_ -from _checks_ so I can be sure they're printed in the right order. -{% endmarkdown %} - - </div> - </div> - <div class="language-defender"> - <div class="language-code"> - -{% highlight perl6 -tio=https://tio.run/##K0gtyjH7/7@4NEnB19HTT6GaSwEIcisVVIoVbBVUtDz99NJTS6y5wMLFiZUKxfkgqbo6BX2bxJy80lw7fWvscgUZiTjkUjLTM0twyOXkl6cW4ZArLSiAyNX@/@@YlJxiaGSsDAA %} -sub MAIN { - my $s = $*IN.get; - - say so $s ~~ /<alnum>/; - say so $s ~~ /<alpha>/; - say so $s ~~ /<digit>/; - say so $s ~~ /<lower>/; - say so $s ~~ /<upper>/; -} -{% endhighlight %} - - </div> - <div class="language-commentary"> - -{% markdown %} -Perl 6 does have regexes available in the main namespace by default, so that -made this challenge a lot easier to work with. `$*IN` in a special variable -that refers to `STDIN`, and the `.slurp` method reads all remaining data from -the buffer. - -The next 5 lines all do a `say`, which acts like `print` in Python 3. The `so` -function coerces a value to become a `Bool`. When a `Bool` is given to `say`, -it will be coerced to a string representation again, and become either `"True"` -or `"False"`. The smartmatch operator `~~` has already been covered in the -previous post, so I recommend you read that as well if you haven't yet. - -In Perl 6, regexes are (usually) delimited by the `/` character. The `<alnum>`, -`<alpha>` etcetera parts are [predefined character classes][classes] in Perl 6 -regexes. These check for exactly what we need in the challenges, so were a good -pick to solve them. - -[classes]: https://docs.perl6.org/language/regexes.html#Predefined_character_classes -{% endmarkdown %} - - </div> - </div> - </div> - <div class="language-announcer"> - -{% markdown %} -### Text Wrap - -You are given a string _s_ and a width _w_. The string should be split over -multiple lines so it is never more wide than _w_. -{% endmarkdown %} - - </div> - <div class="language-arena"> - <div class="language-challenger"> - <div class="language-code"> - -{% highlight python3 tio=https://tio.run/##K6gsycjPM/7/PzO3IL@oRKEktaKkvCixgIsrJTVNAcTSKC4pysxL11HITayIL89MKcnQtOJSAIKi1JLSojwFpZg8Jb2s/Mw8DZhePRzaNP8XAIVKNMDSmXkFpSUamjoKIBEoBwj@Ozo5u7i6uXt4enn7@Pr5BwQGBYeEhoVHREZxmQIA %} -import textwrap - -def wrap(string, max_width): - return "\n".join(textwrap.wrap(string, max_width)) -{% endhighlight %} - - </div> - <div class="language-commentary"> - -{% markdown %} -This challenge introduces the first Python module: `textwrap`. This makes the -challenge very easy to solve as well, using the `wrap` function exposed by the -module. This function makes a list of strings, each no longer than the given -width. I then join these together with newlines to get the desired output. -{% endmarkdown %} - - </div> - </div> - <div class="language-defender"> - <div class="language-code"> - -{% highlight perl6 tio=https://tio.run/##K0gtyjH7/7@4NEmhvCixQEFDpbikKDMvXUdBpTwzpSRDU6GaSwEIoMJ6yfm5SRpQKb2s/Mw8DaWYPCVNrtr/IO0aKlqefnrpqSVA7VCWnmdeiaZecWLlf0cnZxdXN3cPTy9vH18//4DAoOCQ0LDwiMgoLlMA %} -sub wrap ($string, $width) { - $string.comb($width).join("\n") -} -{% endhighlight %} - - </div> - <div class="language-commentary"> - -{% markdown %} -For the Perl 6 solution, I have not used an additional module, as all the -functionality are in the core namespace. I actually made a module in Perl 6 for -a less primitive wrapping functionality, called [`String::Fold`][string::fold]. - -In this solution, I use `comb` with the `$width` argument. This returns a list -of strings, each no longer than the given width, just like Python's -`textwrap.wrap`. I can then join these together with newlines as well to get -the same result. - -[string::fold]: https://modules.perl6.org/dist/String::Fold:cpan:TYIL -{% endmarkdown %} - - </div> - </div> - </div> - <div class="language-announcer"> - -{% markdown %} -### Designer Door Mat - -This challenge is more complex than previous challenges. The task at hand is to -"draw" a certain "design" as the output. For the input, you are given both a -height _y_ and a width _x_, however _x_ must always be _y_ × 3, so you can -ignore the second argument. - -This one is much simpler to explain using two examples. The first example is -the output if the input were `7 21`. -{% endmarkdown %} - -{% highlight text %} ----------.|.--------- -------.|..|..|.------ ----.|..|..|..|..|.--- --------WELCOME------- ----.|..|..|..|..|.--- -------.|..|..|.------ ----------.|.--------- -{% endhighlight %} - -{% markdown %} -In the second example, the input is `11 33`. -{% endmarkdown %} - -{% highlight text %} ----------------.|.--------------- -------------.|..|..|.------------ ----------.|..|..|..|..|.--------- -------.|..|..|..|..|..|..|.------ ----.|..|..|..|..|..|..|..|..|.--- --------------WELCOME------------- ----.|..|..|..|..|..|..|..|..|.--- -------.|..|..|..|..|..|..|.------ ----------.|..|..|..|..|.--------- -------------.|..|..|.------------ ----------------.|.--------------- -{% endhighlight %} - - </div> - <div class="language-arena"> - <div class="language-challenger"> - <div class="language-code"> - -{% highlight python3 tio=https://tio.run/##zZDNS8MwGMbv@SseMwbJlrbrehCEnWS3DS@CBxGpNFsCJQ1p5xD832s@Nh2I7GpyCHmf3/u8H/ZjUJ2pxnFyg@LQu@JNm0Kad9gkEKKk3qsBK2gzMKaNPQyM571ttX/58@KFk6NuBuWJEzpDRVTd7s45p3CGkqPAkhMywWNnERiy6xxabaRH4Wqzl6wUUcHc83cE/pjOvDZ1r2TvLRmL@MwbRctIfKuxYOonA7tI9E3xc/mQYV1A6bQPl2IKRjPqqYQL0PwzD/8fCx@7IHgcY6ubppVxAPLbMRZKSSxoqa8idV5xkfSn9eb@Ybum4gpOUslNd5Tur92FsMBCIPvfuxvHWyzLLw %} -#! /usr/bin/env python3 - -height = int((input().split())[0]) -width = height * 3 -half = int((height - 1) / 2) - -# Top half -for line in range(1, half + 1): - non_dashes = ((line * 2) - 1) - dashes = int((width - (non_dashes * 3)) / 2) - - print("%s%s%s" % ("-" * dashes, ".|." * non_dashes, "-" * dashes)) - -# Middle line -print("%s%s%s" % ( - "-" * (int(width / 2) - 3), - "WELCOME", - "-" * (int(width / 2) - 3) -)) - -# Lower half -for line in range(half, 0, -1): - non_dashes = ((line * 2) - 1) - dashes = int((width - (non_dashes * 3)) / 2) - - print("%s%s%s" % ("-" * dashes, ".|." * non_dashes, "-" * dashes)) -{% endhighlight %} - - </div> - <div class="language-commentary"> - -{% markdown %} -I split the code up in a top half, middle line and lower half, to make it -easier to reason about. The `for` loops contain some logic to get the right -output on every line. I found out that `range` supports a third argument, -allowing me to count down with it as well, which was perfect for this -situation. -{% endmarkdown %} - - </div> - </div> - <div class="language-defender"> - <div class="language-code"> - -{% highlight perl6 tio=https://tio.run/##zZBBTsMwEEX3PsXgdpEgxVFSqSwiVqiLSi1skFiiIE@xpeBEdtpQhZyDo3CAHix0UpKGSuzxypp5/8/XL9Bm87adXEG4dTZ80SZEs4OCxoy97WGqUL@qEm5her28Fy7b2kJUuZVOKEylWJoy6bhKy1IR9iM4fMLstFFptgkGG68HAoh8OHxBnDA2gce8AALZJrcQCfFLVTM4PvIyuQlk6hS6zuqZzsQ@eSUDM9qfQgXH30hIyfzhNKlcugde84DDey9vai4@RDc4S5sLhiesofBrLWWGkGmDbOzl9QnoFqWc@c3TYnX3sF7U8DfCu0ZWeYX23Il3UYovLO7QOvzX7bTtDcTRNw %} -#! /usr/bin/env perl6 - -my $height = $*IN.slurp.words.head.Int; -my $width = $height × 3; -my $half-height = ($height - 1) ÷ 2; - -# Top half -for 1..$half-height { - my $non-dashes = ($_ × 2) - 1; - my $dashes = ($width - ($non-dashes × 3)) ÷ 2; - - say "{"-" x $dashes}{".|." x $non-dashes}{"-" x $dashes}"; -} - -# Middle line -say "{"-" x (($width ÷ 2) - 3)}WELCOME{ "-" x (($width ÷ 2) - 3)}"; - -# Lower half -for (1..$half-height).reverse { - my $non-dashes = ($_ × 2) - 1; - my $dashes = ($width - ($non-dashes × 3)) ÷ 2; - - say "{"-" x $dashes}{".|." x $non-dashes}{"-" x $dashes}"; -} -{% endhighlight %} - - </div> - <div class="language-commentary"> - -{% markdown %} -As usual, the code is functionally the same. I must admit I like the functional -style to get an `Int` from the first argument much more than the way I do it in -Python, though. - -A thing I learned is that the `..` operator that generates a sequence does not -have a way to make a sequence that counts down, so I had to use `.reverse` on a -sequence that counts up. I had expected this to Just Work as I expected and -count down if the left hand side would be larger than the right hand side. - -You may notice some fancy Unicode characters in the source, namely `×` for -multiplication, and ÷ for division. Perl 6 allows Unicode characters in the -source files, which can oftentimes lead to prettier code. In this particular -instance, there's no big difference in code readability, though. And for those -who don't yet have a modern editor that can make Unicode characters, do not -worry, as the ASCII equivalents (`*` and `/` respectively) still work as well. -{% endmarkdown %} - - </div> - </div> - </div> - <div class="language-announcer"> - -{% markdown %} -### String Formatting - -In this challenge, you are to produce a table with four columns. The columns -should contain the decimal, octal, hexadecimal and binary values of the row -numbers. The function receives an int _number_. The table should contain that -many rows, starting with row number 1. -{% endmarkdown %} - - </div> - <div class="language-arena"> - <div class="language-challenger"> - <div class="language-code"> - -{% highlight python3 tio=https://tio.run/##jZFNCoMwEIX3nmIQhIQG0XYn9CwSTawpJoYYqVJ6dht/W7uQDlll3vveTKJ7W9bqMgyMF6CNUDYtaiOptZwh1cqMG5x44ErSLn0IZku4QsUV8p9Rkr38cJavWuxNYncJAoQCQ9WNo5jA3IcTxAtvLMZzIWnliI72YQkcmnvbWLRl4s1S53Y1JPV/lpJ3dJeUdDtjqzU36ACQCUVNv3iz49DNND0m8oMG1uNDAGgZhMyLkO/hyBKE8fD7EyNKKN1ahF33HL0B %} -def print_formatted(number): - max_width = len("{0:b}".format(number)) - - for i in range(1, number + 1): - decimal = "{0}".format(i).rjust(max_width) - octal = "{0:o}".format(i).rjust(max_width) - hexadecimal = "{0:x}".format(i).upper().rjust(max_width) - binary = "{0:b}".format(i).rjust(max_width) - - print("%s %s %s %s" % (decimal, octal, hexadecimal, binary)) -{% endhighlight %} - - </div> - <div class="language-commentary"> - -{% markdown %} -In the Python 3 solution I first calculate the max width I need to take into -account. Then I loop from 1 until _number_ to get the right amount of rows. -Each iteration, I format the number correctly, and then print it out using a -printf format string. - -The hardest part of this challenge was to get formatting right the way -Hackerrank wanted it. But I guess that was the entire point of the challenge. -{% endmarkdown %} - - </div> - </div> - <div class="language-defender"> - <div class="language-code"> - -{% highlight perl6 tio=https://tio.run/##XY9NCsIwEIX3PcUQIiRiB1tEBPEA3bhxK0iq0VZMWpIUK1KvXmNb/HurGWbeN29KaS7ztrVVCqXJtQuPhVHCOXkARnWlUmk43APwUjegStThNT@4DFYwjDEVVrKY48YZ3GfC2OV7vYeF1nn0yVsYGd0/jMYSqGuYcTwXuWYECIcHkK0my6BDeDtEiMOhIcZLv2Dskh8Z3U2A7vo8C/6po/lXE3Pe52uCpv17mdFxssaTdJhox9t4@gQ %} -sub print-formatted ($number) { - my $max-width = $number.base(2).Str.chars; - my $format-string = ("%{$max-width}s" xx 4).join(" ") ~ "\n"; - - for 1..$number { - $format-string.printf($_, $_.base(8), $_.base(16), $_.base(2)); - } -} -{% endhighlight %} - - </div> - <div class="language-commentary"> - -{% markdown %} -The Perl 6 solution starts of the same, in that it first calculates the max -width I need to take into account. Next, however, I generate the format string -using the `$max-width` to make the `printf` subroutine pad it for me. The `xx` -operator makes a total of 4 such strings, and puts them into a list, which I -can then `join` together with a space character, and add a `\n` at the end of -it (the `~` operator is for string concatenation). - -I'm assuming something similar is possible in Python 3 as well, and I would -like to have an example so I can compare it more fairly. - -In the Perl 6 solution I am also able to make use of the `base` method to -convert the numbers into the right base, something I could not find for Python -3. -{% endmarkdown %} - - </div> - </div> - </div> -</div> - -{% markdown %} -## Wrap-up - -This time I did not do all of the challenges, as the post would probably get -too long. I still did 8 of them, and might do the rest of the string challenges -in a later part anyway. - -I still find Perl 6 to produce much cleaner code, which is shown best with the -first challenge. In Perl 6 (`$line.split(" ").join("-")`), I can read from left -to right to see what I'm doing: I have a `$line`, which I split, and then join. -In the Python variant (`"-".join(line.split(" "))`), it is much less clear what -the actual item I'm working on is, as it's hidden inbetween the `join` and -`split` calls. - -Of course, I'm still not an expert on Python 3 code, so I'm sure that there are -many parts that could be written in a cleaner fashion. I'm still open for -feedback to improve my Python 3 skills (hence I'm publishing these posts), so -please let me know if you know better ways to solve some challenges. -{% endmarkdown %} diff --git a/_posts/2019-02-03-how-to-sign-pgp-keys.html b/_posts/2019-02-03-how-to-sign-pgp-keys.html deleted file mode 100644 index a8a9618..0000000 --- a/_posts/2019-02-03-how-to-sign-pgp-keys.html +++ /dev/null @@ -1,165 +0,0 @@ ---- -title: How to sign PGP keys -layout: post -tags: PGP Tutorial -description: > - A small tutorial on how to sign others PGP keys. ---- - -{% markdown %} -Having attended [FOSDEM](https://fosdem.org/2019/) last weekend, I have been -asked to help some people out with signing PGP keys. As it is an international -gathering of users and developers of all levels of expertise, it's a great event -to get your key out in to the wild. While helping people out, I figured it might -be even easier next time around to just refer to a small tutorial on my blog -instead. -{% endmarkdown %} - -{% markdown %} -## Creating a PGP key - -The first step to sign keys, is to have a PGP key. If you already have one, -you're good to go to the next part of this tutorial. If you don't, you can check -out the `gpg` manual on how to create a key, or read about key creation in my -[article on using PGP with a Yubikey][yubikey-pgp-article]. While I would -strongly suggest reading at least some material, `gpg` does quite a good job of -guiding you through the process without prior knowledge, so you can just get -started with `gpg --generate-key` as well. - -[yubikey-pgp-article]: {{ "/post/2018/09/04/setting-up-pgp-with-a-yubikey/#creating-pgp-keys" | prepend: site.baseurl | prepend: site.url }} -{% endmarkdown %} - -{% markdown %} -## Create key slips - -A *key slip* is a small piece of paper containing some basic information about -the PGP key. They're exchanged when people meet, so they don't have to -immediately sign the key, but can do it safely at home. When you're signing in a -group, this may be faster to work with. Another benefit is that some people -don't have their private keys with them. They can then just collect the key slips -from the people who's key they want to sign, and sign it whenever they are in -possession of their private key again. - -A key slip doesn't have to contain much. A key ID, fingerprint, email address and -a name is plenty. For reference, my key slips look as follows: -{% endmarkdown %} - -{% highlight text %} -Patrick Spek <p.spek@tyil.nl> rsa4096/0x7A6AC285E2D98827 - 1660 F6A2 DFA7 5347 322A 4DC0 7A6A C285 E2D9 8827 -{% endhighlight %} - -{% markdown %} -## Verifying the owner - -Before you sign anyone's public key, you should verify that the person is -actually who they say they are. You can easily do this by asking for government -issued identification, such as an ID card, driver's license or passport. What -constitutes good proof is up to you, but in general people expect at least one -form of government issued identification. - -If the person can't verify who they are, you should *not* sign their key! -{% endmarkdown %} - -{% markdown %} -## Retrieving their key - -Once you have verified the person is who they say they are, and you have -received their key slip containing their key ID, you can look up their key -online. You can let `gpg` do all the work for you in searching and downloading -the key, using the `--search` switch. For instance, to retrieve my key, do the -following: -{% endmarkdown %} - -{% highlight sh %} -gpg --search-keys 0x7A6AC285E2D98827 -{% endhighlight %} - -{% markdown %} -If a result has been found, you are prompted to enter the numbers of the keys -you want to download. Make sure you download the right key, in case multiple -have been found! - -After retrieving the key, you can see it in the list of all the keys `gpg` knows -about using `gpg --list-keys`. -{% endmarkdown %} - -{% markdown %} -## Signing their key - -To actually sign their key, and show that you trust that the key belongs to the -person's name attached to it, you can use `gpg --sign-key`: -{% endmarkdown %} - -{% highlight sh %} -gpg --sign-key 0x7A6AC285E2D98827 -{% endhighlight %} - -{% markdown %} -You will be prompted whether you are sure you want to sign. You should answer -this with a single `y` to continue. - -After signing it, you'll have signed a PGP key! You can verify this by looking -at the signatures on a given key with `--list-sigs 0x7A6AC285E2D98827`. This should -contain your name and key ID. -{% endmarkdown %} - -{% markdown %} -## Exchanging the signed key - -While you could publish the updated public key with your signature on it, you -should **not** do this! You should encrypt the updated public key and send it to -the person that owns the private key, and they should upload it themselves. One -reason for this is that it allows you to safely verify that they do in fact -actually own the private key as well, without ever asking them explicitly to -show you their private key. - -To export the public key, use `--export`: -{% endmarkdown %} - -{% highlight sh %} -gpg --armor --export 0x7A6AC285E2D98827 > pubkey-tyil.asc -{% endhighlight %} - -{% markdown %} -The `--armor` option is used to export the key as base64, instead of binary -data. - -You can attach this file to an email, and let your email client encrypt the -entire email and all attachments for they key ID. How you can do this depends on -your email client, so you should research how to do this properly in the -documentation for it. - -However, it's also possible to encrypt the public key file before adding it as -an attachment, in case you don't know how to let your email client do it (or if -you don't trust your email client to do it right). - -You can use the `--encrypt` option for this, and add a `--recipient` to encrypt -it for a specific key. -{% endmarkdown %} - -{% highlight sh %} -gpg --encrypt --recipient 0x7A6AC285E2D98827 < pubkey-tyil.asc > pubkey-tyil.pgp -{% endhighlight %} - -{% markdown %} -Now you can use this encrypted key file and share it with the owner of the key. -If the person you send it to really is the owner of the key, they can use the -private key to decrypt the file, import it with `gpg --import` and then publish -it with `gpg --send-keys` -{% endmarkdown %} - -{% markdown %} -## Winding down - -Once all this is done, other people should have sent you your signed pubkey as -well, and you should have published your updated key with the new signatures. -Now you can start using PGP signatures and encryption for your communication -with the world. People who have not signed your key can see that there's other -people that do trust your key, and they can use that information to deduce that -whatever's signed with your key really came from you, and that anything they -encrypt with your public key can only be read by you. - -With this [trust](https://en.wikipedia.org/wiki/Web_of_trust), you can make -communication and data exchange in general more secure. -{% endmarkdown %} diff --git a/_posts/2019-04-11-perl6-nightly-docker-images.html b/_posts/2019-04-11-perl6-nightly-docker-images.html deleted file mode 100644 index 78c868e..0000000 --- a/_posts/2019-04-11-perl6-nightly-docker-images.html +++ /dev/null @@ -1,148 +0,0 @@ ---- -title: Perl 6 nightly Docker images -layout: post -tags: Perl6 Docker Raku -description: > - An overview of my work on nightly Perl 6 Docker images, and a few examples - on how I'm using them. ---- - -{% markdown %} -Due to the slow release of Rakudo Star (which actually did release a new -version last month), I had set out to make Docker images for personal use based -on the regular Perl 6 releases. But, as I discovered some [memory related -issues](https://github.com/rakudo/rakudo/issues/1501), and [another branch with -some possible fixes](https://github.com/MoarVM/MoarVM/pull/1072), I changed my -mind to make them nightlies based on the `master` branches of all related -projects instead. This way I could get fixes faster, and help testing when -needed. -{% endmarkdown %} - -{% markdown %} -These nightlies are now up and running, available on [Docker -Hub](https://hub.docker.com/r/tyil/perl6) for anyone to use! You can also find -[the Dockerfiles I'm using on git.tyil.nl](https://git.tyil.nl/docker/perl6), -in case you're interested or have suggestions to further improve the process. -{% endmarkdown %} - -{% markdown %} -The timing of the (public) release of these images could have been better, -though. About two weeks ago, other nightlies were released as well, by Tony -O'Dell, as has been noted in the [Perl 6 Weekly -post](https://p6weekly.wordpress.com/2019/03/25/2019-12-cool-truck/). While I -greatly appreciate his efforts, I was not going to just abandon all the work -I've put into my images. Instead I've tried to make smaller images, and provide -different bases than him. Maybe we can eventually learn from each other's images -and improve Docker support for the entire community together. -{% endmarkdown %} - -{% markdown %} -The easiest thing to work on was providing different bases. For now, this means -I have images with the following four base images: - -- Alpine -- Debian -- Ubuntu -- Voidlinux - -This way, people can have more options with regards to using the distribution -tooling that they're more comfortable with. One could also opt to use a more -familiar or better supported base image for development and testing out their -module, and use a smaller image for production releases. -{% endmarkdown %} - -{% markdown %} -As to the size of the images, Tony's `tonyodell/rakudo-nightly:latest` is about -1.42GB at the time of writing this post. My images range from 43.6MB -(`alpine-latest`) to 165MB (`voidlinux-latest`). Though this is not a -completely fair comparison, as my images have stripped out a lot of the tooling -used (and often required) to build some Perl 6 modules, making them unusable in -their default shape for many projects. -{% endmarkdown %} - -{% markdown %} -To remedy this particular issue, I've also created *-dev* images. These images -come with a number of additional packages installed to allow `zef` to do its -work to get dependencies installed without requiring end-users to search for -those packages. This should reduce complexity when using the images for -end-users. If we take the dev images into account when comparing sizes, my -images range from 256MB (`alpine-dev-latest`) to 1.27GB -(`voidlinux-dev-latest`). That's much closer to the `rakudo-nightly` image. -{% endmarkdown %} - -{% markdown %} -If you're interested in trying these images out, you may be interested in the -way I'm using these images myself as reference. Currently, my [CPAN upload -notifier bot](https://git.tyil.nl/perl6/app-cpan-uploadannouncer-irc) is using -these nightly images in its -[`Dockerfile`](https://git.tyil.nl/perl6/app-cpan-uploadannouncer-irc/src/branch/master/Dockerfile). -{% endmarkdown %} - -{% highlight dockerfile %} -FROM tyil/perl6:debian-dev-latest as install - -RUN apt update && apt install -y libssl-dev uuid-dev - -COPY META6.json META6.json - -RUN zef install --deps-only --/test . -{% endhighlight %} - -{% markdown %} -As you can see from the `Dockerfile`, I start out by using a `-dev` image, and -name that stage `install`. I'm still contemplating to include `libssl-dev` into -the `-dev` images, as it seems to pop up a lot, but for now, it's not part of -the `-dev` images, so I install it manually. Same goes for `uuid-dev`. Then I -copy in the `META6.json`, and instruct `zef` to install all the dependencies -required. -{% endmarkdown %} - -{% highlight dockerfile %} -FROM tyil/perl6:debian-latest - -ENV PERL6LIB=lib - -WORKDIR /app - -RUN mkdir -p /usr/share/man/man1 -RUN mkdir -p /usr/share/man/man7 -RUN apt update && apt install -y libssl-dev postgresql-client - -COPY bin bin -COPY lib lib -COPY --from=install /usr/local /usr/local - -RUN mkdir -p /var/docker/meta -RUN date "+%FT%TZ" > /var/docker/meta/build-date - -CMD [ "perl6", "bin/bot" ] -{% endhighlight %} - -{% markdown %} -Then I start a new stage. I set the `$PERL6LIB` environment variable so I don't -have to use `-Ilib` at the end, and set a `WORKDIR` to have a clean directory -to work in. Next, I set up the *runtime dependencies* of the application. -{% endmarkdown %} - -{% markdown %} -I then continue to copy in the `bin` and `lib` directories, containing the -application itself, and copy over `/usr/local` from the `install` stage. -`/usr/local` is where Perl 6 is installed, and `zef` installs all its -dependencies into. This way, the `-dev` image can be used for building all the -dependencies as needed, and only the finished dependencies end up in the final -image that's going to run in production. -{% endmarkdown %} - -{% markdown %} -Lastly, I set the build date and time of the image in a file, so the -application can refer to it later on. It is displayed when the IRC bot replies -to a `.bots` command, so I can verify that the running bot is the one I just -built. And finally, the `CMD` instruction runs the application. -{% endmarkdown %} - -{% markdown %} -I hope this displays how the images can be used for your applications, and the -reasoning as to why I made them the way they are. If you have any suggestions -or issues, feel free to contact me in whatever way suits you best. You can find -some contact details on the homepage of my blog. -{% endmarkdown %} diff --git a/_posts/2019-07-22-the-powerful-tooling-of-gentoo.html b/_posts/2019-07-22-the-powerful-tooling-of-gentoo.html deleted file mode 100644 index aa90e80..0000000 --- a/_posts/2019-07-22-the-powerful-tooling-of-gentoo.html +++ /dev/null @@ -1,189 +0,0 @@ ---- -title: The Power(ful Tooling) of Gentoo -layout: post -tags: Gentoo -description: > - Why do people (like me) love Gentoo so much? Part of it is stability and - configurability, but there is also an amazing set of tooling available to - make your life administering your machines much more enjoyable. ---- - -{% markdown %} -People often ask me for my reasons to use [Gentoo](https://gentoo.org/). Many -perceive it as a "hard" distro that takes a lot of time. While it does come -with a learning curve, I don't perceive it as particularly "hard", as the -documentation is very thorough and the community is very helpful. And the -tooling you get to maintain your system is far beyond what I've come across -with any other GNU+Linux distribution. -{% endmarkdown %} - -{% markdown %} -This blog post will highlight some of the key features I love about Gentoo. -There are certainly many more perks that I don't (yet) use, so please feel free -to inform me of other cool things that I missed. -{% endmarkdown %} - -{% markdown %} -## Configurability - -One of the main reasons for preferring Gentoo is due to the ease of configuring -it to work just the way you want. - -A great example for this would be with `init` choices. Many distributions only -support [systemd](https://en.wikipedia.org/wiki/Systemd) these days. As I'm not -a big fan of this particular system, I want to change this. But even asking a -question about this will get you a lot of hatred in most distribution -communities. In Gentoo, however, changing init is supported and well -documented, allowing you to pick from a range of possible inits. - -### `USE` flags - -One of the core concepts of Gentoo are the [`USE` -flags](https://wiki.gentoo.org/wiki/USE_flag). These allow you to easily alter -the software you're compiling to use the features you want. They can also be -used to indicate which library you would like to use to make use of a certain -feature, if there are multiple implementations available. - -### `make.conf` - -Like most distros that work with self-compiled packages, Gentoo has a -`make.conf` file available to specify some default arguments in to use while -compiling. Unlike most other distros, Gentoo's `make.conf` also allows for some -configuration of the `emerge` utility. - -For instance, I use my `make.conf` to ensure `emerge` always asks for -confirmation before performing actions. I also ensure that the build system, -`portage`, is heavily sandboxed when building packages. - -Additionally, like all configuration files in `/etc/portage`, it can be made -into a directory. In this case, all files in the directory will be loaded in -alphabetical order. This allows for easier management using tools like -[Ansible](https://www.ansible.com/). - -### Ease of patching - -Another feature I find very useful of Gentoo, is the ease of applying my own -patches to software. If you have a custom patch for a package that you want to -be applied, all you have to do is drop it in a directory in -`/etc/portage/patches`. The directory is should be in is the same as the -package's name the patch is intended for. For instance, I have the following -patch in `/etc/portage/patches/www-client/firefox`: -{% endmarkdown %} - -{% highlight diff %} -diff --git a/browser/extensions/moz.build b/browser/extensions/moz.build -index 6357998..c5272a2 100644 ---- a/browser/extensions/moz.build -+++ b/browser/extensions/moz.build -@@ -5,15 +5,10 @@ - # file, You can obtain one at http://mozilla.org/MPL/2.0/. - - DIRS += [ -- 'activity-stream', - 'aushelper', - 'followonsearch', - 'formautofill', - 'jaws-esr', -- 'onboarding', -- 'pdfjs', -- 'pocket', -- 'screenshots', - 'webcompat', - ] -{% endhighlight %} - -{% markdown %} -Whenever a new Firefox is released and built, this patch will be applied on it -to remove some of the features I dislike. - -## Ebuilds and overlays - -In Gentoo vocabulary, `ebuild` files are the files that describe how a package -is to be built, which `USE` flags it supports and everything else relating to a -package. An overlay is a repository of ebuild files. Everyone can make their -own, and easily add 5 lines in their `repos.conf` to use it. In most cases, -they're just git repositories. - -The documentation on everything around ebuilds is superb, in my experience, -especially compared to other distros. It is incredibly easy to get started -with, since it's made to be usable with very little effort. While being simple, -it's also very flexible: All default behaviours can be overwritten if needed to -get a package to build. - -## Binary packages - -Yes, you read that right. [Binary -packages](https://wiki.gentoo.org/wiki/Binary_package_guide)! Contrary to -popular belief, Gentoo *does* support this. You can instruct `emerge` to build -binary packages of all the packages it compiles, which can then be re-used on -other systems. It does need to be compiled in such a way that the other machine -can use it, of course. You can't simply exchange the packages of an x64 machine -with and ARM machine, for instance. You can set up a [cross build -environment](https://wiki.gentoo.org/wiki/Cross_build_environment) to get that -particular usecase going, though. - -If you want to easily share the binary packages you build with one machine, you -can set up a -[binhost](https://wiki.gentoo.org/wiki/Binary_package_guide#Setting_up_a_binary_package_host), -and have `emerge` pull the binary packages on the other systems as needed using -`--usepkg`. There actually is a [binhost provided by Gentoo -itself](http://packages.gentooexperimental.org/), but it seems to only contain -important packages used to restore systems into a working state. - -## Tooling - -Some of the core tooling available to any Gentoo user has already been talked -about. But there's some additional tooling you can install to make your life -even better. - -### `genkernel` - -One of the hardest tasks to newcomers to Gentoo is often to compile a kernel. -Of course, Gentoo has an answer for this, `genkernel`. The defaults `genkernel` -will give you are reasonably sane if you just want to have a kernel that works. -Of course, you can still edit the kernelconfig before compilation starts. It -will also build an `initramfs` when requested, that goes along with the kernel. -When things have been made, the kernel and initramfs will be moved to `/boot`, -and a copy of the working kernelconfig is saved to `/etc/kernels`. All you need -to remember is to update your preferred bootloader's configuration to include -your new kernel. - -### `eix` - -[`eix`](https://wiki.gentoo.org/wiki/Eix) is a utility most Gentoo users use to -update the Portage repositories and search for available packages. The -interface is considered more convenient, and it's a bit faster at getting your -results. - -To get a quick overview of which packages are in need of updates, you can run -`eix -uc` (*u*pdates, *c*ompact). To sync the Portage tree and all overlays, -`eix-sync` is the way to go. This will ensure the cache used by `eix` also gets -updated. - -In addition to having a cleaner interface and being faster, it also comes with -additional tools for keeping your system sane. The most notable to me is -`eix-test-obsolete`. - -This utility will report any installed packages that are no longer provided by -any repository (orphaned packages). It will also report all configuration lines -that affect such packages. This is really valuable in keeping your -configuration maintainable. - -### `glsa-check` - -The `glsa-check` utility is part of the `app-portage/gentoolkit` package. When -ran, it will produce a list of all packages which have known vulnerabilities. -It will use the [GLSA database](https://security.gentoo.org/glsa) for the list -of known vulnerabilities. This can be much easier than subscribing to a mailing -list and having to check every mail to see if a vulnerability affects you. - -### `qlop` - -`qlop` is another utility that comes with `app-portage/gentoolkit`. This -program parses the logs from `emerge` to give provide you with some -information. I use this mostly to see compile times of certain packages using -`qlop -Htvg <package-name>`. Using this, I can more easily deduce if I want my -desktop (with a stronger CPU) to compile a certain package, or if it'll be -faster to just compile it on my laptop. - -{% endmarkdown %} diff --git a/_posts/2019-08-10-the-soc-controversy.html b/_posts/2019-08-10-the-soc-controversy.html deleted file mode 100644 index b18cad2..0000000 --- a/_posts/2019-08-10-the-soc-controversy.html +++ /dev/null @@ -1,121 +0,0 @@ ---- -title: The SoC Controversy -layout: post -tags: Perl6 Conference CodeOfConduct Raku -description: > - For a while now, there's been a controversy about Standards or Codes of - Conduct. This has also sprung up in the context of PerlCon. I'd like to - address the issue for myself. ---- - -{% admonition_md Disclaimer %} -Please keep in mind that the opinion shared in this blog post is mine and mine -alone. I do not speak for any other members of the PerlCon organization team. -Please do not address anyone but me for the positions held in this post. -{% endadmonition_md %} - -{% markdown %} -Those that know me are probably aware that I generally dislike to make -political posts on my personal blog. I'd rather stick to technological -arguments, as there's less problems to be found with regards to personal -feelings and all that. However, as I'm growing older (and hopefully more -mature), I find it harder to keep politics out of my life as I interact with -online communities. This becomes especially true as I plan to assist with -organizing [PerlCon -2020](https://wiki.perlcon.eu/doku.php/proposals/2020/amsterdam). -{% endmarkdown %} - -{% markdown %} -PerlCon 2019 ended yesterday, and I had a lot of fun. I'd like to thank the -organizer, Andrew Shitov, once more for doing an amazing job. Especially so, as -he has been harassed for weeks, for trying to organize the conference. The -reason behind the harassment was partly due to his decision to not have an SoC, -or "Standards of Conduct", for PerlCon 2019. -{% endmarkdown %} - -{% markdown %} -During his final announcements at the end of the conference, he noted that this -is still happening, even in person at the conference itself. This toxic -behavior towards him has made him decide to no longer involve himself in -organizing a conference for the Perl community. I personally think this is a -loss for everyone involved in the community, and one that was completely -avoidable by having humane discussion instead of going for Twitter harassment. -{% endmarkdown %} - -{% markdown %} -For what it's worth, I think Twitter is also the worst possible place on the -Internet for any reasonable discussion, as it puts a very low limit on the -amount of characters you are allowed to spend on a single post. This makes it -downright impossible for any discussion, and seems to always lead to petty -name-calling. This is one of the reasons why [I'm instead using a Pleroma -instance](https://soc.fglt.nl/main/public) for my social media presence on the -Internet. If anyone is on the Internet with the intent of having interesting -discussion, I'd highly recommend to use some entrance into the Fediverse. The -instance I'm using is open for sign-ups! -{% endmarkdown %} - -{% markdown %} -But I digress. The SoC controversy is what made me want to write this blog -post. I wonder why this even is a controversy. Why do people think it is -impossible to co-exist without some document describing explicitly what is and -is not allowed? I would hope that we're all adults, and can respect one another -as such. -{% endmarkdown %} - -{% markdown %} -I wonder, was there any certain event at PerlCon 2019 that would've been -avoided if there *was* a SoC provided? I certainly did not, at any point, feel -that people were being harmful to one another, but maybe I'm just blind to it. -If anyone has concrete examples of events that happened during PerlCon 2019 -that a SoC could've prevented, I would be genuinely interested in hearing about -them. If I am to assist in organizing PerlCon 2020, and I want to be able to -present a good argument on the SoC discussion, I'll need concrete examples of -real problems that have occurred. -{% endmarkdown %} - -{% markdown %} -Of course, I also consider the opposite of this discussion. Can the SoC be used -to *cause* harm, in stead of deter it? For this, I actually have clear -evidence, and the answer is a resounding **yes**. The harassment brought upon -Andrew was originally caused by an event that transpired at The Perl Conference -in Pittsburgh (2019). A video was removed, and a speaker harassed, for -dead-naming someone. Until that event, I wasn't even aware of the term, but -apparently it's grounds for removal of your presentation from the conference -archives. -{% endmarkdown %} - -{% markdown %} -A similar event happened with The Perl Conference in Glasgow (2018), where a -talk was also removed from the archives for a supposedly offensive joke that -was made. This also sparked a heavy discussion on IRC back then, with people -from all sides pitching in with their opinion. -{% endmarkdown %} - -{% markdown %} -From my perspective, the people shouting the loudest in these discussions -aren't interested in making the world a better place where we can live in -harmony, but to punish the offender for their behavior. I don't think we -should strive towards punishment, but towards understanding, if anything. Just -being angry, shouting at people (either in real life, or over the Internet) -isn't going to solve any underlying problem. It is more likely to cause more -issues in the long run, where people will just be more divided, and will want -to get continuous revenge upon the other side. -{% endmarkdown %} - -{% markdown %} -Additionally, I think that the existence of an SoC or likewise document is a -sign towards outsiders that your community can't behave itself maturely. They -need special rules laid out to them, after all. Like most rules, they are -codified because issues have arisen in the past, and keep on arising. I don't -think the Perl community is too immature to behave itself. I trust in the good -faith of people, and to me it feels like a SoC does the exact opposite. -{% endmarkdown %} - -{% markdown %} -I hope this blog post does it's job to invite you kindly to share your opinions -with me, either on [IRC, email or on the Fediverse]({{ -"#communication-channels" | prepend: site.baseurl | prepend: site.url }}). I'd -gladly start a discussion on the positive and negative effects the SoC has, and the problems -it solves and creates. I think a civil discussion is in order here, to best -prepare us for PerlCon 2020. -{% endmarkdown %} diff --git a/_posts/2019-10-17-getting-thigs-done-with-app-gtd.html b/_posts/2019-10-17-getting-thigs-done-with-app-gtd.html deleted file mode 100644 index 2f67be0..0000000 --- a/_posts/2019-10-17-getting-thigs-done-with-app-gtd.html +++ /dev/null @@ -1,194 +0,0 @@ ---- -title: Getting Things Done with App::GTD -layout: post -tags: Perl6 Raku GettingThingsDone -description: > - My $day-job has introduced me to this concept of "Getting Things Done", and I - have been convinced to give it a shot. However, I could not find a good free - software program to assist me with following this lifestyle. Thus I brought - App::GTD to the world. ---- - -{% markdown %} -A couple months ago, I was given a workshop at work about "getting things -done". There I was told that there exists a concept called "[Getting Things -Done](https://en.wikipedia.org/wiki/Getting_Things_Done)", or "GTD" for short, -to help you, well, get things done. A number of web-based tools were introduced -to assist us with following the rules laid out in the concept. -{% endmarkdown %} - -{% markdown %} -## The problem - -The tools that were introduced did their job, and looked reasonably shiny. -However, most required a constant Internet connection. I like my tools to be -available offline, and optionally synced together. There was one local -application and a couple cloud-synced applications that I found, so this -problem could've been resolved. However, my other problem with all these -programs was that they're all proprietary. Those who've read more of my blog -may have realized by now that I strongly prefer free software whenever -possible. -{% endmarkdown %} - -{% markdown %} -Being unable to find any free software programs to fulfill my needs, I took a -look at the features I would need, and tried to adapt other programs to fit -those particular needs. I quickly learned that it's inconvenient at best to try -and mold generic task keeping programs into the specifics of GTD. But, it did -give me a reasonable idea of what features I needed for basic usage. It -occurred to me that it shouldn't be terribly hard to just write something of my -own. So I did. -{% endmarkdown %} - -{% markdown %} -## The solution, `App::GTD` - -Introducing [`App::GTD`](https://gitlab.com/tyil/raku-app-gtd), a brand new -project written in the [Raku programming language](https://raku.org/). While -still in its early phases, it seems to be usable on a day-to-day basis for me -and another colleague. In its bare basics, it's just another to-do list, but -the commands it gives you incorporate the concepts of GTD. There's an inbox -that you fill up through the day, a list of next items to work on, and projects -to structure larger tasks in. -{% endmarkdown %} - -{% admonition_md Note %} -The Raku programming language used to be called the Perl 6 programming -language. They function the same, but the name was changed for various reasons -I will not get into here. -{% endadmonition_md %} - -{% markdown %} -This program can be installed using `zef`, though I'm planning an `ebuild` for -Gentoo (and derivatives) too. Once installed, you can use `gtd` from your -shell. Doing so without arguments will show the usage information. The most -important will be `gtd add`, `gtd next` and `gtd done`. Most of these commands -require an `id` argument. The IDs required are displayed in front of the items -when listing them with commands like `inbox` or `next`. -{% endmarkdown %} - -{% markdown %} -## Daily life with `gtd` - -Once you have `gtd` installed, you don't *need* to do any configuration, as the -defaults should work fine for most people. This means you can start using it -immediately if you want to try it out! -{% endmarkdown %} - -{% markdown %} -The most common invocation will be with the `add` sub-command. Whenever -something pops up that needs doing, you add it to your inbox using it. -{% endmarkdown %} - -{% highlight sh %} -gtd add Buy eggs -gtd add "update cpan-raku's help command" -{% endhighlight %} - -{% markdown %} -These items go to your inbox, and don't need to be long, so long as *you* -understand what you meant by it. You can see that you also don't need to use -quotes around the item you want to add. All arguments after `add` will be -joined together as a string again, but some shells may perform their magic on -certain things. This is why I quoted the second call, but not the first. -{% endmarkdown %} - -{% markdown %} -All these things that you write down like this need to be sorted out at some -point. I do this every day in the morning, before I get to my regular tasks at -work. To get started, I want to see an overview of your inbox, for which the -`inbox` sub-command is intended. Running it will give you a list of all the -items in your inbox, including their ID and the date they were added. -{% endmarkdown %} - -{% highlight text %} -$ gtd inbox -[1] Buy eggs (2019-10-17) -[2] update cpan-raku's help command (2019-10-17) -{% endhighlight %} - -{% markdown %} -Now I can go through the list, and decide which actions I should undertake -specifically. These are called "next items", and the sub-command is named -`next`. Called without arguments it will give you an overview of your next -items, but when given an ID it will move an inbox item to your list of next -items. You can optionally also specify a new name for the item, to be more -clear about what needs doing. -{% endmarkdown %} - -{% highlight text %} -$ gtd next -You're all out of Next actions! - -$ gtd next 1 -"Buy eggs" has been added as a Next item. - -$ gtd next 2 "Add usage and repo info to cpan-raku, whenever it's messaged with 'help'" -"Add usage and repo info to cpan-raku, whenever it's messaged with 'help'" has -been added as a Next item. -{% endhighlight %} - -{% markdown %} -You can now see that your inbox is empty when using `inbox`, and see a list of -the next items you created with `next`. -{% endmarkdown %} - -{% highlight text %} -$ gtd inbox -Your inbox is empty! - -$ gtd next -[1] Buy eggs (2019-10-17) -[2] Add usage and repo info to cpan-raku, whenever it's messaged with 'help' (2019-10-17) -{% endhighlight %} - -{% markdown %} -Now all that's left is to do all the things you've created next items for. When -done, you can remove the entry from your next items using `done`. This command -also works on items in your inbox, so small tasks that require no next item(s) -can be marked as done immediately. -{% endmarkdown %} - -{% highlight text %} -$ gtd done 1 -"Buy eggs" has been removed from your list. - -$ gtd done 2 -"Add usage and repo info to cpan-raku, whenever it's messaged with 'help'" has -been removed from your list. - -$ gtd next -You're all out of Next actions! -{% endhighlight %} - -{% markdown %} -## Future plans - -The basics are here, but there are some things I'd very much like to add. First -and foremost, I want to be have a context to add to items, and a single context -the program operates in. This way, I can more clearly separate work and -personal tasks, which now just share one global context. - -Additionally, I've read about a new YouTube tutorial about using `ncurses` in -Raku, which I hope can guide me through making an `ncurses` application for -this as well. Perhaps I can find time to make a `GTK` application out of it as -well. - -I've already mentioned wanting to create a Gentoo `ebuild` for the application, -but this will require packaging all the module dependencies as well. This comes -with a number of hurdles that I'm trying to iron out before starting on this -endeavor. If you are on Gentoo (or a derivative) and want to assist in any way, -please contact me. - -Another thing I've taken into account when structuring the application is the -possibility for other data back-end. `gtd` is currently storing it's -information in `JSON` files in a filesystem directory, which comes with various -drawbacks. It may be beneficial to also support databases such as SQLite or -PostgreSQL. However, this is not a high priority for me right now, as it would -slow down the speed at which I can make improvements to the general program. - -I hope that `App::GTD` can help others to get things done as well. The program -is all but finished, but it should be usable for people besides me and my -colleague by now. If you have any suggestions or questions about the program, -do not hesitate to seek me out! -{% endmarkdown %} diff --git a/_posts/2020-01-08-running-cgit-on-gentoo.md b/_posts/2020-01-08-running-cgit-on-gentoo.md deleted file mode 100644 index fc45e33..0000000 --- a/_posts/2020-01-08-running-cgit-on-gentoo.md +++ /dev/null @@ -1,303 +0,0 @@ ---- -title: Running cgit on Gentoo -layout: post -tags: git cgit Gentoo -social: - mastodon: https://soc.fglt.nl/notice/9rG9O32VTSYnlL451U -description: > - Recently, I've setup cgit on my desktop, running Gentoo. This post covers the - installation and configuration I've undertaken to get it running as desired. ---- - -[cgit](https://git.zx2c4.com/cgit/about/), a web interface for git -repositories, allows you to easily share your projects' source code over a web -interface. It's running on my desktop right now, so you can [see for -yourself](https://home.tyil.nl/git) what it looks like. On -[Gentoo](https://www.gentoo.org/), the ebuild for this software can be found as -`www-apps/cgit`. However, after installation, a number of configuration steps -should be performed to make it accessible on `$HOSTNAME/git`, and index your -repositories. This post will guide you through the steps I took. - -## Filesystem layout - -In my setup, my (bare) git repositories reside in `$HOME/.local/git`. But, some -of the repositories should not be public, such as the -[`pass`](https://www.passwordstore.org/) store. So, a different directory -for cgit to look in exists, at `$HOME/.local/srv/cgit`. This directory contains -symlinks to the actual repositories I want publicly available. - -## Installing the required software - -For this to work, there is more than just cgit to install. There are a number -of ways to set this up, but I chose for Nginx as web server, and `uwsgi` as the -handler for the fastcgi requests. - -{% highlight sh %} -emerge dev-python/pygments www-apps/cgit www-servers/nginx www-servers/uwsgi -{% endhighlight %} - -## Configuring all elements - -After installation, each of these packages needs to be configured. - -### cgit - -The configuration file for cgit resides in `/etc/cgitrc`. After removing all -the comments, the contents of my `/etc/cgitrc` can be found below. - -{% highlight text %} -# Fixes for running cgit in a subdirectory -css=/git/cgit.css -logo=/git/cgit.png -virtual-root=/git -remove-suffix=1 - -# Customization -root-desc=All public repos from tyil -enable-index-owner=0 -cache-size=1000 -snapshots=tar.gz tar.bz2 -clone-prefix=https://home.tyil.nl/git -robots=index, follow - -readme=master:README.md -readme=master:README.pod6 - -# Add filters before repos (or filters won't work) -about-filter=/usr/lib64/cgit/filters/about-formatting.sh -source-filter=/usr/lib64/cgit/filters/syntax-highlighting.py - -# Scan paths for repos -scan-path=/home/tyil/.local/srv/cgit -{% endhighlight %} - -You should probably update the values of `root-desc`, `clone-prefix` and -`scan-path`. The first describes the small line of text at the top of the web -interface. `clone-prefix` is the prefix URL used for `git clone` URLs. The -`scan-path` is the directory `cgit` will look for repositories in. - -Additionally, the `readme=master:README.pod6` only positively affects -your setup if you also use my [Raku](https://raku.org/) customizations, -outlined in the next section. - -For more information on the available settings and their impact, consult `man -cgitrc`. - -#### Raku customizations - -Since I love working with Raku, I made some changes and a couple modules to get -`README.pod6` files rendered on the *about* tab on projects. You should ensure -the `cgit` user can run `raku` and has the -[`Pod::To::Anything`](https://home.tyil.nl/git/raku/Pod::To::Anything/) and -[`Pod::To::HTML::Section`](https://home.tyil.nl/git/raku/Pod::To::HTML::Section/) -modules installed (including any dependencies). How to achieve this depends on -how you installed Raku. Feel free to send me an email if you need help on this -part! - -Once this works, however, the remaining step is quite simple. The -`about-filter` configuration item in `/etc/cgitrc` points to a small shell -script that invokes the required program to convert a document to HTML. In my -case, this file is at `/usr/lib64/cgit/filters/about-formatting.sh`. Open up -this file in your favorite `$EDITOR` and add another entry to the `case` for -[Pod6](https://docs.raku.org/language/pod) to call Raku. - -{% highlight sh %} -case "$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]')" in - *.markdown|*.mdown|*.md|*.mkd) exec ./md2html; ;; - *.pod6) exec raku --doc=HTML::Section; ;; - *.rst) exec ./rst2html; ;; - *.[1-9]) exec ./man2html; ;; - *.htm|*.html) exec cat; ;; - *.txt|*) exec ./txt2html; ;; -esac -{% endhighlight %} - -#### Highlighting style - -The `syntax-highlighting.py` filter carries the responsibility to get your code -highlighted. This uses the Python library [pygments](https://pygments.org/), -which comes with a number of styles. cgit uses *Pastie* by default. To change -this, open the Python script, and look for the `HtmlFormatter`, which contains -a `style='Pastie'` bit. You can change `Pastie` for any other style name. These -styles are available in my version (2.4.2): - -- default -- emacs -- friendly -- colorful -- autumn -- murphy -- manni -- monokai -- perldoc -- pastie -- borland -- trac -- native -- fruity -- bw -- vim -- vs -- tango -- rrt -- xcode -- igor -- paraiso-light -- paraiso-dark -- lovelace -- algol -- algol_nu -- arduino -- rainbow_dash -- abap -- solarized-dark -- solarized-light -- sas -- stata -- stata-light -- stata-dark - -For those interested, I use the `emacs` theme. - -### uwsgi - -Next up, `uwsgi`. This needs configuration, which on Gentoo exists in -`/etc/conf.d/uwsgi`. However, this file itself shouldn't be altered. Instead, -make a copy of it, and call it `/etc/conf.d/uwsgi.cgit`. The standard file -exists solely as a base template. For brevity, I left out all the comments in -the contents below. - -{% highlight sh %} -UWSGI_SOCKET= -UWSGI_THREADS=0 -UWSGI_PROGRAM= -UWSGI_XML_CONFIG= -UWSGI_PROCESSES=1 -UWSGI_LOG_FILE= -UWSGI_CHROOT= -UWSGI_DIR=/home/tyil -UWSGI_PIDPATH_MODE=0750 -UWSGI_USER= -UWSGI_GROUP= -UWSGI_EMPEROR_PATH= -UWSGI_EMPEROR_PIDPATH_MODE=0770 -UWSGI_EMPEROR_GROUP= -UWSGI_EXTRA_OPTIONS="--ini /etc/uwsgi.d/cgit.ini" -{% endhighlight %} - -That covers the service configuration file. When things don't work the way you -expect, specify a path in `UWSGI_LOG_FILE` to see its logs. Additionally, you -may want to alter the value of `UWSGI_DIR`. This specifies the working -directory from which the process starts. - -Now comes the application configuration, which will be read from -`/etc/uwsgi.d/cgit.ini`, according to `UWSGI_EXTRA_OPTIONS`. Create that file -with the following contents. - -{% highlight ini %} -[uwsgi] -master = true -plugins = cgi -socket = 127.0.0.1:1234 -uid = cgit -gid = cgit -procname-master = uwsgi cgit -processes = 1 -threads = 2 -cgi = /usr/share/webapps/cgit/1.2.1/hostroot/cgi-bin/cgit.cgi -{% endhighlight %} - -Note that the `cgi` value contains the version number of `www-apps/cgit`. You -may need to come back after an upgrade and update it accordingly. - -As last step for `uwsgi` configuration, a service script, to manage it with -`rc-service`. These scripts all exist in `/etc/conf.d`, and the package -installed a script called `uwsgi` in there. Just like with the `conf.d` -variant, its just a template. This time, however, don't make a copy of it, but -a symlink. It does not need to be edited, but the name must be the same as the -`conf.d` entry name. That would be `uwsgi.cgit`. - -{% highlight sh %} -cd /etc/conf.d -ln -s uwsgi uwsgi.cgit -{% endhighlight %} - -Now you can start the service with `rc-service uwsgi.cgit start`. If a -consequent `status` notes the state as *Started*, you're all good. If the state -says *Crashed*, you should go back and double-check all configuration files. -When those are correct and you can't figure out why, feel free to reach out to -me via email. - -{% highlight sh %} -rc-service uwsgi.cgit start -rc-service uwsgi.cgit service - -# Start this after boot -rc-update add uwsgi.cgit -{% endhighlight %} - -### nginx - -The final element to make it accessible, the web server, `nginx`. How you -organize the configuration files here is largely up to you. Explaining how to -set up nginx from scratch is beyond the scope of this post. Assuming you know -how to configure this, add the following `location` blocks to the `server` -definition for the vhost you want to make `cgit` available on. - -{% highlight nginx %} -location "/git" { - alias /usr/share/webapps/cgit/1.2.1/htdocs; - try_files $uri @cgit; -} - -location @cgit { - include uwsgi_params; - - gzip off; - - uwsgi_modifier1 9; - uwsgi_pass 127.0.0.1:1234; - - fastcgi_split_path_info ^(/git/?)(.+)$; - uwsgi_param PATH_INFO $fastcgi_path_info; -} -{% endhighlight %} - -Once saved, you can reload `nginx`, and the `$HOSTNAME/git` endpoint can be -reached, and should display an cgit page, detailing there are no repositories. -That can be easily solved by making some available in `$HOME/.local/srv/cgit`, -through the power of symlinks. - -## Symlinking the repositories - -Go nuts with making symlinks to the various repositories you have gathered over -the years. You don't need to use bare repositories, `cgit` will also handle -regular repositories that you actively work in. As with the `nginx` -configuration, explaining how to make symlinks is out of scope. In dire -situations, consult `man ln`. - -### `git-mkbare` - -While making the symlinks is easy, I found that it sheepishly boring to do. I go -to `$HOME/.local/git`, make a directory, `cd` to it, and create a bare -repository. Then off to `$HOME/.local/srv/cgit` to make a symlink back to the -newly created bare repository. I think you can see this will get tedious very -quickly. - -So, to combat this, I made a small shell script to do all of that for me. I -called it `git-mkbare`, and put it somewhere in my `$PATH`. This allows me to -call it as `git mkbare repo-name`. It will ask for a small description as well, -so I that can also be skipped as a manual task. This script may be of use to -you if you want to more quickly start a new project. - -You can find this script [in my dotfiles -repository](https://home.tyil.nl/git/dotfiles/tree/.local/bin/git-mkbare). - -## Wrapping up - -Now you should have cgit available from your site, allowing you to share the -sources of all your projects easily with the world. No need to make use of a -(proprietary) third-party service! - -If you have questions or comments on my setup, or the post in general, please -contact me through email or irc. diff --git a/_posts/2020-05-30-setting-up-pgp-wkd.md b/_posts/2020-05-30-setting-up-pgp-wkd.md deleted file mode 100644 index 147f8c0..0000000 --- a/_posts/2020-05-30-setting-up-pgp-wkd.md +++ /dev/null @@ -1,107 +0,0 @@ ---- -title: Setting Up a PGP Webkey Directory -layout: post -tags: PGP GPG WKD Security -social: - email: mailto:~tyil/public-inbox@lists.sr.ht&subject=Setting Up a PGP Webkey Directory - mastodon: https://soc.fglt.nl/notice/9vaBwcOO6ynNYfT7Lc -description: > - A friend on IRC asked me how I made my PGP key available in a webkey - directory. This post will detail my path, so you can easily set it up for - yourself. ---- - -A little while ago, a friend on IRC asked me how I set up a PGP webkey -directory on my website. For those that don't know, a webkey directory is a -method to find keys through `gpg`'s `--locate-key` command. This allows people -to find my key using this command: - -{% highlight sh %} -gpg --locate-key p.spek@tyil.nl -{% endhighlight %} - -This is a very user-friendly way for people to get your key, as compared to -using long IDs. - -This post will walk you through setting it up on your site, so you can make -your key more easily accessible to other people. - -## Set up the infrastructure - -For a webkey directory to work, you simply need to have your key available at a -certain path on your website. The base path for this is -`.well-known/openpgpkey/`. - -{% highlight sh %} -mkdir -p .well-known/openpgpkey -{% endhighlight %} - -The webkey protocol will check for a `policy` file to exist, so you must create -this too. The file can be completely empty, and that's exactly how I have it. - -{% highlight sh %} -touch .well-known/openpgpkey/policy -{% endhighlight %} - -The key(s) will be placed in the `hu` directory, so create this one too. - -{% highlight sh %} -mkdir .well-known/openpgpkey/hu -{% endhighlight %} - -## Adding your PGP key - -The key itself is just a standard export of your key, without ASCII armouring. -However, the key does need to have its file **name** in a specific format. -Luckily, you can just show this format with `gpg`'s `--with-wkd-hash` option. - -{% highlight sh %} -gpg --with-wkd-hash -k p.spek@tyil.nl -{% endhighlight %} - -This will yield output that may look something like this: - -{% highlight text %} -pub rsa4096/0x7A6AC285E2D98827 2018-09-04 [SC] - Key fingerprint = 1660 F6A2 DFA7 5347 322A 4DC0 7A6A C285 E2D9 8827 -uid [ultimate] Patrick Spek <p.spek@tyil.nl> - i4fxxwcfae1o4d7wnb5bop89yfx399yf@tyil.nl -sub rsa2048/0x031D65902E840821 2018-09-04 [S] -sub rsa2048/0x556812D46DABE60E 2018-09-04 [E] -sub rsa2048/0x66CFE18D6D588BBF 2018-09-04 [A] -{% endhighlight %} - -What we're interested in is the `uid` line with the hash in the local-part of -the email address, which would be `i4fxxwcfae1o4d7wnb5bop89yfx399yf@tyil.nl`. -For the filename, we only care about the local-part itself, meaning the export -of the key must be saved in a file called `i4fxxwcfae1o4d7wnb5bop89yfx399yf`. - -{% highlight sh %} -gpg --export 0x7A6AC285E2D98827 > .well-known/openpgpkey/hu/i4fxxwcfae1o4d7wnb5bop89yfx399yf -{% endhighlight %} - -## Configuring your webserver - -Lastly, your webserver may require some configuration to serve the files -correctly. For my blog, I'm using [`lighttpd`](https://www.lighttpd.net/), for -which the configuration block I'm using is as follows. - -{% highlight lighttpd %} -$HTTP["url"] =~ "^/.well-known/openpgpkey" { - setenv.add-response-header = ( - "Access-Control-Allow-Origin" => "*", - ) -} -{% endhighlight %} - -It may be worthwhile to note that if you do any redirection on your domain, -such as adding `www.` in front of it, the key lookup may fail. The error -message given by `gpg` on WKD lookup failures is... poor to say the least, so -if anything goes wrong, try some verbose `curl` commands and ensure that the -key is accessible at the right path in a single HTTP request. - -## Wrapping up - -That's all there's to it! Adding this to your site should be relatively -straightforward, but it may be a huge convenience to anyone looking for your -key. If you have any questions or feedback, feel free to reach out to me! diff --git a/_posts/2020-06-21-lately-in-raku.md b/_posts/2020-06-21-lately-in-raku.md deleted file mode 100644 index fed891a..0000000 --- a/_posts/2020-06-21-lately-in-raku.md +++ /dev/null @@ -1,157 +0,0 @@ ---- -title: Lately in Raku -layout: post -tags: Raku -social: - email: mailto:~tyil/public-inbox@lists.sr.ht&subject=Lately in Raku - mastodon: https://soc.fglt.nl/notice/9wIq8QpmdRpsP4Qrr6 -description: > - A post on some Raku stuff I've been working on lately. ---- - -I've been working on some Raku projects, but each of them is *just* too small -to make an individual blog post about. So, I decided to just pack them together -in a slightly larger blog post instead. - -## Binary Rakudo Star builds for GNU+Linux and FreeBSD - -A friend on IRC asked if it was possible to get Rakudo Star precompiled for -ARM, since compiling it on his machine took forever. I took a look around for -potential build services, and settled for [Sourcehut](https://builds.sr.ht/). - -I added build instructions for amd64 FreeBSD, GNU+Linux, musl+Linux, and ARM -GNU+Linux. Tarballs with precompiled binaries get build whenever I push to the -Rakudo Star mirror on Sourcehut, and are uploaded to -[dist.tyil.nl/tmp](https://dist.tyil.nl/tmp/). Currently, these are not -considered to be an official part of Rakudo Star, but if interest increases and -more people can test these packages, I can include them in official releases. - -## `IRC::Client` plugins - -IRC bots are great fun, and the -[`IRC::Client`](https://github.com/raku-community-modules/perl6-IRC-Client) -module allows for easy extension through *plugins*. For my own IRC bot, -[musashi](https://git.sr.ht/~tyil/raku-local-musashi), I've created two new -plugins, which are now available in the Raku ecosystem for anyone to use. - -### `IRC::Client::Plugin::Dicerolls` - -The first plugin I've created can do dice rolls, D&D style. You can roll any -number of dice, with any number of sides, and add (or subtract) bonusses from -these. - - <@tyil> .roll 1d20 - <+musashi> 1d20 = 1 - <@tyil> .roll 5d20 - <+musashi> 5d20 = 3 + 19 + 8 + 6 + 11 = 47 - <@tyil> .roll 1d8+2d6+10 - <+musashi> 1d8+2d6+10 = 4 + 6 + 4 + 10 = 24 - -Since this is ripe for abuse, the plugin allows to set limits, and sets some -defaults for the limits as well. This should help prevent your bot from getting -killed for spam. - -### `IRC::Client::Plugin::Reminders` - -Everyone forgets things, and there's various tools helping people remember -things in various situations. For IRC based situations, I created a reminder -plugin for `IRC::Client`. - - 10:19 <@tyil> musashi: remind me to write a blog post in 10 minutes - 10:19 <+musashi> Reminding you to write a blog post on 2020-06-21T08:29:00Z (UTC) - 10:29 <+musashi> tyil: Reminder to write a blog post - -It's not very sophisticated yet, working only with numbers and certain -identifiers (minutes, hours, days, weeks), but I may add more useful -identifiers later on such as "tomorrow", or "next Sunday". Contributions for -such extended functionality are obviously also very welcome! - -There's [a small -issue](https://git.sr.ht/~tyil/raku-irc-client-plugin-reminders/tree/master/lib/IRC/Client/Plugin/Reminders.rakumod#L69) -with logging in a `start` block. It seems the dynamic variable `$*LOG` is no -longer defined within it. If anyone has an idea why, and how I could fix this, -please let me know! - -## Template program for D&D - -Another little utility I made for D&D purposes. My DM asked me how hard it'd be -to create a program to fill out a number of templates he made, so he could use -them in the game with another party. He was able to hand me a list of variables -in the form of a CSV, so I set out to use that. With some help from `Text::CSV` -and `Template::Mustache`, I had a working solution in a couple minutes, with -all the required things nicely fit into a single file. - -I had not used `$=pod` before in Raku, and I'm quite happy with how easy it is -to use, though I would like a cleaner way to refer to a Pod block by name. - -{% highlight perl6 %} -{% raw %} -#!/usr/bin/env raku - -use v6.d; - -use Template::Mustache; -use Text::CSV; - -#| Read a CSV input file to render contracts with. -sub MAIN () { - # Set the directory to write the contracts to. - my $output-dir = $*PROGRAM.parent(2).add('out'); - - # Make sure the output directory exists - $output-dir.mkdir; - - # Load the template - my $template = $=pod - .grep({ $_.^can('name') && $_.name eq 'template' }) - .first - .contents - .map(*.contents) - .join("\n\n") - ; - - # Parse STDIN as CSV - my @records = Text::CSV - .new - .getline_all($*IN) - .skip - ; - - # Create a contract out of each record - for @records -> @record { - $output-dir.add("contract-{@record[0]}.txt").spurt( - Template::Mustache.render($template, { - contractor => @record[2], - date => @record[1], - description => @record[6], - item => @record[3], - location => @record[5], - price => @record[4] - }) ~ "\n" - ); - } -} - -=begin template -As per our verbal agreement this contract will detail the rewards, rights, and -obligations of both parties involved. - -The contractor, to be known henceforth as {{ contractor }}. -The contractee, to be known henceforth as the Association. - -{{ contractor }} requests the delivery of an object identified as the "{{ item }}" -to be delivered by the Association at the location specified for the agreed -upon compensation. The Association shall deliver the object within two weeks of -the signing of this contract and receive compensation upon delivery. - -The location is to be known as "{{ location }}", described as "{{ description }}". -The compensation agreed upon is {{ price }} pieces of Hondia standard -gold-coin currency, or an equivalent in precious gemstones. - -Written and signed on the {{ date }}. - -For the association, Lan Torrez -For the {{ contractor }} -=end template -{% endraw %} -{% endhighlight %} diff --git a/_posts/2020-07-15-config-3.0.md b/_posts/2020-07-15-config-3.0.md deleted file mode 100644 index 9fb33c0..0000000 --- a/_posts/2020-07-15-config-3.0.md +++ /dev/null @@ -1,181 +0,0 @@ ---- -title: Config 3.0 -layout: post -tags: Raku Programming -social: - email: mailto:~tyil/public-inbox@lists.sr.ht&subject=Config 3.0 - mastodon: https://soc.fglt.nl/notice/9x8QT2TxD2dSlEYse8 -description: > - I've made a reasonably sized change to Raku's Config module, resulting in a - major version bump. This article details my reasoning behind it, and shows - some examples on how I think I solved the issues at hand. ---- - -For those who don't know, the -[`Config`](https://modules.raku.org/dist/Config:cpan:TYIL) module for the Raku -programming language is a generic class to hold... well... configuration data. -It supports -[`Config::Parser`](https://modules.raku.org/search/?q=Config%3A%3AParser) -modules to handle different configuration file formats, such as `JSON`, `YAML` -and `TOML`. - -Up until now, the module didn't do much for you other than provide an interface -that's generally the same, so you won't need to learn differing methods to -handle differing configuration file formats. It was my first Raku module, and -as such, the code wasn't the cleanest. I've written many new modules since -then, and learned about a good number of (hopefully better) practices. - -For version 3.0, I specifically wanted to remove effort from using the `Config` -module on the developer's end. It should check default locations for -configuration files, so I don't have to rewrite that code in every other module -all the time. Additionally, configuration using environment variables is quite -popular in the current day and age, especially for Dockerized applications. So, -I set out to make an automated way to read those too. - -## The Old Way - -First, let's take a look at how it used to work. Generally, I'd create the -default configuration structure and values first. - -{% highlight perl6 %} -use Config; - -my $config = Config.new.read({ - foo => "bar", - alpha => { - beta => "gamma", - }, - version => 3, -}); -{% endhighlight %} - -And after that, check for potential configuration file locations, and read any -that exist. - -{% highlight perl6 %} -$config.read($*HOME.add('config/project.toml').absolute); -{% endhighlight %} - -The `.absolute` call was necessary because I wrote the initial `Config` version -with the `.read` method not supporting `IO::Path` objects. A fix for this has -existed for a while, but wasn't released, so couldn't be relied on outside of -my general development machines. - -If you wanted to add additional environment variable lookups, you'd have to -check for those as well, and perhaps also cast them as well, since environment -variables are all strings by default. - -## Version 3.0 - -So, how does the new version improve this? For starters, the `.new` method of -`Config` now takes a `Hash` as positional argument, in order to create the -structure, and optionally types *or* default values of your configuration -object. - -{% highlight perl6 %} -use Config; - -my $config = Config.new({ - foo => Str, - alpha => { - beta => "gamma", - }, - version => 3, -}, :name<project>); -{% endhighlight %} - -{% admonition_md note %} -`foo` has been made into the `Str` *type object*, rather than a `Str` *value*. -This was technically allowed in previous `Config` versions, but it comes with -actual significance in 3.0. -{% endadmonition_md %} - -Using `.new` instead of `.read` is a minor syntactic change, which saves 1 word -per program. This isn't quite that big of a deal. However, the optional `name` -argument will enable the new automagic features. The name you give to `.new` is -arbitrary, but will be used to deduce which directories to check, and which -environment variables to read. - -### Automatic Configuration File Handling - -By setting `name` to the value `project`, `Config` will consult the -configuration directories from the [XDG Base Directory -Specification](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html). -It uses one of my other modules, -[`IO::Path::XDG`](https://modules.raku.org/dist/IO::Path::XDG:cpan:TYIL), for -this, together with -[`IO::Glob`](https://modules.raku.org/dist/IO::Glob:cpan:HANENKAMP). -Specifically, it will check my `$XDG_CONFIG_DIRS` and `$XDG_CONFIG_HOME` (in -that order) for any files that match the globs `project.*` or -`project/config.*`. - -If any files are found to match, they will be read as well, and the -configuration values contained therein, merged into `$config`. It will load the -appropriate `Config::Parser` implementation based on the file's extension. I -intend to add a number of these to future Rakudo Star releases, to ensure most -default configuration file formats are supported out of the box. - -### Automatic Environment Variable Handling - -After this step, it will try out some environment variables for configuration -values. Which variables are checked depends on the structure (and `name`) of -the `Config` object. The entire structure is squashed into a 1-dimensional list -of fields. Each level is replaced by an `_`. Additionally, each variable name -is prefixed with the `name`. Lastly, all the variable names are uppercased. - -For the example `Config` given above, this would result in the following -environment variables being checked. - -{% highlight sh %} -$PROJECT_FOO -$PROJECT_ALPHA_BETA -$PROJECT_VERSION -{% endhighlight %} - -If any are found, they're also cast to the appropriate type. Thus, -`$PROJECT_FOO` would be cast to a `Str`, and so would `$PROJECT_ALPHA_BETA`. In -this case that doesn't do much, since they're already strings. But -`$PROJECT_VERSION` would be cast to an `Int`, since it's default value is also -of the `Int` type. This should ensure that your variables are always in the -type you expected them to be originally, no matter the user's configuration -choices. - -## Debugging - -In addition to these new features, `Config` now also makes use of my -[`Log`](https://modules.raku.org/dist/Log:cpan:TYIL) module. This module is -made around the idea that logging should be simple if module developers are to -use it, and the way logs are represented is up to the end-user. When running an -application in your local terminal, you may want more human-friendly logs, -whereas in production you may want `JSON` formatted logs to make it fit better -into other tools. - -You can tune the amount of logging performed using the `$RAKU_LOG_LEVEL` -environment variable, as per the `Log` module's interface. When set to `7` (for -"debug"), it will print the configuration files that are being merged into your -`Config` and which environment veriables are being used as well. - -{% admonition_md note %} -A downside is that the application using `Config` for its configuration must -also support `Log` to actually make the new logging work. Luckily, this is -quite easy to set up, and there's example code for this in `Log`'s README. -{% endadmonition_md %} - -## Too Fancy For Me - -It could very well be that you don't want these features, and you want to stick -to the old ways as much as possible. No tricks, just plain and simple -configuration handling. This can be done by simply ommitting the `name` -argument to `.new`. The new features depend on this name to be set, and won't -do anything without it. - -Alternatively, both the automatic configuration file handling and the -environment variable handling can be turned off individually using `:!from-xdg` -and `:!from-env` arguments respectively. - -## In Conclusion - -The new `Config` module should result in cleaner code in modules using it, and -more convenience for the developer. If you find any bugs or have other ideas -for improving the module, feel free to send an email to -`https://lists.sr.ht/~tyil/raku-devel`. diff --git a/_posts/2020-07-19-freebsd-mailserver-part-6-system-updates.md b/_posts/2020-07-19-freebsd-mailserver-part-6-system-updates.md deleted file mode 100644 index 5094eba..0000000 --- a/_posts/2020-07-19-freebsd-mailserver-part-6-system-updates.md +++ /dev/null @@ -1,342 +0,0 @@ ---- -title: "FreeBSD Email Server - Part 6: System Updates" -layout: post -tags: Tutorial FreeBSD Email -social: - email: mailto:~tyil/public-inbox@lists.sr.ht&subject=FreeBSD Email Server - mastodon: https://soc.fglt.nl/notice/9xF5VVpcK1NJR0kgOO -description: > - Updates are important. This article aims to help you to learn from my - mistakes, so your updates will go smooth. ---- - -Four years have past, and my FreeBSD email server has keps on running without -any problems. However, some people on IRC have recently been nagging me to -support TLSv1.3 on my mailserver. Since the installation was done 4 years ago, -it didn't do 1.3 yet, just 1.2. I set out to do a relatively simple system -update, which didn't go as smooth as I had hoped. This tutorial post should -help you avoid the mistakes I made, so your updates *will* go smooth. - -{% admonition_md info %} -The rest of this tutorial assumes you're running as the `root` user. -{% endadmonition_md %} - -## Preparations - -Before we do anything wild, let's do the obvious first step: backups. Since -this is a FreeBSD server, it uses glorious -[ZFS](https://en.wikipedia.org/wiki/ZFS) as the filesystem, which allows us to -make use of -[snapshots](https://docs.oracle.com/cd/E23824_01/html/821-1448/gbciq.html). -Which subvolumes to make snapshots off depends on your particular setup. In my -case, my actual email data is stored on `zroot/srv`, and all the services and -their configurations are in `zroot/usr/local`. My database's data is stored on -`zroot/postgres/data96`. Additionally, I want to make a snapshot of -`zroot/usr/ports`. - -{% highlight sh %} -zfs snapshot -r zroot/srv@`date +%Y%m%d%H%M%S`-11.0-final -zfs snapshot -r zroot/usr/local@`date +%Y%m%d%H%M%S`-11.0-final -zfs snapshot -r zroot/postgres@`date +%Y%m%d%H%M%S`-11.0-final -zfs snapshot -r zroot/usr/ports@`date +%Y%m%d%H%M%S`-11.0-final -{% endhighlight %} - -This will make a snapshot of each of these locations, for easy restoration in -case any problems arise. You can list all your snapshots with `zfs list -t -snapshot`. - -Your server is most likely hosted at a provider, not in your home. This means -you won't be able to just physically access it and retrieve the harddrive if -things go really bad. You might not be able to boot single-user mode either. -Because of this, you might not be able to restore the snapshots if things go -*really* bad. In this case, you should also make a local copy of the important -data. - -The services and their configuration can be recreated, just follow the earlier -parts of this series again. The email data, however, cannot. This is the data -in `/srv/mail`. You can make a local copy of all this data using `rsync`. - -{% highlight sh %} -rsync -av example.org:/srv/mail/ ~/mail-backup -{% endhighlight %} - -There's one more thing to do, which I learned the hard way. Set your login -shell to a simple one, provided by the base system. The obvious choice is -`/bin/sh`, but some people may wrongly prefer `/bin/tcsh` as well. During a -major version update, the ABI changes, which will temporarily break most of -the user-installed packages, including your shell. - -{% highlight sh %} -chsh -{% endhighlight %} - -{% admonition_md warning %} -Be sure to change the shell for whatever user you're using to SSH into this -machine too, if any! -{% endadmonition_md %} - -## Updating the Base System - -With the preparations in place in case things get royally screwed up, the -actual updates can begin. FreeBSD has a dedicated program to handle updating -the base system, `freebsd-update`. First off, fetch any updates, and make sure -all the updates for your current version are applied. - -{% highlight sh %} -freebsd-update fetch install -{% endhighlight %} - -Afterwards, set the new system version you want to update to. In my case, this -is `12.1-RELEASE`, but if you're reading this in the future, you most certainly -want a newer version. - -{% highlight sh %} -freebsd-update -r 12.1-RELEASE upgrade -{% endhighlight %} - -This command will ask you to review the changes and confirm them as well. It -should generally be fine, but this is your last chance to make any backups or -perform other actions to secure your data! If you're ready to continue, install -the updates to the machine. - -{% highlight sh %} -freebsd-update install -{% endhighlight %} - -At this point, your kernel has been updated. Next you must reboot to start -using the new kernel. - -{% highlight sh %} -reboot -{% endhighlight %} - -Once the system is back online, you can continue installing the rest of the -updates. - -{% highlight sh %} -freebsd-update install -{% endhighlight %} - -When this command finishes, the base system has been updated and should be -ready for use. Next up is updating all the software you installed manually. - -## Updating User-Installed Packages - -Unlike GNU+Linux distributions, FreeBSD has a clear distinction between the -*base system* and *user installed software*. The base system has now been -updated, but everything installed through `pkg` or ports is still at the old -version. If you performed a major version upgrade (say, FreeBSD 11.x to 12.x), -the ABI has changed and few, if any, of the user-installed packages still work. - -### Binary Packages using `pkg` - -Binary packages are the most common packages used. These are the packages -installed through `pkg`. Currently, `pkg` itself doesn't even work. Luckily, -FreeBSD has `pkg-static`, which is a statically compiled version of `pkg` -intended to fix this very problem. Let's fix up `pkg` itself first. - -{% highlight sh %} -pkg-static install -f pkg -{% endhighlight %} - -That will make `pkg` itself work again. Now you can use `pkg` to update package -information, and upgrade all packages to a version that works under this -FreeBSD version. - -{% highlight sh %} -pkg update -pkg upgrade -{% endhighlight %} - -#### PostgreSQL - -A particular package that was installed through `pkg`, PostgreSQL, just got -updated to the latest version. On FreeBSD, the data directory used by -PostgreSQL is dependent on the version you're running. If you try to list -databases now, you'll notice that the `mail` database used throughout the -tutorial is gone. The data directory is still there, so you *could* downgrade -PostgreSQL again, restart the database, run a `pgdump`, upgrade, restart and -import. However, I find it much cleaner to use FreeBSD jails to solve this -issue. - -{% admonition_md info %} -My original installation used PostgreSQL 9.6, you may need to update some -version numbers accordingly! -{% endadmonition_md %} - -I generally put my jails in a ZFS subvolume, so let's create one of those -first. - -{% highlight sh %} -zfs create -o mountpoint=/usr/jails zroot/jails -zfs create zroot/jails/postgres96 -{% endhighlight %} - -This will create a new subvolume at `/usr/jails/postgres96`. Using -`bsdinstall`, a clean FreeBSD installation usable by the jail can be set up -here. This command will give you some popups you may remember from installing -FreeBSD initially. This time, you can uncheck *all* boxes, to get the most -minimal system. - -{% highlight sh %} -bsdinstall jail /usr/jails/postgres96 -{% endhighlight %} - -When `bsdinstall` finishes, you can configure the jail. This is done in -`/etc/jail.conf`. If this file doesn't exist, you can create it. Make sure the -following configuration block is written to the file. - -{% highlight cfg %} -postgres96 { - # Init information - exec.start = "/bin/sh /etc/rc"; - exec.stop = "/bin/sh /etc/rc.shutdown"; - exec.clean; - - # Set the root path of the jail - path = "/usr/jails/$name"; - - # Mount /dev - mount.devfs; - - # Set network information - host.hostname = $name; - ip4.addr = "lo0|127.1.1.1/32"; - ip6.addr = "lo0|fd00:1:1:1::1/64"; - - # Required for PostgreSQL to function - allow.raw_sockets; - allow.sysvipc; -} -{% endhighlight %} - -Now you can start up the jail, so it can be used. - -{% highlight sh %} -service jail onestart postgres96 -{% endhighlight %} - -Using the host system's `pkg`, you can install PostgreSQL into the jail. - -{% highlight sh %} -pkg -c /usr/jails/postgres96 install postgresql96-server -{% endhighlight %} - -Now you just need to make the data directory available to the jail, which you -can most easily do using -[`nullfs`](https://www.freebsd.org/cgi/man.cgi?query=nullfs&sektion=&n=1). - -{% highlight sh %} -mount -t nullfs /var/db/postgres/data96 /usr/jails/postgres96/var/db/postgres/data96 -{% endhighlight %} - -Now everything should be ready for use inside the jail. Let's head on in using -`jexec`. - -{% highlight sh %} -jexec postgres96 -{% endhighlight %} - -Once inside the jail, you can start the PostgreSQL service, and dump the `mail` -database. - -{% highlight sh %} -service postgresql onestart -su - postgres -pg_dump mail > ~/mail.sql -{% endhighlight %} - -This will write the dump to `/usr/jails/postgres96/var/db/postgres/mail.sql` on -the host system. You can leave the jail and close it down again. - -{% highlight sh %} -exit -exit -service jail onestop postgres96 -{% endhighlight %} - -This dump can be imported in your updated PostgreSQL on the host system. -Connect to the database first. - -{% highlight sh %} -su - postgres -psql -{% endhighlight %} - -Then, recreate the user, database and import the data from the dump. - -{% highlight sql %} -CREATE USER postfix WITH PASSWORD 'incredibly-secret!'; -CREATE DATABASE mail WITH OWNER postfix; -\c mail -\i /usr/jails/postgres96/var/db/postgres/mail.sql -\q -{% endhighlight %} - -The `mail` database is now back, and ready for use! - -### Packages from Ports - -With all the binary packages out of the way, it's time to update packages from -ports. While it is very possible to just go to each port's directory and -manually update each one individually, I opted to use `portupgrade`. This will -need manual installation, but afterwards, we can rely on `portupgrade` to do -the rest. Before doing anything with the ports collection, it should be -updated, which is done using `portsnap`. - -{% highlight sh %} -portsnap fetch extract -{% endhighlight %} - -Once this is done, you can go to the `portupgrade` directory and install it. - -{% highlight sh %} -cd /usr/ports/ports-mgmt/portupgrade -make install clean -{% endhighlight %} - -Now, to upgrade all other ports. - -{% highlight sh %} -portupgrade -a -{% endhighlight %} - -Be sure to double-check the compilation options that you are prompted about! If -you're missing a certain option, you may miss an important feature that is -required for your mailserver to work appropriately. This can be easily fixed by -recompiling, but a few seconds checking now can save you an hour figuring it -out later! - -## Tidying Up - -Now that all user-installed software has been updated too, it's time to -finalize the update by running `freebsd-update` for a final time. - -{% highlight sh %} -freebsd-update install -{% endhighlight %} - -You can return to your favourite shell again. - -{% highlight sh %} -chsh -{% endhighlight %} - -And you can clean up the ports directories to get some wasted space back. - -{% highlight sh %} -portsclean -C -{% endhighlight %} - -I would suggest making a new snapshot as well, now that you're on a relatively -clean and stable state. - -{% highlight sh %} -zfs snapshot -r zroot/srv@`date +%Y%m%d%H%M%S`-12.1-clean -zfs snapshot -r zroot/usr/local@`date +%Y%m%d%H%M%S`-12.1-clean -zfs snapshot -r zroot/postgres@`date +%Y%m%d%H%M%S`-12.1-clean -zfs snapshot -r zroot/usr/ports@`date +%Y%m%d%H%M%S`-12.1-clean -{% endhighlight %} - -And that concludes your system update. Your mailserver is ready to be neglected -for years again! |