Static website hosting in 2019

TL;DR: use Gitlab+Netlify.

With the push for back to static, I found myself looking for static hosting again. And with some specific requirements:

  • DNS on Cloudflare
  • SSL support for 4th level subdomains (sub.sub.domain.com). Which means SSL termination can’t happen on Cloudflare – their free plan only allows to proxy 3rd level subdomains, and business one is too expensive
  • Not necessarily free, but unexpensive, at least.

And here’s what I came up with:

  1. First idea: AWS S3. Looks good at first site, but the problem is that it doesn’t support SSL without Cloudfront. So you have to create a CF distribution for each site, then invalidate it each time it’s updated… too complicated for a simple static website.
  2. Second: Github pages. Very good, except that it doesn’t allow to publish sites from private repos, unless you pay for your account. Pass.
  3. Gitlab pages, on the other hand, do allow that. Sadly, they have no automatic SSL renewal, they actually recommend to “When you finish setting up, just put in your calendar to remember to renew the certificate in time“. Seems ridiculous for a service that is all about DevOps and IaC.
  4. And finally I found the holy grail: Netlify. It has everything that’s on the list, plus automatic Jekyll (and other site generators) support. And their own CDN. So all you have to do to deploy is push to website repo.

So for now, Netlify is my static hosting platform of choice. Still, there are some gotchas and things where it can be improved:

  1. It’s a lot of clicking to create a website: create, pick repo, receive Netlify’s default subdomain, add a CNAME to DNS, enable SSL in settings, add redirect from default subdomain to the main domain in netlify.toml. All this is manual. That is fine while you have one or two sites, but when it’s a dozen, you really need to employ infrastructure as code. An Ansible role where you could just specify a list of domains and corresponding repos to have them all created automatically would be perfect.
  2. Netlify gets full access to your Gitlab (Github, etc) account. In some cases, that might be acceptable. In others, you might want to create a dedicated account for it instead and grant access only to the website repos.
  3. Make no mistake – if you’re not paying for the product, you are the product. (Which is why S3 was my first go-to option: I’d rather pay a little, than nothing at all. Alas, S3 just didn’t cut it.)

Ansible filter list based on attributes

Suppose we want to find only users who have a middle name from this:

users:
  - name: John
    surname: Johnson
  - name: Alice
    surname: Wonderland
  - name: Bob
    surname: Rabbit
    middlename: Bebop

Old, verbose way would be:

- debug: msg="{{ item.middlename }}"
  loop: "{{ users }}"
  when: item.middlename is defined

This results in “skipping” lines polluting the output, however. To avoid this, use

- debug: msg="{{ item.middlename }}"
  loop: "{{ users | selectattr('middlename','defined') | list }}"

This, however, grows unwieldy when you need to chain many conditions. But remember that you can break lines, that’ll work just as well:

- debug: msg="{{ item.middlename }}"
  loop: "{{ users
    | selectattr('middlename','defined')
    | list }}"

VScode extension highighting tips

Reusing parts of another grammar

Could be hard to find, but actually it’s pretty simple:

"patterns": [
  {
     "include": "source.mylanguage#action-body", //Here it is
     "name": "meta.mylanguage.another_language"
  }
]

Custom language highlighting in hovers

You need to inject your language into markdown (package.json):

"grammars": [
  "scopeName": "markdown.mylanguage",
  "path": "./syntaxes/mylanguage-markdown-injection.tmLanguage.json",
  "injectTo": [
    "text.html.markdown"
  ],
  "embeddedLanguages": {
    "meta.embedded.block.mylanguage": "mylanguage"
  }
]

(Also see an excellent example).

Navbar menu for a large static website

Say you need to make a navigation menu bar for a large static (HTML) website. Very large, maybe thousands of pages.

First problem would be – how to put this menu onto every page. Remember, the site is static, files can’t be changed. Fortunately, even static pages can be modified with a combination of SSI (server side includes) and Nginx sub_filter directive. So you can generate a separate HTML file for the menu (SmartMenus is a good choice for that) and then insert it at web server level. Nginx example:

ssi on;
sub_filter '</body>' '<!--# include file="/navbar.html" --> </body>';
sub_filter_once on;
sub_filter_last_modified off;

Where navbar.html is SmartMenus-style list. Very clean and pluggable solution. Can be styled, disabled/enabled when necessary, etc.

It will work fine more for small/medium-sized sites, maybe hundreds of pages. However, once we’re talking thousands, you’ll probably notice that menu becomes bigger than the pages themselves. It could be a a few Mb. And once it gets inserted into every 50 Kb page, you’ll notice that every page load is painfully slow – no surprise here. And even after reload, the pages will be clunky – having to parse several Mb of HTML with Javascript is expensive.

An obvious workaround to that would be to load the menu once and cache it. But it’s not possible if it gets inserted into every page directly.

Well… it sounds like a job for iframe, right? Put one on top, and display the menu there. Of course, wrong. I don’t want to bore you with details, let’s just say that iframes were ugly (from aestetic and programming perspectives) in the 90s, and they still are. Just avoid them.

It’s hard to think of an alternative, though. But the trick is that we don’t really need the full menu all the time. What we need is appearance of having a full menu all the time, and actual functionality ony when it’s being used.

With this in mind, we can try to chain menu load:

  1. Create a small menu with only top-level items, put it into a <div>
    <--! navbar_top.html>
    
    <div id="sm-menu-wrapper">
      <div class="sm_navbar">
        <ul class="sm sm-blue" id="sm-main-menu">
          (Normal SmartMenus HTML list)
        </ul>
      </div>
    </div>
  2. Insert it into every page with SSI.
    sub_filter '</body>' '<!--# include file="/navbar_top.html" --> </body>';
  3. On page load, run a small Javascript which replaces this <div> with contents of the file which contains full menu.
    <!-- SmartMenus jQuery init -->
    $(function() {
        $('#sm-menu-wrapper').load('/navbar.html', function() {
          $('#sm-main-menu').smartmenus({
            subMenusSubOffsetX: 4,
            subMenusSubOffsetY: -5
          });
        });
    });
  4. Set max expires time for the full menu file.

Now, actual pages are not bloated and will load fast, and after load, menu will be changed to full version from the cached file.

It’s almost perfect, but (in SmartMenus case) not quite. The reason is that items that don’t have submenus look different from those that don’t. Because of that, top bar flickers on page load, which is distracting. To avoid that, make sure that short menu version looks exactly the same as the full one when unexpanded. For example for SmartMenus, you’ll have to add class=”sub-arrow”> to menu items with (future) children.

And that’s about it. Without changing a single file, we added huge responsive AJAX navigation menu bar to every page, without any real overhead on server or user machine.

Some parts were left out, for example SmartMenus also needs some JS and CSS inserted, but that’s trivial with power of SSI and sub_filter. To see an example of the solution in action, check this link.

Python threading keyboard interrupt

Proper async thread termination:

pool = ThreadPool(threads_number)

try:
  results = []
  for root, dirs, files in os.walk('.'):
  for f in files:
    results.append( pool.apply_async(process_file, (f, root, file_counter, numfiles)) )
    file_counter += 1
    pool.close()
  except KeyboardInterrupt:
    pool.terminate()
  finally:
    pool.join()

 

Change resolution from command line in Ubuntu 18.04’s Wayland

Wayland doesn’t allow applications to change resolution, and there’s no official utility to do that. There’s a third party display-config script, but it stopped working with latest changes in Mutter API. Here’s a short workaround until a better solution comes up:

#!/bin/bash

set -xeu -o pipefail

resolution="$1"
scale="$2"

function get_serial() {
  serial="$(gdbus call --session --dest org.gnome.Mutter.DisplayConfig \
  --object-path /org/gnome/Mutter/DisplayConfig \
  --method org.gnome.Mutter.DisplayConfig.GetResources | awk '{print $2}' | tr -d ',')"
  echo $serial
}

serial=$(get_serial)
gdbus call --session --dest org.gnome.Mutter.DisplayConfig \
  --object-path /org/gnome/Mutter/DisplayConfig \
  --method org.gnome.Mutter.DisplayConfig.ApplyMonitorsConfig \
  $serial 1 "[(0, 0, $scale, 0, true, [('eDP-1', '$resolution', [] )] )]" "[]"

It should be called like this:

mutter-display-config.sh [email protected] 1

Notice that you need to replace “eDP-1” with your monitor id. And you need to provide a supported resolution. You can get those by running dbus-monitor and changing resolution from GUI.

Curating cron emails

As you might know, cron captures all the output of executed tasks and mails them to the user under which the tasks are executed. The problem is that often this mail just piles up somewhere in /var/mail directory, without being ever reviewed. It’s not a good practice, akin to sweeping the trash under the carpet.

The good practice is:

  1. Use a separate log file for each cron task.
  2. Redirect all output to the log, and additionally print stderr to terminal:
    exec 3>&1
    exec > >(tee -a ${LOGFILE} >/dev/null) 2> >(tee -a ${LOGFILE} >&3)
  3. Capture all that mail and one by one work out all the quirks in your scripts.

Capturing is usually done with something like “root: [email protected]” in /etc/aliases. However, it’s error prone, as various packages add their own accounts and that mail won’t go to root. Manually adding entries is also subject to human errors. So what we’re looking for is wildcard match. Unfortunately, that is not trivial with some email servers. I suggest you do yourself a favor and install Exim, instead of Sendmail or Postfix.

In Exim config (/etc/exim/exim.conf or RedHat based distros, /etc/exim4/exim4.conf.template on Debian based) find system_aliases stanza, and make the change:

-data = ${lookup{$local_part}lsearch{/etc/aliases}}
+data = ${lookup{$local_part}wildlsearch{/etc/aliases}}

Then, add wildcard to /etc/aliases:

*: [email protected]

Restart Exim, and prepare for polishing your cronjobs.

Unfortunately, even with this, cron tasks are still prone to various issues, so for important jobs it’s better to use something more reliable. (But you should capture cron emails, still. They often point out non-obvious issues.)

Typing diacritics Mac-style in Ubuntu 17.10

One can say many bad things about Mac OS, but what they got right for sure is typing diacritics, such as ñ or é. You don’t have to add layouts or switch languages, just use alt+n to get dead tilde and alt-e for dead acute accent.

But in Ubuntu, it’s not possible to type diacritics at all in default US layout. Instead, you have to switch in “U.S. international”, which has all the keys messed up – such as, apostrophe key is now dead acute. And to type an actual , you need to use ‘ + space. It’s almost as if user friendliness is a sin on Linux.

Fortunately, the process of adding a layout is well described in this question on SO. One difference is that Ubuntu 17.10 uses Wayland, which doesn’t have *.xkm, so no cache cleaning required. Just create symbols file, append to evdev.xml and then switch layout to get it working.

(Update: apparently Ubuntu 18.04 with Xorg also doesn’t have *.xkm files. But you might need to re-login for language entry to display correctly in the top bar).

$ cat /usr/share/X11/xkb/symbols/us_dia
default partial alphanumeric_keys
xkb_symbols "basic" {
  name[Group1]= "US English with diacritics that do not hate you (Mac style)";
  include "us(basic)"
  key  { [ n, N, dead_tilde ] }; //ñ, etc
  key  { [ e, E, dead_acute ] }; //é, etc
  key  { [ slash, question, U00BF ] }; // ¿ - Inverted Question Mark
  key  { [ 1, exclam, U00A1 ] }; // ¡ - Inverted Exclamation Mark
  key  { [ u, U, dead_diaeresis ] }; // ü, etc
  include "level3(lalt_switch)"
};

$ diff -u /usr/share/X11/xkb/rules/evdev.xml.2018-06-06 /usr/share/X11/xkb/rules/evdev.xml
--- /usr/share/X11/xkb/rules/evdev.xml.2018-06-06 2018-06-06 21:08:14.900404106 +0700
+++ /usr/share/X11/xkb/rules/evdev.xml 2018-06-08 16:06:16.927002821 +0700
@@ -1318,6 +1318,19 @@
    </model>
  </modelList>
   <layoutList>
+
+     <layout>
+       <configItem>
+         <name>us_dia</name>
+         <shortDescription>en</shortDescription>
+         <description>US English with diacritics that do not hate you (Mac style)</description>
+         <languageList>
+           <iso639Id>eng</iso639Id>
+         </languageList>
+       </configItem>
+       <variantList></variantList>
+     </layout>
+
     <layout>
       <configItem>
         <name>us</name>

Update 2: added inverted punctuation characters and diaeresis.

Forgotten KeePassX password

Stages of having forgotten KeePassX master password:

  1. Denial
  2. Anger
  3. Depression
  4. BRUTFORCE

Faling to open your KeePassX database, while being quite sure that you enter correct password is a very frustrating experience.

Fortunately, while it’s hard to brute an unknown password, it’s much easier to do that when you almost have it right, but have doubts about 1 or 2 characters. See an example script on Github.