VScode extension highighting tips

Reusing parts of another grammar

Could be hard to find, but actually it’s pretty simple:

"patterns": [
  {
     "include": "source.mylanguage#action-body", //Here it is
     "name": "meta.mylanguage.another_language"
  }
]

Custom language highlighting in hovers

You need to inject your language into markdown (package.json):

"grammars": [
  "scopeName": "markdown.mylanguage",
  "path": "./syntaxes/mylanguage-markdown-injection.tmLanguage.json",
  "injectTo": [
    "text.html.markdown"
  ],
  "embeddedLanguages": {
    "meta.embedded.block.mylanguage": "mylanguage"
  }
]

(Also see an excellent example).

Navbar menu for a large static website

Say you need to make a navigation menu bar for a large static (HTML) website. Very large, maybe thousands of pages.

First problem would be – how to put this menu onto every page. Remember, the site is static, files can’t be changed. Fortunately, even static pages can be modified with a combination of SSI (server side includes) and Nginx sub_filter directive. So you can generate a separate HTML file for the menu (SmartMenus is a good choice for that) and then insert it at web server level. Nginx example:

ssi on;
sub_filter '</body>' '<!--# include file="/navbar.html" --> </body>';
sub_filter_once on;
sub_filter_last_modified off;

Where navbar.html is SmartMenus-style list. Very clean and pluggable solution. Can be styled, disabled/enabled when necessary, etc.

It will work fine more for small/medium-sized sites, maybe hundreds of pages. However, once we’re talking thousands, you’ll probably notice that menu becomes bigger than the pages themselves. It could be a a few Mb. And once it gets inserted into every 50 Kb page, you’ll notice that every page load is painfully slow – no surprise here. And even after reload, the pages will be clunky – having to parse several Mb of HTML with Javascript is expensive.

An obvious workaround to that would be to load the menu once and cache it. But it’s not possible if it gets inserted into every page directly.

Well… it sounds like a job for iframe, right? Put one on top, and display the menu there. Of course, wrong. I don’t want to bore you with details, let’s just say that iframes were ugly (from aestetic and programming perspectives) in the 90s, and they still are. Just avoid them.

It’s hard to think of an alternative, though. But the trick is that we don’t really need the full menu all the time. What we need is appearance of having a full menu all the time, and actual functionality ony when it’s being used.

With this in mind, we can try to chain menu load:

  1. Create a small menu with only top-level items, put it into a <div>
    <--! navbar_top.html>
    
    <div id="sm-menu-wrapper">
      <div class="sm_navbar">
        <ul class="sm sm-blue" id="sm-main-menu">
          (Normal SmartMenus HTML list)
        </ul>
      </div>
    </div>
  2. Insert it into every page with SSI.
    sub_filter '</body>' '<!--# include file="/navbar_top.html" --> </body>';
  3. On page load, run a small Javascript which replaces this <div> with contents of the file which contains full menu.
    <!-- SmartMenus jQuery init -->
    $(function() {
        $('#sm-menu-wrapper').load('/navbar.html', function() {
          $('#sm-main-menu').smartmenus({
            subMenusSubOffsetX: 4,
            subMenusSubOffsetY: -5
          });
        });
    });
  4. Set max expires time for the full menu file.

Now, actual pages are not bloated and will load fast, and after load, menu will be changed to full version from the cached file.

It’s almost perfect, but (in SmartMenus case) not quite. The reason is that items that don’t have submenus look different from those that don’t. Because of that, top bar flickers on page load, which is distracting. To avoid that, make sure that short menu version looks exactly the same as the full one when unexpanded. For example for SmartMenus, you’ll have to add class=”sub-arrow”> to menu items with (future) children.

And that’s about it. Without changing a single file, we added huge responsive AJAX navigation menu bar to every page, without any real overhead on server or user machine.

Some parts were left out, for example SmartMenus also needs some JS and CSS inserted, but that’s trivial with power of SSI and sub_filter. To see an example of the solution in action, check this link.

Python threading keyboard interrupt

Proper async thread termination:

pool = ThreadPool(threads_number)

try:
  results = []
  for root, dirs, files in os.walk('.'):
  for f in files:
    results.append( pool.apply_async(process_file, (f, root, file_counter, numfiles)) )
    file_counter += 1
    pool.close()
  except KeyboardInterrupt:
    pool.terminate()
  finally:
    pool.join()

 

Change resolution from command line in Ubuntu 18.04’s Wayland

Wayland doesn’t allow applications to change resolution, and there’s no official utility to do that. There’s a third party display-config script, but it stopped working with latest changes in Mutter API. Here’s a short workaround until a better solution comes up:

#!/bin/bash

set -xeu -o pipefail

resolution="$1"
scale="$2"

function get_serial() {
  serial="$(gdbus call --session --dest org.gnome.Mutter.DisplayConfig \
  --object-path /org/gnome/Mutter/DisplayConfig \
  --method org.gnome.Mutter.DisplayConfig.GetResources | awk '{print $2}' | tr -d ',')"
  echo $serial
}

serial=$(get_serial)
gdbus call --session --dest org.gnome.Mutter.DisplayConfig \
  --object-path /org/gnome/Mutter/DisplayConfig \
  --method org.gnome.Mutter.DisplayConfig.ApplyMonitorsConfig \
  $serial 1 "[(0, 0, $scale, 0, true, [('eDP-1', '$resolution', [] )] )]" "[]"

It should be called like this:

mutter-display-config.sh [email protected] 1

Notice that you need to replace “eDP-1” with your monitor id. And you need to provide a supported resolution. You can get those by running dbus-monitor and changing resolution from GUI.

Curating cron emails

As you might know, cron captures all the output of executed tasks and mails them to the user under which the tasks are executed. The problem is that often this mail just piles up somewhere in /var/mail directory, without being ever reviewed. It’s not a good practice, akin to sweeping the trash under the carpet.

The good practice is:

  1. Use a separate log file for each cron task.
  2. Redirect all output to the log, and additionally print stderr to terminal:
    exec 3>&1
    exec > >(tee -a ${LOGFILE} >/dev/null) 2> >(tee -a ${LOGFILE} >&3)
  3. Capture all that mail and one by one work out all the quirks in your scripts.

Capturing is usually done with something like “root: [email protected]” in /etc/aliases. However, it’s error prone, as various packages add their own accounts and that mail won’t go to root. Manually adding entries is also subject to human errors. So what we’re looking for is wildcard match. Unfortunately, that is not trivial with some email servers. I suggest you do yourself a favor and install Exim, instead of Sendmail or Postfix.

In Exim config (/etc/exim/exim.conf or RedHat based distros, /etc/exim4/exim4.conf.template on Debian based) find system_aliases stanza, and make the change:

-data = ${lookup{$local_part}lsearch{/etc/aliases}}
+data = ${lookup{$local_part}wildlsearch{/etc/aliases}}

Then, add wildcard to /etc/aliases:

*: [email protected]

Restart Exim, and prepare for polishing your cronjobs.

Unfortunately, even with this, cron tasks are still prone to various issues, so for important jobs it’s better to use something more reliable. (But you should capture cron emails, still. They often point out non-obvious issues.)

Typing diacritics Mac-style in Ubuntu 17.10

One can say many bad things about Mac OS, but what they got right for sure is typing diacritics, such as ñ or é. You don’t have to add layouts or switch languages, just use alt+n to get dead tilde and alt-e for dead acute accent.

But in Ubuntu, it’s not possible to type diacritics at all in default US layout. Instead, you have to switch in “U.S. international”, which has all the keys messed up – such as, apostrophe key is now dead acute. And to type an actual , you need to use ‘ + space. It’s almost as if user friendliness is a sin on Linux.

Fortunately, the process of adding a layout is well described in this question on SO. One difference is that Ubuntu 17.10 uses Wayland, which doesn’t have *.xkm, so no cache cleaning required. Just create symbols file, append to evdev.xml and then switch layout to get it working.

(Update: apparently Ubuntu 18.04 with Xorg also doesn’t have *.xkm files. But you might need to re-login for language entry to display correctly in the top bar).

$ cat /usr/share/X11/xkb/symbols/us_dia
default partial alphanumeric_keys
xkb_symbols "basic" {
  name[Group1]= "US English with diacritics that do not hate you (Mac style)";
  include "us(basic)"
  key  { [ n, N, dead_tilde ] }; //ñ, etc
  key  { [ e, E, dead_acute ] }; //é, etc
  key  { [ slash, question, U00BF ] }; // ¿ - Inverted Question Mark
  key  { [ 1, exclam, U00A1 ] }; // ¡ - Inverted Exclamation Mark
  key  { [ u, U, dead_diaeresis ] }; // ü, etc
  include "level3(lalt_switch)"
};

$ diff -u /usr/share/X11/xkb/rules/evdev.xml.2018-06-06 /usr/share/X11/xkb/rules/evdev.xml
--- /usr/share/X11/xkb/rules/evdev.xml.2018-06-06 2018-06-06 21:08:14.900404106 +0700
+++ /usr/share/X11/xkb/rules/evdev.xml 2018-06-08 16:06:16.927002821 +0700
@@ -1318,6 +1318,19 @@
    </model>
  </modelList>
   <layoutList>
+
+     <layout>
+       <configItem>
+         <name>us_dia</name>
+         <shortDescription>en</shortDescription>
+         <description>US English with diacritics that do not hate you (Mac style)</description>
+         <languageList>
+           <iso639Id>eng</iso639Id>
+         </languageList>
+       </configItem>
+       <variantList></variantList>
+     </layout>
+
     <layout>
       <configItem>
         <name>us</name>

Update 2: added inverted punctuation characters and diaeresis.

Forgotten KeePassX password

Stages of having forgotten KeePassX master password:

  1. Denial
  2. Anger
  3. Depression
  4. BRUTFORCE

Faling to open your KeePassX database, while being quite sure that you enter correct password is a very frustrating experience.

Fortunately, while it’s hard to brute an unknown password, it’s much easier to do that when you almost have it right, but have doubts about 1 or 2 characters. See an example script on Github.

Using Jenkins as an automation dashboard, part 2

Continuing on using Jenkins as a dashboard, I’d like to highlight some things that are essential to ensure reliable job execution:

  1. Always use “set -xeu -o pipefail” and the first line of your bash script (or its equivalent if you use another language for the build script. You must be sure that every non-zero exit code triggers a notification.
    Some tools may not conform to standard and return 0 exit code even when there are errors. In that case, use Log parser plugin, to search for errors and warnings in the output and possibly fail the build based on that.
  2. Always use Build timeout plugin, and set reasonable timeouts for the builds, with failure on timeout. It will ensure that you don’t have hanged jobs.
    Also add Timestamper plugin, which will log execution time of every command. Useful for debugging.
  3. One more pitfall is that a slave may get disconnected. In that case, no job will run and no notification will be sent. Make sure it doesn’t happen with a special job.

This cover almost all possible cases when a job doesn’t run and the administrator doesn’t get a notification. (2 remaining issues are reported to Jenkins Jira: one, two – vote and comment if you want them to be fixed).

That’s it, you’ve been warned. If you still use cron for critical tasks such as backups, don’t act surprised next time when you discover they’ve been failing for half a year.

Using Jenkins as an automation dashboard, part 1

email_crontab

The most common automation tool on Unix-like system is cron. It seems to be convenient at first, but it has a number of drawbacks. There are just too many things to worry about:

  1. If some command fails, there’s no notification.
  2. A command might not fail, but hang. No notification about that either.
  3. And in addition to that, a slow running or hanged command might cause multiple instances of the script simultaneuosly. Which, in some cases, might be disastrous. So you have to add lock files, pid files, etc.
  4. Even if you do add a notification in the script, it might not be sent due to, say, mail server being down.
  5. You need to store script logs. And to rotate them, too.
  6. And, if something goes wrong, and you do get a notification, then you have to log into the problem server, find the log, search it for problems, to see what’s really happened.

All this isn’t impossible to solve with enough scripting. But it’s tedious and time consuming.

It would be rather nice to have a bird’s eye view of your tasks instead, wouldn’t it? Current state, logs, history of launches, launch on demand, execution time control, chaining tasks, etc – all one click away in a browser? Well, Jenkins CI is just that. It solves all those problems.

Move any task that you care about to Jenkins, and sleep well – you will know when there are issues. It’s so much more convenient than scavenging servers for logs.

Stay tuned for second part – proper Jenkins configuration for automation.