Category: automation

Ansible filter list based on attributes

Suppose we want to find only users who have a middle name from this:

users:
  - name: John
    surname: Johnson
  - name: Alice
    surname: Wonderland
  - name: Bob
    surname: Rabbit
    middlename: Bebop

Old, verbose way would be:

- debug: msg="{{ item.middlename }}"
  loop: "{{ users }}"
  when: item.middlename is defined

This results in “skipping” lines polluting the output, however. To avoid this, use

- debug: msg="{{ item.middlename }}"
  loop: "{{ users | selectattr('middlename','defined') | list }}"

This, however, grows unwieldy when you need to chain many conditions. But remember that you can break lines, that’ll work just as well:

- debug: msg="{{ item.middlename }}"
  loop: "{{ users
    | selectattr('middlename','defined')
    | list }}"

Curating cron emails

As you might know, cron captures all the output of executed tasks and mails them to the user under which the tasks are executed. The problem is that often this mail just piles up somewhere in /var/mail directory, without being ever reviewed. It’s not a good practice, akin to sweeping the trash under the carpet.

The good practice is:

  1. Use a separate log file for each cron task.
  2. Redirect all output to the log, and additionally print stderr to terminal:
    exec 3>&1
    exec > >(tee -a ${LOGFILE} >/dev/null) 2> >(tee -a ${LOGFILE} >&3)
  3. Capture all that mail and one by one work out all the quirks in your scripts.

Capturing is usually done with something like “root: [email protected]” in /etc/aliases. However, it’s error prone, as various packages add their own accounts and that mail won’t go to root. Manually adding entries is also subject to human errors. So what we’re looking for is wildcard match. Unfortunately, that is not trivial with some email servers. I suggest you do yourself a favor and install Exim, instead of Sendmail or Postfix.

In Exim config (/etc/exim/exim.conf or RedHat based distros, /etc/exim4/exim4.conf.template on Debian based) find system_aliases stanza, and make the change:

-data = ${lookup{$local_part}lsearch{/etc/aliases}}
+data = ${lookup{$local_part}wildlsearch{/etc/aliases}}

Then, add wildcard to /etc/aliases:

*: [email protected]

Restart Exim, and prepare for polishing your cronjobs.

Unfortunately, even with this, cron tasks are still prone to various issues, so for important jobs it’s better to use something more reliable. (But you should capture cron emails, still. They often point out non-obvious issues.)

Forgotten KeePassX password

Stages of having forgotten KeePassX master password:

  1. Denial
  2. Anger
  3. Depression
  4. BRUTFORCE

Faling to open your KeePassX database, while being quite sureĀ that you enter correct password is a very frustrating experience.

Fortunately, while it’s hard to brute an unknown password, it’s much easier to do that when you almost have it right, but have doubts about 1 or 2 characters. See an example script on Github.

Using Jenkins as an automation dashboard, part 2

Continuing on using Jenkins as a dashboard, I’d like to highlight some things that are essential to ensure reliable job execution:

  1. Always use “set -xeu -o pipefail” and the first line of your bash script (or its equivalent if you use another language for the build script. You must be sure that every non-zero exit code triggers a notification.
    Some tools may not conform to standard and return 0 exit code even when there are errors. In that case, use Log parser plugin, to search for errors and warnings in the output and possibly fail the build based on that.
  2. Always use Build timeout plugin, and set reasonable timeouts for the builds, with failure on timeout. It will ensure that you don’t have hanged jobs.
    Also add Timestamper plugin, which will log execution time of every command. Useful for debugging.
  3. One more pitfall is that a slave may get disconnected. In that case, no job will run and no notification will be sent. Make sure it doesn’t happen with a special job.

This cover almost all possible cases when a job doesn’t run and the administrator doesn’t get a notification. (2 remaining issues are reported to Jenkins Jira: one, two – vote and comment if you want them to be fixed).

That’s it, you’ve been warned. If you still use cron for critical tasks such as backups, don’t act surprised next time when you discover they’ve been failing for half a year.

Using Jenkins as an automation dashboard, part 1

email_crontab

The most common automation tool on Unix-like system is cron. It seems to be convenient at first, but it has a number of drawbacks. There are just too many things to worry about:

  1. If some command fails, there’s no notification.
  2. A command might not fail, butĀ hang. No notification about that either.
  3. And in addition to that, a slow running or hanged command might cause multiple instances of the script simultaneuosly. Which, in some cases, might be disastrous. So you have to add lock files, pid files, etc.
  4. Even if you do add a notification in the script, it might not be sent due to, say, mail server being down.
  5. You need to store script logs. And to rotate them, too.
  6. And, if something goes wrong, and you do get a notification, then you have to log into the problem server, find the log, search it for problems, to see what’s really happened.

All this isn’t impossible to solve with enough scripting. But it’s tedious and time consuming.

It would be rather nice to have a bird’s eye view of your tasks instead, wouldn’t it? Current state, logs, history of launches, launch on demand, execution time control, chaining tasks, etc – all one click away in a browser? Well, Jenkins CI is just that. It solves all those problems.

Move any task that you care about to Jenkins, and sleep well – you will know when there are issues. It’s so much more convenient than scavenging servers for logs.

Stay tuned for second part – proper Jenkins configuration for automation.