<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="4.3.4">Jekyll</generator><link href="https://devopsx.com/feed.xml" rel="self" type="application/atom+xml" /><link href="https://devopsx.com/" rel="alternate" type="text/html" /><updated>2025-11-15T07:46:39+00:00</updated><id>https://devopsx.com/feed.xml</id><title type="html">DevOps eXtraordinaire</title><subtitle>All things Linux, DevOps and automation</subtitle><author><name>Ilya Ivanov</name><email>ws9@devopsx.com</email></author><entry><title type="html">Basic auth from victoria-metrics-k8s-stack to remote VictoriaMetrics</title><link href="https://devopsx.com/victoria-metrics-k8s-stack-basic-auth" rel="alternate" type="text/html" title="Basic auth from victoria-metrics-k8s-stack to remote VictoriaMetrics" /><published>2025-02-22T08:00:00+00:00</published><updated>2025-02-22T08:00:00+00:00</updated><id>https://devopsx.com/victoria-metrics-k8s-stack-basic-auth</id><content type="html" xml:base="https://devopsx.com/victoria-metrics-k8s-stack-basic-auth"><![CDATA[<p>Victoria Metrics has a <a href="https://github.com/VictoriaMetrics/helm-charts/tree/master/charts/victoria-metrics-k8s-stack">complete Helm chart</a> to install everything together into a k8s cluster. It’s all great, except that it’s not very smart to install a monitoring solution into the same system that it’s supposed to monitor.</p>

<p>Fortunately, Victoria also provides a <a href="https://github.com/VictoriaMetrics/VictoriaMetrics/tree/master/deployment/docker">docker compose stack</a> that can easily be launched elsewhere. That shouldn’t pose any issues, so we’ll assume you have it running.</p>

<p>But you’ll probably want to add some authentication, just in case. Easiest way to do that is basic auth, with, say, Caddy.
Adjust <a href="https://github.com/VictoriaMetrics/VictoriaMetrics/blob/master/deployment/docker/docker-compose.yml">compose.yml</a> like this</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  <span class="na">caddy</span><span class="pi">:</span>
    <span class="na">image</span><span class="pi">:</span> <span class="s">caddy:2.9.1</span>
    <span class="na">ports</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s2">"</span><span class="s">80:80"</span>
      <span class="pi">-</span> <span class="s2">"</span><span class="s">443:443"</span>
    <span class="na">volumes</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s">./config/caddy/Caddyfile:/etc/caddy/Caddyfile</span>
      <span class="pi">-</span> <span class="s">./data/caddy/data:/data</span>
      <span class="pi">-</span> <span class="s">./data/caddy/config:/config</span>
    <span class="na">restart</span><span class="pi">:</span> <span class="s">unless-stopped</span>
</code></pre></div></div>

<p>Also, delete all other <code class="language-plaintext highlighter-rouge">ports:</code> stanzas from compose file. We don’t want them exposed, everything should go through Caddy, with auth.</p>

<p>Create a password hash:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>docker compose <span class="nb">exec </span>caddy caddy hash-password <span class="nt">--plaintext</span> <span class="s1">'mypassword123'</span>
<span class="nv">$2a$14$JEhbEw5WRBhnJhxP8t</span>/ZOOtcpBehikc1XFfMBVEOP82UYjVg8U6r.
</code></pre></div></div>

<p>And <code class="language-plaintext highlighter-rouge">./config/caddy/Caddyfile</code> can look like this</p>
<pre><code class="language-caddy">grafana.mydomain.com {
  reverse_proxy grafana:3000
}

victoriametrics.mydomain.com {
  reverse_proxy victoriametrics:8428
  basic_auth {
    k8s   $2a$14$JEhbEw5WRBhnJhxP8t/ZOOtcpBehikc1XFfMBVEOP82UYjVg8U6r.
  }
}
</code></pre>

<p>Now, <code class="language-plaintext highlighter-rouge">docker compose up -d</code>, try and log into victoriametrics.mydomain.com, verify that it’s working.</p>

<p>On the k8s side, <code class="language-plaintext highlighter-rouge">values.yaml</code> for victoria-metrics-k8s-stack is straighforward enough, disable everything handled by the remote VM instance:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">alertmanager</span><span class="pi">:</span>
  <span class="na">enabled</span><span class="pi">:</span> <span class="kc">false</span>
<span class="na">grafana</span><span class="pi">:</span>
  <span class="na">enabled</span><span class="pi">:</span> <span class="kc">false</span>
<span class="na">defaultDashboards</span><span class="pi">:</span>
  <span class="na">enabled</span><span class="pi">:</span> <span class="kc">true</span>
<span class="na">vmsingle</span><span class="pi">:</span>
  <span class="na">enabled</span><span class="pi">:</span> <span class="kc">false</span>
<span class="na">vmagent</span><span class="pi">:</span>
  <span class="na">spec</span><span class="pi">:</span>
    <span class="na">remoteWrite</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="na">url</span><span class="pi">:</span> <span class="s">https://victoriametrics.mydomain.com/api/v1/write</span>
<span class="na">victoria-metrics-operator</span><span class="pi">:</span>
  <span class="na">admissionWebhooks</span><span class="pi">:</span>
    <span class="na">certManager</span><span class="pi">:</span>
      <span class="na">enabled</span><span class="pi">:</span> <span class="kc">true</span>
<span class="na">vmalert</span><span class="pi">:</span>
  <span class="na">enabled</span><span class="pi">:</span> <span class="kc">false</span>
</code></pre></div></div>

<p>The annoying part is getting basic auth working. You may find references to <code class="language-plaintext highlighter-rouge">VM_remoteWrite_basicAuth_username</code>/<code class="language-plaintext highlighter-rouge">VM_remoteWrite_basicAuth_password</code>, <code class="language-plaintext highlighter-rouge">extraargs</code>, <code class="language-plaintext highlighter-rouge">envflag.prefix: "VM_"</code> and similar. Forget it, it doesn’t work.</p>

<p>First, create a secret:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Secret</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">victoria-basic-auth</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">Opaque</span>
<span class="na">stringData</span><span class="pi">:</span>
  <span class="na">username</span><span class="pi">:</span> <span class="s">k8s</span>
  <span class="na">password</span><span class="pi">:</span> <span class="s">mypassword123</span>
</code></pre></div></div>

<p>Then, pass it like this:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">alertmanager</span><span class="pi">:</span>
  <span class="na">enabled</span><span class="pi">:</span> <span class="kc">false</span>
<span class="na">grafana</span><span class="pi">:</span>
  <span class="na">enabled</span><span class="pi">:</span> <span class="kc">false</span>
<span class="na">defaultDashboards</span><span class="pi">:</span>
  <span class="na">enabled</span><span class="pi">:</span> <span class="kc">true</span>
<span class="na">vmsingle</span><span class="pi">:</span>
  <span class="na">enabled</span><span class="pi">:</span> <span class="kc">false</span>
<span class="na">vmagent</span><span class="pi">:</span>
  <span class="na">spec</span><span class="pi">:</span>
    <span class="na">remoteWrite</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="na">url</span><span class="pi">:</span> <span class="s">https://victoriametrics.mydomain.com/api/v1/write</span>
        <span class="na">basicAuth</span><span class="pi">:</span>
          <span class="na">username</span><span class="pi">:</span>
            <span class="na">name</span><span class="pi">:</span> <span class="s">victoria-basic-auth</span>
            <span class="na">key</span><span class="pi">:</span> <span class="s">username</span>
          <span class="na">password</span><span class="pi">:</span>
            <span class="na">name</span><span class="pi">:</span> <span class="s">victoria-basic-auth</span>
            <span class="na">key</span><span class="pi">:</span> <span class="s">password</span>
<span class="na">victoria-metrics-operator</span><span class="pi">:</span>
  <span class="na">admissionWebhooks</span><span class="pi">:</span>
    <span class="na">certManager</span><span class="pi">:</span>
      <span class="na">enabled</span><span class="pi">:</span> <span class="kc">true</span>
<span class="na">vmalert</span><span class="pi">:</span>
  <span class="na">enabled</span><span class="pi">:</span> <span class="kc">false</span>
</code></pre></div></div>

<p>And that’s it, VMagent in k8s should now be able to send metrics properly. Verify with <code class="language-plaintext highlighter-rouge">kubectl -n vm logs vmagent-vm-victoria-metrics-k8s-stack-6fb7bb75cb-fewyf vmagent</code> or similar.</p>]]></content><author><name>Ilya Ivanov</name><email>ws9@devopsx.com</email></author><category term="kubernetes" /><category term="victoriametrics" /><summary type="html"><![CDATA[Victoria Metrics has a complete Helm chart to install everything together into a k8s cluster. It’s all great, except that it’s not very smart to install a monitoring solution into the same system that it’s supposed to monitor.]]></summary></entry><entry><title type="html">Github Torrent Action</title><link href="https://devopsx.com/github-torrent-action/" rel="alternate" type="text/html" title="Github Torrent Action" /><published>2024-08-25T09:11:00+00:00</published><updated>2024-08-25T09:11:00+00:00</updated><id>https://devopsx.com/github-torrent-action</id><content type="html" xml:base="https://devopsx.com/github-torrent-action/"><![CDATA[<p>Here’s something I’ve been meaning to get to for a while, and finally got a chance. Enter <a href="https://github.com/devopsx/action-torrent">GitHub Torrent Action</a>!</p>

<p>This lightweight action allows you to create and publish torrents for your release assets. Can come in handy if the files are large.</p>

<p>While GitHub does not limit bandwidth usage for releases, it doesn’t mean that they guarantee fast download speeds at all times. The action itself uses the assets’ HTTP links as web seeds, so, download time wise, the result should be a net win anyway.</p>

<p>Example configuration:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">name</span><span class="pi">:</span> <span class="s">Release</span>

<span class="na">on</span><span class="pi">:</span>
  <span class="na">push</span><span class="pi">:</span>
    <span class="na">tags</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s2">"</span><span class="s">v*"</span>

<span class="na">jobs</span><span class="pi">:</span>
  <span class="na">release</span><span class="pi">:</span>
    <span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
    <span class="na">permissions</span><span class="pi">:</span>
      <span class="na">contents</span><span class="pi">:</span> <span class="s">write</span>
    <span class="na">steps</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Checkout repository</span>
        <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v4</span>

        <span class="c1"># Create torrents first</span>
      <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Create torrents</span>
        <span class="na">uses</span><span class="pi">:</span> <span class="s">devopsx/action-torrent@v1</span>
        <span class="na">with</span><span class="pi">:</span>
          <span class="na">token</span><span class="pi">:</span> <span class="s">${{ secrets.GITHUB_TOKEN }}</span>
          <span class="na">files</span><span class="pi">:</span> <span class="pi">|</span>
            <span class="s">dist/index.js</span>
            <span class="s">src/*</span>

        <span class="c1"># And then upload torrents together with corresponding assets</span>
      <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Release</span>
        <span class="na">uses</span><span class="pi">:</span> <span class="s">softprops/action-gh-release@v2</span>
        <span class="na">with</span><span class="pi">:</span>
          <span class="na">token</span><span class="pi">:</span> <span class="s">${{ secrets.GITHUB_TOKEN }}</span>
          <span class="na">name</span><span class="pi">:</span> <span class="s">${{ github.ref_name }}</span>
          <span class="na">fail_on_unmatched_files</span><span class="pi">:</span> <span class="kc">true</span>
          <span class="na">files</span><span class="pi">:</span> <span class="pi">|</span> <span class="c1"># same as above, but with addition of torrents/* directory</span>
            <span class="s">dist/index.js</span>
            <span class="s">src/*</span>
            <span class="s">torrents/*</span>
</code></pre></div></div>

<p>Here we create a torrent for <code class="language-plaintext highlighter-rouge">dist/index.js</code>, and for every file in <code class="language-plaintext highlighter-rouge">src/</code>, then release the assets and the corresponding torrents together.</p>

<p>This the “local” mode, meaning torrents are generated before the release. There’s also a “remote” mode, which finds release assets automatically. You’ll find more about it in the <a href="https://github.com/devopsx/action-torrent?tab=readme-ov-file#remote">readme</a>.</p>

<p>Good luck, and keep seeding!</p>]]></content><author><name>Ilya Ivanov</name><email>ws9@devopsx.com</email></author><category term="github" /><category term="torrent" /><category term="magnet" /><category term="github-actions" /><category term="automation" /><summary type="html"><![CDATA[Here’s something I’ve been meaning to get to for a while, and finally got a chance. Enter GitHub Torrent Action!]]></summary></entry><entry><title type="html">Install Karpenter with ArgoCD</title><link href="https://devopsx.com/install-karpenter-with-argocd" rel="alternate" type="text/html" title="Install Karpenter with ArgoCD" /><published>2024-02-15T16:13:00+00:00</published><updated>2024-02-15T16:13:00+00:00</updated><id>https://devopsx.com/argocd-karpenter</id><content type="html" xml:base="https://devopsx.com/install-karpenter-with-argocd"><![CDATA[<p>Today we are going to install <a href="https://karpenter.sh">Karpenter</a> with <a href="https://argo-cd.readthedocs.io/en/stable/">ArgoCD</a>.</p>

<p><em>This article assumes that you already have ArgoCD working.</em></p>

<p>The first step to installing a new application with Argo always is to look for an exising Helm chart. If there is one, installing it usually is a no-brainer. Just add up an application, provide repository and chart urls, done.</p>

<p>However, the Karpenter team made a <a href="https://github.com/aws/karpenter-provider-aws/issues/3375#issuecomment-1496572209">decision</a> to drop Helm repository support at <code class="language-plaintext highlighter-rouge">https://charts.karpenter.sh/</code> and instead publish it in an <a href="https://helm.sh/docs/topics/registries/">OCI repository</a> (which turns out to be a thing) hosted on Amazon ECR.</p>

<p>The problem is, ArgoCD <a href="https://github.com/aws/karpenter-provider-aws/issues/4493">will not</a> resolve <code class="language-plaintext highlighter-rouge">oci://</code> url for Helm repository. At least, <a href="https://github.com/aws/karpenter-provider-aws/issues/3375">not without authentication</a>. Even if it’s a public repository. Joyful.</p>

<p>Of course, if you have private charts in ECR, <a href="https://github.com/argoproj/argo-cd/issues/8097">there’s not much choice</a>, you need a token. Someone even put together an <a href="https://github.com/smcavallo/argocd-ecr-updater">automation</a> for refreshing ECR tokens.</p>

<p>However, if you want just the public ones, there’s a workaround. We will chain the charts and let Helm do the heavy lifting.</p>

<p>First, create a wrapper application:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">argoproj.io/v1alpha1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Application</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">karpenter-root</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">argocd</span>
  <span class="na">finalizers</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="s">resources-finalizer.argocd.argoproj.io</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">destination</span><span class="pi">:</span>
    <span class="na">namespace</span><span class="pi">:</span> <span class="s">karpenter</span>
    <span class="na">server</span><span class="pi">:</span> 
  <span class="na">project</span><span class="pi">:</span> <span class="s">default</span>
  <span class="na">source</span><span class="pi">:</span>
    <span class="na">repoURL</span><span class="pi">:</span> 
    <span class="na">targetRevision</span><span class="pi">:</span> <span class="s">HEAD</span>
    <span class="na">path</span><span class="pi">:</span> <span class="s">helm/karpenter-root</span>
    <span class="na">helm</span><span class="pi">:</span>
      <span class="na">valuesObject</span><span class="pi">:</span>
        <span class="na">karpenter</span><span class="pi">:</span>
          <span class="na">serviceAccount</span><span class="pi">:</span>
            <span class="na">annotations</span><span class="pi">:</span>
              <span class="na">eks.amazonaws.com/role-arn</span><span class="pi">:</span> <span class="s2">"</span><span class="s">arn:aws:iam::something/something"</span>
          <span class="na">settings</span><span class="pi">:</span>
            <span class="na">clusterName</span><span class="pi">:</span> <span class="s2">"</span><span class="s">mycluster"</span>
            <span class="na">interruptionQueue</span><span class="pi">:</span> <span class="s2">"</span><span class="s">Karpenter-mycluster"</span>
  <span class="na">syncPolicy</span><span class="pi">:</span>
    <span class="na">automated</span><span class="pi">:</span>
      <span class="na">prune</span><span class="pi">:</span> <span class="kc">true</span>
      <span class="na">selfHeal</span><span class="pi">:</span> <span class="kc">true</span>
    <span class="na">syncOptions</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s">CreateNamespace=true</span>
</code></pre></div></div>

<p>This configuration expects a custom Helm chart in <code class="language-plaintext highlighter-rouge">helm/karpenter-root</code> in the same repository. Also note some minimal settings passed to Helm as <code class="language-plaintext highlighter-rouge">valuesObject</code>.</p>

<p>Our Helm chart in <code class="language-plaintext highlighter-rouge">helm/karpenter-root/Chart.yaml</code> is going to be simplistic:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v2</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">karpenter-root</span>
<span class="na">description</span><span class="pi">:</span> <span class="s">Karpenter parent chart</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">application</span>

<span class="na">version</span><span class="pi">:</span> <span class="s">0.1.0</span>
<span class="na">appVersion</span><span class="pi">:</span> <span class="s2">"</span><span class="s">1.0.0"</span>

<span class="na">dependencies</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">karpenter</span>
    <span class="na">version</span><span class="pi">:</span> <span class="s2">"</span><span class="s">v0.34.0"</span>
    <span class="c1"># TAKE THAT, KARPENTER TEAM! NO TOKENS FOR YOU!!</span>
    <span class="na">repository</span><span class="pi">:</span> <span class="s2">"</span><span class="s">oci://public.ecr.aws/karpenter"</span>
</code></pre></div></div>

<p>Also add values to <code class="language-plaintext highlighter-rouge">helm/karpenter-root/values.yaml</code>:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">karpenter</span><span class="pi">:</span>
  <span class="na">serviceAccount</span><span class="pi">:</span>
    <span class="na">annotations</span><span class="pi">:</span>
      <span class="na">eks.amazonaws.com/role-arn</span><span class="pi">:</span> <span class="s2">"</span><span class="s">none"</span>
  <span class="na">settings</span><span class="pi">:</span>
    <span class="na">clusterName</span><span class="pi">:</span> <span class="s2">"</span><span class="s">none"</span>
    <span class="na">interruptionQueue</span><span class="pi">:</span> <span class="s2">"</span><span class="s">none"</span>
</code></pre></div></div>

<p>To pass more values, you’ll need to add them to both Application definition and <code class="language-plaintext highlighter-rouge">values.yaml</code>, but that’s a small price to pay.</p>

<p>That’s all, folks!</p>]]></content><author><name>Ilya Ivanov</name><email>ws9@devopsx.com</email></author><category term="kubernetes" /><category term="argocd" /><category term="karpenter" /><category term="aws" /><category term="eks" /><summary type="html"><![CDATA[Today we are going to install Karpenter with ArgoCD.]]></summary></entry><entry><title type="html">Traefik ingress: redirect HTTP to HTTPS</title><link href="https://devopsx.com/traefik-ingress-redirect-http-to-https/" rel="alternate" type="text/html" title="Traefik ingress: redirect HTTP to HTTPS" /><published>2024-01-15T07:18:00+00:00</published><updated>2024-01-15T07:18:00+00:00</updated><id>https://devopsx.com/traefik-ingress</id><content type="html" xml:base="https://devopsx.com/traefik-ingress-redirect-http-to-https/"><![CDATA[<p>Let’s say we need to set up HTTP to HTTPS redirection in Traefik ingress.</p>

<p><em>This article assumes that you already have Traefik ingress installed and HTTPS working.</em></p>

<p>A typical virtual host for service <code class="language-plaintext highlighter-rouge">frontend</code> with domain <code class="language-plaintext highlighter-rouge">sub.domain.example</code> looks something like this:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">networking.k8s.io/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Ingress</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">ingress-traefik</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">frontend</span>
  <span class="na">annotations</span><span class="pi">:</span>
    <span class="na">traefik.ingress.kubernetes.io/router.entrypoints</span><span class="pi">:</span> <span class="s">websecure</span>
    <span class="na">kubernetes.io/ingress.class</span><span class="pi">:</span> <span class="s">traefik</span>
  <span class="na">labels</span><span class="pi">:</span>
    <span class="na">app</span><span class="pi">:</span> <span class="s">frontend</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">tls</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="na">hosts</span><span class="pi">:</span>
        <span class="pi">-</span> <span class="s">sub.domain.example</span>
      <span class="na">secretName</span><span class="pi">:</span> <span class="s">tls</span>
  <span class="na">rules</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="na">host</span><span class="pi">:</span> <span class="s">sub.domain.example</span>
      <span class="na">http</span><span class="pi">:</span>
        <span class="na">paths</span><span class="pi">:</span>
          <span class="pi">-</span> <span class="na">path</span><span class="pi">:</span> <span class="s">/</span>
            <span class="na">pathType</span><span class="pi">:</span> <span class="s">Prefix</span>
            <span class="na">backend</span><span class="pi">:</span>
              <span class="na">service</span><span class="pi">:</span>
                <span class="na">name</span><span class="pi">:</span> <span class="s">frontend</span>
                <span class="na">port</span><span class="pi">:</span>
                  <span class="na">number</span><span class="pi">:</span> <span class="m">80</span>
</code></pre></div></div>

<p>Now, you’d like to have an automatic HTTP =&gt; HTTPS redirect for this host. Unfortunately, Trafik ingress does not provide an easy to do that with annotations. Instead, it offers <a href="https://doc.traefik.io/traefik/middlewares/overview/">middlewares</a>, in particular <a href="https://doc.traefik.io/traefik/middlewares/http/redirectscheme/">RedirectScheme</a>.</p>

<p>However, RedirectScheme is a) indiscriminate and b) in some cases relies on <code class="language-plaintext highlighter-rouge">X-Forwarded</code> headers, which isn’t a solid choice.</p>

<p>Instead, we can use <a href="https://doc.traefik.io/traefik/middlewares/http/redirectregex/">RedirectRegex</a> middleware. Gating with virtual host name will allow to apply it selectively.</p>

<p>First, create the middleware in <strong>Traefik’s</strong> namespace (usually <code class="language-plaintext highlighter-rouge">traefik</code>):</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># http =&gt; https redirect middleware</span>
<span class="nn">---</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">traefik.io/v1alpha1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Middleware</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">redirect-to-https</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">traefik</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">redirectRegex</span><span class="pi">:</span>
    <span class="na">regex</span><span class="pi">:</span> <span class="s2">"</span><span class="s">^http://(.*)"</span>
    <span class="na">replacement</span><span class="pi">:</span> <span class="s2">"</span><span class="s">https://$1"</span>
    <span class="na">permanent</span><span class="pi">:</span> <span class="kc">true</span>
</code></pre></div></div>

<p>Second, add another ingress configuration to the application namespace, listening on port 80. It’s pretty much a copy of the HTTPS one, just different entrypoint and no SSL configuration:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nn">---</span>
<span class="na">apiVersion</span><span class="pi">:</span> <span class="s">networking.k8s.io/v1</span>
<span class="na">kind</span><span class="pi">:</span> <span class="s">Ingress</span>
<span class="na">metadata</span><span class="pi">:</span>
  <span class="na">name</span><span class="pi">:</span> <span class="s">ingress-traefik-http</span>
  <span class="na">namespace</span><span class="pi">:</span> <span class="s">frontend</span>
  <span class="na">annotations</span><span class="pi">:</span>
    <span class="na">traefik.ingress.kubernetes.io/router.entrypoints</span><span class="pi">:</span> <span class="s">web</span>
    <span class="na">traefik.ingress.kubernetes.io/router.middlewares</span><span class="pi">:</span> <span class="s">traefik-redirect-to-https@kubernetescrd</span>
    <span class="na">kubernetes.io/ingress.class</span><span class="pi">:</span> <span class="s">traefik</span>
<span class="na">spec</span><span class="pi">:</span>
  <span class="na">rules</span><span class="pi">:</span>
    <span class="pi">-</span> <span class="na">host</span><span class="pi">:</span>
      <span class="na">http</span><span class="pi">:</span> <span class="s">sub.domain.example</span>
        <span class="s">paths</span><span class="err">:</span>
          <span class="pi">-</span> <span class="na">path</span><span class="pi">:</span> <span class="s">/</span>
            <span class="na">pathType</span><span class="pi">:</span> <span class="s">Prefix</span>
            <span class="na">backend</span><span class="pi">:</span>
              <span class="na">service</span><span class="pi">:</span>
                <span class="na">name</span><span class="pi">:</span> <span class="s">frontend</span>
                <span class="na">port</span><span class="pi">:</span>
                  <span class="na">number</span><span class="pi">:</span> <span class="m">80</span>
</code></pre></div></div>

<p>That’s it! Should work now.</p>]]></content><author><name>Ilya Ivanov</name><email>ws9@devopsx.com</email></author><category term="kubernetes" /><category term="ingress" /><category term="traefik" /><category term="https" /><summary type="html"><![CDATA[Let’s say we need to set up HTTP to HTTPS redirection in Traefik ingress.]]></summary></entry><entry><title type="html">How to get realip to work with ingress-nginx and ELB</title><link href="https://devopsx.com/how-to-get-realip-to-work-with-ingress-nginx-and-elb/" rel="alternate" type="text/html" title="How to get realip to work with ingress-nginx and ELB" /><published>2023-11-28T05:34:00+00:00</published><updated>2023-11-28T05:34:00+00:00</updated><id>https://devopsx.com/How-to-enable-proxy-protocol</id><content type="html" xml:base="https://devopsx.com/how-to-get-realip-to-work-with-ingress-nginx-and-elb/"><![CDATA[<p><strong>Environment</strong>: AWS EKS, <a href="https://github.com/kubernetes/ingress-nginx">Ingress-Nginx</a> controller installed, and an Ingress resource is configured, passing requests to a backend http service.</p>

<p><strong>Problem</strong>: backend service can’t see real client IP addresses. It sees internal VPC addresses instead.</p>

<p><strong>Goal</strong>: let ingress-nginx pass client ip addresses to backend.</p>

<p>By default, ingress-nginx-controller installed into EKS will create an ELB. It also allows to use NLB and ALB.
But that requires installation of <a href="https://github.com/kubernetes-sigs/aws-load-balancer-controller">AWS load balancer controller</a>, and who wants an additional dependency that can break during an upgrade?
So we’ll stick with ELB.</p>

<p>The created ELB passes traffic at L4. To pass client IPs, we will need to configure <a href="https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#proxy-protocol">proxy-protocol</a>. Here’s <a href="https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable-proxy-protocol.html">AWS documentation</a>. ELB uses protocol v1, while ALB uses v2, but Nginx seemingly speaks both, so that shouldn’t be a problem.</p>

<p>Check your ingress controller</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl <span class="nt">-n</span> ingress-nginx get svc
NAME                                            TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT<span class="o">(</span>S<span class="o">)</span>                      AGE
ingress-nginx-controller-controller             LoadBalancer   172.20.124.111   c43c1730102072835b21c6af3cede412-886572035.us-east-1.elb.amazonaws.com   80:32131/TCP,443:31739/TCP   15d
</code></pre></div></div>

<p>Note <code class="language-plaintext highlighter-rouge">c43c1730102072835b21c6af3cede412</code> - that is the load balancer name. You can see it in AWS console in EC2 load balancers.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">export </span><span class="nv">lbname</span><span class="o">=</span>c43c1730102072835b21c6af3cede412
</code></pre></div></div>

<p>As per AWS doc, create policy</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aws elb create-load-balancer-policy <span class="nt">--load-balancer-name</span> <span class="nv">$lbname</span> <span class="nt">--policy-name</span> ProxyProtocolEnable <span class="nt">--policy-type-name</span> ProxyProtocolPolicyType <span class="nt">--policy-attributes</span> <span class="nv">AttributeName</span><span class="o">=</span>ProxyProtocol,AttributeValue<span class="o">=</span><span class="nb">true</span>
</code></pre></div></div>

<p>Now the tricky part, enable policy. Note that policy should be set on <strong>target ports</strong>. Not 80 and 443, but the paired ones.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>aws elb set-load-balancer-policies-for-backend-server <span class="nt">--load-balancer-name</span> <span class="nv">$lbname</span> <span class="nt">--instance-port</span> 32131 <span class="nt">--policy-names</span> ProxyProtocolEnable
aws elb set-load-balancer-policies-for-backend-server <span class="nt">--load-balancer-name</span> <span class="nv">$lbname</span> <span class="nt">--instance-port</span> 31739 <span class="nt">--policy-names</span> ProxyProtocolEnable
</code></pre></div></div>

<p>Enable proxy protocol in ingress-nginx controller</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl <span class="nt">-n</span> ingress-nginx edit cm ingress-nginx-controller-controller
</code></pre></div></div>

<p>Ensure <code class="language-plaintext highlighter-rouge">use-proxy-protocol: "true"</code> is set:</p>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">apiVersion</span><span class="pi">:</span> <span class="s">v1</span>
<span class="na">data</span><span class="pi">:</span>
  <span class="na">use-proxy-protocol</span><span class="pi">:</span> <span class="s2">"</span><span class="s">true"</span>
<span class="nn">...</span>
</code></pre></div></div>

<p>At this point, ELB should pass client IPs to ingress-nginx. To verify, check ingress-nginx logs.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>kubectl <span class="nt">-n</span> ingress-nginx get pods
NAME                                                  READY   STATUS    RESTARTS   AGE
ingress-nginx-controller-controller-931aa6241-3a1f1   1/1     Running   0          18d

curl <span class="nt">-s</span> http://my-backend-service

kubectl <span class="nt">-n</span> ingress-nginx logs ingress-nginx-controller-controller-931aa6241-3a1f1 <span class="nt">--tail</span><span class="o">=</span>3
</code></pre></div></div>]]></content><author><name>Ilya Ivanov</name><email>ws9@devopsx.com</email></author><category term="aws" /><category term="kubernetes" /><category term="ingress-nginx" /><category term="nginx" /><category term="elb" /><summary type="html"><![CDATA[Environment: AWS EKS, Ingress-Nginx controller installed, and an Ingress resource is configured, passing requests to a backend http service.]]></summary></entry><entry><title type="html">Terraform, Lambda, source_code_hash</title><link href="https://devopsx.com/terraform-lambda-source-code-hash/" rel="alternate" type="text/html" title="Terraform, Lambda, source_code_hash" /><published>2023-06-01T11:41:00+00:00</published><updated>2023-06-01T11:41:00+00:00</updated><id>https://devopsx.com/Terraform-Lambda</id><content type="html" xml:base="https://devopsx.com/terraform-lambda-source-code-hash/"><![CDATA[<p>Oh, no! Terraform deploys Lambda again!</p>

<div class="language-diff highlighter-rouge"><div class="highlight"><pre class="highlight"><code>~ resource "aws_lambda_function" "myfunc" {
      id                             = "myfunc"
      ...
    ~ source_code_hash               = "eqFDS7dB1KoQEAC6CNQ2ZAY1tV0ghf836eCQZwnWlpc=" -&gt; "vnjjxeDfV108KVscceyBBxsRGYin4CPOWqgWvgWTVOo="
</code></pre></div></div>

<h2 id="problem">Problem</h2>

<p>Terraform and AWS Lambda have a long history of bad blood: <a href="https://github.com/hashicorp/terraform-provider-aws/issues/17989">1</a>, <a href="https://github.com/hashicorp/terraform-provider-aws/issues/7385">2</a>. The problem is that Lambda appears to be perpetually in dirty state, due to <code class="language-plaintext highlighter-rouge">source_code_hash</code> differences.</p>

<p>The root cause of is this is difference in packaging on different machines and bad documentation. Well, and an asinine design choice on AWS part.</p>

<ol>
  <li><code class="language-plaintext highlighter-rouge">source_code_hash</code> gets <a href="https://github.com/hashicorp/terraform-provider-aws/issues/7385#issuecomment-728596589">overwritten</a> by AWS-provided data upon response.</li>
  <li>The documentation for <a href="https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_function#source_code_hash">source_code_hash</a> (aka <code class="language-plaintext highlighter-rouge">output_base64sha256</code>, <code class="language-plaintext highlighter-rouge">filebase64sha256</code>) lies:
    <blockquote>
      <p>(String) The base64-encoded SHA256 checksum of output archive file.</p>
    </blockquote>
  </li>
</ol>

<p>Why would you even want to base64-encode a hash? The purpose of base64 encoding is to do away with non-printable chars, which a hash doesn’t have.</p>

<p>Turns out, what they actually do is compute sha256, then take the resulting text <em>string</em> and treat its characters as <em>binary</em> values, then base64 that: <code class="language-plaintext highlighter-rouge">sha256sum lambda.zip | xxd -r -p | base64</code>.</p>

<p>The problem is, recent <code class="language-plaintext highlighter-rouge">zip</code> versions store file permissions, and different <code class="language-plaintext highlighter-rouge">umask</code> values on different machines result in different permissions, which in turn produces different archives with different hashes.</p>

<h2 id="solution">Solution</h2>

<p>To alleviate this, set <a href="https://registry.terraform.io/providers/hashicorp/archive/latest/docs/data-sources/file#output_file_mode">output_file_mode</a> in the corresponding <code class="language-plaintext highlighter-rouge">archive_file</code>:</p>

<div class="language-terraform highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">data</span> <span class="s2">"archive_file"</span> <span class="s2">"myfile"</span> <span class="p">{</span>
  <span class="nx">type</span>             <span class="o">=</span> <span class="s2">"zip"</span>
  <span class="nx">source_file</span>      <span class="o">=</span> <span class="s2">"</span><span class="p">${</span><span class="nx">path</span><span class="p">.</span><span class="k">module</span><span class="p">}</span><span class="s2">/src/myfunc/index.js"</span>
  <span class="nx">output_path</span>      <span class="o">=</span> <span class="s2">"myfunc.zip"</span>
  <span class="nx">output_file_mode</span> <span class="o">=</span> <span class="s2">"0644"</span>
<span class="p">}</span>

<span class="k">resource</span> <span class="s2">"aws_lambda_function"</span> <span class="s2">"myfunc"</span> <span class="p">{</span>
  <span class="nx">description</span>                    <span class="o">=</span> <span class="s2">"Example description"</span>
  <span class="nx">function_name</span>                  <span class="o">=</span> <span class="s2">"myfunc"</span>
  <span class="nx">filename</span>                       <span class="o">=</span> <span class="s2">"myfunc.zip"</span>
  <span class="nx">source_code_hash</span>               <span class="o">=</span> <span class="k">data</span><span class="p">.</span><span class="nx">archive_file</span><span class="p">.</span><span class="nx">myfile</span><span class="p">.</span><span class="nx">output_base64sha256</span>
  <span class="p">...</span>
</code></pre></div></div>

<p>If it doesn’t seem to help, add a newline to each of your lambda functions and deploy again. Sometimes AWS seems to return old <code class="language-plaintext highlighter-rouge">source_code_hash</code> if the new version only differs in file permissions (or not deploy the new version at all).
The newline will force re-deploy, and after that deployment from any machine should be working as intended.</p>]]></content><author><name>Ilya Ivanov</name><email>ws9@devopsx.com</email></author><category term="aws" /><category term="terraform" /><category term="lambda" /><summary type="html"><![CDATA[Oh, no! Terraform deploys Lambda again!]]></summary></entry><entry><title type="html">Intel GPU hang</title><link href="https://devopsx.com/intel-gpu-hang/" rel="alternate" type="text/html" title="Intel GPU hang" /><published>2022-06-01T10:41:00+00:00</published><updated>2022-06-01T10:41:00+00:00</updated><id>https://devopsx.com/Intel-GPU-hang</id><content type="html" xml:base="https://devopsx.com/intel-gpu-hang/"><![CDATA[<p>The dreaded GPU hang error:</p>

<pre><code class="language-log">[  242.891878] i915 0000:00:02.0: [drm] Resetting rcs0 for preemption time out
[  242.891888] i915 0000:00:02.0: [drm] x.exe[10957] context reset due to GPU hang
[  242.911578] i915 0000:00:02.0: [drm] GPU HANG: ecode 9:1:87f93cf9, in x.exe [10957]
</code></pre>

<p>Below is a list of possible workarounds to try.</p>

<h2 id="environment-variables">Environment variables</h2>

<h3 id="older-driver">Older driver</h3>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">export </span><span class="nv">MESA_LOADER_DRIVER_OVERRIDE</span><span class="o">=</span>i965
</code></pre></div></div>
<p><a href="https://gitlab.freedesktop.org/mesa/mesa/-/issues/2552">Source</a>.</p>

<h3 id="intel-debug">Intel debug</h3>
<p>Don’t ask…</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">export </span><span class="nv">INTEL_DEBUG</span><span class="o">=</span>reemit
</code></pre></div></div>
<p><a href="https://gitlab.freedesktop.org/mesa/mesa/-/issues/3748">Source</a>.</p>

<h2 id="increase-driver-timeout">Increase driver timeout</h2>
<p><code class="language-plaintext highlighter-rouge">echo 10000 | sudo tee /sys/class/drm/card0/engine/rcs0/preempt_timeout_ms</code>. If this works, add to startup scripts. <a href="https://gitlab.freedesktop.org/mesa/mesa/-/issues/3746">Source</a>.</p>

<h2 id="kernel-parameters">Kernel parameters</h2>

<p>On Ubuntu, edit <code class="language-plaintext highlighter-rouge">/etc/default/grub</code> and then run <code class="language-plaintext highlighter-rouge">update-grub</code> and reboot. On other distros, you’ll figure it out :).</p>

<ol>
  <li><code class="language-plaintext highlighter-rouge">GRUB_CMDLINE_LINUX_DEFAULT="i915.enable_psr=0 i915.enable_fbc=1"</code>. <a href="https://gitlab.freedesktop.org/drm/intel/-/issues/3496">Source</a>.</li>
  <li><code class="language-plaintext highlighter-rouge">GRUB_CMDLINE_LINUX_DEFAULT="drm.debug=0 drm.vblankoffdelay=1 i915.semaphores=0 i915.modeset=1 i915.use_mmio_flip=1 i915.powersave=1 i915.enable_ips=1 i915.disable_power_well=1 i915.enable_hangcheck=1 i915.enable_cmd_parser=1 i915.fastboot=0 i915.enable_ppgtt=1 i915.reset=0 i915.lvds_use_ssc=0 i915.enable_psr=0"</code>
  <a href="https://johnlewis.ie/tentative-fixwork-around-for-i915-gpu-hangs/">Source</a>.</li>
  <li><code class="language-plaintext highlighter-rouge">GRUB_CMDLINE_LINUX_DEFAULT="intel_idle.max_cstate=1 i915.enable_dc=0 ahci.mobile_lpm_policy=1"</code>. <a href="https://wiki.archlinux.org/title/intel_graphics#Crash/freeze_on_low_power_Intel_CPUs">Source</a>.</li>
  <li><code class="language-plaintext highlighter-rouge">GRUB_CMDLINE_LINUX_DEFAULT="i915.mitigations=off"</code>. <a href="https://gitlab.freedesktop.org/drm/intel/-/issues/5432">Source</a>.</li>
  <li><code class="language-plaintext highlighter-rouge">GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"</code>. <a href="https://bbs.archlinux.org/viewtopic.php?id=256520&amp;p=3">Source</a>.</li>
</ol>

<h2 id="driver-parameters">Driver parameters</h2>
<p>On Ubuntu, put into <code class="language-plaintext highlighter-rouge">/usr/share/X11/xorg.conf.d/intel.conf</code> or similar.</p>

<div class="language-conf highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">Section</span> <span class="s2">"Device"</span>
  <span class="n">Identifier</span> <span class="s2">"Intel Graphics"</span>
  <span class="n">Driver</span> <span class="s2">"intel"</span>
<span class="c">#  Option "TripleBuffer" "false"
#  Option "VSync" "false"
#  Option "PageFlip" "false"
#  Option "DRI" "2"
#  Option "AccelMethod" "UXA"
</span><span class="n">EndSection</span>
</code></pre></div></div>
<p>Try each option separately. Reboot after each change. <a href="https://man.archlinux.org/man/intel.4#CONFIGURATION_DETAILS">Source</a>.</p>]]></content><author><name>Ilya Ivanov</name><email>ws9@devopsx.com</email></author><category term="linux" /><category term="intel" /><category term="gpu" /><summary type="html"><![CDATA[The dreaded GPU hang error:]]></summary></entry><entry><title type="html">Jenkins Job Builder Github action</title><link href="https://devopsx.com/jenkins-job-builder-github-action/" rel="alternate" type="text/html" title="Jenkins Job Builder Github action" /><published>2021-07-18T22:11:00+00:00</published><updated>2021-07-18T22:11:00+00:00</updated><id>https://devopsx.com/Jenkins-Job-Builder-Github-action</id><content type="html" xml:base="https://devopsx.com/jenkins-job-builder-github-action/"><![CDATA[<p>When codifying Jenkins jobs, you have two options:</p>
<ul>
  <li><a href="https://www.digitalocean.com/community/tutorials/how-to-automate-jenkins-job-configuration-using-job-dsl">Job DSL + seed job</a>. More native to Jenkins, but requires Groovy scripting.</li>
  <li><a href="https://jenkins-job-builder.readthedocs.io/en/latest/">Jenkins Job Builder</a>. A third party solution, but configs are in yaml, so it’s easier to get started.</li>
</ul>

<p>If you prefer DSL, can stop reading now. JJB users - read on.</p>

<p>Since we’re automating all, it’d be great to have the configuration applied on a repo push. Of course, it’s possible to configure a seed job for JJB as well, but point and click is annoying. How about a <a href="https://github.com/devopsx/action-jjb">github action</a> instead?</p>

<p>Configuration is very simple:</p>

<ol>
  <li>Create Jenkins <a href="https://www.jenkins.io/blog/2018/07/02/new-api-token-system/">API token</a>.</li>
  <li>Use it to create <code class="language-plaintext highlighter-rouge">JENKINS_TOKEN</code> <a href="https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository">secret</a> in repo settings.</li>
  <li>Add workflow yaml, something like this:</li>
</ol>

<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">name</span><span class="pi">:</span> <span class="s">jjb</span>

<span class="na">on</span><span class="pi">:</span>
  <span class="na">push</span><span class="pi">:</span>
    <span class="na">paths</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s">jenkins/jobs/**</span>                             <span class="c1"># job definitions here, searched recursively</span>

<span class="na">jobs</span><span class="pi">:</span>
  <span class="na">jjb</span><span class="pi">:</span>
    <span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span>
    <span class="na">steps</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v4</span>
      <span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">devopsx/action-jjb@master</span>
        <span class="na">with</span><span class="pi">:</span>
          <span class="na">jjb_user</span><span class="pi">:</span> <span class="s">$</span>
          <span class="na">jjb_password</span><span class="pi">:</span> <span class="s">$</span>
          <span class="na">jjb_dir</span><span class="pi">:</span> <span class="s">jenkins/jobs</span>                      <span class="c1"># Same dir with definitions as in push stanza</span>
          <span class="na">jjb_ini</span><span class="pi">:</span> <span class="s">jenkins/jenkins_jobs.ini</span>          <span class="c1"># See example</span>
</code></pre></div></div>

<p>That’s it! Now your builds will be automatically updated each time you push.</p>]]></content><author><name>Ilya Ivanov</name><email>ws9@devopsx.com</email></author><category term="jenkins" /><category term="automation" /><category term="github" /><summary type="html"><![CDATA[When codifying Jenkins jobs, you have two options: Job DSL + seed job. More native to Jenkins, but requires Groovy scripting. Jenkins Job Builder. A third party solution, but configs are in yaml, so it’s easier to get started.]]></summary></entry><entry><title type="html">Ryzen: adventures in RedmiLand</title><link href="https://devopsx.com/ryzen-adventures-in-redmiland/" rel="alternate" type="text/html" title="Ryzen: adventures in RedmiLand" /><published>2020-11-19T14:11:00+00:00</published><updated>2020-11-19T14:11:00+00:00</updated><id>https://devopsx.com/Ryzen-adventures-in-RedmiLand</id><content type="html" xml:base="https://devopsx.com/ryzen-adventures-in-redmiland/"><![CDATA[<p>Greetings, fellow netizen. Today we’re going to install Ubuntu on Redmibook 16 Ryzen edition (Ryzen 4700U).</p>

<ul>
  <li><a href="#ubuntu-version">Ubuntu version</a></li>
  <li><a href="#installation">Installation</a></li>
  <li><a href="#display-brightness">Display brightness</a></li>
  <li><a href="#bluetooth">Bluetooth</a></li>
  <li><a href="#wifi">WiFi</a></li>
  <li><a href="#night-light">Night light</a></li>
  <li><a href="#fn-button">Fn button</a></li>
  <li><a href="#still-doesnt-work">Still doesn’t work</a></li>
</ul>

<h2 id="ubuntu-version">Ubuntu version</h2>

<p>First off, choosing Ubuntu version. There’s 20.04 LTS and 20.10 at this point. But the only important difference between them is kernel version, which is 5.4 in 20.04 and 5.8 in 20.10. The new kernel can be installed on LTS too, though, so we’re going to pick 20.04.</p>

<h2 id="installation">Installation</h2>

<p>Installation works properly. No issue here, just follow the standard procedure. The only strange thing is that it won’t connect to 5Ghz wifi. 2.4 Ghz works.</p>

<h2 id="display-brightness">Display brightness</h2>

<p>Right after first login, trying the brightness buttons, it turns out that they don’t work. A quick search turns up <a href="https://askubuntu.com/questions/1249152/ubuntu-20-04-lenovo-ideapad-5-ryzen-4800u-display-brightness-not-working">this</a>. Apparently, this needs kernel 5.5, and some sources even say 5.7.</p>

<p>Well, no problem. Installing a newer kernel:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt <span class="nb">install </span>linux-generic-hwe-20.04-edge
</code></pre></div></div>
<p>This gets us to 5.8 (at this point).</p>

<p>Reboot and see that brightness buttons now work!</p>

<h2 id="bluetooth">Bluetooth</h2>

<p>And right after that, another hiccup. A bluetooth mouse won’t get detected by the laptop. However, a headset works fine. Trying to do a <a href="https://wiki.archlinux.org/index.php/Bluetooth#Device_does_not_show_up_in_scan">lescan</a> yields no result.
Time to search again. Bugzilla has some <a href="https://bugzilla.kernel.org/show_bug.cgi?id=208965">info</a> with apparently the same problem. Specifically, it mentions Realtek <code class="language-plaintext highlighter-rouge">RTL8822CE</code> device.</p>

<p>Let’s see if we have that one:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lsusb
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 0cb5:c547 Realtek Bluetooth Radio
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
</code></pre></div></div>

<p>Looks like it’s one of the devices mentioned in bugzilla: <code class="language-plaintext highlighter-rouge">0cb5:c547</code>, meaning <code class="language-plaintext highlighter-rouge">RTL8822CE</code>. And here’s a <a href="https://marc.info/?l=linux-bluetooth&amp;m=160383198717911&amp;w=2">patch</a> for it. Of course, it’s not in the kernel yet, so we’ll have to do some compilation.</p>

<p>Instead of compiling the whole kernel, as suggested in the discussion, let’s try to compile <code class="language-plaintext highlighter-rouge">btusb</code> module only. Should be faster and less work.
We’ll still need to the sources for the whole kernel, of course.</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt <span class="nb">install </span>git build-essential kernel-package fakeroot libncurses5-dev libssl-dev ccache bison flex <span class="c"># some packages for building</span>
wget https://github.com/torvalds/linux/archive/v5.8.tar.gz <span class="c"># since we're on 5.8 kernel</span>
<span class="nb">tar </span>zxf v5.4.tar.gz
<span class="nb">cd </span>linux-5.4/drivers/bluetooth
vi btusb.c
</code></pre></div></div>

<p>Apply the patch, save it.</p>

<p>Then, as described <a href="https://unix.stackexchange.com/questions/199350/bluetooth-btusb-how-to-replace-module-version-with-a-newer-one">here</a> (you should still be in <code class="language-plaintext highlighter-rouge">linux-5.4/drivers/bluetooth</code>):</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>make <span class="nt">-C</span> /lib/modules/<span class="si">$(</span><span class="nb">uname</span> <span class="nt">-r</span><span class="si">)</span>/build <span class="nv">M</span><span class="o">=</span><span class="nv">$PWD</span> modules
</code></pre></div></div>

<p>Now, custom <code class="language-plaintext highlighter-rouge">btusb.ko</code> is in the same directory. Let’s try to load it:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cp</span> /usr/lib/modules/<span class="si">$(</span><span class="nb">uname</span> <span class="nt">-r</span><span class="si">)</span>/kernel/drivers/bluetooth/btusb.ko ~/btusb.ko.bak
<span class="nb">cp </span>btusb.ko /usr/lib/modules/<span class="si">$(</span><span class="nb">uname</span> <span class="nt">-r</span><span class="si">)</span>/kernel/drivers/bluetooth/btusb.ko
modprobe <span class="nt">-r</span> btusb
modprobe btusb
...could not insert btusb.ko: Operation not permitted
</code></pre></div></div>

<p>Oops. What’s wrong? Dmesg to rescue:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Lockdown: modprobe: Loading of unsigned module is restricted; see man kernel_lockdown.7
</code></pre></div></div>

<p>Riiight, so the module is not signed, and the kernel refuses to load it. Unfortunately, the only way to do that is to disable UEFI Secure Boot.
So:</p>
<ol>
  <li>Reboot.</li>
  <li>Enter BIOS (<code class="language-plaintext highlighter-rouge">F2</code>)</li>
  <li>Set a supervisor password if there isn’t one yet.</li>
  <li>Disable Secure Boot.</li>
  <li>Save and exit.</li>
</ol>

<p>Check dmesg again:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dmesg | <span class="nb">grep</span> <span class="nt">-i</span> blue
<span class="o">[</span>    1.987803] usb 1-4: Product: Bluetooth Radio
<span class="o">[</span>    9.839189] Bluetooth: Core ver 2.22
<span class="o">[</span>    9.839205] Bluetooth: HCI device and connection manager initialized
<span class="o">[</span>    9.839208] Bluetooth: HCI socket layer initialized
<span class="o">[</span>    9.839210] Bluetooth: L2CAP socket layer initialized
<span class="o">[</span>    9.839212] Bluetooth: SCO socket layer initialized
<span class="o">[</span>    9.858744] Bluetooth: hci0: RTL: examining <span class="nv">hci_ver</span><span class="o">=</span>0a <span class="nv">hci_rev</span><span class="o">=</span>000c <span class="nv">lmp_ver</span><span class="o">=</span>0a <span class="nv">lmp_subver</span><span class="o">=</span>8822
<span class="o">[</span>    9.860923] Bluetooth: hci0: RTL: rom_version <span class="nv">status</span><span class="o">=</span>0 <span class="nv">version</span><span class="o">=</span>3
<span class="o">[</span>    9.860925] Bluetooth: hci0: RTL: loading rtl_bt/rtl8822cu_fw.bin
<span class="o">[</span>    9.864089] Bluetooth: hci0: RTL: loading rtl_bt/rtl8822cu_config.bin
...
</code></pre></div></div>

<p>Great, it loads <code class="language-plaintext highlighter-rouge">rtl8822</code> driver, which is what we need.</p>

<p>Try to pair with the mouse again (don’t forget to put it into pairing mode!). It works even without <code class="language-plaintext highlighter-rouge">hcitool lescan</code>. Yay!</p>

<p>And just so we don’t get any funny bussiness with accidental kernel upgrade, let’s pin kernel version:</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>apt-mark hold linux-generic-hwe-20.04-edge
apt-mark showhold
  linux-generic-hwe-20.04-edge
</code></pre></div></div>

<h2 id="wifi">WiFi</h2>

<ol>
  <li>Still can’t connect to 5 Ghz WiFi.</li>
  <li>The mouse feels choppy. No amount of tinkering with Bluetooth settings helps. It could be due to interference of Bluetooth with 2.4 Ghz wifi.</li>
</ol>

<p>So let’s get on that. Without going into long details, let me say that the only thing that actually works is an alternative driver from this <a href="https://github.com/juanro49/rtl88x2ce-dkms">repo</a>.
However, the version from PatoJad apt repo failed to install on kernel 5.8 (the fix is on github, but it appears that the debs weren’t rebuilt with it yet). So, installing from deb directly:</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget https://github.com/juanro49/rtl88x2ce-dkms/releases/download/5.7.3_35403_1/rtl88x2ce-dkms_35403_amd64.deb
<span class="nb">sudo </span>dpkg <span class="nt">-i</span> ./rtl88x2ce-dkms_35403_amd64.deb
<span class="nb">sudo </span>modprobe rtl88x2ce
</code></pre></div></div>
<p>…
No change. Apparently, we need to unload old modules.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">$ </span><span class="nb">sudo </span>apt <span class="nb">install </span>hwinfo
<span class="nv">$ </span>hwinfo <span class="nt">--wlan</span>
10: PCI 100.0: 0282 WLAN controller
  <span class="o">[</span>Created at pci.386]
  Unique ID: yWPJ.iGV+_7vRbHF
  Parent ID: e6j0.JhR1dXh1J9E
  SysFS ID: /devices/pci0000:00/0000:00:01.2/0000:01:00.0
  SysFS BusID: 0000:01:00.0
  Hardware Class: network
  Model: <span class="s2">"Realtek WLAN controller"</span>
  Vendor: pci 0x10ec <span class="s2">"Realtek Semiconductor Co., Ltd."</span>
  Device: pci 0xc822 
  SubVendor: pci 0x1d2e 
  SubDevice: pci 0xc823 
  Driver: <span class="s2">"rtw_8822ce"</span>
  Driver Modules: <span class="s2">"rtw88_8822ce"</span>
...
</code></pre></div></div>
<p>Right, <code class="language-plaintext highlighter-rouge">rtw_8822ce</code> is still used for the device. Checking <code class="language-plaintext highlighter-rouge">/etc/modprobe.d</code>, looks like <a href="https://github.com/juanro49/rtl88x2ce-dkms/blob/master/rtw88_blacklist.conf">blackisting</a> is also not applied by the deb package. Let’s do that manually:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget https://raw.githubusercontent.com/juanro49/rtl88x2ce-dkms/master/rtw88_blacklist.conf
sudo cp rtw88_blacklist.conf /etc/modprobe.d/rtw88_blacklist.conf
</code></pre></div></div>

<p>And now, reboot. (You can do some <code class="language-plaintext highlighter-rouge">modprobe</code> commands to enable it without reboot, but anyway it’s better to check that everything works from scratch).</p>

<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>hwinfo <span class="nt">--wlan</span>
10: PCI 100.0: 0282 WLAN controller                             
  <span class="o">[</span>Created at pci.386]
  Unique ID: yWPJ.iGV+_7vRbHF
  Parent ID: e6j0.JhR1dXh1J9E
  SysFS ID: /devices/pci0000:00/0000:00:01.2/0000:01:00.0
  SysFS BusID: 0000:01:00.0
  Hardware Class: network
  Model: <span class="s2">"Realtek WLAN controller"</span>
  Vendor: pci 0x10ec <span class="s2">"Realtek Semiconductor Co., Ltd."</span>
  Device: pci 0xc822 
  SubVendor: pci 0x1d2e 
  SubDevice: pci 0xc823 
  Driver: <span class="s2">"rtl88x2ce"</span>
  Driver Modules: <span class="s2">"rtl88x2ce"</span>
...
</code></pre></div></div>

<p>Now we’re talking! Connect to 5Ghz - works! And mouse movement is no longer choppy, too!</p>

<p>Although, now for some reason there’s 2 WLAN devices reported in <code class="language-plaintext highlighter-rouge">hwinfo</code> output. And the Gnome shows 2 wifi devices as well. Well, nevermind as long as it works.</p>

<h2 id="night-light">Night light</h2>

<p>Night light worked strangely. For some reason it would get enabled and the right away go back to daylight setting. Then I installed <a href="http://jonls.dk/redshift/">Redshift</a>, which worked fine. Then I disabled Redshift and tried Night light again. This time, there was no problem. Magic?</p>

<p>Anyway, if NL doesn’t work for you, just use RS instead:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>sudo apt install redshift redshift-gtk
</code></pre></div></div>

<h2 id="fn-button">Fn button</h2>

<p>If <code class="language-plaintext highlighter-rouge">F1-F12</code> work as if <code class="language-plaintext highlighter-rouge">Fn</code> is always pressed, use <code class="language-plaintext highlighter-rouge">Fn+ESC</code> combination to switch the default mode.</p>

<h2 id="still-doesnt-work">Still doesn’t work</h2>
<p>Suspend/hibernation. It’s a <a href="https://gitlab.freedesktop.org/drm/amd/-/issues/1230">known issue</a> that will supposedly get resolved in kernel 5.10.
Of course, then we’ll likely have to jump through the hoops with Bluetooth again, and possibly <a href="https://github.com/juanro49/rtl88x2ce-dkms/issues/3">WiFi</a> too.</p>]]></content><author><name>Ilya Ivanov</name><email>ws9@devopsx.com</email></author><category term="linux" /><category term="redmibook" /><category term="usb" /><category term="realtek" /><category term="drivers" /><category term="bluetooth" /><category term="wifi" /><category term="ubuntu" /><category term="amd" /><category term="ryzen" /><summary type="html"><![CDATA[Greetings, fellow netizen. Today we’re going to install Ubuntu on Redmibook 16 Ryzen edition (Ryzen 4700U).]]></summary></entry><entry><title type="html">Travis, Wine, Innosetup: Error creating window.</title><link href="https://devopsx.com/travis-wine-innosetup-error-creating-window/" rel="alternate" type="text/html" title="Travis, Wine, Innosetup: Error creating window." /><published>2020-03-04T02:05:00+00:00</published><updated>2020-03-04T02:05:00+00:00</updated><id>https://devopsx.com/Travis-Wine-and-Innosetup-Error-creating-window</id><content type="html" xml:base="https://devopsx.com/travis-wine-innosetup-error-creating-window/"><![CDATA[<p>Given:</p>

<ol>
  <li>You’re using Travis.</li>
  <li>You using Linux runners.</li>
  <li>You want to generate and publish a Windows installer for your app.</li>
</ol>

<p>Seems straightforward enough. Just</p>
<ul>
  <li>install Wine
    <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">addons</span><span class="pi">:</span>
  <span class="na">apt</span><span class="pi">:</span>
    <span class="na">packages</span><span class="pi">:</span>
      <span class="pi">-</span> <span class="s">wine32</span>
      <span class="pi">-</span> <span class="s">wine-stable</span>
</code></pre></div>    </div>
  </li>
  <li>install Innosetup
    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wget <span class="nt">-q</span> https://jrsoftware.org/download.php/is.exe
wine is.exe /VERYSILENT /NORESTART
</code></pre></div>    </div>
  </li>
  <li>then run it normally
    <div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>wine <span class="s2">"</span><span class="nv">$inno_bin</span><span class="s2">"</span> my_install_script.iss
</code></pre></div>    </div>
  </li>
</ul>

<p>However, the catch is that Innosetup really want a display for itself, even when installing silently. So, add</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">services</span><span class="pi">:</span>
  <span class="pi">-</span> <span class="s">xvfb</span>
</code></pre></div></div>
<p>to <code class="language-plaintext highlighter-rouge">.travis.yml</code>.</p>

<p>According to all docs, that should be about enough. Still, it didn’t work for me:</p>
<pre><code class="language-log">2020-03-03 22:12:41.068   Successfully imported the DLL function. Delay loaded? No
2020-03-03 22:12:41.069   Exception message:
2020-03-03 22:12:41.069   Message box (OK):
                          Error creating window.
2020-03-03 22:12:41.070   User chose -1.
2020-03-03 22:12:41.070   Deinitializing Setup.
2020-03-03 22:12:41.071   Log closed.
</code></pre>

<p>After much trial and error, I discovered that Innosetup just would not install on a <code class="language-plaintext highlighter-rouge">language: minimal</code> Bionic runner. Apparently it was still missing some libraries. Setting <code class="language-plaintext highlighter-rouge">language: java</code> in <code class="language-plaintext highlighter-rouge">.travis.yml</code>, however, allowed the build to pass.</p>

<p>Most important are the things they don’t tell you…</p>]]></content><author><name>Ilya Ivanov</name><email>ws9@devopsx.com</email></author><category term="travis" /><category term="innosetup" /><category term="wine" /><summary type="html"><![CDATA[Given:]]></summary></entry></feed>