This article assumes that you already have ArgoCD working.
The first step to installing a new application with Argo always is to look for an exising Helm chart. If there is one, installing it usually is a no-brainer. Just add up an application, provide repository and chart urls, done.
However, the Karpenter team made a decision to drop Helm repository support at https://charts.karpenter.sh/
and instead publish it in an OCI repository (which turns out to be a thing) hosted on Amazon ECR.
The problem is, ArgoCD will not resolve oci://
url for Helm repository. At least, not without authentication. Even if it’s a public repository. Joyful.
Of course, if you have private charts in ECR, there’s not much choice, you need a token. Someone even put together an automation for refreshing ECR tokens.
However, if you want just the public ones, there’s a workaround. We will chain the charts and let Helm do the heavy lifting.
First, create a wrapper application:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: karpenter-root
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
destination:
namespace: karpenter
server:
project: default
source:
repoURL:
targetRevision: HEAD
path: helm/karpenter-root
helm:
valuesObject:
karpenter:
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: "arn:aws:iam::something/something"
settings:
clusterName: "mycluster"
interruptionQueue: "Karpenter-mycluster"
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
This configuration expects a custom Helm chart in helm/karpenter-root
in the same repository. Also note some minimal settings passed to Helm as valuesObject
.
Our Helm chart in helm/karpenter-root/Chart.yaml
is going to be simplistic:
apiVersion: v2
name: karpenter-root
description: Karpenter parent chart
type: application
version: 0.1.0
appVersion: "1.0.0"
dependencies:
- name: karpenter
version: "v0.34.0"
# TAKE THAT, KARPENTER TEAM! NO TOKENS FOR YOU!!
repository: "oci://public.ecr.aws/karpenter"
Also add values to helm/karpenter-root/values.yaml
:
karpenter:
serviceAccount:
annotations:
eks.amazonaws.com/role-arn: "none"
settings:
clusterName: "none"
interruptionQueue: "none"
To pass more values, you’ll need to add them to both Application definition and values.yaml
, but that’s a small price to pay.
That’s all, folks!
]]>This article assumes that you already have Traefik ingress installed and HTTPS working.
A typical virtual host for service frontend
with domain sub.domain.example
looks something like this:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-traefik
namespace: frontend
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: websecure
kubernetes.io/ingress.class: traefik
labels:
app: frontend
spec:
tls:
- hosts:
- sub.domain.example
secretName: tls
rules:
- host: sub.domain.example
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: frontend
port:
number: 80
Now, you’d like to have an automatic HTTP => HTTPS redirect for this host. Unfortunately, Trafik ingress does not provide an easy to do that with annotations. Instead, it offers middlewares, in particular RedirectScheme.
However, RedirectScheme is a) indiscriminate and b) in some cases relies on X-Forwarded
headers, which isn’t a solid choice.
Instead, we can use RedirectRegex middleware. Gating with virtual host name will allow to apply it selectively.
First, create the middleware in Traefik’s namespace (usually traefik
):
# http => https redirect middleware
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: redirect-to-https
namespace: traefik
spec:
redirectRegex:
regex: "^http://(.*)"
replacement: "https://$1"
permanent: true
Second, add another ingress configuration to the application namespace, listening on port 80. It’s pretty much a copy of the HTTPS one, just different entrypoint and no SSL configuration:
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-traefik-http
namespace: frontend
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: web
traefik.ingress.kubernetes.io/router.middlewares: traefik-redirect-to-https@kubernetescrd
kubernetes.io/ingress.class: traefik
spec:
rules:
- host:
http: sub.domain.example
paths:
- path: /
pathType: Prefix
backend:
service:
name: frontend
port:
number: 80
That’s it! Should work now.
]]>Problem: backend service can’t see real client IP addresses. It sees internal VPC addresses instead.
Goal: let ingress-nginx pass client ip addresses to backend.
By default, ingress-nginx-controller installed into EKS will create an ELB. It also allows to use NLB and ALB. But that requires installation of AWS load balancer controller, and who wants an additional dependency that can break during an upgrade? So we’ll stick with ELB.
The created ELB passes traffic at L4. To pass client IPs, we will need to configure proxy-protocol. Here’s AWS documentation. ELB uses protocol v1, while ALB uses v2, but Nginx seemingly speaks both, so that shouldn’t be a problem.
Check your ingress controller
kubectl -n ingress-nginx get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller-controller LoadBalancer 172.20.124.111 c43c1730102072835b21c6af3cede412-886572035.us-east-1.elb.amazonaws.com 80:32131/TCP,443:31739/TCP 15d
Note c43c1730102072835b21c6af3cede412
- that is the load balancer name. You can see it in AWS console in EC2 load balancers.
export lbname=c43c1730102072835b21c6af3cede412
As per AWS doc, create policy
aws elb create-load-balancer-policy --load-balancer-name $lbname --policy-name ProxyProtocolEnable --policy-type-name ProxyProtocolPolicyType --policy-attributes AttributeName=ProxyProtocol,AttributeValue=true
Now the tricky part, enable policy. Note that policy should be set on target ports. Not 80 and 443, but the paired ones.
aws elb set-load-balancer-policies-for-backend-server --load-balancer-name $lbname --instance-port 32131 --policy-names ProxyProtocolEnable
aws elb set-load-balancer-policies-for-backend-server --load-balancer-name $lbname --instance-port 31739 --policy-names ProxyProtocolEnable
Enable proxy protocol in ingress-nginx controller
kubectl -n ingress-nginx edit cm ingress-nginx-controller-controller
Ensure use-proxy-protocol: "true"
is set:
apiVersion: v1
data:
use-proxy-protocol: "true"
...
At this point, ELB should pass client IPs to ingress-nginx. To verify, check ingress-nginx logs.
kubectl -n ingress-nginx get pods
NAME READY STATUS RESTARTS AGE
ingress-nginx-controller-controller-931aa6241-3a1f1 1/1 Running 0 18d
curl -s http://my-backend-service
kubectl -n ingress-nginx logs ingress-nginx-controller-controller-931aa6241-3a1f1 --tail=3
~ resource "aws_lambda_function" "myfunc" {
id = "myfunc"
...
~ source_code_hash = "eqFDS7dB1KoQEAC6CNQ2ZAY1tV0ghf836eCQZwnWlpc=" -> "vnjjxeDfV108KVscceyBBxsRGYin4CPOWqgWvgWTVOo="
Terraform and AWS Lambda have a long history of bad blood: 1, 2. The problem is that Lambda appears to be perpetually in dirty state, due to source_code_hash
differences.
The root cause of is this is difference in packaging on different machines and bad documentation. Well, and an asinine design choice on AWS part.
source_code_hash
gets overwritten by AWS-provided data upon response.output_base64sha256
, filebase64sha256
) lies:
(String) The base64-encoded SHA256 checksum of output archive file.
Why would you even want to base64-encode a hash? The purpose of base64 encoding is to do away with non-printable chars, which a hash doesn’t have.
Turns out, what they actually do is compute sha256, then take the resulting text string and treat its characters as binary values, then base64 that: sha256sum lambda.zip | xxd -r -p | base64
.
The problem is, recent zip
versions store file permissions, and different umask
values on different machines result in different permissions, which in turn produces different archives with different hashes.
To alleviate this, set output_file_mode in the corresponding archive_file
:
data "archive_file" "myfile" {
type = "zip"
source_file = "${path.module}/src/myfunc/index.js"
output_path = "myfunc.zip"
output_file_mode = "0644"
}
resource "aws_lambda_function" "myfunc" {
description = "Example description"
function_name = "myfunc"
filename = "myfunc.zip"
source_code_hash = data.archive_file.myfile.output_base64sha256
...
If it doesn’t seem to help, add a newline to each of your lambda functions and deploy again. Sometimes AWS seems to return old source_code_hash
if the new version only differs in file permissions (or not deploy the new version at all).
The newline will force re-deploy, and after that deployment from any machine should be working as intended.
[ 242.891878] i915 0000:00:02.0: [drm] Resetting rcs0 for preemption time out
[ 242.891888] i915 0000:00:02.0: [drm] x.exe[10957] context reset due to GPU hang
[ 242.911578] i915 0000:00:02.0: [drm] GPU HANG: ecode 9:1:87f93cf9, in x.exe [10957]
Below is a list of possible workarounds to try.
export MESA_LOADER_DRIVER_OVERRIDE=i965
Don’t ask…
export INTEL_DEBUG=reemit
echo 10000 | sudo tee /sys/class/drm/card0/engine/rcs0/preempt_timeout_ms
. If this works, add to startup scripts. Source.
On Ubuntu, edit /etc/default/grub
and then run update-grub
and reboot. On other distros, you’ll figure it out :).
GRUB_CMDLINE_LINUX_DEFAULT="i915.enable_psr=0 i915.enable_fbc=1"
. Source.GRUB_CMDLINE_LINUX_DEFAULT="drm.debug=0 drm.vblankoffdelay=1 i915.semaphores=0 i915.modeset=1 i915.use_mmio_flip=1 i915.powersave=1 i915.enable_ips=1 i915.disable_power_well=1 i915.enable_hangcheck=1 i915.enable_cmd_parser=1 i915.fastboot=0 i915.enable_ppgtt=1 i915.reset=0 i915.lvds_use_ssc=0 i915.enable_psr=0"
Source.GRUB_CMDLINE_LINUX_DEFAULT="intel_idle.max_cstate=1 i915.enable_dc=0 ahci.mobile_lpm_policy=1"
. Source.GRUB_CMDLINE_LINUX_DEFAULT="i915.mitigations=off"
. Source.GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on"
. Source.On Ubuntu, put into /usr/share/X11/xorg.conf.d/intel.conf
or similar.
Section "Device"
Identifier "Intel Graphics"
Driver "intel"
# Option "TripleBuffer" "false"
# Option "VSync" "false"
# Option "PageFlip" "false"
# Option "DRI" "2"
# Option "AccelMethod" "UXA"
EndSection
Try each option separately. Reboot after each change. Source.
]]>If you prefer DSL, can stop reading now. JJB users - read on.
Since we’re automating all, it’d be great to have the configuration applied on a repo push. Of course, it’s possible to configure a seed job for JJB as well, but point and click is annoying. How about a github action instead?
Configuration is very simple:
JENKINS_TOKEN
secret in repo settings.name: jjb
on:
push:
paths:
- jenkins/jobs/** # job definitions here, searched recursively
jobs:
jjb:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: devopsx/gha-jjb@master
with:
jenkins_token: ${{ secrets.JENKINS_TOKEN }}
jjb_dir: jenkins/jobs # same dir with definitions as in push stanza
jjb_ini: jenkins/jenkins_jobs.ini
That’s it! Now your builds will be automatically updated each time you push.
]]>First off, choosing Ubuntu version. There’s 20.04 LTS and 20.10 at this point. But the only important difference between them is kernel version, which is 5.4 in 20.04 and 5.8 in 20.10. The new kernel can be installed on LTS too, though, so we’re going to pick 20.04.
Installation works properly. No issue here, just follow the standard procedure. The only strange thing is that it won’t connect to 5Ghz wifi. 2.4 Ghz works.
Right after first login, trying the brightness buttons, it turns out that they don’t work. A quick search turns up this. Apparently, this needs kernel 5.5, and some sources even say 5.7.
Well, no problem. Installing a newer kernel:
sudo apt install linux-generic-hwe-20.04-edge
This gets us to 5.8 (at this point).
Reboot and see that brightness buttons now work!
And right after that, another hiccup. A bluetooth mouse won’t get detected by the laptop. However, a headset works fine. Trying to do a lescan yields no result.
Time to search again. Bugzilla has some info with apparently the same problem. Specifically, it mentions Realtek RTL8822CE
device.
Let’s see if we have that one:
lsusb
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 0cb5:c547 Realtek Bluetooth Radio
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Looks like it’s one of the devices mentioned in bugzilla: 0cb5:c547
, meaning RTL8822CE
. And here’s a patch for it. Of course, it’s not in the kernel yet, so we’ll have to do some compilation.
Instead of compiling the whole kernel, as suggested in the discussion, let’s try to compile btusb
module only. Should be faster and less work.
We’ll still need to the sources for the whole kernel, of course.
sudo apt install git build-essential kernel-package fakeroot libncurses5-dev libssl-dev ccache bison flex # some packages for building
wget https://github.com/torvalds/linux/archive/v5.8.tar.gz # since we're on 5.8 kernel
tar zxf v5.4.tar.gz
cd linux-5.4/drivers/bluetooth
vi btusb.c
Apply the patch, save it.
Then, as described here (you should still be in linux-5.4/drivers/bluetooth
):
make -C /lib/modules/$(uname -r)/build M=$PWD modules
Now, custom btusb.ko
is in the same directory. Let’s try to load it:
cp /usr/lib/modules/$(uname -r)/kernel/drivers/bluetooth/btusb.ko ~/btusb.ko.bak
cp btusb.ko /usr/lib/modules/$(uname -r)/kernel/drivers/bluetooth/btusb.ko
modprobe -r btusb
modprobe btusb
...could not insert btusb.ko: Operation not permitted
Oops. What’s wrong? Dmesg to rescue:
Lockdown: modprobe: Loading of unsigned module is restricted; see man kernel_lockdown.7
Riiight, so the module is not signed, and the kernel refuses to load it. Unfortunately, the only way to do that is to disable UEFI Secure Boot. So:
F2
)Check dmesg again:
dmesg | grep -i blue
[ 1.987803] usb 1-4: Product: Bluetooth Radio
[ 9.839189] Bluetooth: Core ver 2.22
[ 9.839205] Bluetooth: HCI device and connection manager initialized
[ 9.839208] Bluetooth: HCI socket layer initialized
[ 9.839210] Bluetooth: L2CAP socket layer initialized
[ 9.839212] Bluetooth: SCO socket layer initialized
[ 9.858744] Bluetooth: hci0: RTL: examining hci_ver=0a hci_rev=000c lmp_ver=0a lmp_subver=8822
[ 9.860923] Bluetooth: hci0: RTL: rom_version status=0 version=3
[ 9.860925] Bluetooth: hci0: RTL: loading rtl_bt/rtl8822cu_fw.bin
[ 9.864089] Bluetooth: hci0: RTL: loading rtl_bt/rtl8822cu_config.bin
...
Great, it loads rtl8822
driver, which is what we need.
Try to pair with the mouse again (don’t forget to put it into pairing mode!). It works even without hcitool lescan
. Yay!
And just so we don’t get any funny bussiness with accidental kernel upgrade, let’s pin kernel version:
sudo apt-mark hold linux-generic-hwe-20.04-edge
apt-mark showhold
linux-generic-hwe-20.04-edge
So let’s get on that. Without going into long details, let me say that the only thing that actually works is an alternative driver from this repo. However, the version from PatoJad apt repo failed to install on kernel 5.8 (the fix is on github, but it appears that the debs weren’t rebuilt with it yet). So, installing from deb directly:
wget https://github.com/juanro49/rtl88x2ce-dkms/releases/download/5.7.3_35403_1/rtl88x2ce-dkms_35403_amd64.deb
sudo dpkg -i ./rtl88x2ce-dkms_35403_amd64.deb
sudo modprobe rtl88x2ce
… No change. Apparently, we need to unload old modules.
$ sudo apt install hwinfo
$ hwinfo --wlan
10: PCI 100.0: 0282 WLAN controller
[Created at pci.386]
Unique ID: yWPJ.iGV+_7vRbHF
Parent ID: e6j0.JhR1dXh1J9E
SysFS ID: /devices/pci0000:00/0000:00:01.2/0000:01:00.0
SysFS BusID: 0000:01:00.0
Hardware Class: network
Model: "Realtek WLAN controller"
Vendor: pci 0x10ec "Realtek Semiconductor Co., Ltd."
Device: pci 0xc822
SubVendor: pci 0x1d2e
SubDevice: pci 0xc823
Driver: "rtw_8822ce"
Driver Modules: "rtw88_8822ce"
...
Right, rtw_8822ce
is still used for the device. Checking /etc/modprobe.d
, looks like blackisting is also not applied by the deb package. Let’s do that manually:
wget https://raw.githubusercontent.com/juanro49/rtl88x2ce-dkms/master/rtw88_blacklist.conf
sudo cp rtw88_blacklist.conf /etc/modprobe.d/rtw88_blacklist.conf
And now, reboot. (You can do some modprobe
commands to enable it without reboot, but anyway it’s better to check that everything works from scratch).
hwinfo --wlan
10: PCI 100.0: 0282 WLAN controller
[Created at pci.386]
Unique ID: yWPJ.iGV+_7vRbHF
Parent ID: e6j0.JhR1dXh1J9E
SysFS ID: /devices/pci0000:00/0000:00:01.2/0000:01:00.0
SysFS BusID: 0000:01:00.0
Hardware Class: network
Model: "Realtek WLAN controller"
Vendor: pci 0x10ec "Realtek Semiconductor Co., Ltd."
Device: pci 0xc822
SubVendor: pci 0x1d2e
SubDevice: pci 0xc823
Driver: "rtl88x2ce"
Driver Modules: "rtl88x2ce"
...
Now we’re talking! Connect to 5Ghz - works! And mouse movement is no longer choppy, too!
Although, now for some reason there’s 2 WLAN devices reported in hwinfo
output. And the Gnome shows 2 wifi devices as well. Well, nevermind as long as it works.
Night light worked strangely. For some reason it would get enabled and the right away go back to daylight setting. Then I installed Redshift, which worked fine. Then I disabled Redshift and tried Night light again. This time, there was no problem. Magic?
Anyway, if NL doesn’t work for you, just use RS instead:
sudo apt install redshift redshift-gtk
If F1-F12
work as if Fn
is always pressed, use Fn+ESC
combination to switch the default mode.
Suspend/hibernation. It’s a known issue that will supposedly get resolved in kernel 5.10. Of course, then we’ll likely have to jump through the hoops with Bluetooth again, and possibly WiFi too.
]]>Seems straightforward enough. Just
addons:
apt:
packages:
- wine32
- wine-stable
wget -q https://jrsoftware.org/download.php/is.exe
wine is.exe /VERYSILENT /NORESTART
wine "$inno_bin" my_install_script.iss
However, the catch is that Innosetup really want a display for itself, even when installing silently. So, add
services:
- xvfb
to .travis.yml
.
According to all docs, that should be about enough. Still, it didn’t work for me:
2020-03-03 22:12:41.068 Successfully imported the DLL function. Delay loaded? No
2020-03-03 22:12:41.069 Exception message:
2020-03-03 22:12:41.069 Message box (OK):
Error creating window.
2020-03-03 22:12:41.070 User chose -1.
2020-03-03 22:12:41.070 Deinitializing Setup.
2020-03-03 22:12:41.071 Log closed.
After much trial and error, I discovered that Innosetup just would not install on a language: minimal
Bionic runner. Apparently it was still missing some libraries. Setting language: java
in .travis.yml
, however, allowed the build to pass.
Most important are the things they don’t tell you…
]]>{% for static_file in site.static_files %}
{% if static_file.path == '/favicon.ico' %}
{% assign favicon = true %}
{% endif %}
{% endfor %}
Both not very attractive, so I dived into docs to see if there’s something better. Here’s the result:
{% assign file = site.static_files | where: "path", jpg_path | first %}
{% if file %}<img src="{{ jpg_path }}">{% endif %}
No idea if it’s faster than the for loop, but at least it’s not as ugly.
]]>But, being lazy (like a good developer normally is), you specify it as “GPLv3 or later”. This seems efficient… takes away the burden of re-licensing it later, when a new GPL version is released.
Stop. Think about it again:
Not sure about you, but I don’t know the answer to the last question. Being more precise, no longer know. As Richard Stallman resigned from the organization.
]]>