Basic auth from victoria-metrics-k8s-stack to remote VictoriaMetrics

Victoria Metrics has a complete Helm chart to install everything together into a k8s cluster. It’s all great, except that it’s not very smart to install a monitoring solution into the same system that it’s supposed to monitor.

Fortunately, Victoria also provides a docker compose stack that can easily be launched elsewhere. That shouldn’t pose any issues, so we’ll assume you have it running.

But you’ll probably want to add some authentication, just in case. Easiest way to do that is basic auth, with, say, Caddy. Adjust compose.yml like this

  caddy:
    image: caddy:2.9.1
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./config/caddy/Caddyfile:/etc/caddy/Caddyfile
      - ./data/caddy/data:/data
      - ./data/caddy/config:/config
    restart: unless-stopped

Also, delete all other ports: stanzas from compose file. We don’t want them exposed, everything should go through Caddy, with auth.

Create a password hash:

docker compose exec caddy caddy hash-password --plaintext 'mypassword123'
$2a$14$JEhbEw5WRBhnJhxP8t/ZOOtcpBehikc1XFfMBVEOP82UYjVg8U6r.

And ./config/caddy/Caddyfile can look like this

grafana.mydomain.com {
  reverse_proxy grafana:3000
}

victoriametrics.mydomain.com {
  reverse_proxy victoriametrics:8428
  basic_auth {
    k8s   $2a$14$JEhbEw5WRBhnJhxP8t/ZOOtcpBehikc1XFfMBVEOP82UYjVg8U6r.
  }
}

Now, docker compose up -d, try and log into victoriametrics.mydomain.com, verify that it’s working.

On the k8s side, values.yaml for victoria-metrics-k8s-stack is straighforward enough, disable everything handled by the remote VM instance:

alertmanager:
  enabled: false
grafana:
  enabled: false
defaultDashboards:
  enabled: true
vmsingle:
  enabled: false
vmagent:
  spec:
    remoteWrite:
      - url: https://victoriametrics.mydomain.com/api/v1/write
victoria-metrics-operator:
  admissionWebhooks:
    certManager:
      enabled: true
vmalert:
  enabled: false

The annoying part is getting basic auth working. You may find references to VM_remoteWrite_basicAuth_username/VM_remoteWrite_basicAuth_password, extraargs, envflag.prefix: "VM_" and similar. Forget it, it doesn’t work.

First, create a secret:

apiVersion: v1
kind: Secret
metadata:
  name: victoria-basic-auth
type: Opaque
stringData:
  username: k8s
  password: mypassword123

Then, pass it like this:

alertmanager:
  enabled: false
grafana:
  enabled: false
defaultDashboards:
  enabled: true
vmsingle:
  enabled: false
vmagent:
  spec:
    remoteWrite:
      - url: https://victoriametrics.mydomain.com/api/v1/write
        basicAuth:
          username:
            name: victoria-basic-auth
            key: username
          password:
            name: victoria-basic-auth
            key: password
victoria-metrics-operator:
  admissionWebhooks:
    certManager:
      enabled: true
vmalert:
  enabled: false

And that’s it, VMagent in k8s should now be able to send metrics properly. Verify with kubectl -n vm logs vmagent-vm-victoria-metrics-k8s-stack-6fb7bb75cb-fewyf vmagent or similar.