Single-node GitOps lab

Handbook

Sebastian Maniak's lab: single-node dockerized k3s on Viper, managed with Argo CD. Everything after bootstrap lives in git — this page is the full map of the lab.

k3s v1.32 Argo CD Vault OSS External Secrets Headlamp Traefik agentgateway Langfuse Hugo
Headlamp
:30080
Argo CD
:30443
Vault UI
:30200
agentgateway
:30100
Langfuse
:30300
kagent
:30500

Rewrites UI links below. Stored only in this browser.
docker exec k3s-viper kubectl get nodes -o wide

SandboxAgents

Isolated sandboxes, not plain Agents. gVisor SandboxAgents on Viper — the page I send people is /agents/. Handbook below is still the full lab map.

01

Overview

k8s-viper is a lab-grade single-node Kubernetes box. Bootstrap once; everything after that is git → Argo CD.

ConcernChoice
Clusterdockerized k3s (k3s-viper, rancher/k3s:v1.32.5-k3s1)
Bootstrapscripts/bootstrap.sh (once / idempotent)
CDArgo CD app-of-apps (argocd/apps/*)
SecretsHashiCorp Vault OSS + External Secrets Operator
DashboardHeadlamp (in-cluster OSS)
IngressNode IP via Traefik (k3s default; not kgateway for cluster edge)
Lab UIsNodePorts 30080 / 30443 / 30200 / 30100 / 30300 / 30500
AI gatewayone Gateway agentgateway-proxy :30100 → OpenAI (/v1) + DGX Spark (/spark) + desktop (/desktop/, /desktop-api/)
AgentsOSS kagent 0.10.0-rc2 + Agent Substrate 0.0.9 (kagent + ate-system); default model via the same gateway
LLM observabilityLangfuse + ClickHouse; OTEL path configured (Vault secret/platform/langfuse-otel)
Git sourcegithub.com/sebbycorp/k8s-viper @ main
SSH / tunnelLAN smaniak@172.16.10.135; ngrok TCP for SSH only (not k8s UIs)
02

Architecture

Imperative install only for k3s + Argo + root app. Platform services are GitOps.

bootstrap.sh  →  dockerized k3s (k3s-viper) + Argo CD + root Application
                      ↓
              argocd/apps/*  (GitOps, auto-sync)
                      ↓
   ingress | vault | ESO | headlamp | argocd-access
   gateway-api | agentgateway | langfuse
   substrate | kagent | kagent-ai | desktop
                      ↓
   Traefik :80/:443  +  NodePorts 30080 / 30443 / 30200 / 30100 / 30300 / 30500
                      ↓
   agentgateway-proxy :30100
        ├─ /v1 · /openai  → OpenAI (Vault)     gpt-5.5 / gpt-5-mini
        ├─ /spark         → Spark vLLM :8000   Qwen/Qwen3.6-35B-A3B-FP8
        ├─ /mcp           → SandboxAgent MCPs (LAN Streamable HTTP)
        ├─ /desktop/      → noVNC desktop viewer
        └─ /desktop-api/  → computer-use HTTP API
                      ↓
   kagent UI :30500  → default model gpt-5.5 via gateway /v1
   Agent Substrate (ate-system) → gVisor workers (kagent-default)
                      ↓
   OTEL collector → Langfuse (configured)
Hard rule: never commit secrets. Values live in Vault; git holds ExternalSecret paths only. Prefer PR → validate → merge over kubectl apply for platform desired state.
03

Quick facts

Typical lab node values. Set your InternalIP above if different.

Host
Viper · user smaniak
Node name
k3s-viper
LAN IP (clients)
172.16.10.135
k3s container
k3s-viper · rancher/k3s:v1.32.5-k3s1
kubectl
docker exec k3s-viper kubectl …
k3s InternalIP
172.17.0.2 (Docker bridge)
k3s / API
v1.32.5+k3s1
GitOps project
viper
Ingress class
traefik
AI Gateway
agentgateway-proxy NodePort 30100 (OpenAI + Spark + desktop)
AI models
gpt-5.5 · gpt-5-mini · Qwen/Qwen3.6-35B-A3B-FP8 (Spark)
Spark host
172.16.10.173:8000 (vLLM)
SSH
smaniak@172.16.10.135 · ngrok TCP for remote SSH
App of apps
argocd/apps
04

UI access

Fixed NodePorts for LAN access without port-forward. Keep these private — no public exposure without TLS and real identity.

UITypeURLAuthGit source
HeadlampNodePortSA tokenplatform/headlamp/
Argo CDNodePortadmin + initial secretplatform/argocd-access/
Vault UINodePortroot / app token after initplatform/vault/values.yaml
agentgatewayNodePortOpenAI via Vault; Spark no auth; desktop lab-open; GET / → 404 expectedplatform/agentgateway-ai/
Desktop viewerNodePortlab-open noVNC — import viper-desktop:dev firstplatform/desktop/
Desktop computer-use APINodePortno auth in process; gateway is the front doorplatform/desktop/
LangfuseNodePortfirst-user signupplatform/langfuse/
kagent UINodePortOSS kagent; dummy OpenAI key; real key on gatewayplatform/kagent-ai/
Argo CD (HTTP)NodePortsame as Argo CDplatform/argocd-access/
whoami demoIngressnoneplatform/ingress/whoami.yaml
Headlamp (host)IngressSA tokenplatform/headlamp/values.yaml
Langfuse (host)Ingressfirst-user signupplatform/langfuse/values.yaml

Auth commands

# Talk to the cluster (kubectl is not on the host PATH)
docker exec k3s-viper kubectl get nodes -o wide

# Argo CD admin password (user: admin)
docker exec k3s-viper kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d; echo

# Headlamp SA token (paste into login form)
docker exec k3s-viper kubectl -n headlamp create token headlamp --duration=12h

Argo UI uses a self-signed cert — accept the browser warning. Vault UI needs init + unseal first.

05

Platform apps

Argo CD Applications. Watch with docker exec k3s-viper kubectl -n argocd get applications.

ApplicationKindNamespaceSourceNotes
rootbootstrapargocdbootstrap/argocd/root-application.yamlApp-of-apps; watches argocd/apps
argocd-projectcoreargocdargocd/project.yamlAppProject viper allow-list
platform-ingressingresskube-system / appsplatform/ingress/Traefik HelmChartConfig + whoami
platform-vaultsecretsvaultplatform/vault/values.yamlVault Helm · Raft ×1 · UI :30200
platform-external-secretssecretsexternal-secretsplatform/external-secrets/ESO operator
platform-headlampdashboardheadlampplatform/headlamp/kustomize helmCharts 0.44.0 · JSON6902 drops hostUsers · :30080
platform-argocd-accessaccessargocdplatform/argocd-access/Argo UI NodePort :30443 / :30081 · argocd-cm --enable-helm
platform-gateway-apicrdsclusterplatform/gateway-api/Gateway API CRDs v1.6.0
platform-agentgateway-crdscrdsagentgateway-systemoci agentgateway-crds 1.4.1agentgateway CRDs
platform-agentgatewayaiagentgateway-systemplatform/agentgateway/values.yamlcontrol plane Helm 1.4.1
platform-agentgateway-aiaiagentgateway-systemplatform/agentgateway-ai/one Gateway · OpenAI + Spark + desktop routes · OTEL · :30100
platform-desktopdesktopdesktopplatform/desktop/computer-use Deployment · noVNC :6080 · API :18790 · import viper-desktop:dev
platform-langfuse-secretssecretslangfuseplatform/langfuse/ExternalSecret from Vault
platform-langfuseailangfuseplatform/langfuse/values.yamlLangfuse 1.5.41 + ClickHouse · :30300
platform-substrate-crdscrdsate-systemoci substrate-crds 0.0.9Agent Substrate CRDs
platform-substrateaiate-systemplatform/substrate-app/kustomize helmCharts 0.0.9 · JSON6902 STS defaults · Valkey ×6 (8.0)
platform-substrate-rbacrbacate-systemplatform/substrate/ate-api extra ClusterRole/Binding hook (empty on 0.0.9)
platform-kagent-crdscrdskagentoci kagent-crds 0.10.0-rc2kagent OSS CRDs (substrate subchart left off)
platform-kagentaikagentplatform/kagent/values.yamlkagent OSS 0.10.0-rc2 · model via agentgateway /v1
platform-kagent-aiaikagentplatform/kagent-ai/dummy kagent-openai · hello-substrate · fortigate · f5-bigip · arista-ceos · UI :30500
06

Namespaces

NamespacePurpose
argocdArgo CD control plane + root app-of-apps
vaultHashiCorp Vault server (Raft single voter)
external-secretsExternal Secrets Operator
headlampKubernetes dashboard UI
agentgateway-systemagentgateway control plane + OpenAI/Spark/desktop proxy + OTEL collector
desktopcomputer-use desktop Deployment (noVNC + HTTP API)
langfuseLangfuse + Postgres/Redis/ClickHouse/MinIO
ate-systemAgent Substrate control plane (ate-api, atelet, Valkey, RustFS)
kagentOSS kagent controller, UI, WorkerPool, hello-substrate, fortigate, f5-bigip, arista-ceos
appsDemo / user workloads (whoami)
kube-systemk3s system: Traefik, CoreDNS, metrics, local-path
defaultKubernetes default (mostly unused by platform)
07

Ingress hosts

Map hosts to the node IP in /etc/hosts, then hit Traefik on :80.

172.16.10.135  whoami.viper.local headlamp.viper.local langfuse.viper.local
HostTargetNamespaceClass
whoami.viper.localwhoami Serviceappstraefik
headlamp.viper.localheadlamp Serviceheadlamptraefik
langfuse.viper.locallangfuse weblangfusetraefik
curl -H 'Host: whoami.viper.local' http://172.16.10.135/
Why Traefik, not kgateway? k3s ships Traefik; v1 only needs simple host Ingress on the node IP. kgateway / Gateway API is deferred as a later platform choice — docs/why-traefik.md.
08

Network & ports

PortServiceHow
80 / 443Traefik ingressLoadBalancer on node IP (k3s ServiceLB). Do not steal for agentgateway — svclb-agentgateway-proxy Pending is cosmetic.
30080HeadlampNodePort
30443Argo CD HTTPSNodePort (argocd-server-nodeport)
30081Argo CD HTTPNodePort (often redirects)
30200Vault UI / APINodePort (vault-ui)
30100agentgateway OpenAI + Spark + desktop proxyNodePort (agentgateway-proxy-nodeport)
30300Langfuse UINodePort (langfuse web service)
30500kagent UINodePort (kagent-ui-nodeport)
6443Kubernetes APIk3s on node (kubeconfig)
09

AI stack

agentgateway fronts OpenAI (/v1, /openai) with credentials from Vault, and DGX Spark Qwen (/spark) at 172.16.10.173:8000 with no auth. Models: gpt-5.5, gpt-5-mini, Qwen/Qwen3.6-35B-A3B-FP8. Langfuse (+ ClickHouse) receives OTEL traces. Full runbook: docs/agentgateway-langfuse.md.

export GW=http://172.16.10.135:30100

curl -sS "$GW/v1/chat/completions" -H 'content-type: application/json' \
  -d '{"model":"gpt-5.5","messages":[{"role":"user","content":"hi"}],"max_completion_tokens":64}'

# small model
curl -sS "$GW/v1/chat/completions" -H 'content-type: application/json' \
  -d '{"model":"gpt-5-mini","messages":[{"role":"user","content":"hi"}],"max_completion_tokens":64}'

# DGX Spark (vLLM)
curl -sS "$GW/spark/v1/chat/completions" -H 'content-type: application/json' \
  -d '{"model":"Qwen/Qwen3.6-35B-A3B-FP8","messages":[{"role":"user","content":"hi"}],"max_tokens":64}'
No client OpenAI key — gateway injects secret/platform/openai from Vault via ExternalSecret. Prefer LAN IP 172.16.10.135 for devices on the subnet.
10

Bootstrap

Run on the target Linux node with outbound HTTPS.

git clone https://github.com/sebbycorp/k8s-viper.git
cd k8s-viper
sudo ./scripts/bootstrap.sh

Optional environment

VariableDefaultMeaning
REPO_URLthis GitHub repoGitOps source for Argo
REPO_REVISIONmainBranch / tag / SHA
ARGOCD_VERSIONv3.5.0Upstream install manifest pin
INSTALL_K3S_SKIP0Set 1 if k3s already installed
sudo REPO_URL=https://github.com/YOU/k8s-viper.git \
     REPO_REVISION=main \
     ARGOCD_VERSION=v3.5.0 \
     ./scripts/bootstrap.sh

# Existing cluster:
sudo INSTALL_K3S_SKIP=1 ./scripts/bootstrap.sh
11

Day-2 GitOps

  1. Edit manifests or Helm values on a branch.
  2. Open a PR — CI runs scripts/validate.sh (no cluster credentials).
  3. Merge to main.
  4. Argo CD auto-syncs (prune + self-heal on platform apps).
./scripts/validate.sh
docker exec k3s-viper kubectl -n argocd get applications

# Build this handbook locally
cd site && hugo --minify
12

Secrets (Vault + ESO)

After Argo syncs Vault and External Secrets, init and unseal once, then wire Kubernetes auth for ESO. Full runbook: docs/vault-eso-setup.md.

Vault pathFieldsUsed by
secret/platform/openaiapi_keyagentgateway ExternalSecret (kagent uses a dummy Secret; gateway injects this key)
secret/platform/langfusesalt, encryption_key, nextauth_secret, DB passwordsLangfuse ExternalSecret
secret/platform/langfuse-otelpublic_key, secret_key, endpointOTEL collector → Langfuse (configured; keys in Vault)
secret/platform/fortigatetoken, hostFortiOS MCP ExternalSecret fortigate-mcp (never commit token)
secret/platform/f5-bigiphost, username, passwordF5 iControl MCP ExternalSecret f5-bigip-mcp (never commit password)
secret/platform/arista-ceosusername, password, hosts_jsonArista eAPI MCP ExternalSecret arista-ceos-mcp (never commit password)
# Unseal (lab helper)
~/.config/k8s-viper/vault-unseal.sh

# UI after unseal → http://172.16.10.135:30200/
Never commit unseal keys, root tokens, or raw secret values. Example store only: platform/external-secrets/cluster-secret-store-vault.example.yaml.
13

Versions

ComponentPinWhere
Argo CDv3.5.0scripts/bootstrap.sh
k3s (lab observed)v1.32.5+k3s1dockerized rancher/k3s:v1.32.5-k3s1
Vault Helm chart0.30.0argocd/apps/platform-vault.yaml
Vault imagehashicorp/vault:1.19.0lab observed
External Secrets chart0.14.4argocd/apps/platform-external-secrets.yaml
Headlamp Helm chart0.44.0platform/headlamp/kustomization.yaml
Headlamp imageghcr.io/headlamp-k8s/headlamp:v0.44.0platform/headlamp/values.yaml
agentgateway / CRDsv1.4.1OCI oci://cr.agentgateway.dev/charts
agentgateway imagesv1.4.1chart pin v1.4.1
Langfuse Helm chart1.5.41argocd/apps/platform-langfuse.yaml
OTEL collectorotel/opentelemetry-collector-contrib:0.132.1platform/agentgateway-ai/otel-collector.yaml
Gateway API CRDsv1.6.0platform/gateway-api/
whoami imagetraefik/whoami:v1.10.2platform/ingress/whoami.yaml
kagent OSS Helm / CRDs0.10.0-rc2OCI oci://ghcr.io/kagent-dev/kagent/helm/kagent
Agent Substrate Helm / CRDs0.0.9platform/substrate-app/kustomization.yaml (helmCharts 0.0.9)
Substrate worker imageghcr.io/kagent-dev/substrate/ateom-gvisor:v0.0.9platform/kagent/values.yaml
desktop imageviper-desktop:devimages/desktop-computer-use/ (intended ghcr.io/sebbycorp/viper-desktop:dev)
FortiOS MCP imagefortigate-mcp:devimages/fortigate-mcp/ (intended ghcr.io/sebbycorp/fortigate-mcp:dev)
Arista cEOS MCP imagearista-ceos-mcp:devimages/arista-ceos-mcp/ (intended ghcr.io/sebbycorp/arista-ceos-mcp:dev)
14

Repo layout

PathRole
scripts/bootstrap.shOnly imperative install path (k3s + Argo + root)
scripts/validate.shLocal/CI checks — no cluster credentials
bootstrap/argocd/Root Application seed
argocd/project.yamlAppProject viper
argocd/apps/Child Application CRs (GitOps tree)
platform/ingress/Traefik config + whoami demo
platform/vault/Vault Helm values
platform/external-secrets/ESO Helm values + store example
platform/headlamp/Headlamp helmCharts + values (JSON6902 drops hostUsers)
platform/argocd-access/Argo CD UI NodePort + argocd-cm --enable-helm
platform/gateway-api/Gateway API CRDs
platform/agentgateway/agentgateway control plane values
platform/agentgateway-ai/OpenAI + Spark + desktop Gateway routes, OTEL
platform/desktop/computer-use desktop Deployment (noVNC + API)
images/desktop-computer-use/viper-desktop:dev image (build on Viper, ctr import)
platform/langfuse/Langfuse Helm + ExternalSecret
platform/substrate-app/Agent Substrate helmCharts + valkey STS JSON6902 (0.0.9)
platform/substrate/extra ate-api ClusterRole/Binding only
platform/kagent/kagent OSS Helm values (0.10.0-rc2)
platform/kagent-ai/dummy OpenAI Secret + hello + fortigate + f5-bigip + arista-ceos SandboxAgents + UI NodePort
images/fortigate-mcp/fortigate-mcp:dev image (build on Viper, ctr import)
images/arista-ceos-mcp/arista-ceos-mcp:dev image (build on Viper, ctr import)
apps/User workloads
docs/Operator runbooks
site/Hugo handbook + Agents showcase (`/agents/`)
.github/workflows/CI validate + Pages deploy
15

Security notes

  • v1 lab assumes private LAN access to NodePorts and Traefik.
  • Headlamp binds cluster-admin for full lab visibility — tighten for multi-user.
  • Leave config.unsafeUseServiceAccountToken off unless behind a real auth proxy.
  • Argo CD serves a self-signed TLS cert on the NodePort.
  • Vault tlsDisable: true in chart values — fine for lab, not production.
  • No OIDC / SSO in v1.
16

Out of scope (v1)

DeferredWhy later
ngrok for k8s UIsngrok TCP is SSH to the box only; UIs stay on the LAN
Multi-node / HA control planeSingle-node powerhouse focus
MetalLBk3s ServiceLB + node IP is enough
Push-based deploy from CIGitOps pull via Argo only
Cloud KMS auto-unsealManual unseal for v1
kgateway as primary cluster IngressTraefik is k3s default; agentgateway is AI data plane only
Substrate Actor wrap for the computer-use desktopFollow-up; first path is a Deployment (docs/desktop-computer-use.md). kagent install is OSS + ateom-gvisor only.
17

Troubleshooting

SymptomCheck
kubectl not found on the hostdocker exec k3s-viper kubectl …
Applications stuck UnknownRepo URL / OCI chart reachable; argocd-repo-server logs
Project errorsappproject viper sourceRepos + destinations
whoami 404Host header / /etc/hosts; docker exec k3s-viper kubectl -n apps get ingress,pods
Headlamp 404 / no UIhttp://172.16.10.135:30080/; Docker published ports
Headlamp token rejectedCreate a fresh SA token — docs/headlamp.md
platform-headlamp OutOfSync / hostUsersJSON6902 removes hostUsers; do not set false. Restart repo-server after --enable-helm
Argo UI unreachablesvc argocd-server-nodeport; app platform-argocd-access
Vault sealed / UI downunseal helper; docs/vault-eso-setup.md
ClusterSecretStore not ReadyVault unsealed + k8s auth role external-secrets
agentgateway GET / → 404Expected — use /v1, /spark, /desktop/, or /desktop-api/health
desktop ImagePullBackOffImport viper-desktop:dev — docs/desktop-computer-use.md
fortigate-mcp ImagePullBackOffImport fortigate-mcp:dev — docs/fortigate-agent.md
arista-ceos-mcp ImagePullBackOffImport arista-ceos-mcp:dev — docs/arista-ceos-agent.md
agentgateway OpenAI 401/404 modelVault openai key; use gpt-5.5 or gpt-5-mini
Spark 502 / no routeBackend 172.16.10.173:8000; model Qwen/Qwen3.6-35B-A3B-FP8
svclb-agentgateway-proxy PendingCosmetic — Traefik owns :80/:443. Use NodePort 30100
Langfuse ImagePullBackOffCluster egress/DNS to Docker Hub
kagent UI unreachablehttp://172.16.10.135:30500/; Docker published 30500; app platform-kagent-ai
hello-substrate not ReadyActorTemplate CRD must be 0.0.9 (valueFrom); WorkerPool kagent-default; gVisor-on-dockerized-k3s — docs/kagent-substrate.md
Re-run bootstrapSafe: skips k3s if healthy; re-applies Argo + root app
18

Docs & links