Handbook
Sebastian Maniak's lab: single-node dockerized k3s on Viper, managed with Argo CD. Everything after bootstrap lives in git — this page is the full map of the lab.
Rewrites UI links below. Stored only in this browser.docker exec k3s-viper kubectl get nodes -o wide
SandboxAgents
Isolated sandboxes, not plain Agents. gVisor
SandboxAgents on Viper — the page I send people is
/agents/.
Handbook below is still the full lab map.
Kubernetes helper for this dockerized k3s lab.
Home FortiGate 80F assistant (fw-maniak-hq).
Executive AWS budget and capacity for us-east-2.
Manager-facing IT tickets on a personal developer instance.
Executive GCP budget and capacity for us-east1.
F5 BIG-IP VIP monitor for 172.16.10.10.
Read-only Arista cEOS operator for the Containerlab spine/leaf demo.
Overview
k8s-viper is a lab-grade single-node Kubernetes box. Bootstrap once; everything after that is git → Argo CD.
| Concern | Choice |
|---|---|
| Cluster | dockerized k3s (k3s-viper, rancher/k3s:v1.32.5-k3s1) |
| Bootstrap | scripts/bootstrap.sh (once / idempotent) |
| CD | Argo CD app-of-apps (argocd/apps/*) |
| Secrets | HashiCorp Vault OSS + External Secrets Operator |
| Dashboard | Headlamp (in-cluster OSS) |
| Ingress | Node IP via Traefik (k3s default; not kgateway for cluster edge) |
| Lab UIs | NodePorts 30080 / 30443 / 30200 / 30100 / 30300 / 30500 |
| AI gateway | one Gateway agentgateway-proxy :30100 → OpenAI (/v1) + DGX Spark (/spark) + desktop (/desktop/, /desktop-api/) |
| Agents | OSS kagent 0.10.0-rc2 + Agent Substrate 0.0.9 (kagent + ate-system); default model via the same gateway |
| LLM observability | Langfuse + ClickHouse; OTEL path configured (Vault secret/platform/langfuse-otel) |
| Git source | github.com/sebbycorp/k8s-viper @ main |
| SSH / tunnel | LAN smaniak@172.16.10.135; ngrok TCP for SSH only (not k8s UIs) |
Architecture
Imperative install only for k3s + Argo + root app. Platform services are GitOps.
bootstrap.sh → dockerized k3s (k3s-viper) + Argo CD + root Application
↓
argocd/apps/* (GitOps, auto-sync)
↓
ingress | vault | ESO | headlamp | argocd-access
gateway-api | agentgateway | langfuse
substrate | kagent | kagent-ai | desktop
↓
Traefik :80/:443 + NodePorts 30080 / 30443 / 30200 / 30100 / 30300 / 30500
↓
agentgateway-proxy :30100
├─ /v1 · /openai → OpenAI (Vault) gpt-5.5 / gpt-5-mini
├─ /spark → Spark vLLM :8000 Qwen/Qwen3.6-35B-A3B-FP8
├─ /mcp → SandboxAgent MCPs (LAN Streamable HTTP)
├─ /desktop/ → noVNC desktop viewer
└─ /desktop-api/ → computer-use HTTP API
↓
kagent UI :30500 → default model gpt-5.5 via gateway /v1
Agent Substrate (ate-system) → gVisor workers (kagent-default)
↓
OTEL collector → Langfuse (configured)ExternalSecret paths only. Prefer PR → validate → merge over
kubectl apply for platform desired state.Quick facts
Typical lab node values. Set your InternalIP above if different.
UI access
Fixed NodePorts for LAN access without port-forward. Keep these private — no public exposure without TLS and real identity.
| UI | Type | URL | Auth | Git source |
|---|---|---|---|---|
| Headlamp | NodePort | SA token | platform/headlamp/ | |
| Argo CD | NodePort | admin + initial secret | platform/argocd-access/ | |
| Vault UI | NodePort | root / app token after init | platform/vault/values.yaml | |
| agentgateway | NodePort | OpenAI via Vault; Spark no auth; desktop lab-open; GET / → 404 expected | platform/agentgateway-ai/ | |
| Desktop viewer | NodePort | lab-open noVNC — import viper-desktop:dev first | platform/desktop/ | |
| Desktop computer-use API | NodePort | no auth in process; gateway is the front door | platform/desktop/ | |
| Langfuse | NodePort | first-user signup | platform/langfuse/ | |
| kagent UI | NodePort | OSS kagent; dummy OpenAI key; real key on gateway | platform/kagent-ai/ | |
| Argo CD (HTTP) | NodePort | same as Argo CD | platform/argocd-access/ | |
| whoami demo | Ingress | none | platform/ingress/whoami.yaml | |
| Headlamp (host) | Ingress | SA token | platform/headlamp/values.yaml | |
| Langfuse (host) | Ingress | first-user signup | platform/langfuse/values.yaml |
Auth commands
# Talk to the cluster (kubectl is not on the host PATH)
docker exec k3s-viper kubectl get nodes -o wide
# Argo CD admin password (user: admin)
docker exec k3s-viper kubectl -n argocd get secret argocd-initial-admin-secret \
-o jsonpath='{.data.password}' | base64 -d; echo
# Headlamp SA token (paste into login form)
docker exec k3s-viper kubectl -n headlamp create token headlamp --duration=12hArgo UI uses a self-signed cert — accept the browser warning. Vault UI needs init + unseal first.
Platform apps
Argo CD Applications. Watch with docker exec k3s-viper kubectl -n argocd get applications.
| Application | Kind | Namespace | Source | Notes |
|---|---|---|---|---|
| root | bootstrap | argocd | bootstrap/argocd/root-application.yaml | App-of-apps; watches argocd/apps |
| argocd-project | core | argocd | argocd/project.yaml | AppProject viper allow-list |
| platform-ingress | ingress | kube-system / apps | platform/ingress/ | Traefik HelmChartConfig + whoami |
| platform-vault | secrets | vault | platform/vault/values.yaml | Vault Helm · Raft ×1 · UI :30200 |
| platform-external-secrets | secrets | external-secrets | platform/external-secrets/ | ESO operator |
| platform-headlamp | dashboard | headlamp | platform/headlamp/ | kustomize helmCharts 0.44.0 · JSON6902 drops hostUsers · :30080 |
| platform-argocd-access | access | argocd | platform/argocd-access/ | Argo UI NodePort :30443 / :30081 · argocd-cm --enable-helm |
| platform-gateway-api | crds | cluster | platform/gateway-api/ | Gateway API CRDs v1.6.0 |
| platform-agentgateway-crds | crds | agentgateway-system | oci agentgateway-crds 1.4.1 | agentgateway CRDs |
| platform-agentgateway | ai | agentgateway-system | platform/agentgateway/values.yaml | control plane Helm 1.4.1 |
| platform-agentgateway-ai | ai | agentgateway-system | platform/agentgateway-ai/ | one Gateway · OpenAI + Spark + desktop routes · OTEL · :30100 |
| platform-desktop | desktop | desktop | platform/desktop/ | computer-use Deployment · noVNC :6080 · API :18790 · import viper-desktop:dev |
| platform-langfuse-secrets | secrets | langfuse | platform/langfuse/ | ExternalSecret from Vault |
| platform-langfuse | ai | langfuse | platform/langfuse/values.yaml | Langfuse 1.5.41 + ClickHouse · :30300 |
| platform-substrate-crds | crds | ate-system | oci substrate-crds 0.0.9 | Agent Substrate CRDs |
| platform-substrate | ai | ate-system | platform/substrate-app/ | kustomize helmCharts 0.0.9 · JSON6902 STS defaults · Valkey ×6 (8.0) |
| platform-substrate-rbac | rbac | ate-system | platform/substrate/ | ate-api extra ClusterRole/Binding hook (empty on 0.0.9) |
| platform-kagent-crds | crds | kagent | oci kagent-crds 0.10.0-rc2 | kagent OSS CRDs (substrate subchart left off) |
| platform-kagent | ai | kagent | platform/kagent/values.yaml | kagent OSS 0.10.0-rc2 · model via agentgateway /v1 |
| platform-kagent-ai | ai | kagent | platform/kagent-ai/ | dummy kagent-openai · hello-substrate · fortigate · f5-bigip · arista-ceos · UI :30500 |
Namespaces
| Namespace | Purpose |
|---|---|
| argocd | Argo CD control plane + root app-of-apps |
| vault | HashiCorp Vault server (Raft single voter) |
| external-secrets | External Secrets Operator |
| headlamp | Kubernetes dashboard UI |
| agentgateway-system | agentgateway control plane + OpenAI/Spark/desktop proxy + OTEL collector |
| desktop | computer-use desktop Deployment (noVNC + HTTP API) |
| langfuse | Langfuse + Postgres/Redis/ClickHouse/MinIO |
| ate-system | Agent Substrate control plane (ate-api, atelet, Valkey, RustFS) |
| kagent | OSS kagent controller, UI, WorkerPool, hello-substrate, fortigate, f5-bigip, arista-ceos |
| apps | Demo / user workloads (whoami) |
| kube-system | k3s system: Traefik, CoreDNS, metrics, local-path |
| default | Kubernetes default (mostly unused by platform) |
Ingress hosts
Map hosts to the node IP in /etc/hosts, then hit Traefik on :80.
172.16.10.135 whoami.viper.local headlamp.viper.local langfuse.viper.local| Host | Target | Namespace | Class |
|---|---|---|---|
| whoami.viper.local | whoami Service | apps | traefik |
| headlamp.viper.local | headlamp Service | headlamp | traefik |
| langfuse.viper.local | langfuse web | langfuse | traefik |
curl -H 'Host: whoami.viper.local' http://172.16.10.135/Network & ports
| Port | Service | How |
|---|---|---|
| 80 / 443 | Traefik ingress | LoadBalancer on node IP (k3s ServiceLB). Do not steal for agentgateway — svclb-agentgateway-proxy Pending is cosmetic. |
| 30080 | Headlamp | NodePort |
| 30443 | Argo CD HTTPS | NodePort (argocd-server-nodeport) |
| 30081 | Argo CD HTTP | NodePort (often redirects) |
| 30200 | Vault UI / API | NodePort (vault-ui) |
| 30100 | agentgateway OpenAI + Spark + desktop proxy | NodePort (agentgateway-proxy-nodeport) |
| 30300 | Langfuse UI | NodePort (langfuse web service) |
| 30500 | kagent UI | NodePort (kagent-ui-nodeport) |
| 6443 | Kubernetes API | k3s on node (kubeconfig) |
AI stack
agentgateway fronts OpenAI (/v1, /openai)
with credentials from Vault, and DGX Spark Qwen (/spark) at
172.16.10.173:8000 with no auth.
Models: gpt-5.5, gpt-5-mini,
Qwen/Qwen3.6-35B-A3B-FP8.
Langfuse (+ ClickHouse) receives OTEL traces.
Full runbook:
docs/agentgateway-langfuse.md.
export GW=http://172.16.10.135:30100
curl -sS "$GW/v1/chat/completions" -H 'content-type: application/json' \
-d '{"model":"gpt-5.5","messages":[{"role":"user","content":"hi"}],"max_completion_tokens":64}'
# small model
curl -sS "$GW/v1/chat/completions" -H 'content-type: application/json' \
-d '{"model":"gpt-5-mini","messages":[{"role":"user","content":"hi"}],"max_completion_tokens":64}'
# DGX Spark (vLLM)
curl -sS "$GW/spark/v1/chat/completions" -H 'content-type: application/json' \
-d '{"model":"Qwen/Qwen3.6-35B-A3B-FP8","messages":[{"role":"user","content":"hi"}],"max_tokens":64}'secret/platform/openai from Vault via ExternalSecret.
Prefer LAN IP 172.16.10.135 for devices on the subnet.Bootstrap
Run on the target Linux node with outbound HTTPS.
git clone https://github.com/sebbycorp/k8s-viper.git
cd k8s-viper
sudo ./scripts/bootstrap.shOptional environment
| Variable | Default | Meaning |
|---|---|---|
| REPO_URL | this GitHub repo | GitOps source for Argo |
| REPO_REVISION | main | Branch / tag / SHA |
| ARGOCD_VERSION | v3.5.0 | Upstream install manifest pin |
| INSTALL_K3S_SKIP | 0 | Set 1 if k3s already installed |
sudo REPO_URL=https://github.com/YOU/k8s-viper.git \
REPO_REVISION=main \
ARGOCD_VERSION=v3.5.0 \
./scripts/bootstrap.sh
# Existing cluster:
sudo INSTALL_K3S_SKIP=1 ./scripts/bootstrap.shDay-2 GitOps
- Edit manifests or Helm values on a branch.
- Open a PR — CI runs
scripts/validate.sh(no cluster credentials). - Merge to
main. - Argo CD auto-syncs (prune + self-heal on platform apps).
./scripts/validate.sh
docker exec k3s-viper kubectl -n argocd get applications
# Build this handbook locally
cd site && hugo --minifySecrets (Vault + ESO)
After Argo syncs Vault and External Secrets, init and unseal once, then wire Kubernetes auth for ESO. Full runbook: docs/vault-eso-setup.md.
| Vault path | Fields | Used by |
|---|---|---|
| secret/platform/openai | api_key | agentgateway ExternalSecret (kagent uses a dummy Secret; gateway injects this key) |
| secret/platform/langfuse | salt, encryption_key, nextauth_secret, DB passwords | Langfuse ExternalSecret |
| secret/platform/langfuse-otel | public_key, secret_key, endpoint | OTEL collector → Langfuse (configured; keys in Vault) |
| secret/platform/fortigate | token, host | FortiOS MCP ExternalSecret fortigate-mcp (never commit token) |
| secret/platform/f5-bigip | host, username, password | F5 iControl MCP ExternalSecret f5-bigip-mcp (never commit password) |
| secret/platform/arista-ceos | username, password, hosts_json | Arista eAPI MCP ExternalSecret arista-ceos-mcp (never commit password) |
# Unseal (lab helper)
~/.config/k8s-viper/vault-unseal.sh
# UI after unseal → http://172.16.10.135:30200/platform/external-secrets/cluster-secret-store-vault.example.yaml.Versions
| Component | Pin | Where |
|---|---|---|
| Argo CD | v3.5.0 | scripts/bootstrap.sh |
| k3s (lab observed) | v1.32.5+k3s1 | dockerized rancher/k3s:v1.32.5-k3s1 |
| Vault Helm chart | 0.30.0 | argocd/apps/platform-vault.yaml |
| Vault image | hashicorp/vault:1.19.0 | lab observed |
| External Secrets chart | 0.14.4 | argocd/apps/platform-external-secrets.yaml |
| Headlamp Helm chart | 0.44.0 | platform/headlamp/kustomization.yaml |
| Headlamp image | ghcr.io/headlamp-k8s/headlamp:v0.44.0 | platform/headlamp/values.yaml |
| agentgateway / CRDs | v1.4.1 | OCI oci://cr.agentgateway.dev/charts |
| agentgateway images | v1.4.1 | chart pin v1.4.1 |
| Langfuse Helm chart | 1.5.41 | argocd/apps/platform-langfuse.yaml |
| OTEL collector | otel/opentelemetry-collector-contrib:0.132.1 | platform/agentgateway-ai/otel-collector.yaml |
| Gateway API CRDs | v1.6.0 | platform/gateway-api/ |
| whoami image | traefik/whoami:v1.10.2 | platform/ingress/whoami.yaml |
| kagent OSS Helm / CRDs | 0.10.0-rc2 | OCI oci://ghcr.io/kagent-dev/kagent/helm/kagent |
| Agent Substrate Helm / CRDs | 0.0.9 | platform/substrate-app/kustomization.yaml (helmCharts 0.0.9) |
| Substrate worker image | ghcr.io/kagent-dev/substrate/ateom-gvisor:v0.0.9 | platform/kagent/values.yaml |
| desktop image | viper-desktop:dev | images/desktop-computer-use/ (intended ghcr.io/sebbycorp/viper-desktop:dev) |
| FortiOS MCP image | fortigate-mcp:dev | images/fortigate-mcp/ (intended ghcr.io/sebbycorp/fortigate-mcp:dev) |
| Arista cEOS MCP image | arista-ceos-mcp:dev | images/arista-ceos-mcp/ (intended ghcr.io/sebbycorp/arista-ceos-mcp:dev) |
Repo layout
| Path | Role |
|---|---|
| scripts/bootstrap.sh | Only imperative install path (k3s + Argo + root) |
| scripts/validate.sh | Local/CI checks — no cluster credentials |
| bootstrap/argocd/ | Root Application seed |
| argocd/project.yaml | AppProject viper |
| argocd/apps/ | Child Application CRs (GitOps tree) |
| platform/ingress/ | Traefik config + whoami demo |
| platform/vault/ | Vault Helm values |
| platform/external-secrets/ | ESO Helm values + store example |
| platform/headlamp/ | Headlamp helmCharts + values (JSON6902 drops hostUsers) |
| platform/argocd-access/ | Argo CD UI NodePort + argocd-cm --enable-helm |
| platform/gateway-api/ | Gateway API CRDs |
| platform/agentgateway/ | agentgateway control plane values |
| platform/agentgateway-ai/ | OpenAI + Spark + desktop Gateway routes, OTEL |
| platform/desktop/ | computer-use desktop Deployment (noVNC + API) |
| images/desktop-computer-use/ | viper-desktop:dev image (build on Viper, ctr import) |
| platform/langfuse/ | Langfuse Helm + ExternalSecret |
| platform/substrate-app/ | Agent Substrate helmCharts + valkey STS JSON6902 (0.0.9) |
| platform/substrate/ | extra ate-api ClusterRole/Binding only |
| platform/kagent/ | kagent OSS Helm values (0.10.0-rc2) |
| platform/kagent-ai/ | dummy OpenAI Secret + hello + fortigate + f5-bigip + arista-ceos SandboxAgents + UI NodePort |
| images/fortigate-mcp/ | fortigate-mcp:dev image (build on Viper, ctr import) |
| images/arista-ceos-mcp/ | arista-ceos-mcp:dev image (build on Viper, ctr import) |
| apps/ | User workloads |
| docs/ | Operator runbooks |
| site/ | Hugo handbook + Agents showcase (`/agents/`) |
| .github/workflows/ | CI validate + Pages deploy |
Security notes
- v1 lab assumes private LAN access to NodePorts and Traefik.
- Headlamp binds
cluster-adminfor full lab visibility — tighten for multi-user. - Leave
config.unsafeUseServiceAccountTokenoff unless behind a real auth proxy. - Argo CD serves a self-signed TLS cert on the NodePort.
- Vault
tlsDisable: truein chart values — fine for lab, not production. - No OIDC / SSO in v1.
Out of scope (v1)
| Deferred | Why later |
|---|---|
| ngrok for k8s UIs | ngrok TCP is SSH to the box only; UIs stay on the LAN |
| Multi-node / HA control plane | Single-node powerhouse focus |
| MetalLB | k3s ServiceLB + node IP is enough |
| Push-based deploy from CI | GitOps pull via Argo only |
| Cloud KMS auto-unseal | Manual unseal for v1 |
| kgateway as primary cluster Ingress | Traefik is k3s default; agentgateway is AI data plane only |
| Substrate Actor wrap for the computer-use desktop | Follow-up; first path is a Deployment (docs/desktop-computer-use.md). kagent install is OSS + ateom-gvisor only. |
Troubleshooting
| Symptom | Check |
|---|---|
| kubectl not found on the host | docker exec k3s-viper kubectl … |
| Applications stuck Unknown | Repo URL / OCI chart reachable; argocd-repo-server logs |
| Project errors | appproject viper sourceRepos + destinations |
| whoami 404 | Host header / /etc/hosts; docker exec k3s-viper kubectl -n apps get ingress,pods |
| Headlamp 404 / no UI | http://172.16.10.135:30080/; Docker published ports |
| Headlamp token rejected | Create a fresh SA token — docs/headlamp.md |
| platform-headlamp OutOfSync / hostUsers | JSON6902 removes hostUsers; do not set false. Restart repo-server after --enable-helm |
| Argo UI unreachable | svc argocd-server-nodeport; app platform-argocd-access |
| Vault sealed / UI down | unseal helper; docs/vault-eso-setup.md |
| ClusterSecretStore not Ready | Vault unsealed + k8s auth role external-secrets |
| agentgateway GET / → 404 | Expected — use /v1, /spark, /desktop/, or /desktop-api/health |
| desktop ImagePullBackOff | Import viper-desktop:dev — docs/desktop-computer-use.md |
| fortigate-mcp ImagePullBackOff | Import fortigate-mcp:dev — docs/fortigate-agent.md |
| arista-ceos-mcp ImagePullBackOff | Import arista-ceos-mcp:dev — docs/arista-ceos-agent.md |
| agentgateway OpenAI 401/404 model | Vault openai key; use gpt-5.5 or gpt-5-mini |
| Spark 502 / no route | Backend 172.16.10.173:8000; model Qwen/Qwen3.6-35B-A3B-FP8 |
| svclb-agentgateway-proxy Pending | Cosmetic — Traefik owns :80/:443. Use NodePort 30100 |
| Langfuse ImagePullBackOff | Cluster egress/DNS to Docker Hub |
| kagent UI unreachable | http://172.16.10.135:30500/; Docker published 30500; app platform-kagent-ai |
| hello-substrate not Ready | ActorTemplate CRD must be 0.0.9 (valueFrom); WorkerPool kagent-default; gVisor-on-dockerized-k3s — docs/kagent-substrate.md |
| Re-run bootstrap | Safe: skips k3s if healthy; re-applies Argo + root app |