Skip to content

Kubernetes

Run Prometheus Proxy on Kubernetes to scrape metrics from services that live inside firewalled or private clusters.

Topology

The two components play different roles, and that determines where each one runs:

  • The agent runs inside each target cluster, alongside the services it scrapes. It opens an outbound gRPC connection to the proxy, so the target cluster never needs an inbound firewall rule.
  • The proxy runs centrally, next to Prometheus. Agents connect in to it on the gRPC port (50051), and Prometheus scrapes it on the HTTP port (8080).
flowchart LR
  subgraph monitoring["Monitoring cluster"]
    prom["Prometheus"]
    proxy["Prometheus Proxy<br/>:8080 HTTP / :50051 gRPC"]
    prom -- "scrape :8080" --> proxy
  end

  subgraph target["Target cluster (behind firewall)"]
    agent["Prometheus Agent"]
    app["my-app :9100/metrics"]
    agent -- "scrape in-cluster" --> app
  end

  agent -- "outbound gRPC :50051" --> proxy

The agent always initiates the connection, so a target cluster only needs egress to the proxy's gRPC endpoint — no inbound ports are exposed.

See it run

The Kubernetes Demo builds this topology on one laptop: two kind clusters, the charts below, and an agent whose targets come from annotated Services, without restarting it.

Namespaces in these examples

The manifests below put the proxy in a monitoring namespace and the agent in default. Adjust the namespace fields and the in-cluster DNS names (e.g. prometheus-proxy.monitoring.svc.cluster.local) to match your own layout.

Installing with Helm

The repository ships a Helm chart for each side of the firewall, in charts/: prometheus-proxy for the monitoring cluster and prometheus-agent for each target cluster. Install them from a clone of the repository:

# In the monitoring cluster, next to Prometheus. agentService adds a LoadBalancer for agents in other clusters.
helm install prometheus-proxy ./charts/prometheus-proxy --namespace monitoring --create-namespace \
  --set agentService.enabled=true

# In each target cluster
helm install prometheus-agent ./charts/prometheus-agent -f agent-values.yaml
# agent-values.yaml
proxy:
  hostname: proxy.example.com:50051   # the proxy's externally reachable gRPC address
config: |
  agent {
    pathConfigs: [
      { name: "My app", path: my_app_metrics, url: "http://my-app.default.svc.cluster.local:8080/metrics" }
    ]
  }

The charts set up what the manifests below do by hand: the admin-endpoint probes, resource requests, a non-root pod with a read-only root filesystem, and optionally a ServiceMonitor (serviceMonitor.enabled), an agent token from a Secret (agentToken.existingSecret), TLS (tls.secretName), and, for the agent, a discovery ConfigMap (discovery.enabled). A config that holds secrets, such as proxy.auth tokens or credentials in an agent's target URLs, can come from a Secret (existingSecret) instead of the ConfigMap the chart renders. Each chart's README lists its values. The rest of this page builds the same deployment from plain manifests.

Deploying the Proxy

Deploy the proxy as a Deployment plus a Service. Admin and metrics endpoints are enabled through environment variables so Kubernetes can probe them and Prometheus can scrape them.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-proxy
  namespace: monitoring
  labels:
    app: prometheus-proxy
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-proxy
  template:
    metadata:
      labels:
        app: prometheus-proxy
    spec:
      containers:
        - name: proxy
          image: pambrose/prometheus-proxy:4.6.0
          ports:
            - { name: http, containerPort: 8080 }     # Prometheus scrapes here
            - { name: grpc, containerPort: 50051 }     # agents connect here
            - { name: metrics, containerPort: 8082 }
            - { name: admin, containerPort: 8092 }
          env:
            - { name: ADMIN_ENABLED, value: "true" }
            - { name: METRICS_ENABLED, value: "true" }
          livenessProbe:
            httpGet: { path: /ping, port: admin }
            initialDelaySeconds: 10
            periodSeconds: 15
          readinessProbe:
            httpGet: { path: /ping, port: admin }   # not /healthcheck, which measures load
            initialDelaySeconds: 10
            periodSeconds: 15
          resources:
            requests: { cpu: 100m, memory: 256Mi }
            limits: { memory: 512Mi }
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus-proxy
  namespace: monitoring
  labels:
    app: prometheus-proxy
spec:
  selector:
    app: prometheus-proxy
  ports:
    - { name: http, port: 8080, targetPort: http }
    - { name: grpc, port: 50051, targetPort: grpc }
    - { name: metrics, port: 8082, targetPort: metrics }

The Service above is a ClusterIP, which is enough when Prometheus and the agents live in the same cluster. Prometheus reaches the proxy at prometheus-proxy.monitoring.svc.cluster.local:8080, and same-cluster agents can use the same DNS name for the gRPC port.

Exposing gRPC to remote agents

When agents run in other clusters, they need to reach the proxy's gRPC port from outside. Expose it with a LoadBalancer Service (or an Ingress controller):

apiVersion: v1
kind: Service
metadata:
  name: prometheus-proxy-grpc
  namespace: monitoring
  labels:
    app: prometheus-proxy
spec:
  type: LoadBalancer
  selector:
    app: prometheus-proxy
  ports:
    - { name: grpc, port: 50051, targetPort: grpc }

gRPC needs HTTP/2 end to end

gRPC runs over HTTP/2. A plain LoadBalancer (L4) forwards it transparently. If you put an Ingress in front instead, it must support HTTP/2 / gRPC backends (for example, the NGINX Ingress nginx.ingress.kubernetes.io/backend-protocol: GRPC annotation). Terminate or pass through TLS as described under TLS.

Deploying the Agent

The agent's path mappings live in a ConfigMap. The proxy hostname is supplied separately through the PROXY_HOSTNAME environment variable, so the same config works regardless of where the proxy is exposed.

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-agent-config
  namespace: default
data:
  agent.conf: |
    agent {
      pathConfigs: [
        {
          name: "My App metrics"
          path: my_app_metrics
          url: "http://my-app.default.svc.cluster.local:8080/metrics"
        }
      ]
    }

Each entry in pathConfigs maps a proxy path (what Prometheus scrapes) to the in-cluster url the agent fetches from. Point url at a Kubernetes Service DNS name such as http://my-app.default.svc.cluster.local:8080/metrics.

Updating targets without restarting the agent

With pathConfigs alone, changing the target list means updating the ConfigMap and restarting the agent pod — which interrupts scraping for every path the agent serves, not just the one that changed. Dynamic target discovery removes the restart: mount a second ConfigMap holding a paths list and point agent.discovery at it. The agent re-reads the file on an interval and reconciles — targets added to the ConfigMap start scraping within the interval, removed ones unregister, and untouched paths never stop.

apiVersion: v1
kind: ConfigMap
metadata:
  name: agent-targets
data:
  targets.conf: |
    paths = [
      { name = "app1", path = "app1_metrics", url = "http://app1.default.svc.cluster.local:9090/metrics" }
      { name = "app2", path = "app2_metrics", url = "http://app2.default.svc.cluster.local:9090/metrics" }
    ]

Mount it into the agent pod (e.g. at /etc/prometheus-proxy/targets.conf) and enable discovery in the agent's main config:

agent {
  discovery {
    enabled = true
    file.path = "/etc/prometheus-proxy/targets.conf"
    reconcileIntervalSecs = 30
  }
}

Discovery polls the file rather than using inotify precisely because of how Kubernetes delivers ConfigMap updates (atomic symlink swaps that file watchers often miss). Note that a kubelet can take up to a minute to propagate a ConfigMap change to the mounted volume — that delay is Kubernetes's, and comes on top of the reconcile interval. A malformed or half-propagated file is skipped, keeping the last-known-good target set, so a bad edit never tears down working targets, even when the agent reconnects to the proxy while the file is still bad. Keep discovery targets in their own ConfigMap, separate from the agent's main config: whatever automation writes the target list then can't touch ports, TLS, or the proxy address.

Standalone Deployment

The common pattern is a single agent Deployment per cluster that scrapes several in-cluster endpoints:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus-agent
  namespace: default
  labels:
    app: prometheus-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus-agent
  template:
    metadata:
      labels:
        app: prometheus-agent
    spec:
      containers:
        - name: agent
          image: pambrose/prometheus-agent:4.6.0
          ports:
            - { name: metrics, containerPort: 8083 }
            - { name: admin, containerPort: 8093 }
          env:
            - name: PROXY_HOSTNAME
              value: "proxy.example.com:50051"   # the proxy's externally reachable address
            - { name: AGENT_CONFIG, value: /app/agent.conf }
            - { name: ADMIN_ENABLED, value: "true" }
            - { name: METRICS_ENABLED, value: "true" }
          volumeMounts:
            - { name: config, mountPath: /app/agent.conf, subPath: agent.conf }
          livenessProbe:
            httpGet: { path: /ping, port: admin }
            initialDelaySeconds: 10
            periodSeconds: 15
          readinessProbe:
            httpGet: { path: /healthcheck, port: admin }
            initialDelaySeconds: 10
            periodSeconds: 15
          resources:
            requests: { cpu: 100m, memory: 256Mi }
            limits: { memory: 512Mi }
      volumes:
        - name: config
          configMap:
            name: prometheus-agent-config

Set PROXY_HOSTNAME to the proxy's externally reachable address (the LoadBalancer host from above), including the gRPC port if it is not the default 50051.

Keep it at one replica, as above, or the agent chart's replicaCount: 1. Every replica runs the same config and registers the same paths: without consolidated mode the last to register serves them and the others sit idle, and with it the proxy joins their answers, so Prometheus gets every series once per replica and drops the duplicates, logging an error for each target on every scrape. Kubernetes replaces a failed agent pod on its own, in about a second in the Kubernetes Demo; see Consolidated Mode.

Sidecar

Alternatively, run the agent as a sidecar next to a single application pod and scrape it over localhost:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  namespace: default
  labels:
    app: my-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
        - name: my-app
          image: my-app:latest
          ports:
            - { name: metrics, containerPort: 9100 }
        - name: prometheus-agent
          image: pambrose/prometheus-agent:4.6.0
          env:
            - { name: PROXY_HOSTNAME, value: "proxy.example.com:50051" }
            - { name: AGENT_CONFIG, value: /app/agent.conf }
          volumeMounts:
            - { name: agent-config, mountPath: /app/agent.conf, subPath: agent.conf }
      volumes:
        - name: agent-config
          configMap:
            name: prometheus-agent-config   # agent.conf url points at http://localhost:9100/metrics

With the sidecar pattern the agent's url should point at http://localhost:9100/metrics rather than a Service DNS name.

Native sidecar containers

On Kubernetes 1.29+ you can run the agent as a native sidecar — an entry in initContainers with restartPolicy: Always — so it starts before the app container and is tracked by the pod's lifecycle. The plain multi-container form above works on all versions.

Prometheus Integration

Prometheus scrapes the proxy, using the agent path as the metrics_path. The path must match the path value from the agent ConfigMap.

Add a job to prometheus.yml pointing at the proxy Service:

scrape_configs:
  - job_name: 'my-app-via-proxy'
    metrics_path: '/my_app_metrics'
    static_configs:
      - targets: ['prometheus-proxy.monitoring.svc.cluster.local:8080']

If you run the Prometheus Operator (e.g. kube-prometheus-stack), use a ServiceMonitor instead:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus-proxy
  namespace: monitoring
  labels:
    release: kube-prometheus-stack   # must match your Prometheus serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app: prometheus-proxy
  endpoints:
    # Add one endpoint per registered proxy path.
    - port: http
      path: /my_app_metrics
      interval: 30s

The release label must match your Prometheus instance's serviceMonitorSelector. Add one entry under endpoints for each registered proxy path.

Dynamic path discovery

Rather than listing every path by hand, enable the proxy's service-discovery endpoint and let Prometheus pull the target list with http_sd_config. See Service Discovery. (This is the Prometheus-facing half; the agent-facing half — picking up target changes inside the cluster without a restart — is dynamic target discovery above. The two compose: discovered paths appear in the service-discovery output like any other. Caveat: if you run an HA proxy pair, scrape with static_config instead — a standby proxy returns an empty discovery list, which http_sd_config treats as "delete these targets".)

TLS

To secure the proxy ↔ agent gRPC channel, bundle your certificates into a Secret and mount it into the proxy and agent containers:

# Bundle your certificate files into a Secret:
kubectl create secret generic prometheus-proxy-tls \
  --namespace monitoring \
  --from-file=cert.pem \
  --from-file=key.pem \
  --from-file=ca.pem
# Container spec excerpt — point the config at the mounted certs:
          env:
            - { name: PROXY_CONFIG, value: /app/tls.conf }
          volumeMounts:
            - { name: tls, mountPath: /app/certs, readOnly: true }
      volumes:
        - name: tls
          secret:
            secretName: prometheus-proxy-tls

The mounted config (tls.conf) references the certificate paths under /app/certs. See TLS Setup for the full configuration, including mutual authentication.

Health Probes & Resources

The proxy and agent Deployments above wire Kubernetes probes to the admin endpoints:

Probe Proxy Agent
livenessProbe GET /ping GET /ping
readinessProbe GET /ping GET /healthcheck

/ping returns pong while the process is up. /healthcheck returns each health check's status, and a non-200 when any is unhealthy.

The proxy's readiness probe is /ping because its /healthcheck measures load. Each agent stays connected to one proxy pod, so taking a busy pod out of the Service doesn't move its load elsewhere: the scrapes for its agents' targets fail until it's ready again. Alert on the proxy's /healthcheck instead. It turns unhealthy when the scrapes or chunked transfers in flight reach 80% of maxInFlightScrapeRequests, or one agent's queue reaches scrapeRequestBacklogUnhealthySize.

Both endpoints are served on the admin port (8092 for the proxy, 8093 for the agent), which requires ADMIN_ENABLED=true. The manifests also set conservative CPU/memory requests and limits — tune them to your scrape volume and payload sizes.

Next Steps

  • Agent Configuration


    Path configs, HTTP client tuning, and scrape options

    Agent Configuration

  • Security & TLS


    Secure the proxy-agent channel with TLS and mutual authentication

    Security

  • Service Discovery


    Let Prometheus discover proxied targets dynamically

    Service Discovery

  • :material-activity:{ .lg .middle } Monitoring


    Scrape the proxy's and agent's own operational metrics

    Monitoring