Skip to content

Running in Production

Guidance for deploying Prometheus Proxy in a production environment, pulling together the security, reliability, and tuning knobs documented elsewhere into one operational checklist.

Security

  • Encrypt the gRPC channel with TLS, and prefer mutual TLS so the proxy authenticates agents and vice-versa. See TLS Setup.
  • Set an agent token (--agent_token / AGENT_TOKEN) as a lightweight app-level control in addition to TLS. An empty token leaves the agent port open and logs a startup warning.
  • Prefer per-agent identities over one shared token. A shared token cannot tell agents apart, so any holder can register — and silently take over — any path, including one already served by another agent. Named identities under proxy.auth scope each agent to its own path patterns and can be revoked individually. See Per-Agent Identities, including the incremental migration path from a shared token.
  • Segment the network so only trusted agents can reach the gRPC port (50051).
  • Keep the scrape port (8080) to Prometheus. It has no authentication, and whoever reaches it can have the agents scrape their targets with their own query string and Authorization header. Bind it to a private address with proxy.http.host and firewall it. See Scrape Port Access.
  • Do not expose the admin or dashboard ports publicly. /threaddump and friends are operational tools, not public endpoints — keep 8092/8093 on an internal network only. The dashboard port (8094) is equally unauthenticated and shows agent names, hostnames, and target URLs; it runs on its own port precisely so it can be firewalled without cutting off the /ping and /healthcheck probes. Bind it to a private address with proxy.dashboard.host. If browsers that also load untrusted pages can reach it, set proxy.dashboard.allowedHosts to the names you use, and behind a reverse proxy list its public origin in proxy.dashboard.allowedOrigins. See Exposure.
  • When forwarding auth headers to targets, require TLS so credentials aren't sent in plaintext between proxy and agent. See Auth Header Forwarding.

Running as a service

In production, run the proxy and agents under a supervisor that starts them at boot and restarts them if they exit. nohup keeps a process running after you log out, but nothing restarts it, so don't rely on it here.

A proxy that can't start, for example because one of its ports is already taken, logs the reason and exits with status 1, so the supervisor's restart policy retries it. Before 4.3.0 it logged the error but kept running, holding its admin and metrics ports, so no restart policy ever fired. See The proxy or agent exits at startup.

  • Kubernetes restarts a failed container and reschedules its pod. See Kubernetes.
  • Docker does it with --restart unless-stopped, or restart: unless-stopped in Compose. See Running in the Background.
  • Homebrew's brew services start runs it under launchd on macOS or systemd on Linux and starts it at login; sudo brew services start starts it at boot instead. See the Quick Start.
  • systemd, for the JARs on a Linux host: a unit like this one for the proxy, installed as /etc/systemd/system/prometheus-proxy.service. It assumes the JAR is in /opt/prometheus-proxy, the config is in /etc/prometheus-proxy, and an unprivileged prometheus user exists (create it with sudo useradd --system --no-create-home prometheus):

    [Unit]
    Description=Prometheus Proxy
    Wants=network-online.target
    After=network-online.target
    
    [Service]
    User=prometheus
    ExecStart=/usr/bin/java -jar /opt/prometheus-proxy/prometheus-proxy.jar --config /etc/prometheus-proxy/proxy.conf
    Restart=on-failure
    RestartSec=5
    # The JVM exits with 143 when systemd stops it with SIGTERM; count that as a clean stop.
    SuccessExitStatus=143
    
    [Install]
    WantedBy=multi-user.target
    

    For an agent, use prometheus-agent.jar and its config, and name the unit prometheus-agent.service. Then:

    sudo systemctl daemon-reload
    sudo systemctl enable --now prometheus-proxy   # start now and at every boot
    systemctl status prometheus-proxy              # check that it is running
    journalctl -u prometheus-proxy -f              # follow its log
    sudo systemctl restart prometheus-proxy        # pick up an edited config
    sudo systemctl disable --now prometheus-proxy  # stop, and no longer start at boot
    

High availability

  • For proxy redundancy, give the agent an ordered list of proxy endpoints. It uses the first that accepts it, and moves to the next on a failed connect or when a proxy rejects its registration (or every one of its static paths); when a registered connection drops it returns to the head of the list, so a recovered primary is picked up on the next reconnect without any manual step.

    agent {
      proxy {
        endpoints = [ "proxy-a.example.com:50051", "proxy-b.example.com:50051" ]
      }
    }
    

    Equivalently --proxy proxy-a.example.com:50051,proxy-b.example.com:50051, or the same comma-separated value in PROXY_HOSTNAME.

    Only one connection is active at a time; the agent registers its paths on whichever proxy it is connected to. Point Prometheus at both proxies with identical target lists — see Scraping an HA pair below, which has a requirement that is easy to get wrong.

    Failing over is quick: a refused connection, or a name that doesn't resolve, moves the agent to the next endpoint at once. Two things can slow a switch:

    • An endpoint whose host is gone, rather than refusing connections, isn't known to be down until the connection attempt times out. That takes agent.grpc.unaryDeadlineSecs (--unary_deadline_secs, 30 seconds by default), the deadline on every unary call, the connection's first among them. Lowering it shortens the wait, at the cost of patience with a slow proxy.
    • The JVM remembers a failed name lookup for 10 seconds (networkaddress.cache.negative.ttl, set in the JDK's java.security). Where a proxy's name stops resolving while it is down, as a stopped container's does on a Docker network, an agent that looked the primary up during the outage skips it for up to 10 seconds after it returns. If the agent's connection to the standby drops in that window, as when the standby is taken down moments after the primary comes back, it skips the primary, waits out the deadline above on the departed standby, and only then reaches the primary. Give a recovered primary a few seconds before stopping the standby.

    All endpoints must share the same TLS configuration. The agent builds one TLS context and one authority override for the whole list, so endpoints with different CAs or certificate SANs fail with an opaque handshake error rather than a clear configuration error.

  • For agent redundancy, use consolidated mode, with each agent scraping its own instance of the service: the proxy merges their answers into each scrape of the path, and keeps serving it, with the remaining agents' series, while one is down. Agents that scrape the same endpoint send every series once per agent, which Prometheus rejects as duplicates, logging an error on every scrape. A rolling upgrade needs no consolidated mode: an agent redeployed with the same identity takes its paths over when it registers.

  • For health probes, point Kubernetes liveness probes at /ping, and readiness probes at /ping on the proxy and /healthcheck on the agent. The proxy's /healthcheck measures load, so alert on it rather than take a busy proxy out of rotation. See Kubernetes.

Scraping an HA pair

Use static_config, not http_sd_config

A standby proxy — one no agent is currently connected to — returns an empty list from its service-discovery endpoint. Under http_sd_config Prometheus treats that as "these targets no longer exist" and deletes them: no up=0, no failed scrape, no alert. The series simply stop, which is the one failure mode an HA pair is supposed to make impossible.

With static_config, the standby returns 404 for a path it does not know, Prometheus records up=0 for that target, and your existing alerting works unchanged.

scrape_configs:
  - job_name: proxied-metrics
    static_configs:
      - targets: ["proxy-a.example.com:8080", "proxy-b.example.com:8080"]

Both proxies are scraped every interval. Exactly one of them has the agent connected and returns the metrics; the other returns 404 and shows as up=0. Deduplicate in the usual Prometheus HA way (distinct external_labels plus a downstream deduplicating reader, or honor_labels).

Sizing & tuning

Start from the defaults and adjust against the backlog and latency metrics:

Parameter Default When to change
agent.http.maxConcurrentClients 1 Raise for many endpoints or slow targets
agent.scrapeTimeoutSecs 15 Raise for slow targets
agent.http.clientTimeoutSecs 90 Lower for fast-failing scrapes
agent.chunkContentSizeKbs 32 Raise for large payloads to cut chunk count
agent.minGzipSizeBytes 512 Lower to compress more aggressively
agent.http.maxContentLengthMBytes 10 Raise for large scrape bodies (guards agent heap)
proxy.internal.maxUnzippedContentSizeMBytes 10 Raise for large decompressed payloads
proxy.internal.maxAgentInactivitySecs 60 Tune stale-agent eviction window

Size the JVM heap for the largest decompressed payload times the scrape concurrency, and watch agent_scrape_backlog_size / proxy_cumulative_agent_backlog_size — a steadily growing backlog means agents can't keep up. See Performance Tuning.

Observability

  • Enable metrics on proxy and agent and scrape their internal /metrics. See Monitoring.
  • Import the dashboards and alert rules from Grafana & Alerting.
  • Alert on success rate, P99 latency, agent count, and backlog growth — grafana/alerts.yml, shown on that page, covers each.
  • Enable the operational dashboard (--dashboard) on an internal network. Grafana tells you that a target is failing; the dashboard shows why — which agent backs the path, whether that agent is still connected, and how its recent scrapes went — including paths whose agent has already gone.

Logging

  • Leave requestLoggingEnabled on if you want per-scrape logs — they emit at DEBUG, so they won't flood INFO on a busy proxy.
  • Use logLevel = "trace" for the most verbose output; the legacy "all" level was removed and now fails fast at startup.

Shutdown

  • Standalone proxy/agent processes shut down cleanly on SIGTERM. Embedded agents should be stopped via EmbeddedAgentInfo.shutdown(), which blocks until terminated. See Embedded Agent.

Pre-flight checklist

  • [ ] Proxy and agents run under a supervisor that restarts them (systemd, Kubernetes, Docker's --restart, or brew services)
  • [ ] TLS (ideally mutual) enabled on the gRPC channel
  • [ ] Agent token set (per-agent identities where teams share a proxy), or mutual TLS in place
  • [ ] gRPC port reachable by agents; admin and dashboard ports not publicly exposed
  • [ ] Metrics enabled and scraped by Prometheus
  • [ ] Dashboards imported and alert rules loaded
  • [ ] maxConcurrentClients / timeouts tuned for your targets
  • [ ] Content-size limits sized for your largest payload
  • [ ] JVM heap sized for peak decompressed payload × concurrency
  • [ ] Liveness probes on /ping; readiness on /ping (proxy) and /healthcheck (agent)
  • [ ] Consolidated mode where you need agent redundancy, with each agent scraping its own instance