Proxy Configuration
The proxy runs outside the firewall alongside Prometheus. It accepts scrape requests from Prometheus on HTTP and communicates with agents via gRPC.
HTTP Service
Configure the HTTP service that Prometheus scrapes:
proxy {
http {
port = 8080 // Listen port for Prometheus scrapes
host = "0.0.0.0" // Listen address for Prometheus scrapes
idleTimeoutSecs = 45 // Seconds an idle connection stays open (-1 = 45)
requestLoggingEnabled = true // Log every proxy metrics request
}
}
proxy.http.idleTimeoutSecs is how long the proxy keeps an idle connection from Prometheus open. -1 gives the
default, 45 seconds. 0 doesn't turn the timeout off: it, and any other negative value, fails the proxy's startup,
since the scrape port would close every connection before answering it.
gRPC Service
Configure the gRPC service that agents connect to:
proxy {
agent.port = 50051 // gRPC listen port for agent connections
grpc {
keepAliveTimeSecs = 7200 // Interval between PING frames
keepAliveTimeoutSecs = 20 // Timeout for PING acknowledgment
permitKeepAliveWithoutCalls = false
maxConnectionIdleSecs = -1 // -1 = unlimited
maxConnectionAgeSecs = -1 // -1 = unlimited
}
}
Agent Authentication
The agent gRPC port is unauthenticated by default. Set a
pre-shared token so the proxy rejects
agents that do not present a matching value (UNAUTHENTICATED). Resolved from --agent_token →
AGENT_TOKEN → proxy.agentToken; empty (the default) disables the check and logs a startup warning
unless mutual TLS is configured. The token is never logged.
Per-Agent Identities
Scope agents to specific paths with proxy.auth: a list of named identities, each with its own token
and allowed path glob patterns. The proxy resolves an agent's token to an identity and rejects any
registerPath for a path that does not match the identity's patterns. An empty paths list allows
all paths. A legacy proxy.agentToken (above) is honored alongside auth as an allow-all identity,
easing migration. See
Per-Agent Identities and Path Authorization
for details and a migration walkthrough.
proxy {
auth = [
{ name = team-a, token = "team-a-token", paths = ["team_a_*"] }
{ name = team-b, token = "team-b-token", paths = ["team_b_*"] }
]
}
Because auth is a list of objects, it is config-file-only — there is no CLI/env equivalent, and the
-D override cannot express a list. Identity names must be unique and tokens non-empty. The name
legacy-agent-token is reserved for the legacy proxy.agentToken, which the logs name that way and the
dashboard shows as shared token, so the proxy refuses to start with an entry of that name, whether or
not proxy.agentToken is set.
Service Discovery
Enable Prometheus HTTP service discovery:
proxy {
service.discovery {
enabled = true
path = "discovery"
targetPrefix = "proxy-host.example.com:8080"
}
}
See Service Discovery for complete details.
Admin Endpoints
Enable admin endpoints for health checks and debugging:
proxy {
admin {
enabled = true
port = 8092
pingPath = "ping"
versionPath = "version"
healthCheckPath = "healthcheck"
threadDumpPath = "threaddump"
debugEnabled = false
}
}
When enabled, the following endpoints are available:
| Endpoint | Description |
|---|---|
GET /ping |
Returns "pong" (liveness check) |
GET /healthcheck |
Returns health status JSON |
GET /version |
Returns version information |
GET /threaddump |
Returns JVM thread dump |
GET /debug |
Proxy debug info (requires debugEnabled = true) |
Dashboard
An operational dashboard on its own port. Off by default.
proxy {
dashboard {
enabled = false // Enable the operational dashboard
port = 8094 // Its own port, NOT the admin port
path = "dashboard" // Served at http://<proxy>:8094/dashboard
host = "0.0.0.0" // Listen address; "127.0.0.1" keeps it on the proxy host
refreshIntervalSecs = 2 // Re-push interval for drifting counters
pathHistorySize = 20 // Scrapes kept per path, for the History column and each agent's Recent scrapes
departedPathRetentionSecs = 3600 // How long a path whose agent has gone stays in the path table
activityLogSize = 500 // Lines kept in the Activity view: agents and paths coming, going, and refused
scrapeNowEnabled = true // A Scrape button on each path, which scrapes it at once and shows the result
maxSessions = 50 // Concurrent WebSocket sessions; more are closed
allowedOrigins = [] // Browser origins allowed besides the dashboard's own host
allowedHosts = [] // Host names the dashboard answers to; empty turns the check off
links = [] // Links from each path into other tools: { name = "...", url = "...", icon = "..." }
}
}
Also settable as --dashboard / --dashboard_port / --dashboard_path / --dashboard_host, or DASHBOARD_ENABLED / DASHBOARD_PORT / DASHBOARD_PATH / DASHBOARD_HOST.
The proxy logs a warning at startup when the dashboard is enabled on all interfaces. A WebSocket from a browser
origin other than the dashboard's own host is refused; behind a reverse proxy that rewrites the Host header, list the
public origin (for example "https://dash.example.com") in allowedOrigins. Being a list, allowedOrigins must be
set in a config file rather than with -D.
allowedHosts (for example ["dash.internal"]) lists the names you reach the dashboard by, which stops DNS rebinding.
It is off while empty, the default, and like allowedOrigins must be set in a config file. See
Exposure for what both lists guard against.
links gives each row of the Paths table links into other tools, each a name and a url template whose
placeholders ({path}, {agent}, {identity}, and {label.<name>}) every row fills from its path, for example
{ name = "Grafana", url = "http://grafana:3000/d/proxy-targets?var-job={label.job}" }. It too must be set in a config
file, and a link the dashboard can't use stops the proxy at startup. See Links.
It runs on its own port rather than the admin port, partly because the admin port is a servlet container
that cannot host WebSockets, and partly so the dashboard can be firewalled without also cutting off the
/ping and /healthcheck endpoints Kubernetes probes target.
Like the admin and metrics endpoints, it has no authentication and no TLS — treat the port as internal. See Dashboard for what it shows.
Metrics
Enable internal metrics collection:
proxy {
metrics {
enabled = true
port = 8082
path = "metrics"
}
admin {
enabled = true
port = 8092
}
}
See Monitoring for the complete metrics reference.
Internal Settings
Configure agent cleanup and scrape request management:
proxy.internal {
staleAgentCheckEnabled = true
maxAgentInactivitySecs = 60 // Evict after 60s of inactivity
staleAgentCheckPauseSecs = 10 // Check every 10s
scrapeRequestTimeoutSecs = 90 // Timeout for scrape requests
}
| Setting | Default | Description |
|---|---|---|
staleAgentCheckEnabled |
true | Enable periodic stale agent cleanup |
maxAgentInactivitySecs |
60 | Seconds of inactivity before agent is evicted |
staleAgentCheckPauseSecs |
10 | Interval between cleanup checks |
scrapeRequestTimeoutSecs |
90 | Timeout for individual scrape requests |
scrapeRequestBacklogUnhealthySize |
25 | Backlog that marks the proxy unhealthy; each agent's queue is capped at twice this |
maxInFlightScrapeRequests |
1000 | Max scrapes in flight across all agents; more get a 503 |
scrapeRequestMapUnhealthySize |
0 | Scrapes in flight that mark the proxy unhealthy (0 = 80% of maxInFlightScrapeRequests) |
chunkContextMapUnhealthySize |
0 | Chunked transfers in progress that mark the proxy unhealthy (0 = 80% of the same) |
maxPathsPerAgent |
20000 | Max paths one agent connection may register; more are refused (0 = unlimited) |
maxPathLength |
512 | Max characters in a registered path (0 = unlimited) |
maxLabelsSizeBytes |
8192 | Max size of a path's labels JSON, in bytes (0 = unlimited) |
maxAgentFieldLength |
256 | Max characters of an agent's name, host, launch ID, or endpoint (0 = unlimited) |
maxAgentProxyEndpoints |
16 | Max entries of an agent's proxy endpoint list (0 = unlimited) |
maxTargetUrlLength |
2048 | Max characters of a path's target URL, as the agent reports it (0 = unlimited) |
maxAgentsPerIdentity |
10000 | Max agents connected as one identity with transportFilterDisabled (0 = unlimited) |
The three path limits are safety nets against a runaway or misbehaving agent, set well above normal use. A path
over a limit is refused at registration, before it is checked against the agent's
identity patterns, and the agent logs the rejection once without retrying it. The proxy
warns when an agent reaches 80% of maxPathsPerAgent.
The agent limits bound what an agent reports about itself and its targets. The proxy only displays these values, in
its logs, service discovery labels, dashboard, and /debug page, but it keeps them, and repeats some of them for
every path the agent serves. A longer name, host name, launch ID, endpoint, or path source is cut to its first
maxAgentFieldLength characters, a longer endpoint list to its first maxAgentProxyEndpoints entries, and a longer
target URL to its first maxTargetUrlLength characters, after its credentials are redacted. The agent and its paths
stay registered, and the proxy logs one warning naming what it cut. The agent redacts a target URL's query values
before it sends it, so a real one is far shorter than 2048 characters.
maxAgentsPerIdentity applies only with the transport filter disabled; see there.
Content Size Limits
proxy.internal {
maxZippedContentSizeMBytes = 5 // Max zipped content size
maxUnzippedContentSizeMBytes = 10 // Max unzipped content size
}
Transport Filter
The transport filter detects agent disconnections immediately. Disable it when using a reverse proxy like Nginx:
Warning
With transportFilterDisabled, the proxy can't see a connection close, only the agent's request
stream end. An agent that disconnects ends that stream, and is removed at once. One that dies
before opening its stream, or whose connection stays half-open, is removed after the inactivity
timeout (default: 60 seconds, proxy.internal.maxAgentInactivitySecs).
Give each agent its own identity
With the filter disabled, the proxy can't tie a call to a connection, so it binds each agent's calls
to the auth identity the agent connected with. Agents that share an identity, including every agent
on the legacy proxy.agentToken, can't be told apart, so give each agent its own proxy.auth token.
See Isolation Between Agents.
Agents per identity
With the filter disabled, every connect adds an agent on the proxy, so the proxy limits the agents
connected as one auth identity to proxy.internal.maxAgentsPerIdentity (10,000). It refuses a connect
past the limit with RESOURCE_EXHAUSTED, rather than drop an agent already connected, and the refused
agent tries again after its reconnect pause. The proxy warns as an identity reaches 80% of the limit,
and when it refuses a connect, at most once a minute for each identity. With agent auth off, or on the
legacy proxy.agentToken, every agent has the same identity, so the limit is a total. Leave room for
agents that count twice for a while: one that reconnects while its old connection is half-open, until
the old one is evicted, and one whose registration is refused, once for each retry until each is
evicted. Set 0 for no limit.
gRPC Reflection
gRPC Reflection is disabled by default, because it lets anyone who reaches the agent port list and describe the proxy's API. Enable it for debugging and tooling:
When agent authentication is configured (proxy.agentToken or proxy.auth), reflection calls need a valid agent
token, like every other call on the agent port.
Log Level
# Set log level in config file:
proxy.logLevel = "debug"
agent.logLevel = "debug"
# Or via CLI:
# java -jar prometheus-proxy.jar --log_level debug
# java -jar prometheus-agent.jar --log_level debug
# Or via environment variable:
# PROXY_LOG_LEVEL=debug
# AGENT_LOG_LEVEL=debug
# Available levels: trace, debug, info, warn, error, off
Full Reference
See the complete configuration schema:
config/config.conf