Advanced Topics
Nginx Reverse Proxy
To use Nginx as a reverse proxy in front of the gRPC service, disable the transport filter on both proxy and agent:
When using Nginx as a reverse proxy, disable the transport filter
on both Proxy and Agent:
java -jar prometheus-proxy.jar --tf_disabled
java -jar prometheus-agent.jar --tf_disabled --config myconfig.conf
Or via environment variable:
TRANSPORT_FILTER_DISABLED=true
Or via config:
proxy.transportFilterDisabled = true
agent.transportFilterDisabled = true
Disconnect detection
With transportFilterDisabled, the proxy can't see a connection close, only the agent's request
stream end. An agent that disconnects ends that stream, and is removed at once. One that dies before
opening its stream, or whose connection stays half-open, is removed after the inactivity timeout
(default: 60 seconds, controlled by proxy.internal.maxAgentInactivitySecs).
In this mode the proxy also limits the agents connected as one auth identity, which is every agent when agent auth is off; see Transport Filter.
Example Nginx and proxy configuration files are available in the repository:
Prometheus Federation
Scrape an existing Prometheus instance via the /federate endpoint:
agent {
pathConfigs: [
{
name: "Federated Prometheus"
path: federated_metrics
url: "http://prometheus-server:9090/federate?match[]={__name__=~\"job:.*\"}"
}
]
}
This leverages Prometheus's built-in federation support, allowing you to pull metrics from another Prometheus server through the proxy.
A complete federation config is available at:
examples/federate.conf
Consolidated Mode
By default, each scrape path is owned by a single agent. If a second agent tries to register the same path, it displaces the first agent, provided both connected with the same auth identity. With per-agent identities, another identity's registration is rejected while the first agent is connected.
The displaced agent stays connected without the path, so two agents configured with the same path settle with the later one serving it. Only a connection the agent itself has since replaced, one of the same launch, is disconnected when it loses its last path. To serve a path from both agents, make them consolidated.
In consolidated mode, multiple agents can register the same path:
When a scrape request arrives for a consolidated path, the proxy sends it to every agent serving the path, waits for all of them, and merges their responses into one. If one agent disconnects, the remaining agents continue serving the path.
Use cases:
- Several instances behind one path -- each agent scrapes its own instance of a service, whose series carry a label that tells the instances apart, and Prometheus scrapes them as one target
- Redundancy -- the path keeps answering, with the remaining agents' series, while one agent is down
- Rolling upgrades -- new agent registers before old one deregisters
Agents that scrape the same endpoint send each series twice
The proxy joins the agents' answers as they are, so when two consolidated agents scrape the same
endpoint, Prometheus gets each of its series once per agent, with the same labels. It keeps the first
sample of each and drops the others, logging Error on ingesting samples with different value but same
timestamp for the target on every scrape and counting them in
prometheus_target_scrapes_sample_duplicate_timestamp_total, while up stays 1. Give each
consolidated agent its own endpoint, and keep the agent Helm chart's replicaCount at 1, since its
replicas all run the same config and scrape the same targets; Kubernetes replaces a failed agent pod.
A rolling upgrade sends each series twice for as long as the old and new agents overlap.
A consolidated path belongs to one identity
With per-agent identities, a consolidated path records the identity that registered it, and only agents presenting that identity may join. Another identity's registration is rejected and retried, so it joins once the agents serving the path disconnect. Agents that share an identity -- all of them, when agent auth is off -- can still join each other's consolidated paths, and a joining agent's output is merged into every scrape of the path, so one that never answers holds each scrape until it times out.
gRPC Reflection
gRPC Reflection is disabled by default, because it lets anyone who
reaches the agent port list and describe the proxy's API. Enable it with proxy.reflectionDisabled = false (or
-Dproxy.reflectionDisabled=false on the command line) to let tools like
grpcurl inspect the service. When agent authentication is configured,
reflection calls need a valid agent token, like every other call on the agent port:
# List available gRPC services:
grpcurl -plaintext localhost:50051 list
# Output:
# ProxyService
# grpc.health.v1.Health
# grpc.reflection.v1alpha.ServerReflection
# Describe the ProxyService:
grpcurl -plaintext localhost:50051 describe ProxyService
# Note: If using TLS, omit the -plaintext flag and provide cert flags.
# When the proxy requires agent tokens, present one:
grpcurl -plaintext -H 'agent-token: <token>' localhost:50051 list
Note
When using grpcurl with the -plaintext option, ensure the proxy is running
without TLS. When TLS is enabled, provide the appropriate certificate flags.
Performance Tuning
Concurrent Scraping
Increase the number of parallel scrapes for high-throughput scenarios:
# Increase concurrent scraping capacity:
java -jar prometheus-agent.jar \
--max_concurrent_clients 10 \
--client_timeout_secs 30 \
--chunk 64 \
--config myconfig.conf
# Tune HTTP client cache:
java -jar prometheus-agent.jar \
--max_cache_size 200 \
--max_cache_age_mins 60 \
--max_cache_idle_mins 20 \
--config myconfig.conf
agent {
http {
maxConcurrentClients = 10 // Parallel scrape limit
clientTimeoutSecs = 30 // HTTP client timeout
clientCache {
maxSize = 200 // More cached clients
maxAgeMins = 60 // Longer cache lifetime
maxIdleMins = 20 // Longer idle tolerance
cleanupIntervalMins = 10 // Less frequent cleanup
}
}
chunkContentSizeKbs = 64 // Larger chunks for big payloads
minGzipSizeBytes = 256 // Compress more aggressively
scrapeTimeoutSecs = 30 // More time for slow targets
}
Key Tuning Parameters
| Parameter | Default | Guidance |
|---|---|---|
maxConcurrentClients |
1 | Increase for many endpoints or slow targets |
clientTimeoutSecs |
90 | Lower for fast-failing scrapes |
chunkContentSizeKbs |
32 | Increase for large payloads to reduce chunk count |
minGzipSizeBytes |
512 | Lower to compress more aggressively |
scrapeTimeoutSecs |
15 | Increase for slow targets |
clientCache.maxSize |
100 | Increase if many unique auth credentials are used |
gRPC Keepalive Tuning
Fine-tune gRPC keepalive behavior for specific network environments:
// Proxy gRPC keepalive settings:
proxy.grpc {
keepAliveTimeSecs = 7200 // Interval between PING frames
keepAliveTimeoutSecs = 20 // Timeout for PING ack
permitKeepAliveWithoutCalls = false
permitKeepAliveTimeSecs = 300 // Min interval between client PINGs
maxConnectionIdleSecs = -1 // -1 = unlimited
maxConnectionAgeSecs = -1 // -1 = unlimited
}
// Agent gRPC keepalive settings:
agent.grpc {
keepAliveTimeSecs = -1 // -1 = use server default
keepAliveTimeoutSecs = 20
keepAliveWithoutCalls = false
}
See the gRPC keepalive guide for detailed tuning advice.
Stale Agent Cleanup
The proxy periodically checks for inactive agents and evicts them:
proxy.internal {
staleAgentCheckEnabled = true
maxAgentInactivitySecs = 60 // Evict after 60s of inactivity
staleAgentCheckPauseSecs = 10 // Check every 10s
scrapeRequestTimeoutSecs = 90 // Timeout for scrape requests
}
Info
When transportFilterDisabled is true, stale agent cleanup is automatically
force-enabled, regardless of the staleAgentCheckEnabled setting. This ensures
leaked agent contexts are eventually cleaned up.
Zipkin Tracing
Both proxy and agent support distributed tracing via Zipkin:
proxy.internal.zipkin {
enabled = true
hostname = "zipkin.example.com"
port = 9411
path = "api/v2/spans"
serviceName = "prometheus-proxy"
}
agent.internal.zipkin {
enabled = true
hostname = "zipkin.example.com"
port = 9411
path = "api/v2/spans"
serviceName = "prometheus-agent"
}
Tracing covers the gRPC calls between agent and proxy: the agent reports a client span for each call, and the proxy
a server span in the same trace, each under its own serviceName. Spans are sent in the background, so a Zipkin
server that can't be reached costs the spans, not the agent or the proxy.
What reaches Zipkin follows from how an agent talks to the proxy. Zipkin joins the agent's client side and the proxy's
server side of each call into one span, so a trace names both services. Connecting and registering are unary calls
(connectAgent, registerAgent, a registerPath for each path, and sendHeartBeat while the agent is idle), each
traced as it completes. Scrapes travel over long-lived streams (readRequestsFromProxy, writeResponsesToProxy, and
writeChunkedResponsesToProxy), and a stream's span completes only when the stream ends, so a healthy, connected agent
reports little: its streams appear when its connection closes, lasting as long as it did. Tracing shows an agent's
connections and registrations, not its individual scrapes.
Start Zipkin before the proxy and agents. Spans sent while it can't be reached are dropped: the reporter warns once,
then logs further failures only at the FINE level, so after that first warning, lost spans leave nothing in the
logs.