Skip to content

Advanced Topics

Nginx Reverse Proxy

To use Nginx as a reverse proxy in front of the gRPC service, disable the transport filter on both proxy and agent:

When using Nginx as a reverse proxy, disable the transport filter
on both Proxy and Agent:

java -jar prometheus-proxy.jar --tf_disabled
java -jar prometheus-agent.jar --tf_disabled --config myconfig.conf

Or via environment variable:
TRANSPORT_FILTER_DISABLED=true

Or via config:
proxy.transportFilterDisabled = true
agent.transportFilterDisabled = true

Disconnect detection

With transportFilterDisabled, the proxy can't see a connection close, only the agent's request stream end. An agent that disconnects ends that stream, and is removed at once. One that dies before opening its stream, or whose connection stays half-open, is removed after the inactivity timeout (default: 60 seconds, controlled by proxy.internal.maxAgentInactivitySecs).

In this mode the proxy also limits the agents connected as one auth identity, which is every agent when agent auth is off; see Transport Filter.

Example Nginx and proxy configuration files are available in the repository:

Prometheus Federation

Scrape an existing Prometheus instance via the /federate endpoint:

agent {
  pathConfigs: [
    {
      name: "Federated Prometheus"
      path: federated_metrics
      url: "http://prometheus-server:9090/federate?match[]={__name__=~\"job:.*\"}"
    }
  ]
}

This leverages Prometheus's built-in federation support, allowing you to pull metrics from another Prometheus server through the proxy.

A complete federation config is available at: examples/federate.conf

Consolidated Mode

By default, each scrape path is owned by a single agent. If a second agent tries to register the same path, it displaces the first agent, provided both connected with the same auth identity. With per-agent identities, another identity's registration is rejected while the first agent is connected.

The displaced agent stays connected without the path, so two agents configured with the same path settle with the later one serving it. Only a connection the agent itself has since replaced, one of the same launch, is disconnected when it loses its last path. To serve a path from both agents, make them consolidated.

In consolidated mode, multiple agents can register the same path:

agent.consolidated = true

When a scrape request arrives for a consolidated path, the proxy sends it to every agent serving the path, waits for all of them, and merges their responses into one. If one agent disconnects, the remaining agents continue serving the path.

Use cases:

  • Several instances behind one path -- each agent scrapes its own instance of a service, whose series carry a label that tells the instances apart, and Prometheus scrapes them as one target
  • Redundancy -- the path keeps answering, with the remaining agents' series, while one agent is down
  • Rolling upgrades -- new agent registers before old one deregisters

Agents that scrape the same endpoint send each series twice

The proxy joins the agents' answers as they are, so when two consolidated agents scrape the same endpoint, Prometheus gets each of its series once per agent, with the same labels. It keeps the first sample of each and drops the others, logging Error on ingesting samples with different value but same timestamp for the target on every scrape and counting them in prometheus_target_scrapes_sample_duplicate_timestamp_total, while up stays 1. Give each consolidated agent its own endpoint, and keep the agent Helm chart's replicaCount at 1, since its replicas all run the same config and scrape the same targets; Kubernetes replaces a failed agent pod. A rolling upgrade sends each series twice for as long as the old and new agents overlap.

A consolidated path belongs to one identity

With per-agent identities, a consolidated path records the identity that registered it, and only agents presenting that identity may join. Another identity's registration is rejected and retried, so it joins once the agents serving the path disconnect. Agents that share an identity -- all of them, when agent auth is off -- can still join each other's consolidated paths, and a joining agent's output is merged into every scrape of the path, so one that never answers holds each scrape until it times out.

gRPC Reflection

gRPC Reflection is disabled by default, because it lets anyone who reaches the agent port list and describe the proxy's API. Enable it with proxy.reflectionDisabled = false (or -Dproxy.reflectionDisabled=false on the command line) to let tools like grpcurl inspect the service. When agent authentication is configured, reflection calls need a valid agent token, like every other call on the agent port:

# List available gRPC services:
grpcurl -plaintext localhost:50051 list

# Output:
#   ProxyService
#   grpc.health.v1.Health
#   grpc.reflection.v1alpha.ServerReflection
# Describe the ProxyService:
grpcurl -plaintext localhost:50051 describe ProxyService

# Note: If using TLS, omit the -plaintext flag and provide cert flags.

# When the proxy requires agent tokens, present one:
grpcurl -plaintext -H 'agent-token: <token>' localhost:50051 list

Note

When using grpcurl with the -plaintext option, ensure the proxy is running without TLS. When TLS is enabled, provide the appropriate certificate flags.

Performance Tuning

Concurrent Scraping

Increase the number of parallel scrapes for high-throughput scenarios:

# Increase concurrent scraping capacity:
java -jar prometheus-agent.jar \
  --max_concurrent_clients 10 \
  --client_timeout_secs 30 \
  --chunk 64 \
  --config myconfig.conf

# Tune HTTP client cache:
java -jar prometheus-agent.jar \
  --max_cache_size 200 \
  --max_cache_age_mins 60 \
  --max_cache_idle_mins 20 \
  --config myconfig.conf
agent {
  http {
    maxConcurrentClients = 10         // Parallel scrape limit
    clientTimeoutSecs = 30            // HTTP client timeout

    clientCache {
      maxSize = 200                   // More cached clients
      maxAgeMins = 60                 // Longer cache lifetime
      maxIdleMins = 20               // Longer idle tolerance
      cleanupIntervalMins = 10        // Less frequent cleanup
    }
  }

  chunkContentSizeKbs = 64            // Larger chunks for big payloads
  minGzipSizeBytes = 256              // Compress more aggressively
  scrapeTimeoutSecs = 30              // More time for slow targets
}

Key Tuning Parameters

Parameter Default Guidance
maxConcurrentClients 1 Increase for many endpoints or slow targets
clientTimeoutSecs 90 Lower for fast-failing scrapes
chunkContentSizeKbs 32 Increase for large payloads to reduce chunk count
minGzipSizeBytes 512 Lower to compress more aggressively
scrapeTimeoutSecs 15 Increase for slow targets
clientCache.maxSize 100 Increase if many unique auth credentials are used

gRPC Keepalive Tuning

Fine-tune gRPC keepalive behavior for specific network environments:

// Proxy gRPC keepalive settings:
proxy.grpc {
  keepAliveTimeSecs = 7200            // Interval between PING frames
  keepAliveTimeoutSecs = 20           // Timeout for PING ack
  permitKeepAliveWithoutCalls = false
  permitKeepAliveTimeSecs = 300       // Min interval between client PINGs
  maxConnectionIdleSecs = -1          // -1 = unlimited
  maxConnectionAgeSecs = -1           // -1 = unlimited
}

// Agent gRPC keepalive settings:
agent.grpc {
  keepAliveTimeSecs = -1              // -1 = use server default
  keepAliveTimeoutSecs = 20
  keepAliveWithoutCalls = false
}

See the gRPC keepalive guide for detailed tuning advice.

Stale Agent Cleanup

The proxy periodically checks for inactive agents and evicts them:

proxy.internal {
  staleAgentCheckEnabled = true
  maxAgentInactivitySecs = 60         // Evict after 60s of inactivity
  staleAgentCheckPauseSecs = 10       // Check every 10s

  scrapeRequestTimeoutSecs = 90       // Timeout for scrape requests
}

Info

When transportFilterDisabled is true, stale agent cleanup is automatically force-enabled, regardless of the staleAgentCheckEnabled setting. This ensures leaked agent contexts are eventually cleaned up.

Zipkin Tracing

Both proxy and agent support distributed tracing via Zipkin:

proxy.internal.zipkin {
  enabled = true
  hostname = "zipkin.example.com"
  port = 9411
  path = "api/v2/spans"
  serviceName = "prometheus-proxy"
}

agent.internal.zipkin {
  enabled = true
  hostname = "zipkin.example.com"
  port = 9411
  path = "api/v2/spans"
  serviceName = "prometheus-agent"
}

Tracing covers the gRPC calls between agent and proxy: the agent reports a client span for each call, and the proxy a server span in the same trace, each under its own serviceName. Spans are sent in the background, so a Zipkin server that can't be reached costs the spans, not the agent or the proxy.

What reaches Zipkin follows from how an agent talks to the proxy. Zipkin joins the agent's client side and the proxy's server side of each call into one span, so a trace names both services. Connecting and registering are unary calls (connectAgent, registerAgent, a registerPath for each path, and sendHeartBeat while the agent is idle), each traced as it completes. Scrapes travel over long-lived streams (readRequestsFromProxy, writeResponsesToProxy, and writeChunkedResponsesToProxy), and a stream's span completes only when the stream ends, so a healthy, connected agent reports little: its streams appear when its connection closes, lasting as long as it did. Tracing shows an agent's connections and registrations, not its individual scrapes.

Start Zipkin before the proxy and agents. Spans sent while it can't be reached are dropped: the reporter warns once, then logs further failures only at the FINE level, so after that first warning, lost spans leave nothing in the logs.