Prometheus Proxy
Enable Prometheus to scrape metrics endpoints behind firewalls.
Prometheus uses a pull model for collecting metrics. This is problematic when a firewall separates the Prometheus server from its metrics endpoints. Prometheus Proxy solves this by using a persistent gRPC connection initiated from inside the firewall, preserving Prometheus's native pull-based architecture.
See it work on your laptop
Five self-contained demos run the proxy, its agents, and Prometheus from the released images with one command, and a guided tour walks you through each:
- the Basics Demo;
- the Network Demo, with a simulated WAN between the agents and the proxies;
- the Kubernetes Demo, with the Helm charts on two kind clusters;
- the Embedded Demo, with the agent embedded in a Ktor app;
- the Spring Boot Demo, with the agent embedded in a Spring Boot app.
All but the Kubernetes Demo need only Docker; it also needs kind, kubectl, and Helm.
How It Works
%%{init: {'flowchart': {'curve': 'linear'}}}%%
graph LR
P[Prometheus] -->|HTTP scrape| Proxy
%% The agent initiates the outbound gRPC connection, so the arrow points back to the proxy. The
%% invisible link pins Proxy left of Agent -- mermaid has no single reverse arrow to do it directly,
%% and curve:linear (above) keeps the resulting back-edge as straight as it can be drawn.
Proxy ~~~ Agent
Agent -->|gRPC stream| Proxy
Agent -->|HTTP fetch| E1[App 1 :9100/metrics]
Agent -->|HTTP fetch| E2[App 2 :9100/metrics]
Agent -->|HTTP fetch| E3[App 3 :9100/metrics]
subgraph Outside Firewall
P
Proxy
end
subgraph Inside Firewall
Agent
E1
E2
E3
end
The system comprises two components:
- Proxy -- runs outside the firewall alongside Prometheus. Accepts scrape requests from Prometheus on HTTP (port 8080) and communicates with agents via gRPC (port 50051).
- Agent -- runs inside the firewall with monitored services. Initiates an outbound gRPC connection to the proxy and responds to scrape requests by fetching metrics from local endpoints.
Key Benefits
- Firewall-friendly -- only requires an outbound connection from the agent
- Preserves pull model -- Prometheus continues to pull metrics as normal
- High performance -- built with Kotlin coroutines and gRPC streaming
- Secure -- optional TLS with mutual authentication, plus per-agent identities that scope each agent to the paths it may register
- Scalable -- one proxy supports many agents, each serving multiple paths
- Highly available -- agents fail over across an ordered list of proxies, and return to a recovered primary at their next reconnect
- Dynamic -- agents pick up target changes from a watched file at runtime, no restart needed
- Bandwidth-conscious -- optional per-path metric filtering drops unwanted families at the agent, before they cross the WAN
- Observable -- a live dashboard shows every agent, path, and recent scrape in one place, and scrapes a path on demand
- Embeddable -- the agent can run inside a JVM app, with its settings built in code, so an app on a cloud platform needs no secret files
- Zero changes to existing Prometheus configuration patterns
Quick Start
Get running in under a minute:
# A network for the two containers, so the agent reaches the proxy by its name
docker network create prometheus-proxy-net
# Start the proxy
docker run --rm --name prometheus-proxy --network prometheus-proxy-net \
-p 8080:8080 -p 50051:50051 --env METRICS_ENABLED=true \
pambrose/prometheus-proxy:4.6.0
# Start the agent, in another terminal
docker run --rm --network prometheus-proxy-net \
--env PROXY_HOSTNAME=prometheus-proxy \
--env AGENT_CONFIG='https://raw.githubusercontent.com/pambrose/prometheus-proxy/master/examples/simple.conf' \
pambrose/prometheus-agent:4.6.0
# Scrape the proxy's and the agent's own metrics through the proxy
curl http://localhost:8080/proxy_metrics
curl http://localhost:8080/agent_metrics
# From a clone of https://github.com/pambrose/prometheus-proxy
# Start the proxy, in the cluster where Prometheus runs
helm install prometheus-proxy ./charts/prometheus-proxy --namespace monitoring --create-namespace
# Start the agent, in the cluster whose services it scrapes
helm install prometheus-agent ./charts/prometheus-agent \
--set proxy.hostname=proxy-host.example.com:50051 \
--set-file config=myapps.conf
See the Quick Start Guide for detailed instructions.
Common Use Cases
| Scenario | Description |
|---|---|
| Enterprise environments | Scrape metrics across corporate firewall boundaries |
| Multi-cloud deployments | Bridge different network segments |
| Secure environments | Monitor internal services without opening inbound ports |
| Federation | Scrape existing Prometheus instances via /federate endpoint |
| Kubernetes | Monitor services across clusters or namespaces |
| Multi-team proxies | Share one proxy across teams, each scoped to its own paths |
| Dynamic fleets | Reconcile churning targets from a generated file, no restarts |
API Reference
Full API documentation (KDocs) is available at KDocs.
Next Steps
-
Demos
Watch the proxy and agents work, with Prometheus and Grafana, on your laptop
-
Architecture
Understand the proxy/agent components, gRPC protocol, and request flow
-
Configuration
Configure the application with HOCON and environment variables
-
Security & TLS
Set up TLS encryption and mutual authentication
-
Monitoring
Built-in metrics and Grafana dashboards
-
Cloud Platforms
Embed the agent in an app on ECS, Cloud Run, or Azure Container Apps