Cloud Platforms
An app on a platform such as ECS, Cloud Run, or Azure Container Apps can't be scraped by a Prometheus outside the
platform without opening its metrics to the internet. An embedded agent reverses the direction: it
connects to a proxy beside Prometheus, and answers each scrape by reading the app's own /metrics. This page sets
that up for a JVM app, with the proxy on AWS, GCP, Azure, DigitalOcean, or Railway. The
Embedded Demo runs the same setup on one machine.
App platform (ECS, Cloud Run, ...) Proxy host (AWS, GCP, Azure, DigitalOcean, Railway)
app: /metrics on 127.0.0.1 prometheus-proxy :50051 agents, over TLS with a token
└ embedded agent ── gRPC, TLS + token ─────► :8080 scrapes, private
Prometheus ── scrapes :8080/my_app_metrics
How It Fits Together
- The agent runs inside the app and connects out to the proxy's agent port, so the app opens no port, and nothing outside reaches into the platform.
- It answers each scrape by reading the app's own
/metricson127.0.0.1. If that endpoint asks for Basic Auth, the agent sends the credentials with each scrape and leaves them out of its logs. - Prometheus scrapes the proxy, on a private network, at the path the agent registered. Keep the proxy's scrape port (8080) off the internet: anyone who reaches it scrapes through the agent. Only the agent port (50051) is public, behind TLS and a token.
- Leave the proxy's dashboard off, as it is by default: it has no authentication.
In the App
Starting the Agent
Build the agent's settings in code from the app's own, with
embeddedAgent { } or EmbeddedAgentConfig.builder(), so no file holds the
token or the password, and start it:
import io.prometheus.agent.EmbeddedAgentInfo
import io.prometheus.agent.embeddedAgent
import io.prometheus.common.ConfigLoadException
// Starts the agent from the app's own settings, in memory, so no config file holds the token or the password. Returns
// null when the agent is off, or can't start: the app's metrics aren't worth an outage.
fun startPrometheusAgent(
appVersion: String,
appPort: Int,
): EmbeddedAgentInfo? {
// host:port, the proxy's agent port; the agent stays off without it
val proxyEndpoint = env("PROMETHEUS_PROXY").ifEmpty { return null }
return try {
embeddedAgent {
// The version tells the old and new revisions apart during a deploy
name = "my-app-$appVersion"
proxy = proxyEndpoint
token = env("PROMETHEUS_AGENT_TOKEN")
// The proxy's certificate, as PEM or its base64, and the name it was issued to
trustCertificatePem(pem = env("PROMETHEUS_PROXY_CERT"), authority = "prometheus-proxy")
// The app's own /metrics, on 127.0.0.1; the agent sends the credentials as Basic Auth
scrapeSelf(
path = "my_app_metrics",
port = appPort,
username = env("METRICS_USER"),
password = env("METRICS_PASSWORD"),
)
}.start(logBanner = false)
} catch (e: ConfigLoadException) {
System.err.println("The Prometheus agent's settings are invalid: ${e.message}")
null
} catch (e: IllegalStateException) {
System.err.println("The Prometheus agent could not start: ${e.message}")
null
}
}
private fun env(name: String) = System.getenv(name).orEmpty().trim()
import io.prometheus.agent.EmbeddedAgentConfig;
import io.prometheus.agent.EmbeddedAgentInfo;
import io.prometheus.common.ConfigLoadException;
public class EmbeddedAgentConfigJavaExample {
// Starts the agent from the app's own settings, in memory, so no config file holds the token or the password.
// Returns null when the agent is off, or can't start: the app's metrics aren't worth an outage.
public static EmbeddedAgentInfo startPrometheusAgent(String appVersion, int appPort) {
// host:port, the proxy's agent port; the agent stays off without it
String proxy = env("PROMETHEUS_PROXY");
if (proxy.isEmpty()) {
return null;
}
try {
return EmbeddedAgentConfig.builder()
// The version tells the old and new revisions apart during a deploy
.name("my-app-" + appVersion)
.proxy(proxy)
.token(env("PROMETHEUS_AGENT_TOKEN"))
// The proxy's certificate, as PEM or its base64, and the name it was issued to
.trustCertificatePem(env("PROMETHEUS_PROXY_CERT"), "prometheus-proxy")
// The app's own /metrics, on 127.0.0.1; the agent sends the credentials as Basic Auth
.scrapeSelf("my_app_metrics", appPort, env("METRICS_USER"), env("METRICS_PASSWORD"))
.build()
.start(false);
} catch (ConfigLoadException e) {
System.err.println("The Prometheus agent's settings are invalid: " + e.getMessage());
return null;
} catch (IllegalStateException e) {
System.err.println("The Prometheus agent could not start: " + e.getMessage());
return null;
}
}
private static String env(String name) {
String value = System.getenv(name);
return value == null ? "" : value.trim();
}
}
- The app's settings are
PROMETHEUS_PROXY(the proxy's agent port, ashost:port),PROMETHEUS_AGENT_TOKEN,PROMETHEUS_PROXY_CERT(the proxy's certificate, as PEM or its base64), andMETRICS_USERandMETRICS_PASSWORDfor its own/metrics. WithoutPROMETHEUS_PROXY, as on a developer's machine, the agent stays off. - The agent reads its own environment variables too, such as
AGENT_TOKENandPROXY_HOSTNAME, and they take precedence over the config, so keep those names out of the app's environment unless you mean the agent to use them. - The app's version in
agent.nametells the old and new revisions apart in the proxy's logs during a deploy. - The agent's admin and metrics servers stay off, as they are unless asked for, since the app has no port to spare for them.
scrapeSelfpercent-encodes the credentials into the scrape URL, so a password with any character works.- A failure to start is logged, and the app carries on without the agent: its metrics aren't worth an outage.
Starting and Stopping It With the App
Start the agent once the app serves /metrics, and stop it when the app stops. In a Ktor module:
fun Application.module() {
routing {
get("/metrics") { call.respondText("my_app_up 1\n") }
}
// Started once the routes are in place, so the agent's first scrape finds /metrics
val agent = startPrometheusAgent(appVersion = "1.0.0", appPort = 8080)
// Stopped once the application has stopped. In development mode, a reload stops the old application, and its
// agent with it, before the new one starts its own.
monitor.subscribe(ApplicationStopped) { agent?.shutdown() }
}
In Spring Boot, a SmartLifecycle bean starts and stops it with the app; see
In Spring Boot, and the Spring Boot Demo, which runs one.
Keep the agent out of the health check the platform restarts the app on. The app serves without its agent, which
reconnects by itself once the proxy is back, so a failed check would only restart a working app. To see whether the
agent is connected, report EmbeddedAgentInfo's isConnected and
proxyEndpoint somewhere that doesn't fail the check, such as a detail of the app's status page; the Spring Boot Demo
reports them in Actuator's health as UNKNOWN, not DOWN.
Give the app's Logback configuration the line the standalone agent has for gRPC's internal logging; see Logging.
Packaging the App
The agent is a library inside the app, so the app's image is an ordinary JVM image. It adds nothing for the agent: no
port, since the agent connects out to the proxy, and no file, since the token and the certificate come in as
environment variables. A two-stage Dockerfile, for a Gradle app named my-app that uses the application plugin:
# Build the app with its own Gradle wrapper, on a JDK, keeping Gradle's downloads between builds
FROM eclipse-temurin:25-jdk-noble AS build
WORKDIR /src
COPY . .
RUN --mount=type=cache,target=/root/.gradle ./gradlew --no-daemon installDist
# Run it on a JRE, as a user that isn't root, named by number
FROM eclipse-temurin:25-jre-noble
RUN useradd --system --uid 1001 --home-dir /app app
COPY --from=build /src/build/install/my-app /app
USER 1001
# Lets Netty, which the agent's gRPC runs on, load its native TLS library and use sun.misc.Unsafe on Java 24 and later,
# and sizes the heap from the container's memory
ENV JAVA_OPTS="-XX:MaxRAMPercentage=75 --enable-native-access=ALL-UNNAMED --sun-misc-unsafe-memory-access=allow"
# The app's own port; the agent opens none, since it connects out to the proxy
EXPOSE 8080
ENTRYPOINT ["/app/bin/my-app"]
- It builds with the app's own Gradle wrapper on a JDK, and runs the result on a JRE as a user that isn't root, named by
number, so a platform that checks for one, such as Kubernetes with
runAsNonRoot, can. Add a.dockerignorethat leaves outbuild/,.gradle/,.kotlin/,.git/, and.idea/, soCOPY . .sends only the sources, and a commit or an IDE doesn't make the next build start over. - The start script
installDistwrites readsJAVA_OPTS. Besides sizing the heap from the container's memory, it carries two flags for Netty, which the agent's gRPC runs on, on Java 24 and later. The stock images run with the second; their jars ask for native access in their manifest instead of the first, which onlyjava -jarreads:--enable-native-access=ALL-UNNAMEDlets Netty load its native TLS library without the warning Java prints, which says a later release will block the call.--sun-misc-unsafe-memory-access=allowlets Netty usesun.misc.Unsafefor its buffers, which it otherwise does without. Java 22 and earlier don't know it and won't start with it, so leave it out there.
- An app built as a fat JAR starts with
java -jar, which doesn't readJAVA_OPTS: give the options inJDK_JAVA_OPTIONS, which thejavalauncher reads, or on its command line. - Any JDK 17 or later runs the agent; the stock images run 25.
The Embedded Demo's coffee shop is built the same way.
What the App Platform Must Give It
-
CPU between requests, and an instance that stays up. The agent holds its connection to the proxy open between scrapes, so a platform that throttles CPU outside requests, or scales the app to zero, stalls it.
- On AWS, run the app as an ECS service on Fargate, whose tasks keep their CPU whether or not they handle requests. App Runner gives an instance CPU only while it handles requests, and bills an idle one for its memory alone, so the agent stalls there.
-
On GCP, Cloud Run needs instance-based billing with a minimum instance:
-
On Azure, Container Apps scales an app to zero by default, so keep a replica running:
On App Service, turn on Always On, which keeps the app loaded when it has no requests.
-
One instance per path. Each agent registers its path as its own, and the last one to register serves it, so a second instance's agent takes the path from the first. Run one instance (a desired count of 1 for an ECS service,
--max-instances=1on Cloud Run,--max-replicas 1on Container Apps), or make the agents consolidated and give each instance's series a label of its own, such as a common tag with the instance's ID: the proxy merges the instances' answers into one scrape, and without the label it would repeat each series once per instance. - Deploys need nothing more. While the old and new revisions overlap, the new revision's agent takes the path over, and the old one stops with its app.
Securing the Link
The agent port faces the internet, so the agents connect over TLS and present a token.
Make a certificate for the proxy, issued to a fixed name rather than to whatever host name the platform gives it, and a token:
openssl req -x509 -newkey rsa:3072 -nodes -days 3650 -subj "/CN=prometheus-proxy" \
-addext "subjectAltName=DNS:prometheus-proxy" -keyout server.key -out server.crt
openssl rand -hex 32 > agent-token
- The proxy takes
server.crtandserver.keyasCERT_CHAIN_PEMandPRIVATE_KEY_PEM, and the token asAGENT_TOKEN. - The agent trusts
server.crt(agent.tls.trustCertCollectionPem), checks it against the fixed name (agent.tls.overrideAuthority = "prometheus-proxy"), and presents the token (agent.agentToken). - A variable field that takes one line takes the certificate and key as base64:
base64 < server.crt | tr -d '\n'. See Certificates as Environment Variables for the forms a PEM setting takes. - Keep
server.keyand the token out of version control. To limit which paths each app may register, give each its own identity instead of one shared token; see Per-agent identities.
Where to Run the Proxy
The proxy and Prometheus run together, on a host whose agent port the apps can reach:
| Platform | Agents reach the proxy through | Prometheus runs |
|---|---|---|
| AWS | An EC2 instance's address, with a security group rule | On the instance, beside the proxy |
| GCP | A Compute Engine VM's address, with a firewall rule | On the VM, beside the proxy |
| Azure | An Azure VM's address, with a network security rule | On the VM, beside the proxy |
| DigitalOcean | A Droplet's address, with a firewall open on 50051 | On the Droplet, beside the proxy |
| Railway | A TCP Proxy on the proxy service's port 50051 | As another service, on the private network |
examples/cloud-proxy holds the
proxy's config and a Prometheus config for each of them, and a Compose file that runs both on a VM:
# A prometheus-proxy and a Prometheus that scrapes through it, on a VM that agents embedded in cloud apps connect to,
# such as an EC2 instance, a Compute Engine instance, an Azure VM, or a DigitalOcean Droplet. The setup is described at
# https://prometheus-proxy.com/cloud-platforms/
#
# Put the proxy's certificate and key, as base64, and the agents' token in a .env file beside this one, and keep it out
# of version control:
#
# CERT_CHAIN_PEM=<base64 < server.crt | tr -d '\n'>
# PRIVATE_KEY_PEM=<base64 < server.key | tr -d '\n'>
# AGENT_TOKEN=<openssl rand -hex 32>
#
# then run `docker compose up --detach` here. Only the agents' port, 50051, is published to the internet; let it, and
# nothing else, through the host's firewall.
services:
proxy:
image: pambrose/prometheus-proxy:4.6.0
ports:
- "50051:50051" # gRPC, for the agents: TLS and the token
environment:
PROXY_CONFIG: /app/proxy.conf
CERT_CHAIN_PEM: ${CERT_CHAIN_PEM:?set CERT_CHAIN_PEM in .env}
PRIVATE_KEY_PEM: ${PRIVATE_KEY_PEM:?set PRIVATE_KEY_PEM in .env}
AGENT_TOKEN: ${AGENT_TOKEN:?set AGENT_TOKEN in .env}
volumes:
- ./proxy.conf:/app/proxy.conf:ro
restart: unless-stopped
prometheus:
image: prom/prometheus:v3.14.0
ports:
- "127.0.0.1:9090:9090" # On this host only: reach it with an SSH tunnel, or run Grafana beside it
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus-data:/prometheus
depends_on:
- proxy
restart: unless-stopped
volumes:
prometheus-data:
// A proxy that agents embedded in cloud apps connect to. Its certificate, key, and the agents' token come from the
// environment (CERT_CHAIN_PEM, PRIVATE_KEY_PEM, AGENT_TOKEN), so this file holds no secret. The setup is described at
// https://prometheus-proxy.com/cloud-platforms/
proxy {
// Prometheus scrapes the apps' metrics here. Keep it off the internet: anyone who reaches it scrapes through the
// agents
http.port = 8080
// The agents connect here, over TLS and with the token: the one port to open to the internet
agent.port = 50051
// The proxy's own metrics, which Prometheus scrapes too: how many agents are connected, and how their scrapes go
metrics {
enabled = true
port = 8082
}
}
# Prometheus beside the proxy. It reaches the proxy on the compose network, where the proxy serves each app's metrics
# at the path its agent registered. The setup is described at
# https://prometheus-proxy.com/cloud-platforms/
global:
scrape_interval: 30s
scrape_configs:
# The app's metrics, which its embedded agent registered as my_app_metrics. Prometheus sends its scrape timeout along,
# and the agent stops the app's scrape at it.
- job_name: my-app
metrics_path: /my_app_metrics
static_configs:
- targets: [ 'proxy:8080' ]
relabel_configs:
# Name the app, not the proxy its metrics come through
- target_label: instance
replacement: my-app
# The proxy's own metrics: proxy_agent_map_size counts the connected agents
- job_name: prometheus-proxy
static_configs:
- targets: [ 'proxy:8082' ]
AWS
-
On an EC2 instance with Docker, copy
examples/cloud-proxyover, put the certificate, key, and token in its.env, and rundocker compose up --detach. Open the agent port in the instance's security group: -
An app on ECS in the same VPC can reach the instance on its private address instead: admit the app's security group (
--source-groupin place of--cidr), so the agent port needn't face the internet at all. Keep the token and TLS anyway. - To run the proxy on ECS on Fargate, as a service with one task, put a Network Load Balancer in front of it with a TCP listener on 50051. A TCP listener forwards the connection as it is, so the proxy does its own TLS, and the agent's heartbeat, which it sends whenever the connection has been idle for a few seconds, keeps the connection inside the listener's idle timeout. Run Prometheus as another service that reaches the proxy's scrape port inside the VPC.
- On EKS, see Kubernetes.
GCP
-
On a Compute Engine VM with Docker, copy
examples/cloud-proxyover, put the certificate, key, and token in its.env, and rundocker compose up --detach. Open the agent port with a firewall rule that applies to the VM's network tag: -
An app on Cloud Run in the same project can reach the VM on its internal address instead, through Direct VPC egress, so the agent port needn't face the internet at all. Keep the token and TLS anyway.
- On GKE, see Kubernetes.
- The proxy can run on Cloud Run too, with constraints. A
Cloud Run service receives traffic on one port, so that port is the agent port (
--port=50051), served over end-to-end HTTP/2 (--use-http2) with TLS from Google's front end, and Prometheus runs as a sidecar that scrapes the proxy onlocalhost:8080. The front end is a reverse proxy, so turn the transport filter off on both sides. The agent's stream ends at the service's request timeout, at most 60 minutes, and the agent reconnects. The proxy needs the same single, always-on instance as the app. A VM is simpler.
Azure
-
On an Azure VM with Docker, copy
examples/cloud-proxyover, put the certificate, key, and token in its.env, and rundocker compose up --detach. Open the agent port in the VM's network security group withaz vm open-port, which admits it from any source; for a narrower one, add the rule withaz network nsg rule createinstead: -
An app on Container Apps, in an environment with a virtual network that reaches the VM's, can reach the VM on its private address instead, so the agent port needn't face the internet at all. Keep the token and TLS anyway.
- On AKS, see Kubernetes.
- The proxy can run on Container Apps too, as an app with one replica. External TCP ingress on port 50051 forwards the agent's connection as it is, so the proxy does its own TLS; Container Apps offers it only in an environment with a virtual network. Run Prometheus as another app in the environment, and give it the proxy's scrape port as an additional, internal TCP port.
DigitalOcean
- On a Droplet with Docker, copy
examples/cloud-proxyover, put the certificate, key, and token in its.env, and rundocker compose up --detach. Give the Droplet a Cloud Firewall that admits only port 50051, and SSH from your own address. - App Platform is a poorer fit. DigitalOcean documents HTTP/2 for its services, which gRPC needs, but its limits page says it doesn't support gRPC. Its edge is also a reverse proxy, which needs the transport filter off on both sides. Try it with your own app before relying on it.
- On DigitalOcean Kubernetes, see Kubernetes.
Railway
- Railway's HTTPS domains don't carry the agent's gRPC through to the container, so give the proxy service a
TCP Proxy on port 50051, and no HTTPS domain. Railway gives it a
host and port, such as
shuttle.proxy.rlwy.net:15140, which is the agent'sPROMETHEUS_PROXY. - The TCP Proxy forwards the connection as it is and adds no TLS, so the proxy does its own: deploy the
pambrose/prometheus-proxyimage, 4.6.0 or later, and setCERT_CHAIN_PEM,PRIVATE_KEY_PEM, andAGENT_TOKENas the service's variables, withMETRICS_ENABLED=truefor the proxy's own metrics. The image needs no config file for this. - Name the service
prometheus-proxy, so Prometheus reaches it atprometheus-proxy.railway.internal:8080on the private network. In an older environment, whose private network carries IPv6 only, set the proxy'sproxy.http.hostto"::", in a config it reads throughPROXY_CONFIG, so Prometheus can reach the scrape port. - Run Prometheus as another service, built from a Dockerfile that copies in a
prometheus.ymllike the example's, with its targets onprometheus-proxy.railway.internal, and a volume for its data.
On the Prometheus Side
The example's prometheus.yml scrapes the app's metrics at the path its agent registered, /my_app_metrics, and
sets instance to the app's name rather than the proxy's address. Prometheus sends its scrape timeout along, and the
agent stops the app's scrape at it.
It scrapes the proxy's own metrics too. When the app stops, or its agent can't connect, the proxy no longer serves the
path, and the app's job goes down: up{job="my-app"} == 0 is the alert to set. proxy_agent_map_size counts the
agents connected. See Grafana & Alerting for dashboards and rules built on the proxy's metrics.