Embedded Agent
If your application runs on the JVM, you can embed the prometheus-agent directly in your app instead of running a separate agent process.
When to Use
- Your application already runs on the JVM (Java, Kotlin, Scala, etc.)
- You want to avoid managing a separate agent process
- You need tight integration between your app and the agent lifecycle
Adding the Dependency
Usage
import io.prometheus.Agent
fun main() {
// Start the agent in the background
val agentInfo = Agent.startAsyncAgent(
configFilename = "agent-config.conf",
exitOnMissingConfig = false,
)
println("Agent started: ${agentInfo.agentName}")
// Your application code runs here...
// The agent runs in background coroutines
// Stop the agent when your application exits
agentInfo.shutdown()
}
import io.prometheus.Agent;
import io.prometheus.agent.EmbeddedAgentInfo;
public class EmbeddedAgentJavaExample {
public static void main(String[] args) {
// Start the agent in the background. false makes a config error throw instead of exiting the JVM; a third
// argument, false, would leave out the startup banner.
EmbeddedAgentInfo agentInfo =
Agent.startAsyncAgent("agent-config.conf", false);
System.out.println("Agent started: " + agentInfo.getAgentName());
// Your application code runs here...
// The agent runs in background threads
// Stop the agent when your application exits
agentInfo.shutdown();
}
}
Agent Configuration
The embedded agent uses the same HOCON configuration as the standalone agent:
// Minimal config for an embedded agent:
agent {
proxy.hostname = "proxy-host.example.com"
pathConfigs: [
{
name: "My App metrics"
path: my_app_metrics
url: "http://localhost:9100/metrics"
}
]
}
Configuring in Code
Since 4.6.0, an application that holds the agent's settings itself, such as a token, the proxy's certificate, or
credentials for its own /metrics, can build them in code, so that no config file holds them on disk.
EmbeddedAgentConfig has a typed method for each setting: build it with embeddedAgent { }
from Kotlin, or EmbeddedAgentConfig.builder() from Java, and call start():
import io.prometheus.agent.EmbeddedAgentInfo
import io.prometheus.agent.embeddedAgent
import io.prometheus.common.ConfigLoadException
// Starts the agent from the app's own settings, in memory, so no config file holds the token or the password. Returns
// null when the agent is off, or can't start: the app's metrics aren't worth an outage.
fun startPrometheusAgent(
appVersion: String,
appPort: Int,
): EmbeddedAgentInfo? {
// host:port, the proxy's agent port; the agent stays off without it
val proxyEndpoint = env("PROMETHEUS_PROXY").ifEmpty { return null }
return try {
embeddedAgent {
// The version tells the old and new revisions apart during a deploy
name = "my-app-$appVersion"
proxy = proxyEndpoint
token = env("PROMETHEUS_AGENT_TOKEN")
// The proxy's certificate, as PEM or its base64, and the name it was issued to
trustCertificatePem(pem = env("PROMETHEUS_PROXY_CERT"), authority = "prometheus-proxy")
// The app's own /metrics, on 127.0.0.1; the agent sends the credentials as Basic Auth
scrapeSelf(
path = "my_app_metrics",
port = appPort,
username = env("METRICS_USER"),
password = env("METRICS_PASSWORD"),
)
}.start(logBanner = false)
} catch (e: ConfigLoadException) {
System.err.println("The Prometheus agent's settings are invalid: ${e.message}")
null
} catch (e: IllegalStateException) {
System.err.println("The Prometheus agent could not start: ${e.message}")
null
}
}
private fun env(name: String) = System.getenv(name).orEmpty().trim()
import io.prometheus.agent.EmbeddedAgentConfig;
import io.prometheus.agent.EmbeddedAgentInfo;
import io.prometheus.common.ConfigLoadException;
public class EmbeddedAgentConfigJavaExample {
// Starts the agent from the app's own settings, in memory, so no config file holds the token or the password.
// Returns null when the agent is off, or can't start: the app's metrics aren't worth an outage.
public static EmbeddedAgentInfo startPrometheusAgent(String appVersion, int appPort) {
// host:port, the proxy's agent port; the agent stays off without it
String proxy = env("PROMETHEUS_PROXY");
if (proxy.isEmpty()) {
return null;
}
try {
return EmbeddedAgentConfig.builder()
// The version tells the old and new revisions apart during a deploy
.name("my-app-" + appVersion)
.proxy(proxy)
.token(env("PROMETHEUS_AGENT_TOKEN"))
// The proxy's certificate, as PEM or its base64, and the name it was issued to
.trustCertificatePem(env("PROMETHEUS_PROXY_CERT"), "prometheus-proxy")
// The app's own /metrics, on 127.0.0.1; the agent sends the credentials as Basic Auth
.scrapeSelf("my_app_metrics", appPort, env("METRICS_USER"), env("METRICS_PASSWORD"))
.build()
.start(false);
} catch (ConfigLoadException e) {
System.err.println("The Prometheus agent's settings are invalid: " + e.getMessage());
return null;
} catch (IllegalStateException e) {
System.err.println("The Prometheus agent could not start: " + e.getMessage());
return null;
}
}
private static String env(String name) {
String value = System.getenv(name);
return value == null ? "" : value.trim();
}
}
- The certificate is PEM text, or its base64
(
agent.tls.trustCertCollectionPem). scrapeSelfpoints the agent at the app's own metrics on127.0.0.1. It percent-encodes a username and password into the scrape URL, whatever their characters; the agent sends them as Basic Auth and leaves them out of its logs.- The settings fall back to the built-in defaults, as a config file's do, and a setting the agent can't use throws
ConfigLoadExceptionfromstart(). - The environment variables of single settings still take precedence, such as
AGENT_TOKENorPROXY_HOSTNAME, as they do over a config file: give your application's own settings other names, unless you mean the agent to read them.AGENT_CONFIGisn't read.
A Config of Your Own
For a setting the builder has no method for, setting(key, value) sets it by its key, relative to agent, such as
setting("scrapeTimeoutSecs", 5). Or build the whole tree as a Typesafe Config rooted at agent, the tree a config
file holds, and start it with Agent.startAsyncAgent(config). ConfigFactory.parseString, ConfigFactory.parseMap,
and ConfigValueFactory.fromMap build one, and a builder's toConfig() gives the one it holds. It's resolved, as a
config file is, so a ${?ENV_VAR} substitution in it works.
Cloud Platforms builds on this to run an embedded agent in an app on ECS on Fargate, Cloud Run, Azure Container Apps, or another platform, with the proxy on AWS, GCP, Azure, DigitalOcean, or Railway, and the Embedded Demo runs it on one machine, in a Ktor app.
In Spring Boot
A Spring Boot app runs the agent from a bean, so it starts and stops with the app. The Spring Boot Demo's roastery does each of these:
- A
SmartLifecyclebean in the last phase, the default, starts the agent once the web server is serving, so the agent's first scrape finds the app's metrics, and stops it first on shutdown, before the web server drains: the agent disconnects, the proxy drops its paths, and no scrape reaches a server that's going away. - Actuator's metrics: with
micrometer-registry-prometheuson the classpath, Actuator serves them at/actuator/prometheus, soscrapeSelftakes"actuator/prometheus"as itsmetricsPath, with the credentials the app's Spring Security asks for. - The settings: a
@ConfigurationPropertiesrecord can hold them, withapplication.ymlreading each from the environment, as${PROMETHEUS_AGENT_TOKEN:}does. Give it atoStringthat leaves the secrets out, so the app can log it. - Spring Boot's dependency management sets the versions of the libraries it knows, some of the agent's among them:
kotlinx-serialization, Jackson 2, Logback, and gRPC run older than the agent's own. The agent runs with them, and
the demo's roastery runs it at Spring Boot's versions on every pull request
(Spring Boot's versions). Were a Spring Boot
release to break it, a property in the app's pom sets that library back:
kotlin-serialization.version,jackson-2-bom.version,logback.version, orgrpc-java.version. - A health indicator can report the agent in Actuator's
/actuator/healthfromEmbeddedAgentInfo.isConnectedandproxyEndpoint: UP while it's connected, and UNKNOWN, not DOWN, when it's off or reconnecting, so a lost proxy leaves the app's health UP. - Logging: see Logging for the line Spring Boot's
application.ymltakes.
API Reference
Agent.startAsyncAgent()
Starts the agent in background threads/coroutines.
| Parameter | Type | Description |
|---|---|---|
configFilename |
String |
Path to the HOCON config file |
exitOnMissingConfig |
Boolean |
Pass false to embed: a config error or invalid option throws ConfigLoadException. true exits the JVM, as the standalone agent does |
logBanner |
Boolean |
Log the startup banner (default: true, which Java callers get by leaving it out too) |
Or, to start the agent from a config built in memory (see A Config of Your Own):
| Parameter | Type | Description |
|---|---|---|
config |
Config |
The agent's configuration, rooted at agent, as a Typesafe Config |
logBanner |
Boolean |
Log the startup banner (default: true, which Java callers get by leaving it out too) |
This form is always embedded: a config error throws ConfigLoadException, and never exits the JVM.
Returns: EmbeddedAgentInfo
Throws: ConfigLoadException, when exitOnMissingConfig is false or the config is in memory, for a config that
can't be loaded or parsed, or for a setting the agent can't use, from the config or an environment variable:
- a value of the wrong type or out of range, such as
scrapeTimeoutSecs = "abc"or0; - an unresolved
${…}substitution; - a malformed proxy endpoint;
- a duplicate
pathConfigsorfiltersentry, or a filter regex that doesn't compile; - an HTTPS trust store (
agent.http.trustStorePath) or TLS certificate file that is missing or malformed, or a TLS certificate or key given as PEM that can't be read, or given as both a file and PEM.
Credentials in a config URL are redacted from its message and from every message in its cause chain. The cause is the
exception that rejected the setting, usually an IllegalArgumentException. When that exception's message held a URL
with credentials, as a Typesafe ConfigException's does for a value read from a config URL, the cause is a stand-in
ConfigLoadException with the redacted message and the original stack trace.
IllegalStateException when the agent fails to start (for example, its admin or metrics port is already in
use), with the startup failure as its cause. A failed start releases everything the agent had opened.
EmbeddedAgentConfig
An embedded agent's settings, built in code (see Configuring in Code). From Kotlin,
embeddedAgent { … } builds one, its block calling the builder's methods; from Java, EmbeddedAgentConfig.builder()
returns the builder, whose methods chain and end with build(). A method called twice keeps its last value, and
scrapeSelf and path add a path each time.
| Builder method | Sets | Description |
|---|---|---|
name(name) |
agent.name |
The agent's name, as the proxy's dashboard and logs show it |
proxy(endpoints…) |
agent.proxy.endpoints |
The proxy's agent port, as host or host:port (the port defaults to 50051), or several, in the order the agent fails over through them |
token(token) |
agent.agentToken |
The token the agent presents to the proxy |
trustCertificatePem(pem, authority) |
agent.tls.trustCertCollectionPem, agent.tls.overrideAuthority |
The certificate the agent trusts the proxy's by, as PEM or its base64, which turns TLS on, and, optionally, the name the proxy's certificate was issued to |
clientCertificatePem(certChainPem, privateKeyPem) |
agent.tls.certChainPem, agent.tls.privateKeyPem |
The agent's own certificate chain and key, as PEM or its base64, for a proxy that requires mutual TLS |
consolidated(consolidated) |
agent.consolidated |
Register the agent's paths in consolidated mode; consolidated() alone turns it on |
agentMetrics(port) |
agent.metrics.enabled, agent.metrics.port |
Turn on the agent's own metrics endpoint, which is off unless asked for |
scrapeSelf(path, port, username, password, metricsPath, labels) |
agent.pathConfigs |
Scrape the app at http://127.0.0.1:<port>/<metricsPath> (default metrics) and serve it at path; the rest are optional |
path(path, url, username, password, labels) |
agent.pathConfigs |
Scrape any target at url and serve it at path; the rest are optional, and the URL then carries no credentials of its own |
setting(key, value) |
the key, under agent |
Any other setting, such as http.clientTimeoutSecs; it takes precedence over a method that sets the same key |
In embeddedAgent { }, name, proxy, token, consolidated, and agentMetricsPort are properties too, so
name = "my-app" does what name("my-app") does; proxy takes several endpoints separated by commas, as
PROXY_HOSTNAME does, and null takes name, proxy, token, or agentMetricsPort out. Java sees only the
methods.
A username and password are percent-encoded into the URL and sent as Basic Auth; both empty, as unset environment
variables give, means none. Labels are a map of names to values. A value the builder can tell is wrong, such as a blank
path or a port out of range, throws ConfigLoadException, as start() does for a setting the agent rejects.
EmbeddedAgentConfig member |
Type | Description |
|---|---|---|
start(logBanner) |
EmbeddedAgentInfo |
Starts the agent, as startAsyncAgent(config) does, and throws as it does; logBanner defaults to true |
toConfig() |
Config |
The settings as the tree a config file holds, rooted at agent |
toString() |
String |
The settings, with the token, passwords, and PEMs shown only as set, and a path's credentials masked, so an app can log them |
EmbeddedAgentInfo
The handle startAsyncAgent() and EmbeddedAgentConfig.start() return.
| Member | Type | Description |
|---|---|---|
launchId |
String |
Unique ID for this agent launch, new each time an agent starts |
agentName |
String |
Name of the agent: agent.name, or Unnamed-<hostname> when none is set |
isConnected |
Boolean |
Whether the agent is connected to a proxy and has registered its paths there, so Prometheus can scrape through it now; read live, for a health check |
proxyEndpoint |
String? |
The proxy it's connected to, as host:port, or null when it isn't |
shutdown() |
Unit |
Stops the agent and blocks until it has stopped; calling it again, or on a failed agent, does nothing |
isConnected is false before the agent's first connection, while it reconnects after losing one, and once it has
stopped. A host's health check can report it, though without failing the app's own health: a lost proxy shouldn't
take the app down.
Call shutdown() from your application's shutdown hook, or wherever it stops its own services. It closes the
connection to the proxy, drains the scrapes in flight, and releases the admin and metrics ports, so the agent's
connection and background work don't outlive your application.
Agent.startSyncAgent()
Starts an agent from command-line arguments, as the standalone agent's main() does, and returns once the agent is
running. It returns no handle: the agent runs on its own non-daemon thread, so a JVM whose main() calls this keeps
running until the agent stops. Use startAsyncAgent() to embed an agent you stop yourself.
| Parameter | Type | Description |
|---|---|---|
args |
Array<String> |
The agent's CLI options, such as --config agent.conf |
exitOnMissingConfig |
Boolean |
true behaves as the standalone agent does, exiting the JVM on a config error. false throws ConfigLoadException instead, and with no --config falls back to the built-in defaults |
Throws: ConfigLoadException, when exitOnMissingConfig is false, in the cases
startAsyncAgent() lists, and for -v or -u; IllegalStateException when the agent fails
to start, with the failure as its cause. It waits up to 30 seconds for the agent to start.
How It Works
When you call startAsyncAgent():
- The agent reads the configuration file, or the config you passed
- Starts its admin and metrics servers, if enabled, and waits for them to come up
- Returns with agent metadata, without waiting for a proxy connection
- Connects to the proxy and registers paths in the background
- Keeps running in the background, processing scrape requests
Your application code runs normally while the agent handles metrics scraping in the background.
The embedded agent's metrics register in the Prometheus Java client 1.x default registry
(io.prometheus.metrics.model.registry.PrometheusRegistry.defaultRegistry). A host that serves its own metrics
from that registry exposes the agent's too; a host still on the 0.x client (io.prometheus:simpleclient) does
not see them. Every agent in the JVM, two at once or one started again after shutdown(), shares one set of agent
metrics, and each series carries the agent's launch_id. An agent's series go when it shuts down. A host metric
with one of the agent metrics' names makes startAsyncAgent() throw.
The agent's logLevel (agent.logLevel, AGENT_LOG_LEVEL, or --log_level) sets the level of the io.prometheus
loggers only, so it doesn't change your application's own logging. A standalone agent sets the root logger's level.
Config file location
The config file path is relative to your application's working directory. Use an absolute path if needed.
Logging
Your application configures the agent's logging. The agent logs through SLF4J to Logback, and the library jar carries
no Logback configuration, so your own logback.xml (or, in Spring Boot, logback-spring.xml and the logging.level
properties) applies to the agent's loggers as it does to the rest. With no configuration at all, Logback logs
everything at DEBUG, gRPC and Netty included.
The standalone agent's configuration sets io.grpc.internal to WARN, so that only gRPC's internal warnings and errors
reach its log; you may want the same:
In Spring Boot, set logging.level.io.grpc.internal=WARN instead.
Upgrading from 4.4.x
startAsyncAgent()throwsConfigLoadExceptionfor a config that loads but holds a setting the agent can't use, as listed underAgent.startAsyncAgent(). Through 4.4.2 those escaped as anIllegalArgumentExceptionor a TypesafeConfigException, and aConfigException's message began with the config URL, credentials included. Code that caught either should catchConfigLoadException; theIllegalArgumentExceptionis now its cause.- The library jar no longer carries a
logback.xml. Through 4.4.2 it did, and Logback, and Spring Boot ahead of yourlogback-spring.xml, took it for your application's configuration. An application with no Logback configuration of its own now gets Logback's default, which logs everything at DEBUG, so add one (see Logging); one whose own configuration was being replaced gets it back. - Maven builds now get Jackson 2.23.0 and netty-tcnative 2.0.86.Final, natives included, through the agent, where they got Jackson 2.12.7 and the 2.0.84.Final natives. A Jackson your build declares itself still wins.
- The agent's generated gRPC classes need protobuf-java 4.36.2 or later, which Maven and Gradle builds now resolve
through it, where they got 3.25.9. Your own code generated by protoc 3.x keeps working on it. A build that pins
protobuf-java to 3.x must move the pin to 4.36.2 or later, or the agent fails when it connects, with a
NoClassDefFoundErrorforcom.google.protobuf.RuntimeVersion.
Upgrading from 4.1.x
- The agent moved from the Prometheus Java client 0.x (
io.prometheus:simpleclient0.16.0) to 1.x (io.prometheus:prometheus-metrics-core1.9.0), and its metrics now register in the 1.x default registry. A host that serves the 0.xCollectorRegistryno longer exposes them. To serve both from one endpoint, serve the 1.x registry and bridge your 0.x metrics into it: addio.prometheus:prometheus-metrics-simpleclient-bridge, at the same version as theprometheus-metrics-corethe agent brings, and callSimpleclientCollector.builder().register()once at startup. simpleclientno longer arrives through the agent. If your application relied on getting it that way, declare it yourself.- With the agent's metrics endpoint on, the
_createdseries of its counters and histograms are gone (setIO_PROMETHEUS_EXPORTER_INCLUDE_CREATED_TIMESTAMPS=trueto bring them back), and with the JVM exports on, the JVM memory metrics are renamed (see JVM Metrics).
Upgrading from 4.0.x
grpc-netty-shadedno longer arrives through the agent, which uses the unshadedgrpc-nettytransport. If your application relied on getting it that way, declare it yourself.startAsyncAgent()now waits for the agent to start and throws if it fails, where it used to return a handle to an agent that had failed (seeAgent.startAsyncAgent()).- When a config URL's error message has to be redacted, the
ConfigLoadException's cause is a stand-in exception with the redacted message and the original stack trace, not Typesafe'sConfigException, so code that checkscause is ConfigExceptionno longer matches it.