Spring Boot Demo
The repository ships a demo of the agent embedded in a Spring Boot app written in Java: a roastery, which roasts the coffee shop's beans, runs the agent inside its own JVM, started by a Spring bean from settings Spring Boot binds. The agent reads the metrics Spring Boot's Actuator serves behind Spring Security, and sends them to a proxy, which Prometheus scrapes.
It shows what's particular to Spring Boot:
- the agent's lifecycle with the app's;
- the agent's connection in Actuator's health;
- Actuator's metrics behind a password;
- the versions of the agent's libraries that Spring Boot's dependency management picks;
- a graceful shutdown that takes the agent down first.
The Embedded Demo's coffee shop shows the rest with an app in Kotlin.
%%{init: {'flowchart': {'curve': 'linear'}}}%%
graph LR
P[Prometheus :9090] -->|scrapes /roastery_metrics| Proxy[Proxy, dashboard :8094]
%% The agent connects out to the proxy, so its arrow points back. The invisible link keeps the app to the right.
Proxy ~~~ Agent
Agent[embedded agent, a SmartLifecycle bean] -->|gRPC over TLS, with a token| Proxy
Agent -->|reads /actuator/prometheus on 127.0.0.1| App[roastery: Spring Boot, Actuator behind Spring Security]
subgraph host[Proxy host]
P
Proxy
end
subgraph roastery[roastery: one container, one JVM]
Agent
App
end
What you need
On macOS or Linux, the demo needs:
- Docker with Compose v2: Docker Desktop, or Docker Engine with the Compose plugin.
git, to clone the repository, and GNUmake. On macOS, the Xcode command-line tools (xcode-select --install) have both. Where GNU make is installed asgmake, typegmakewherever these instructions saymake.curl, which the demo uses to wait for each service to answer.- A web browser.
make up runs the latest release's proxy image, from Docker Hub, and builds the roastery inside Docker with Maven,
against the latest release's library, from Maven Central, so it needs no JDK or Maven. make up-local builds the
library, the proxy, and the roastery from the checkout, which takes a JDK 17 or later and Maven 3.9 or later.
Running it
Clone the repository at the latest release, then run the demo's commands from its own directory, where make help lists
its targets:
git clone -c advice.detachedHead=false --depth 1 --branch 4.6.0 https://github.com/pambrose/prometheus-proxy.git
cd prometheus-proxy/demos/spring-boot
make up # builds the roastery, starts everything, prints the URLs
make tour # walks through the parts below, pausing before each step
make down # stops it and removes its containers and network
The first make up pulls the images and builds the roastery, which takes a few minutes. After that it starts in under
a minute.
| What | URL |
|---|---|
| The roastery's status page | http://localhost:8490 |
| Proxy dashboard | http://localhost:8494/dashboard |
| Prometheus | http://localhost:9490 |
| The roastery's metrics, through the proxy | http://localhost:8480/roastery_metrics |
The ports are the Basics Demo's plus 400, so the demos can run side by side. Every port binds to
127.0.0.1 only. To move one that is already taken, set SPRING_BOOT_DEMO_APP_PORT, SPRING_BOOT_DEMO_DASHBOARD_PORT,
SPRING_BOOT_DEMO_PROMETHEUS_PORT, or SPRING_BOOT_DEMO_PROXY_PORT, for example SPRING_BOOT_DEMO_APP_PORT=8491 make up.
What's running
| Service | Role |
|---|---|
roastery |
A Spring Boot app in Java that roasts a batch of beans every two seconds and ships some to the coffee shop every five, counting both in Micrometer meters. Actuator serves them, with the JVM's and Tomcat's, at /actuator/prometheus, behind Spring Security's Basic Auth. The agent runs inside it, and reads /actuator/prometheus on 127.0.0.1. |
proxy |
The stock proxy image, which takes its TLS certificate and key, and the agent's token, from environment variables, with no file mounted. Its dashboard is on. |
prometheus |
Scrapes the roastery's path through the proxy every 5s, and the proxy's own metrics. |
Where the agent is configured
The roastery starts its agent from a bean, PrometheusAgentLifecycle, in
PrometheusAgentLifecycle.java.
It builds the agent's settings with EmbeddedAgentConfig.builder(), the Java form of
embeddedAgent { }:
/**
* Runs the prometheus-proxy agent embedded in the roastery, from its {@link AgentSettings}, in memory: no config file
* holds the token, the proxy's certificate, or the password for {@code /actuator/prometheus}. The agent connects to
* the proxy, so the app opens no port for Prometheus, and answers each scrape by reading Actuator's
* {@code /actuator/prometheus} on 127.0.0.1.
*
* <p>As a {@link SmartLifecycle} in the last phase, it starts once the web server is serving, so the agent's first
* scrape finds Actuator, and stops first when the app shuts down, before the web server drains: the agent disconnects
* from the proxy, which drops the roastery's path, and no scrape reaches a server that's going away.
*
* <p>The agent stays off without a proxy, or when it can't start: the roastery's metrics aren't worth an outage.
*/
@Component
public class PrometheusAgentLifecycle implements SmartLifecycle {
private static final Logger logger = LoggerFactory.getLogger(PrometheusAgentLifecycle.class);
private final AgentSettings settings;
private final BuildProperties build;
private final Environment environment;
private volatile EmbeddedAgentInfo agent;
public PrometheusAgentLifecycle(AgentSettings settings, BuildProperties build, Environment environment) {
this.settings = settings;
this.build = build;
this.environment = environment;
}
@Override
public void start() {
if (!settings.enabled()) {
logger.info("roastery.monitoring.proxy-endpoint isn't set, so the Prometheus agent stays off");
return;
}
// The port the web server took, which Spring Boot sets once it's serving
int port = environment.getRequiredProperty("local.server.port", Integer.class);
logger.info("Starting the Prometheus agent with {}", settings);
try {
agent =
EmbeddedAgentConfig.builder()
// The version tells the old and new deployments apart on the proxy's dashboard and in its logs
.name("roastery-" + build.getVersion())
.proxy(settings.proxyEndpoint())
.token(settings.token())
// The certificate the proxy's is signed with, as PEM or its base64, and the name it was issued to
.trustCertificatePem(settings.certificate(), settings.authority())
// Actuator's metrics; the agent sends the credentials as Basic Auth
.scrapeSelf(
"roastery_metrics",
port,
settings.metricsUser(),
settings.metricsPassword(),
"actuator/prometheus",
Map.of())
.build()
.start(false);
logger.info(
"Prometheus agent {} started; it connects to the proxy at {}",
agent.getAgentName(),
settings.proxyEndpoint());
} catch (ConfigLoadException e) {
logger.error("The Prometheus agent's settings are invalid, so it stays off: {}", e.getMessage());
} catch (IllegalStateException e) {
logger.error("The Prometheus agent could not start, so it stays off: {}", e.getMessage());
}
}
@Override
public void stop() {
EmbeddedAgentInfo started = agent;
if (started != null) {
logger.info("Stopping the Prometheus agent {}", started.getAgentName());
started.shutdown();
agent = null;
}
}
// Running while the agent is: one that's off, or failed to start, has nothing to stop
@Override
public boolean isRunning() {
return agent != null;
}
/** The running agent's handle, for its health indicator, or null when it's off. */
public EmbeddedAgentInfo agent() {
return agent;
}
}
- The settings:
AgentSettings, a@ConfigurationPropertiesrecord, binds them fromroastery.monitoringinapplication.yml, whose values read the environment the coffee shop's agent reads:PROMETHEUS_PROXY,PROMETHEUS_AGENT_TOKEN,PROMETHEUS_PROXY_CERT, and the rest. ItstoStringleaves the secrets out, so the app can log it. - The password:
SecurityConfiglets only the agent's user read/actuator/prometheus; the status page and/actuator/healthare open.scrapeSelfpercent-encodes the credentials into the scrape URL. - Its health:
PrometheusAgentHealthIndicatorputs the agent in/actuator/health, asprometheusAgent, fromEmbeddedAgentInfo.isConnectedandproxyEndpoint: UP while it's connected, with the proxy's address, and UNKNOWN, not DOWN, when it's off or reconnecting, so a lost proxy leaves the roastery's health UP../demo.sh healthshows it.
In Spring Boot has the same points for an app of your own.
Spring Boot's versions of the agent's libraries
Spring Boot's dependency management, which an app takes from spring-boot-starter-parent, sets the version of every
library it knows, whatever version a dependency asks for. Some of the agent's libraries are among them, and the
roastery runs kotlinx-serialization, Jackson 2, Logback, and gRPC older than the agent is built with.
./demo.sh versions prints both, from the jars the running roastery holds and the project's version catalog.
The agent runs with Spring Boot's versions, as the roastery shows. On every pull request, the project builds the
roastery and runs its
RoasteryAgentTest,
which starts the agent against a proxy and scrapes through it, so a Spring Boot release that breaks the agent fails
there. Were one to break it, a property in the app's pom sets that library back: kotlin-serialization.version,
jackson-2-bom.version, logback.version, or grpc-java.version.
A graceful shutdown
./demo.sh stop-app stops the roastery with SIGTERM, as a platform does on a deploy or a scale-down. The agent, in the
last lifecycle phase, stops first, and the roastery's log shows the order:
Stopping the Prometheus agent roastery-1.0
Commencing graceful shutdown. Waiting for active requests to complete
Graceful shutdown complete
./demo.sh start-app starts it again, and the agent registers its path once more.
Scenario reference
| Command | Effect |
|---|---|
./demo.sh settings |
Shows the agent's settings as Spring Boot bound them, and the agent starting |
./demo.sh peek |
Tries /actuator/prometheus without the password, then reads it through the proxy |
./demo.sh health |
Shows the agent in Actuator's /actuator/health: connected, and to which proxy |
./demo.sh versions |
Compares the versions of the agent's libraries in the roastery with the agent's own |
./demo.sh stop-app / start-app |
Stops the roastery with SIGTERM, or starts it again |
make status |
The containers, the roastery's state, and the URLs |
make logs SERVICE=roastery |
Follows one service's log |
Going further
- The demo's README has its guided tour
and lists the files that set it up. The roastery's code is in
demos/spring-boot/app. - Embedded Agent covers the agent in a Spring Boot app, and the API the roastery uses.
- The Embedded Demo runs a Kotlin app with the agent embedded under load, with Grafana, alerts, a deploy, the proxy going away, and secrets from the environment.
- Cloud Platforms sets an app with the agent embedded up for real, and Packaging the App covers its image.