Skip to content

Network Demo

The Basics Demo shows the proxy, its dashboard, and Grafana with a perfect network between the agents and the proxy. This second demo puts a simulated WAN there, the link through the firewall that the product exists to cross, and shows what a far, thin, dead, flaky, or missing link does, and how the proxy and agents cope. Every screenshot on this page comes from it.

The WAN is Toxiproxy, with four links: one from each agent to each proxy. Each agent lists its link to the primary proxy first and its link to the standby second, so the dashboards name the link an agent connected through.

%%{init: {'flowchart': {'curve': 'linear'}}}%%
graph LR
  G[Grafana :3100] --> P[Prometheus :9190]
  P -->|discovery and scrapes| Proxy[Proxy, dashboard :8194]
  P -->|discovery and scrapes| Standby[Standby proxy, dashboard :8195]
  %% Each agent connects out, through the WAN, so its arrows point back. The invisible links keep the WAN and the
  %% agents to the right of the proxies, as on the Basics Demo page.
  Proxy ~~~ WAN
  Standby ~~~ WAN
  WAN ~~~ HQ
  WAN ~~~ Branch
  WAN[WAN: Toxiproxy] -->|primary links| Proxy
  WAN -.->|standby links| Standby
  HQ[agent-hq] -->|gRPC| WAN
  Branch[agent-branch] -->|gRPC| WAN
  HQ -->|HTTP fetch| App[network-app: simulated services]
  Branch -->|HTTP fetch| App

  subgraph Outside Firewall
    G
    P
    Proxy
    Standby
  end

  subgraph Inside Firewall
    HQ
    Branch
    App
  end

What you need

The proxy and agent are the latest release's images, from Docker Hub, the same ones the Basics Demo runs, so there's nothing to build. On macOS or Linux, the demo needs only:

  • Docker with Compose v2: Docker Desktop, or Docker Engine with the Compose plugin.
  • git, to clone the repository, and GNU make. On macOS, the Xcode command-line tools (xcode-select --install) have both. Where GNU make is installed as gmake, type gmake wherever these instructions say make.
  • curl, which the demo uses to wait for each service to answer.
  • A web browser.

Running it

Clone the repository at the latest release, then run the demo's commands from its own directory, where make help lists its targets:

git clone -c advice.detachedHead=false --depth 1 --branch 4.6.0 https://github.com/pambrose/prometheus-proxy.git
cd prometheus-proxy/demos/network
make up      # pulls the images, starts everything, prints the URLs
make tour    # walks through every scenario below, pausing before each step
make down    # stops it and removes its containers and network

The first make up pulls the proxy, agent, Prometheus, Grafana, Zipkin, Toxiproxy, and Python images, which takes a few minutes. After that it starts in under a minute, and runs without a network connection.

What URL
Proxy dashboard http://localhost:8194/dashboard
Standby proxy's dashboard http://localhost:8195/dashboard
Grafana (no sign-in needed; admin / admin to edit) http://localhost:3100
Prometheus http://localhost:9190 (alerts at /alerts)
The proxy's service discovery http://localhost:8180/discovery
Zipkin http://localhost:9511/zipkin/

Each port is the Basics Demo's plus 100, so the two demos can run side by side. Every port binds to 127.0.0.1 only: the dashboards have no authentication and Grafana keeps its default admin password, which suits a laptop and nothing shared. To move a port that is already taken, set NETWORK_DEMO_DASHBOARD_PORT, NETWORK_DEMO_STANDBY_DASHBOARD_PORT, NETWORK_DEMO_GRAFANA_PORT, NETWORK_DEMO_PROMETHEUS_PORT, NETWORK_DEMO_PROXY_PORT, or NETWORK_DEMO_ZIPKIN_PORT, for example NETWORK_DEMO_GRAFANA_PORT=3101 make up.

What's running

Service Role
proxy The proxy, outside the firewall, with its dashboard and service discovery on.
proxy-standby The standby: the same config, with its own dashboard. It serves nothing until an agent can't connect to the primary.
wan Toxiproxy: the four links, hq-to-proxy, hq-to-standby, branch-to-proxy, and branch-to-standby. The proxies see every agent at its address, as they would behind a NAT.
agent-hq The agent at headquarters: two small services and its own metrics. The scenarios that degrade one agent's links leave its links alone, so it is the steady reference.
agent-branch The agent at a branch office: a small shipping service, its own metrics, and an inventory service whose payload is about 496,000 characters, about 56 KB gzipped, so the agent sends it in chunks.
network-app The simulated services. Their traffic follows a 5-minute "day", so the graphs move.
prometheus Scrapes every agent's paths through whichever proxy advertises them, every 5 seconds with a 4-second timeout, and both proxies' own metrics directly, from its own side of the WAN. It loads a few fast demo alerts.
grafana Opens on the demo's dashboard: scrape time per agent, target health, payload size, connects, connected agents per proxy, evictions, bytes a filter saved, and firing alerts.
zipkin Receives a span from every proxy and agent for each gRPC call between them.

Scenarios

demo.sh, in demos/network, drives the scenarios below. Run it from that directory, where make up started the demo; each command prints what to watch. The scenarios that change one agent's links change agent-branch's unless they're given agent-hq, and ./demo.sh heal ends each scenario: it removes every change to the WAN, brings back every link and proxy, and returns any agent on the standby to the primary.

./demo.sh far         # agent-branch's links add 250 ms, give or take 50 ms, each way
Grafana, with agent-branch's scrape time stepping up Grafana, with agent-branch's scrape time stepping up
agent-branch's scrape time steps up by a round trip; agent-hq's stays flat

agent-branch's scrapes take about half a second, one round trip more, while agent-hq's stay well under a tenth of a second. Nothing fails: the agent's connection is already open, so a scrape needs no new handshake through the firewall, and costs the round trip and nothing more.

./demo.sh thin        # agent-branch's links carry 8 KB/s toward the proxies

The inventory payload, about 56 KB gzipped, takes 7 seconds on this link, and Prometheus gives up at 4. Every path shares the link, so it backs up behind the inventory, and within about 15 seconds the shipping service and the agent's own metrics fail too. agent-hq, on its own links, is untouched.

agent-branch on the thin link, every scrape too late but its idle time near zero agent-branch on the thin link, every scrape too late but its idle time near zero
Most of a minute on the thin link: every scrape too late for Prometheus, and agent-branch still connected

agent-branch stays connected all the while. Its answers keep arriving, too late for Prometheus, and each one tells the proxy the agent is alive, so its idle time stays near zero and the proxy never evicts it: a saturated link is slow, not dead. A link that goes silent shows one that is.

./demo.sh filter      # restarts agent-branch with a metric filter for the inventory
agent-branch's recent scrapes, with the inventory filtered agent-branch's recent scrapes, with the inventory filtered
The filtered inventory, about 199,000 characters, crossing the thin link in about 3 seconds

Most of the inventory's payload is a per-item price list no dashboard uses. The filter drops that family at the agent, before the payload is compressed and chunked, so the inventory falls from about 496,000 characters to about 199,000, 21 KB gzipped. Each scrape takes about 3 seconds on the thin link, inside Prometheus's 4, and every path is green again within about 5 seconds. Grafana's Filter bytes saved at the agent, from agent_filter_bytes_saved_total, climbs to about 60 KB a second: what never crosses the WAN costs nothing to send, store, or query. The restarted agent connects at once; its old connection stays on the agent list, with no paths, for about 20 seconds more, until what it had queued drains through the thin link. ./demo.sh heal and then ./demo.sh unfilter restore the link and the full payload. heal alone, without the filter, brings every path back within a second or two, since what the thin link had queued drains at full speed.

The primary proxy stops, and the agents fail over to the standby

./demo.sh proxy-down  # stops the primary proxy
agent-branch on the standby, after failing over agent-branch on the standby, after failing over
The standby's dashboard: agent-branch via its link to the standby, marked failed over

The primary's connections close, so both agents know at once, and within about 4 seconds they're on the standby, with no manual step: its dashboard shows each one via its link to the standby, (2 of 2), marked failed over, from the agent's ordered endpoints. Prometheus discovers through both proxies, so within about 15 seconds every service is up again in Grafana, scraped through the standby; only prometheus-proxy, the primary's own metrics, stays down, and NetworkTargetDown fires for it alone.

./demo.sh proxy-up brings the primary back, but the agents stay on the standby: an agent returns to the head of its list only when its connection drops. ./demo.sh heal closes each agent's connection through its standby link once, and they return to the primary within seconds.

./demo.sh cut         # agent-branch's link to the primary goes silent, both ways

Here the connection doesn't close. The link stops passing anything, the agent's own goodbye included, which is what a stateful firewall or NAT that drops a connection does, and nothing tells either end.

The primary's dashboard, hearing nothing from agent-branch The primary's dashboard, hearing nothing from agent-branch
The primary hears nothing from agent-branch: its idle time climbs toward the eviction

Within about 10 seconds agent-branch's scrapes time out. Its heartbeat fails three times, with a 5-second deadline each, so at about 20 seconds the agent gives up on the connection and dials the head of its list again: the same dead link, where the connect waits out its 30-second deadline (agent.grpc.unaryDeadlineSecs). Meanwhile the primary, which has heard nothing, evicts the agent after 30 seconds of silence (proxy.internal.maxAgentInactivitySecs, which the demo sets to 30), and NetworkAgentEvicted fires. At about 50 seconds agent-branch connects through its link to the standby, failed over, while the primary is still up. agent-hq is untouched throughout.

./demo.sh flaky       # agent-branch's link to the primary drops its connection about every 15 seconds

The link closes each connection once it has carried 170 KB toward the proxy, about 15 seconds of agent-branch's traffic. Each time, the agent reconnects within milliseconds and stays on the primary, since a connection that registered and then dropped retries from the head of the list. Grafana's Agent connects, by agent, from agent_connect_count_total, climbs, and most scrapes succeed; the one in flight as a connection drops fails as agent_disconnected.

Every proxy and agent reports a Zipkin span for each gRPC call between them, under its own service name. In Zipkin, the search is serviceName = agent-branch. Open http://localhost:9511/zipkin/?serviceName=agent-branch, which fills that search in, and click Run Query.

A reconnect's connectagent call, traced from agent-branch to the proxy
One reconnect's connectagent call, from agent-branch to the proxy

Each reconnect is a connectagent trace, then registeragent and a registerpath for each path. Zipkin joins the agent's client side and the proxy's server side of each call into one span, so a trace names both services, and its annotations mark when the agent sent the call and the proxy answered it. A connection's scrape streams are traced too, and appear as the connection closes, each about 15 seconds long: as long as the connection lasted.

The whole WAN goes down: an alert fires, and the agents reconnect

./demo.sh wan-down    # every link goes down
Grafana with every link down Grafana with every link down
Every link down: no agents connected, and NetworkNoAgentsConnected firing

Both agents leave both dashboards at once, and retry every 3 seconds, alternating between their two links. A proxy's /discovery lists only the paths of the agents connected to it, so within about 15 seconds their targets vanish from Prometheus rather than staying red: on Target health their rows go red for a scrape or two, then stop. Prometheus scrapes the proxies directly, on its own side of the WAN, so it still sees them: at about 20 seconds NetworkNoAgentsConnected, built on proxy_agent_map_size, fires.

./demo.sh wan-up brings the links back. Within a few seconds each agent reconnects, through whichever of its links it tries next, so one may land on the standby, and the alert resolves, leaving a gap in the graphs where nothing could be scraped. ./demo.sh heal returns any agent on the standby to the primary.

Scenario reference

Command Effect How it works
demo.sh far [agent] Adds 250 ms of latency, give or take 50 ms, each way on both of the agent's links A Toxiproxy latency toxic each way on both links
demo.sh thin [agent] Caps both of the agent's links at 8 KB/s toward the proxies A Toxiproxy bandwidth toxic toward the proxies on both links
demo.sh filter / unfilter Restarts agent-branch with or without the inventory filter Recreates agent-branch's container with config/agent-branch-filtered.conf, whose agent.filters drops inventory_item_price_dollars, or with config/agent-branch.conf
demo.sh proxy-down / proxy-up [proxy] Stops or starts a proxy (default proxy, the primary; or proxy-standby) docker compose stop / start; an agent whose connection closes dials its next endpoint
demo.sh cut [agent] Silences the agent's link to the primary, both ways, closes included Two timeout toxics with a timeout of 0, which drop all data, and two slow_close toxics, which hold the closes
demo.sh flaky [agent] Closes each connection through the agent's link to the primary after about 15 seconds of its traffic A limit_data toxic, which closes each connection after a set number of bytes
demo.sh wan-down / wan-up Takes every link down, or brings every link back Disables or enables all four Toxiproxy links; a disabled link closes its connections and refuses new ones
demo.sh heal Removes every change to the WAN, brings back every link and proxy, and returns any agent on the standby to the primary Removes the toxics, enables the links, starts the proxies, and disables an agent's link to the standby for a second, so it reconnects to the primary
make tour [PART=n] Walks through every scenario, from part 1 or the one given, pausing before each step
make status The containers, each link's state and changes, and the URLs
make logs [SERVICE=name] Follows the logs, of every service or the one given
make open Opens both dashboards, Grafana, and Prometheus's alerts in the browser

The optional [agent] is agent-branch, the default, or agent-hq. heal and unfilter undo every scenario. For a clean slate, make down && make up.

Going further

  • The demo's README has these scenarios as a guided tour, explaining how each one works, and lists the files that set the demo up, including which Toxiproxy toxic each scenario uses.
  • The Kubernetes Demo deploys the same proxy and agent with their Helm charts on two kind clusters, and shows Kubernetes driving the agent's targets and replacing its pod. It needs kind, kubectl, and Helm as well as Docker.
  • Running in Production covers a standby proxy and the agent's ordered endpoints.
  • Advanced Topics covers how the proxy evicts a silent agent.