Build an AI SRE with Grok

A glowing orange horizon under a black sky

You build two agents on Grok that watch a small live service: a small, fast watcher that looks every 2 minutes, and a bigger root-cause agent it hands off to when something looks wrong. Then you break the service and watch them find out why.

Time: about 45 minutes.

Before you start

Do the setup. You need the tares-demo folder from it.
Docker is running.
You have your xAI API key at hand.

The demo service

The tares-demo folder starts three containers on your laptop:

api-server, the service you watch, on port 8080. It has three endpoints, /api/orders, /api/stats and /api/users, and a fault switch that ./inject.sh flips.
traffic-generator, which calls the three endpoints every second, so there is always something to look at.
Prometheus, on port 9090. It reads api-server’s metrics every 5 seconds and checks three alert rules.

What api-server gives out:

Metrics on /metrics: requests by endpoint and status, request time, active database connections, and whether its database (postgres) is up.
Logs: one line per request, in its Docker container’s output.
Alerts, fired by Prometheus: HighErrorRate (more than 5% of requests fail), HighLatency (p99 over 1 second) and DependencyDown (postgres down).

The fault switch has three settings, one per alert:

./inject.sh error_spike: 40% of requests return 500.
./inject.sh latency: every request takes 1.5 seconds.
./inject.sh dependency_outage: postgres is down and every request returns 503.
./inject.sh clear puts it back to normal.

How Tares sees it

The AI SRE demo template (step 4) creates three sources. Each one pulls one kind of data into Tares:

demo_metrics asks Prometheus every 5 seconds for four numbers: the 5xx rate, the p99 latency, active database connections and postgres up.
demo_logs reads the api-server container’s log every 5 seconds. It drops the normal 200 lines, so only errors come in.
demo_alerts asks Prometheus every 10 seconds which alerts are firing.

All three label their events service=api-server. That label is what joins them: Tares puts every event with it on one timeline, for the entity service=api-server. Your agents read that timeline, and so can any agent you connect over MCP.

What you build

A wake-up (watch) that fires every 2 minutes and hands the timeline to an agent.
A watcher on a small, fast Grok model. Each time, it says “nothing to do” or “investigate”.
A root-cause agent on a bigger Grok model. Nothing wakes it on a schedule: only the watcher’s “investigate” does. It writes what it finds onto the project page.

Steps

Start the demo service

In the tares-demo folder:


docker compose up -d
curl -s localhost:8080/api/stats

You should see {"ok":true,"endpoint":"/api/stats"}.

Start Tares

In a new terminal:


tares up

The console opens at http://127.0.0.1:8787 . Leave this terminal running.

By default Tares listens on port 8787 and keeps its data in ~/.tares.
Already have a Tares on 8787, or want a clean one? Pick another port and folder: tares up --port 8797 --data-dir ~/tares-workshop. Read 8797 wherever this guide says 8787.

Add Grok as a model provider

Tares talks to any model that speaks the OpenAI chat-completions format, and the xAI API does. In the console:

Open Settings, then Model providers, and click Add provider.
Pick Other OpenAI-compatible, then fill in:
- name: xAI
- base URL: https://api.x.ai/v1
- API key: your xAI key
Click Save. On the new xAI row, click Make default, so the agents and the template run on Grok without you picking a provider each time.

You should see the xAI row CONFIGURED and marked default, with the Grok models it read from xAI. No models listed? Click Refresh models.

Start the demo project

Open the templates page directly: http://127.0.0.1:8787/projects/new/templates . (In the console it is Projects → Create new → Start from a template, at the bottom.)
On the AI SRE demo card, click Set up.
Tares finds your running demo by itself. You should see “Start the demo stack DONE”, and the Prometheus URL, api-server URL and Log container filled in. Click Start.

You land on the project page, with Cause an incident and Clear the fault at the top. This created the three sources above, plus a trigger and an agent of its own.

Turn off the agent the template ships

The template comes with a ready-made agent, incident-first-look. It wakes whenever Prometheus fires an alert and writes its own finding. In this guide you build that job yourself, with a watcher and a root-cause agent. If incident-first-look stays on, breaking the service in step 10 gives you two findings, and you cannot tell which one your agents wrote. So turn it off; you can turn it back on any time.

On the project page, click Setup.
In the 3. Agents card, click Change, then Disable.

You should see incident-first-look marked OFF. You are now in Advanced setup; the rest of the steps happen here.

The project page now shows Needs attention: “No agent is turned on, so nothing looks at what arrives.” That is expected. You turn on your own agents later in this guide, and the warning goes away.

Create the root-cause agent first

In Advanced setup, click the + next to AGENTS, then:

name: rca-from-watcher
trigger: No trigger: only a handoff starts it
start from: Root cause after a handoff (this fills the prompt)
model: grok-4.6

Leave the rest as default, then click Create agent. You should see “Nothing starts it yet”. That is right: the watcher will start it.

Create the wake-up that runs every 2 minutes

Click the + next to TRIGGERS, then:

name: watch
in plain words: every 2 minutes, the watcher looks at api-server
sources: add demo_alerts, demo_logs and demo_metrics
entity label: service
fires: on a schedule
run every: 2m

Click Create trigger. You should see watch with “every 2m” and “no agent”.

Create the watcher

Click the + next to AGENTS, then:

name: watcher
trigger: watch
start from: Triage (on a schedule) (this fills the prompt and the verdict investigate)

at the end of the prompt, add this paragraph. It tells the watcher what normal looks like for this service:


For api-server, also read the last few demo_metrics lines:
p99 api-server (normal about 5 ms), dependency postgres up (normal 1),
db connections active (normal 4 to 5). A firing alert, a p99 above 50 ms,
or postgres up at 0 is a problem even when no count grew.

model: grok-4.3
under When it concludes, Add a handoff: verdict investigate, agent rca-from-watcher, cooldown 5m

Click Create agent. New agents start OFF: click Enable. You should see “watcher ON” and “investigate goes to rca-from-watcher · once per entity per 5m”.

Wait for one quiet run

The setup is done. Click Back to the project to see what the agents find. The watcher’s first run comes within 2 minutes.

When it has run, under What the agents found you should see No action, “No anomalies detected”, with the numbers the watcher checked.

Break the service

In the tares-demo folder:


./inject.sh error_spike

From now on 40% of api-server’s requests fail. What follows, in order:

About 30 seconds later, Prometheus fires HighErrorRate, and the alert and the error log lines land on the timeline.
At the next 2-minute mark, the watcher reads the timeline, concludes investigate and hands off.
About 20 seconds after that, a Needs action finding appears under What the agents found, marked “watcher, then rca-from-watcher”. It says what is failing, since when, the likely cause and a next step.

From the inject to the finding takes 2 to 4 minutes.

Give your own agent the same timeline

The timeline your Grok agents read is open to any agent over MCP: the one you are building today, Claude Code, Cursor. In a new terminal:


tares mcp --transport streamable-http

It listens at http://127.0.0.1:8788/mcp. Leave it running, and add it to your client as an MCP server of type HTTP. For Claude Code:


claude mcp add --transport http tares http://localhost:8788/mcp

Then ask your agent:


Use the tares MCP server. What is wrong with the entity service=api-server right now?

Compare its answer with the finding: do both name the same failing endpoints, the same start time and the same cause?

What just happened

Tares put the service’s metrics, error logs and alerts on one timeline for service=api-server.
Every 2 minutes the watch wake-up handed that timeline to the watcher, a small, fast Grok model. Most runs it said “nothing to do”.
When the errors started, the watcher concluded investigate. That handoff woke rca-from-watcher on a bigger Grok model, which read the evidence and wrote the root cause back onto the project page.
Your own agent read the same timeline over MCP, so you can check the Grok agents’ work.

Use it on your own services

Nothing in the demo is special: it is three sources, one trigger and two agents. To watch a service of yours, swap the sources and keep the rest.

Add your sources. In the console, Sources → Add source, or ask an agent to set them up over MCP. Give every source the same entity label, for example service, so each service gets one timeline.
- Metrics: the Prometheus connector with your own PromQL, reading service from the metric’s labels.
- Logs: Docker logs per container, or OpenTelemetry if your services already send it.
- Alerts: Prometheus alerts or Alertmanager . Your alerting stays where it is; Tares reads what fires.
- Something else: any JSON endpoint with HTTP API , or anything that can POST with a webhook .
Add your deploys. Most real incidents start with a change, and the demo has none. Make a webhook source and add one curl to your deploy pipeline, so “deploy at 14:02, errors from 14:04” shows on the timeline.
Point the wake-up at them. Add your sources to watch (or make a new trigger with the same settings), and rewrite the watcher’s added paragraph with what normal looks like for your service.
Decide where findings go. Keep them on the project page, or send them to a Slack channel.

The AI SRE guide walks through each of these.

If something is off

Runs fail with “xai 401” or “xai 403”. The key is wrong, revoked or out of credit. You see it in Advanced setup → TRIGGERS → watch → Recent firings. Paste a working key in Settings → Model providers → xAI → Edit.
The xAI row says “no models endpoint at this base URL”. The base URL must end in /v1: https://api.x.ai/v1.
A run says “provider ‘xai’ lists no models yet”. Click Refresh models on the xAI row, then pick the model on the agent again.
The runs went to Claude, not Grok. Tares found an Anthropic key (an ANTHROPIC_API_KEY in the shell that started tares up, say) and it is the default. Click Make default on the xAI row, or pick xAI on each agent.
Runs show no cost. Tares prices Anthropic and OpenAI models itself; for Grok, check the spend at console.x.ai .
The project page says “Waiting for the first event” for logs and alerts. That is normal before you break anything: only error log lines and fired alerts come in.
No alert after 60 seconds. Check curl -s localhost:9090/api/v1/alerts. If it is empty, check the inject worked: curl -s localhost:8080/api/stats.
The watcher never concludes investigate. Open watcher and look at its last run. If it called the errors normal, make the added paragraph stricter, or try ./inject.sh dependency_outage.
The watcher concluded investigate but no root cause appears. Check the handoff verdict is spelled investigate in both places, and that the cooldown has passed (5 minutes per service).
The watcher never runs. Check it says ON and that watch is not PAUSED.
Cannot connect to the Docker daemon: start Docker Desktop, then step 1 again.
Clicking a + does not open the form. Click it again, or reload the page.

Try next: ./inject.sh latency and ./inject.sh dependency_outage. Run ./inject.sh clear between them, and wait 5 minutes for the cooldown.

Clean up

Stop the fault: ./inject.sh clear.
Turn the wake-up off: Advanced setup → TRIGGERS → watch → Pause. You should see PAUSED: “it is not evaluated and does not fire”. This stops the model calls.
Stop the demo: docker compose down in the tares-demo folder.
Stop Tares, and the MCP endpoint if you started it, with Ctrl+C in their terminals. Your data stays in ~/.tares.