Table of Contents
- What counts as progress for an AI agent?
- Which tools track AI agent progress best?
- Which progress tool fits which kind of agent job?
- How do you track a fleet of agents without fooling yourself?
- What does tracking AI agent progress cost?
- How do you review agent progress in 20 minutes a day?
- Do you need a dedicated observability platform to track AI agents?
- Frequently asked questions about tools to track AI agent progress
Do not index
The tools to track AI agent progress are git diffs, named output files, timestamped job logs, cron history, and an alive check, all running on a host that stays on when you close your laptop. Agent 37 hosts OpenClaw and Hermes agents from $3.99 a month as of October 2026, with a browser terminal, a file browser, a live Linux desktop, and a built-in task board for assigning work and reviewing outputs, so the evidence sits next to the agent instead of inside a chat window. Tokens are a separate bill, and usually the larger one.
Progress is a diff, a file, or a failed run with a time on it. A chat bubble that says "done" is not a scoreboard. It is a diary entry, and nobody audits a diary.
That one idea is the whole method. The rest of this post covers which tools produce those artifacts, which tool fits which kind of job, what the setup costs once the agent gets chatty, and how we review a fleet in 20 minutes a day without rebuilding state from a transcript.
What counts as progress for an AI agent?
Progress for an AI agent is an artifact a second person can open without asking the agent to explain itself: a committed diff, a file at a path chosen before the run, a cron entry with a start time and an exit code, or a process that is still alive after a restart. A model summary does not count.
We hold to four artifacts, in this order:
- Git. Branch, commit, pull request. For coding agents this is the scoreboard, and nothing else comes close. The git log documentation is the only manual you need to read a night of agent work in 30 seconds.
- Files. A report, a screenshot, a CSV, written to a path you chose before the run. If the path was chosen after the run, the agent chose it, and the agent will choose the path that makes it look busiest.
- Job log. Start, end, exit code, tool calls. One stream per instance, never one stream for the fleet.
- Alive. The process is up after a restart. If it died, the other three artifacts are stale, and a stale artifact looks exactly like a fresh one until you check the timestamp.
This is not a contrarian take. The NIST AI Risk Management Framework puts measurement and documented evidence at the center of trustworthy AI, which is a formal way of saying the same thing: if you cannot point at the artifact, you cannot say the system worked.
The reason this matters more for agents than for ordinary software is that an agent is the only process that will write its own glowing status report. A cron job that fails prints an error. An agent that fails often prints "Task completed successfully" followed by three paragraphs about what it would have done.
[Book a 30 minute call with Agent 37](https://cal.com/vishnukool/30min)
Which tools track AI agent progress best?
Start with four tools you already have or get free with hosting: git for code, the instance file browser for artifacts, the terminal and job logs for failures, and a weekly export from your model provider for token spend. Add cron history and an alive check, then buy a dedicated tracker only after these feel slow.
Most teams buy a dashboard first and still cannot find yesterday's branch. The table below is the order we recommend, with the honest failure mode of each tool.
Tool | What it proves | Where it lives | When it is the wrong tool |
Git / PR | Code actually changed | git log, pull request view | Ops bots that never touch a repo |
Instance files | The job wrote the artifact | File browser, scp, sftp | When five agents share one folder |
Terminal and logs | What failed, and when | Browser TTY, journal, log files | When logs from 5 agents are one soup |
Cron history | The schedule ran | crontab, task board, job history | When success means exit 0 and an empty file |
Live desktop | A browser login still holds | Linux desktop via noVNC | Lean bots with no browser |
Provider usage page | Token dollars | Anthropic or OpenAI console | When you only stare at the host invoice |
Task board | Who was assigned what, what was reviewed | Host dashboard | One-off jobs with no owner |
Streaming events | What the agent is doing right now | Server-Sent Events from the agent API | After-the-fact audits, since a stream is not a ledger |
On Agent 37 the first five rows sit on the instance you already pay for. The browser management tools include a full TTY terminal, a visual file browser, and a live Linux desktop via noVNC, all opened from a browser tab, and the task board lets you assign work, review outputs, and schedule recurring jobs without touching a server. Pricing is public at Agent 37 pricing, and the overview lives at agent37.com.
Two details matter for tracking specifically. First, API keys go directly to the provider, so the provider usage page stays the one true source for token spend and nobody in between is rewriting the numbers. Second, if you build on the Cloud API, every message can stream back as Server-Sent Events with named events for text, reasoning, and tool activity, which is the closest thing to watching the agent's hands move in real time that we have found.
The live desktop deserves its own sentence. If progress depends on a logged-in site, the desktop is the tracker. A CPU chart will never show a session that quietly logged out at 3 a.m., and the agent will keep reporting success against a login wall until someone opens the screen.
DIY is better when you already live in Grafana and the agent logs are just another service. Budget 4 hours for the glue. At $50 an hour that is $200, which is more than a year of the $3.99 managed plan ($47.88). That math is not an argument against Grafana. It is an argument against building Grafana glue for a fleet of two.
Which progress tool fits which kind of agent job?
Coding jobs score on a pull request or diff, report and research jobs score on a file at a path you named, browser jobs score on a live desktop plus a screenshot, and chat room jobs score on a message timestamp. Pick one scoreboard per job in writing before the run, and treat the wrong artifact as a failure.
Using one tool for all three job types is how a fleet looks busy and ships nothing. The scoreboard goes in writing, in one line, before the run starts.
Job | Scoreboard | Fail condition |
Overnight repo work | PR or diff | No commit in 6 hours |
Report or research | File at a path you named | Empty file, exit 0 |
Browser task | Live desktop plus screenshot file | Logged-out session |
Slack or web room | Message timestamp in that room | Webhook 404 |
Morning briefing from email and calendar | File or message delivered before a set time | Nothing by 07:00 |
Shared inbox drafting | Draft count in the drafts folder | Zero drafts, or a draft sent without confirmation |
Price or feed watcher | Alert message with a timestamp | Silence during a known change |
The last three rows are the jobs that actually run on agent hosting: briefings compiled before you wake up, an agent that watches a shared inbox and drafts replies, scheduled checks on prices and pipelines that message you when something changes. None of them touch a repo, so git is useless for them, and all of them fail silently if the host sleeps, so the alive check is mandatory.
A second tool is allowed after the first one has been wrong twice. Not before. We would rather read an ugly git log than a polished board that cannot name the last file written.
If you run a QM-style multi-agent fleet, progress shows up as room files and Slack threads rather than one personal git repo. We host that too at QM agent hosting on Agent 37. It is early, so read the project's
SECURITY.md before you hand it credentials.How do you track a fleet of agents without fooling yourself?
Give every trust boundary its own tracker stream, name the output path before the run, reject runs that produce no artifact, split host dollars from token dollars weekly, and alert on silence rather than on errors. Shared progress boards hide who spent the money and who touched the file, which is exactly what you need to know.
One tracker stream per trust boundary means Engineer A's coding agent and Client B's bot do not share a log file "for convenience." On Agent 37 each agent runs in its own isolated container, so the boundary is already drawn at the infrastructure level; the tracking only has to respect it.
Rules we would not bend:
- Name the output path before the run. Put it in the task description. If the agent writes somewhere else, the run failed.
- Reject runs with no artifact. A cheerful summary is not a deliverable. Re-queue the job and read the log.
- Split host dollars from token dollars every week. The host line is flat. The token line is the one that moves, and it is the one that explains most surprises.
- Alert on silence. No commit and no file in 6 hours for a job that should have run is a failure, even if no error was logged. Errors are easy. Silence is where fleets die.
- Keep a human confirm on external email and on spend over a cap you write down. We use $50 in a day as the "stop and look" line for a young fleet. Raise it when you have data, never before.
The silence rule is the one people skip, and it is the one Gartner is implicitly describing when it predicts that over 40% of agentic AI projects will be canceled by the end of 2027, largely over cost and unclear business value. A fleet with no artifacts and a growing token bill is unclear business value by definition.
We have run the tracker three ways. Laptop notes: gone when the lid closed, and the progress vanished overnight. A $5 VPS and a text file: fine until the disk filled, and the progress vanished overnight. A managed instance with git plus the file browser: the reboot did not eat the diff, and the progress was still there in the morning. The difference was never the tool. It was whether the machine under the tool stayed on and kept its files.
Related ops reading: how to monitor AI agents covers alerting and health checks in more depth, and how to manage multiple AI agents covers fleet shape and ownership.
What does tracking AI agent progress cost?
The tracker is cheap and the tokens are not. As of October 2026, an always-on managed Agent 37 instance starts at $3.99 a month and a Cloud API instance at about $4.76 a month, metered by the minute. Five agents spending $20 a month each in tokens cost $100 a month, which no progress tool caused.
Your progress tool did not cause the $100. Your missing cap did. Here is where the money actually goes.
Option | Price as of October 2026 | Tracking note |
Laptop or free sandbox | $0 | Sleeps or recycles when idle. Progress stalls and files vanish. |
Agent 37 managed instance | From $3.99/mo | Always-on files, browser terminal, task board, bring your own API keys. |
Agent 37 Cloud API instance | From about $4.76/mo | Provisioned by one API call, metered per minute, white-label capable. |
Hyperscaler VM | Near $25/mo | Keeps state, but you wire storage, schedulers, and networking yourself. |
DIY Grafana glue | About $200 one time at $50/hour | Worth it when you already run Grafana for other services. |
Tokens | Separate, often the largest line | Caps and a weekly provider export are the tracker here. |
Cloud API rates are $0.80 per vCPU, $0.70 per GB of RAM, and $0.09 per GB of disk per 730-hour month, metered by the minute, and compute stops billing when a sandbox is stopped. Idle disk for a 4 GB volume works out to about $0.36 a month at that rate, which is what keeping yesterday's artifacts costs you. Higher managed tiers add more resources [confirm current tier prices and specs on the pricing page]. The full comparison against 31 other providers is on the pricing page, and the Cloud side starts with a $5 credit when you add a card, with nothing charged until you use it.
The hyperscaler row is the one founders underestimate. The hosting cost is small either way, but the hours spent rebuilding storage, cron, and SSH on a bare VM are the hours you did not spend reading artifacts. Vision, one of our customers, reported saving about $2K a month after switching from AWS and Hetzner to one machine per coding session, and the point of that story for tracking is that every session's repo, logs, and caches stayed on a machine that nobody had to babysit.
If you want the longer explanation of why agents need persistent compute, durable memory, and scheduling rather than ordinary web hosting, we wrote it up at what AI agent hosting actually requires.
[Book a 30 minute call with Agent 37](https://cal.com/vishnukool/30min)
How do you review agent progress in 20 minutes a day?
One page covers it: which instances are alive, the last artifact each one produced, yesterday's token spend against the cap, and any open incident. Walk the list, open the artifact paths, compare spend, then write one line on what a human must confirm. If it takes two hours, the tools are in the wrong place.
The daily loop, in order:
- List instances. Mark any that slept or restarted. On a managed host this is the first column of the task board; on a VPS it is
uptimeover SSH.
- Open the artifact path for each job. If it is empty, the job failed, even if the log says success. Exit 0 with an empty file is the most common lie in agent operations.
- Check yesterday's token total against your cap. Open the provider usage page, not the host invoice. If the number jumped and no artifact got bigger, something looped.
- Write one line on what a human must confirm before the agent emails anyone or spends money. This is the line you will thank yourself for in a month.
Once a week, add a fifth step: export token spend per agent and put it next to the artifact count per agent. The ratio is the only agent productivity metric we trust. McKinsey's State of AI research keeps finding that most organizations are still experimenting with agents rather than scaling them, and in our experience the teams that do scale are the ones who can answer "what did this agent produce for the tokens it burned" in one glance.
If the daily review takes two hours, you are reconstructing state from chat. Move the artifacts to a host you can open from a browser, and the review shrinks back to 20 minutes.
Do you need a dedicated observability platform to track AI agents?
Not on day one. Host logs, git, and a file browser cover one to five agents. Reach for Grafana, Datadog, LangSmith, Langfuse, or AgentOps when you already operate one of them, when more than one person needs the same view, or when the fleet is large enough that four hours of glue pays back.
These platforms are good at one thing the four basic artifacts are not: tracing. A trace follows a single request through every model call, tool call, and retry, with timing on each step, and the OpenTelemetry traces specification is the open standard most of them speak. When an agent is slow or expensive and you do not know which step is at fault, a trace tells you in seconds.
What they are bad at is the thing this post is about. A trace will show you that the agent called the file-write tool and the tool returned success. It will not show you that the file is empty, that the branch was never pushed, or that the browser session logged out. Those checks still come from git, the file browser, and the desktop.
So the order is: artifacts first, traces when the fleet is big enough to need them. On Agent 37 the logs on each instance are plain files on a Linux machine, so when you do add Datadog or Grafana, shipping them is a standard agent install over SSH, and nothing about the hosting changes.
Frequently asked questions about tools to track AI agent progress
Is a chat transcript a good way to track AI agent progress?
No. Keep the transcript for debugging, because it shows what the agent believed it was doing. Score the run on a diff, a file at a named path, or a timestamped failure. A transcript is written by the thing being measured, and it will always describe success in the present tense. The artifact is the only witness that cannot be talked into a better story.
Can several people watch the same agent's progress?
Yes, but never by sharing one login "temporarily." Point everyone at the same git remote and the same task board, and give each person their own access. If you are building a product where every customer gets an agent, Agent 37 Cloud provisions one isolated instance per user behind a single API, so each user watches their own agent and nobody reads anyone else's logs.
How do I track progress for an agent that is not a coding agent?
Swap git for the artifact that job was supposed to produce: the file it was meant to write, the message it was meant to send, the draft it was meant to leave in a folder. Name that artifact before the run and set a time by which it must exist. The alive check and the token spend check stay exactly the same, because they do not care what the job was.
Do I need a separate observability vendor?
Not on day one. Use the host logs, git, and the file browser, which cover a fleet of one to five agents with no extra spend. Add Grafana, Datadog, LangSmith, or Langfuse when you already operate one of them and the fleet is large enough that about 4 hours of integration work pays back in time saved. Traces help with slow or expensive runs; they do not replace artifact checks.
What is the cheapest always-on setup for tracking an AI agent?
As of October 2026, a managed Agent 37 instance from $3.99 a month, bring-your-own API keys so token spend shows up on the provider's usage page, and a git remote you already have. That gives you files, a terminal, a task board, and an alive process for less than a cup of coffee, with tokens tracked separately where they belong.
Start here: agent37.com from $3.99/mo.
[Book a 30 minute call with Agent 37](https://cal.com/vishnukool/30min)