Compliance and legal ops teams pull records from government portals, such as a county court search or a Secretary of State lookup. They often run the search across a batch of case numbers. The sites require login and often a CAPTCHA, then pagination across many records.
Claude can read a docket entry. Running that search at batch scale, with a session that lasts, is a different problem.
- Claude reasons well about a single record, but government portals demand thousands of identical, repeatable actions. That's the wrong job for a model that re-reasons every step.
- Court, county, and licensing sites resist automation with logins, CAPTCHAs, session timeouts, and inconsistent HTML. Computer Use re-solves these obstacles from scratch on every run.
- Computer Use models produce incorrect behavior 20 to 60 percent of the time, which is a liability when compliance and legal teams need identical, auditable results.
- The fix keeps Claude for judgment and hands login, navigation, and extraction to compiled agents that run like software instead of reasoning through every step.
Where Claude is helpful for scraping government and public records
Claude is good at the parts of records work that require judgment. Reading a docket entry and deciding whether it's a dismissal or a continuance. Classifying a business license as active, suspended, or expired based on ambiguous status text. Drafting a summary memo once the underlying data is in hand. These are reasoning tasks, and reasoning is what a language model is for.
Scraping government and public records, though, is mostly not a reasoning task. It's a mechanical task performed at volume: log into a county clerk system, run the same search a few hundred times with different case numbers, page through results, download a PDF, log the outcome, repeat tomorrow. The judgment call happens once, maybe twice, per record. Everything else is navigation, clicking, typing, and waiting for a page to load.
That split matters because Claude in chat, in Computer Use, or via MCP treats every step as a fresh reasoning problem. It looks at the page, decides what to click, clicks it, looks again. For a single lookup, that's fine. For a recurring pull across a state's court system or a county assessor's parcel database, it means paying the reasoning cost, and absorbing the error rate, on every action, every day.
Why scraping government and public records breaks
Government and public-records sites weren't built with automation in mind, and several were built to actively resist it. The failure modes are specific, and they compound.
Login and SSO. Court e-filing systems, licensing boards, and secretary of state portals frequently sit behind credentialed logins or single sign-on. Claude in a chat or Computer Use session has no durable place to keep a session alive between runs, so it re-authenticates, or fails to, every time.
CAPTCHA and anti-bot defenses. Many county and state systems throttle or challenge repeated automated access. A model reasoning step by step has no reliable way to clear these gates.
Session drift. Even after a successful login, sessions on government sites expire quickly. A search that starts fine can lose its authenticated state midway through a multi-page pull, and the model often doesn't notice until the output looks wrong.
JS-rendered content and inconsistent HTML. There's no standard county clerk website. Each jurisdiction renders search results differently, some behind heavy JavaScript, some as static tables from another decade. A model prompted generically has to re-orient on every new portal.
Pagination and infinite scroll. Records searches routinely return results across dozens of pages. Missing page 12 of 40 is a silent failure, not an error message.
File downloads behind UI. The actual record, a filed judgment, a deed, a license certificate, is often a PDF reachable only after several clicks past the search results, not through any accessible link.
Rate limits and blocks. Repeated automated traffic from the same session or IP gets throttled or blocked outright, especially on court systems with limited infrastructure.
Silent partial success. Perhaps the costliest failure mode is that the run appears to complete, but a fraction of records were skipped, mis-paginated, or pulled from a stale session. Nothing in a chat transcript flags that for you.
Can't we just use Computer Use?
The first Computer Use demo for a records pull usually works. Claude opens the portal, finds the search box, enters a case number, and returns a result. It's genuinely impressive to watch, and that's exactly why teams keep pushing it toward production.
The hidden costs show up at scale. Computer Use models re-reason every step rather than executing a fixed plan, which means they produce incorrect behavior 20 to 60 percent of the time in Airtop's own testing, and those errors recur on every run, not just the first one. There's no persistent session vault, so login state has to be rebuilt constantly. Sessions run one at a time, so pulling records for a few hundred cases in parallel isn't an option. And the cost adds up: a compiled agent completing a comparable multi-step task ran in 1 minute 21 seconds for $0.063, against 7 minutes 58 seconds and $6.26 for an LLM-per-step run of the same job.
None of this means Computer Use is broken. It means Computer Use is built for one-off, supervised tasks, not for a recurring, unattended pull across thousands of government records where the result has to hold up under audit. This is the same failure pattern described in why code-first agents outperform LLM-first agents on reliability, and it shows up in any gated-data workflow, not just public records.
What works for scraping government and public records
The pattern that holds up separates judgment from execution. Claude, or any reasoning model, is well suited to deciding what a record means once it's retrieved: is this filing a judgment against the entity, is this license status current. That decision genuinely benefits from a model in the loop.
Everything before that decision, logging into the portal, running the search, paging through results, downloading the file, belongs to a runtime built for repetition. This is the deterministic AI approach: reason once at build time, then run the compiled result, calling a model only where the work is genuinely variable.
Airtop agents authenticate to court and licensing portals using credentials held in a secure vault, hold the session across searches, and run up to 100 sessions in parallel, so a pull across an entire jurisdiction doesn't queue behind a single browser tab. Broken runs on a redesigned county site heal themselves rather than failing outright, and every run leaves full traces and video, which matters when a compliance team needs a session they can replay. Most government sites have no API and are only reachable through the user interface, so this isn't a workaround. It's the only path in. Teams that build and maintain their own browser infrastructure for this instead report spending 40 to 60 percent of engineering time on infrastructure rather than logic.
Claude still owns the plan and the judgment calls. You can connect Claude Code directly to agents that handle the login and extraction, or pair Claude with a browser layer built for gated portals, so the handoff from reasoning to execution stays in the same workflow you already use. If you'd rather skip the code entirely, describe the records workflow you need in plain language and let the agent builder compile it.
Public-records use cases that deterministic agents can solve
Records sites are search forms, dockets, and PDFs with little patience for a fresh Computer Use loop. Claude reads the document. The site needs a compiled browser.
Pull new filings for a saved party or case list
Docket sites paginate and throttle. Claude can summarize a filing. It cannot rerun the same search every morning without a session that already knows the form. A compiled agent can.
Download recorded documents from a county portal
The PDF is behind a search, a viewer, and sometimes a payment or captcha wall. That path should be compiled, with a replay, so the pull is repeatable. Claude still does the legal reading.
Recheck licenses against a board's public lookup
Status changes do not arrive as a webhook. A chat will not remember 400 license numbers. A compiled agent will, and Claude only sees the ones that flipped.
Archive a snapshot before a portal rotates search results
Some boards drop older hits. A scheduled compiled run keeps the file. That is the difference between a research assistant and a records pipeline.
Keep Claude for the reading, give records a browser
Claude is the right layer for interpreting a filing and deciding what matters. The portal still needs a browser that can sign in, page through results, and pull the same fields on the next run. Pair them and the records job stops depending on a chat session that will not be there in the morning.
Get a records pipeline that runs the same way every morning
Scraping government and public records is a repetition problem wearing a reasoning problem's clothes, and treating it like the latter is where Claude-only workflows stall. Keep Claude for the calls that need judgment, and let compiled agents own the login, the pagination, and the download. Try it for free and spin up your first agent in five minutes.
FAQs
How do I know if a records site needs a browser agent instead of an API?
If the portal has no documented API and the only way to reach case files, licenses, or parcel data is by clicking through search forms and result pages, you're in browser-agent territory. This is common on court and county sites, which is common when those records only live in the page itself.
Can Claude still be part of a government records pipeline?
Yes, but it should own judgment, not navigation. Have Claude classify filing types or summarize a retrieved record, and let a compiled agent handle login, search, pagination, and download. Pair Claude with a browser layer built for gated portals to keep both roles in one workflow.
What happens when a county site changes its HTML or layout?
A hard-coded scraper breaks immediately. Compiled agents built for this are designed to self-heal on DOM changes rather than failing the whole run, which matters when you're pulling from dozens of jurisdictions that each redesign on their own schedule.
How do I prove to a compliance team that a records pull was accurate?
You need a run-level audit trail, not just the final output. Agents that log full traces and video of every session let you show exactly which pages were visited and which records were retrieved, which is harder to produce from a chat transcript.
Will CAPTCHAs and rate limits still block automated record pulls?
Session persistence and realistic browser behavior reduce how often you trigger these defenses, but no approach eliminates them entirely on every government site. The practical fix is a runtime that authenticates once, holds the session, and retries gracefully instead of re-triggering the challenge on every step.
Can I run records pulls for multiple counties or states at the same time?
Yes, if your runtime supports parallel sessions. Compiled agents that run up to 100 simultaneous sessions can pull from several jurisdictions concurrently, versus a single Computer Use session working through one portal at a time.
Do I need to write code to set up a recurring records-scraping workflow?
No. You can describe the records workflow you need in plain language and have it compiled into a repeatable agent, or connect Claude Code directly to agents that handle the login and extraction if you're already building in code.
