You asked Claude for the member directory behind your login, and got back 40 rows of confident, plausible, empty output. Or the login held for a demo and never held again. Most teams hit this when a lead list stops being a one-off pull and becomes a job that has to rerun. Asking Claude, pointing Computer Use at the page, or wiring up an MCP server can handle the reasoning. It does not give you a browser that stays signed in.
- Claude is excellent at deciding who belongs on a list. It does not have a browser that stays signed in.
- Common failures include SSO, 2FA, session expiry, pagination, CAPTCHA, and empty payloads that look complete.
- Computer Use can complete a one-off demo. A scheduled list with an auditable replay needs a durable browser runtime.
- Keep Claude for judgment and compiled browser steps for execution.
Why scraping authenticated lead lists breaks in Claude-only workflows
The login wall. Claude in a chat window has no browser and no credential vault, so SSO or a one-time passcode ends the run before row one. A working authenticated scrape treats sign-in as its own step: submit credentials from a vault, handle the OTP, then confirm the login worked. The Login and Scrape Data from a Web Portal template does all of that before it touches the table. Scraping sales leads from a private business directory is one of the template's uses.
Session drift. A lead list is a recurring job, so credentials and cookies have to survive between runs. You sign in once, stay signed in, with credentials and sessions never exposed in agent code or logs. Session persistence has to be built into the runtime.
Pagination, CAPTCHA, and rate limits. Directories hand over a slice of the table at a time, so something has to click Next through every page and stitch the rows together. Social and directory sources add CAPTCHA and rate limits. CAPTCHA solving, anti-bot evasion, and residential proxies cover the authenticated web, bot-protected web, and legacy vendor portals.
Partial success that looks complete. A loop that lands on an empty table or a gated row can still return a clean-looking answer, and a plausible empty list goes straight into outreach. Compiling reduces how often a run returns bad rows that look fine. It does not stop that from happening, and no other setup stops it either.
The demo that won't rerun. This pattern is an "LLM-per-step" agent. Such an agent "treats every execution as if it were the first one. You are paying a language model to relearn the task, forever."
Can’t we just use Computer Use?
The first run usually works. Point Computer Use at a portal and watch it sign in and read the table.
On later runs, someone has to own session persistence, proxies and fingerprints, CAPTCHA, credential storage, retries, observability, scheduling, and concurrency. The comparison of compiled agents against LLM-per-step agents uses Claude Code on Opus 4.7 as the example. The LLM-per-step column lists the limits: local browser automation only, one session at a time, no password vault, re-reasoning every run, and low visibility when a step breaks.
The lead-generation benchmark is itself a lead-list task. Find engagement on a target LinkedIn post, enrich the people who engaged, qualify them against an ICP, and push the qualified ones to outreach.
The compiled agent that runs like software finishes it in 1 minute 21 seconds for $0.063. Claude Code on Opus 4.7 takes 7 minutes 58 seconds for $6.26. Multiply that by a list you refresh every morning, and the upkeep lands on the GTM engineer who built it.
What works for scraping authenticated lead lists
Planning is a judgment task, so do it with a reasoning model. But execution is a repetitive task, so it should be compiled. The handoff has five steps:
- Claude defines the ICP, the output schema, and the qualification rules.
- A cloud browser signs in, navigates the gated list, clicks through pagination, and extracts the rows.
- Structured JSON comes back.
- Claude scores, dedupes, and drafts.
- The run goes on a schedule, with full logs and a video replay.
Claude does not provide step 2. "Authenticated cloud browsing" means browsers that run in the cloud, not on your computer, with credentials in a vault and up to 100 sessions in parallel. Those are agents that log in, browse, and act. On lead lists specifically, it's how you find leads and signals behind authenticated logins.
You have two ways to build it yourself. Describe the job in plain English and Agent Builder compiles and tests it. Or keep your current setup and run Airtop agents from Claude Code, so Claude still does the reasoning. Marketers who'd rather hand off the build can ask Mark to access data locked behind logins.
Data enrichment use cases that deterministic agents can solve
Keep CRM records current
Accounts and contacts already in the CRM need someone to log into a gated data source weekly and refresh titles, company size, and other details that go stale over time — this is upkeep on first-party data, not net-new harvesting. A chat can't own this because it has no way to remember to check back in a week and re-run the same refresh; someone has to bring it the account list and ask, every time. A compiled agent can, because it authenticates and pulls the update on its own schedule, keeping records current without anyone re-triggering the work.
Find ICP matches in gated directories
Industry directories and association member lists need to be searched regularly for companies matching an Ideal Customer Profile, so "who's new and qualified this week" shows up without a manual export each time. Claude can run one search when asked, but it can't authenticate into a gated directory and check back on its own the next day or week. A compiled agent solves this by logging in and pulling matches on a daily or weekly run, surfacing new qualified companies automatically.
Watch gated launch directories daily
Gated launch platforms and accelerator or portfolio directories need daily watching for newly launched companies and founders that fit an ICP, since timing is what makes this signal valuable. A one-off conversation can't watch anything — it only looks when someone asks it to, by which point a same-day launch may already be stale. A compiled agent fixes that by running the check every day on its own, so new launches surface while they're still timely.
Refresh recruiting shortlists as people appear
Talent platforms and professional networks need to be searched against a live role spec to build and refresh candidate shortlists as new people appear. Claude can't maintain a shortlist over time because it has no way to re-check a talent platform days later without being asked again. A compiled agent keeps the list current instead, logging in on a daily or weekly cadence and adding new matches as they show up — sourcing for real openings, not a one-time blast.
Keep org charts at target accounts current
A defined list of target accounts needs their contacts and reporting lines extracted from a research tool, kept current as people change roles. This decays fast in a chat, since nothing re-runs the extraction after the first pass and org charts drift out of date within weeks. A compiled agent handles the drift by re-authenticating and refreshing the org chart on a weekly schedule, so the mapping stays accurate rather than going stale.
Pull buying-intent while the signal is fresh
Companies showing intent — hiring for a relevant function, running ads, adopting a technology — need to be pulled from gated sources while the signal is still fresh. A chat conversation can surface this once, but intent data has a short shelf life and nothing brings Claude back to check again the next day. A compiled agent runs the pull daily or weekly on its own, catching the signal close to when it happens instead of after it's gone cold.
Scrape reviews for at-risk and switching accounts
Gated review platforms need to be scraped weekly for accounts reviewing you and your competitors, with sentiment attached, to flag at-risk or switching accounts. Claude has no way to revisit those review platforms on a recurring basis without someone re-asking each week. A compiled agent does the scraping and sentiment pass on its own weekly schedule, turning it into a standing feed of warm signals instead of a one-time snapshot.
Build partner and supplier lists from directories
B2B marketplaces and partner directories need to be searched against defined criteria to build qualified lists of potential partners or suppliers for ops and BD pipelines. A chat can produce one list, but the pipeline goes stale the moment new entrants appear and nobody re-runs the search. A compiled agent keeps the list current by re-running the search on a weekly schedule, so ops and BD pipelines don't rely on someone remembering to refresh them.
Move event registrants into the CRM
Registrant and attendee lists from first-party, opt-in event platforms need to move into the CRM per event or on a weekly basis so leads don't pile up unrouted. Claude can move one list when asked, but nothing prompts it to check for new registrants after each event or each week without a person following up. A compiled agent closes that gap by pulling the list and routing it into the CRM on schedule, so the handoff happens without anyone chasing it.
Catch funding news while it still counts
Recently funded companies, grant recipients, and new investment listings need to be pulled from gated financial databases while the funding news is still fresh — "recently funded" only works as a timing signal if it's current. A chat can pull this once, but it can't keep re-checking financial databases day after day without being asked. A compiled agent runs the extraction daily or weekly on its own, keeping the funding signal timely instead of letting it go stale between manual checks.
Keep Claude, and give your lead list a cloud browser
Claude remains the reasoning layer: the ICP, the scoring, the copy that goes out. The list behind the login needs a browser that signs in, paginates, extracts, and behaves the same way on the next run.
Try Airtop on an authenticated lead list
Describe the sign-in, the pagination, and the fields you want. A cloud browser can hold the session while Claude keeps the ICP, the scoring, and the copy. Spin up your first agent in five minutes: Try it for free.
FAQs about scraping authenticated lead lists and Claude
Can Claude scrape a lead list that sits behind a login?
Not on its own. A chat window gives Claude no browser to hold the session and no vault to hold the password. A single sign-on redirect or a one-time passcode stops the run at the door. Computer Use can drive a browser on your machine and will sometimes clear the login, but that session ends when the local browser closes.
Authenticated cloud browsing covers this: browsers that run in the cloud, not on your computer, with credentials kept in a vault. Sign-in becomes a step the run owns. The Login and Scrape Data from a Web Portal template shows the sequence: sign in, handle the OTP, confirm the login worked, then read the table.
Is Computer Use enough to scrape authenticated lead lists?
For a one-off look at a directory, yes. Exploring a source you've never touched is where an agentic loop earns its cost, since you only have to deal with the ambiguity once.
On later runs the operational costs show up. The comparison of compiled agents against LLM-per-step agents lists what you take on: local browser automation only, one session at a time, and no password vault. Add re-reasoning on every run and low visibility when a step breaks. A list that refreshes on a schedule needs those handled in the runtime.
How do you keep the session alive when the list refreshes on a schedule?
Credentials and cookies have to be part of the run itself, rather than something you paste in after a failure. You sign in once, stay signed in, with credentials and sessions stored securely and never exposed in agent code or logs.
Three things then happen inside the run. The agent pulls credentials from the vault, clears two-factor prompts, and checks whether it's already signed in before trying again. Extraction starts from a known state, so an empty table means the list is empty rather than a session that expired without a warning.
What does an LLM-per-step lead-list run cost compared with a compiled agent?
Find engagement on a target LinkedIn post, enrich the people who engaged, qualify them against an ICP, and push the qualified ones to outreach. The compiled agent finishes in 1 minute 21 seconds for $0.063. Claude Code on Opus 4.7 takes 7 minutes 58 seconds for $6.26.
An LLM-in-the-loop agent's cost is linear in steps and runs, while a code-first agent's cost is dominated by infrastructure. On a weekly refresh the extra cost is easy to miss. Refresh several sources every morning and a $6-per-run loop is the whole budget.
What is the right way to combine Claude with a browser agent for this?
Give Claude the ICP and scoring. Give the browser the login and the table. Then, pass the rows back as JSON so Claude can score them without driving the browser.
You can run Airtop agents from Claude Code, so the reasoning stays in the tool you already use. The logged-in browser steps run in the cloud on a schedule, with full logs and a video replay.
