Site Mapping

Overview

BabelWrap can automatically explore any website, understand its structure, and generate typed MCP tools. Instead of your agent manually navigating page-by-page — create session, navigate, fill, click, extract — that's 5+ API calls per flow. With site mapping, it calls a single function like linkedin_search_jobs(query="python developer", location="NYC") that executes the entire multi-step flow and returns structured results.

An AI-powered explorer agent visits the site, discovers entities (jobs, products, users), page types (search, listing, detail, form), and action flows — then generates replayable "recipes" that are converted into typed tools.

How It Works

The mapping pipeline has four phases:

1. Explore — An AI agent (Claude Sonnet 4) browses the target website (limited to ~20 pages). It discovers entities, page types, and records step-by-step recipes for each action.
2. Validate — Each recipe is validated: structural checks for all recipes, live test execution for safe read-only recipes (search, get, list).
3. Generate Tools — Validated recipes become typed tool definitions with a domain prefix. E.g., a search_jobs recipe on linkedin.com becomes linkedin_search_jobs(query: str, location: str = "").
4. Register & Persist — Tools are registered on the MCP server (so Claude Desktop, Cursor, etc. can discover them) and the site model is saved to PostgreSQL. On every server restart, all ready site models are reloaded automatically.

Recipes

A recipe is a replayable, parameterized sequence of browser actions. Each recipe has:

• name — e.g., search_jobs
• params — typed, with defaults (e.g., query: str, location: str = "")
• steps — ordered actions: navigate, fill, click, submit, extract, scroll, conditional
• returns — schema of the output data

Parameter interpolation: steps use ${param_name} which gets replaced with actual values at execution time.

{
  "name": "search_jobs",
  "description": "Search for job listings",
  "params": [
    {
      "name": "query",
      "type": "str",
      "required": true,
      "description": "Search keywords"
    },
    {
      "name": "location",
      "type": "str",
      "required": false,
      "description": "Location filter",
      "default": ""
    }
  ],
  "steps": [
    {"action": "navigate", "target": "https://linkedin.com/jobs"},
    {"action": "fill", "target": "search field", "value": "${query}"},
    {"action": "fill", "target": "location field", "value": "${location}"},
    {"action": "click", "target": "Search button"},
    {"action": "extract", "extract_query": "all job listings: title, company, location, url"}
  ],
  "returns": {"type": "list", "fields": ["title", "company", "location", "url"]}
}

Self-Healing

When a recipe step fails (e.g., a button was renamed after a website redesign), the executor:

1. Takes a snapshot of the current page
2. Sends the snapshot + error to an LLM, asking for the corrected target
3. Retries with the corrected target
4. Persists the fix back to the database

This means mapped sites stay working even when websites update their layouts. The original target is preserved for audit.

Authentication

BabelWrap's recipe executor supports 4 layers of authentication:

1. Cookie injection — Pass auth_cookies when mapping a site. Cookies are injected into every browser session.
2. Stored credentials — Site models can store login credentials for automatic use.
3. Login redirect detection — If the first navigation lands on a login page (detected by URL patterns + password fields), the executor automatically runs the site's login recipe.
4. Expired cookie refresh — If a step fails and the page looks like a login page, the executor re-authenticates, retries the step, and saves fresh cookies for future executions.

The Catalog

Every mapped site is added to a public catalog. Benefits:

• Free reuse — If someone has already mapped linkedin.com, any user can use all its generated tools at no cost.
• Browse — GET /v1/catalog to see all available site models.
• Search — GET /v1/catalog/search?domain=linkedin to find a specific site.
• Instant access — When you call POST /v1/sites/map for a site that's already in the catalog, it returns immediately with status: "ready".

Pricing

• Mapping a new site: $10 per site (one-time, via Stripe Checkout). Requires the usage-based plan.
• Refreshing a site: $10 to re-map a site that has changed significantly.
• Using pre-mapped sites: Free. Browse the catalog and use any available tools.
• Executing generated tools: Each tool execution counts as regular actions (billed at $0.01/action or within free tier).

Getting Started

1. Call POST /v1/sites/map with {"start_url": "https://example.com"}.
2. If the site is already in the catalog, you get a status: "ready" response with the site's entities, recipes, and tools.
3. If not, you receive a checkout_url. Complete the $10 payment via Stripe.
4. After payment, the mapping pipeline runs asynchronously (typically 2–5 minutes).
5. The site appears in the catalog. Use tools via MCP or POST /v1/sites/{id}/tools/{name}.

See the Sites & Catalog API Reference for complete endpoint documentation.

For a hands-on walkthrough, see Using Mapped Sites.