Skip to main content
This guide walks the full flow end to end over the API: find or create an API from a URL, wait for it to build, then call it. Every request goes to https://api.parse.bot with your API key in the X-API-Key header.
Prefer to do this in the browser? The Dashboard guide walks the same flow through the UI — paste a URL, test endpoints, revise, and grab a snippet. Use this page if you want to drive it programmatically.

​
Prerequisites

Set your key as an environment variable so you can copy-paste the examples:
export PARSE_API_KEY="pmx_your_key_here"

​
Step 0 (optional): Check the marketplace first

Thousands of sites already have a pre-built API. Searching first is free, instant, and needs no auth: 
curl "https://api.parse.bot/marketplace/apis?q=books.toscrape.com"
If you find a match, you can use it right away — there are four ways, depending on whether you want your own copy:
Route
Call the canonical directly (any API key)POST /scraper/{canonical_scraper_id}/{endpoint_name}
Subscribe — your own pinned copy, in My APIsPOST /marketplace/apis/{id}/subscribe
Fork privately — isolated, desynced copyPOST /marketplace/apis/{id}/fork
Dispatch by URL — clone it instantlyPOST /dispatch (Step 1 below)
Get the canonical_scraper_id and endpoint list from GET /marketplace/apis/{id}. See the Marketplace guide for the details and trade-offs. If there’s no match, dispatch your URL (below) and Parse builds it from scratch.

​
Step 1: Create an API

Submit a URL and, optionally, describe the data you want: 
curl -X POST https://api.parse.bot/dispatch \
  -H "X-API-Key: $PARSE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://books.toscrape.com",
    "task": "get book titles, prices, and ratings"
  }'
Response: 
{
  "task_id": "abc123-...",
  "matched": false,
  "may_require_auth": false
}
  • matched: true — an existing API already covered this site; it’s ready to call now.
  • matched: false — a build job was queued. Poll task_id until it completes (Step 2).
  • may_require_auth: true — the site likely needs a login. Authenticated APIs are built from the dashboard, not the API — see Authenticated APIs.
contributes_to_marketplace (optional, default true). When true, your API stays linked to the shared canonical for its domain: you can pull upstream improvements, and the build is free. Set it to false to build a fully private, desynced API (this is a charged build and the choice is permanent). Most users should leave it true.

​
Step 2: Poll for completion

curl https://api.parse.bot/dispatch/tasks/abc123-... \
  -H "X-API-Key: $PARSE_API_KEY"
Poll every 3–5 seconds until status is completed. A typical build takes anywhere from a few seconds (marketplace match) to a couple of minutes (a fresh build on a complex site). The status field moves through this lifecycle:
StatusMeaning
queuedAccepted, waiting for a worker.
runningThe agent is actively building. progress shows live detail.
needs_inputThe agent needs something from you (e.g. a sample search term). Respond via POST /dispatch/{task_id} — see below.
completed✅ Done. generated_api is populated and the endpoints are callable.
failed❌ Build failed. error explains why.
cancelledYou cancelled it via POST /dispatch/tasks/{task_id}/cancel.
A completed task includes a generated_api object with everything needed to call it: 
{
  "id": "abc123-...",
  "url": "https://books.toscrape.com",
  "status": "completed",
  "generated_api": {
    "marketplace_id": "mp-789",
    "scraper_id": "scraper-456",
    "name": "Books to Scrape",
    "source_url": "https://books.toscrape.com",
    "execution_base_url": "https://api.parse.bot/scraper/scraper-456",
    "endpoints": [
      {
        "method": "POST",
        "endpoint_name": "get_books",
        "description": "Get book listings with titles, prices, and ratings",
        "input_params": {
          "page": { "type": "integer", "description": "Page number" }
        },
        "return_schema": { "...": "..." }
      }
    ]
  }
}
The agent occasionally needs a hint — most often a realistic input to test an endpoint (e.g. a search term). The user_input_prompt field describes what it’s asking for. Answer it and the build resumes:
curl -X POST https://api.parse.bot/dispatch/abc123-... \
  -H "X-API-Key: $PARSE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"user_response": {"search_term": "python"}}'

​
Step 3: Call an endpoint

Use the scraper_id and an endpoint_name from the spec. Send parameters in the JSON body for POST endpoints, or as query-string params for GET endpoints: 
curl -X POST https://api.parse.bot/scraper/scraper-456/get_books \
  -H "X-API-Key: $PARSE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"page": 1}'
A successful call returns the endpoint’s own JSON — the shape defined by its return_schema, with no wrapper: 
[
  { "title": "A Light in the Attic", "price": 51.77, "rating": 3 },
  { "title": "Tipping the Velvet", "price": 53.74, "rating": 1 }
]
Each execution also returns rate-limit and credit headers (X-RateLimit-Remaining, X-RateLimit-Reset, X-Credits-Remaining, …) so you can pace your calls. When a call fails, the HTTP status tells you whose fault it is — a 502 means the target site failed, a 500 means the scraper bugged out. See Errors & Troubleshooting.

​
Step 4: Revise your API

Need a change? Describe it in plain English: 
curl -X POST https://api.parse.bot/dispatch/tasks/abc123-.../revise \
  -H "X-API-Key: $PARSE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"revision": "add an endpoint to search books by genre"}'

{ "task_id": "revision-task-id", "revision_type": "revision" }
Poll the new task_id exactly like Step 2. Your scraper_id stays the same — new and updated endpoints become callable when the revision completes.

​
Step 5: Export or connect an agent

# OpenAPI 3.1
curl https://api.parse.bot/dispatch/tasks/abc123-.../export/openapi \
  -H "X-API-Key: $PARSE_API_KEY"

# MCP tool definitions
curl https://api.parse.bot/dispatch/tasks/abc123-.../export/mcp \
  -H "X-API-Key: $PARSE_API_KEY"
Or skip exporting entirely and connect an AI agent straight to the hosted MCP server — it gets this API (and all your others) as callable tools. See MCP Server. To call your APIs from Python with fully typed clients, use the Python SDK.

​
What’s next

Python SDK

Typed Python clients generated by the parse CLI

Code examples

Full Python & JavaScript clients with a polling helper

Marketplace

Search pre-built APIs before you build

API Updates

Pull upstream improvements into your APIs

Errors & Troubleshooting

Read the execution envelope — whose fault a 422 / 502 / 503 is