Hosted tool

Crawl website

Waiting on a key

Collect bounded documents from a website.

web.crawlversion 1

Price

0.01

credits per page

Availability

Waiting on a key

Waiting on its upstream credential.

Access

Waiting on a key

Category Web Tools

Input

{
  "additionalProperties": false,
  "properties": {
    "limit": {
      "maximum": 100,
      "minimum": 1,
      "type": "integer"
    },
    "url": {
      "maxLength": 2048,
      "minLength": 1,
      "type": "string"
    }
  },
  "required": [
    "url"
  ],
  "type": "object"
}

Result

array<{url:string,text:string}>

Default fields: url text

Limits

Synchronous deadline	15 s
Maximum result	65,536 bytes
Result pages	100
Long-job deadline	600 s
Maximum upstream read	1,048,576 bytes

Example call

Provisional: the invoke contract ships with the execution lane

curl https://haidercode.ai/v1/tools/web.crawl/invoke \
  -H "Authorization: Bearer <HK_KEY>" \
  -H "Content-Type: application/json" \
  -H 'Idempotency-Key: crawl-1' \
  -d '{"arguments":{"url":"…"},"max_credits":"0.05"}'

Bound every paid call with max_credits, and repeat the same Idempotency-Key to recover a result rather than pay twice.

webpages

Back to the list