← All integrations
Import.io + AWS Lambda

Process Import.io runs in AWS Lambda

Give the Import.io extractor a Lambda function URL as its webhook. On each successful run the function checks a shared secret and lands the latest results where you want them.

Connects via
WebhookEndpoint
Setup
15 min
Available on
Webhooks: every planEndpoints: every plan
Start free trialTalk to sales
Python
import gzip, os, urllib.request
import boto3

s3 = boto3.client("s3")

def handler(event, context):
    if event.get("headers", {}).get("x-webhook-secret") != os.environ["WEBHOOK_SECRET"]:
        return {"statusCode": 401}
    with urllib.request.urlopen(os.environ["IMPORTIO_JSON_URL"]) as r:
        body = r.read()
    if body[:2] == b"\x1f\x8b":  # gzip-compressed
        body = gzip.decompress(body)
    s3.put_object(Bucket=os.environ["BUCKET"],
                  Key=f"importio/{context.aws_request_id}.ndjson", Body=body)
    return {"statusCode": 204}

How it connects.

Source

The web

Any public page, rendered, through managed proxies

Engine

Import.io

Rendering, proxy routing, captcha handling, extraction

Path

Webhook

POST on every successful run

Destination

AWS Lambda

A function URL that lands every run in S3 or DynamoDB.

Also connects via data endpoints.

What teams do with it.

Land every run in S3

An append-only archive of every run, keyed by time.

Fan out with EventBridge

Publish a run-complete event and let other services react.

Load a table

Upsert rows into DynamoDB, Aurora or Redshift from the same function.

Set it up.

  1. Create the function and enable a function URL.
  2. Set BUCKET, WEBHOOK_SECRET and IMPORTIO_JSON_URL as environment variables.
  3. Add the function URL as the extractor’s webhook with an X-Webhook-Secret header.

Questions.

Why fetch the latest-run endpoint instead of reading the payload?

It keeps the function independent of payload shape and the download doesn’t count as a query. The payload also links to the run’s own JSON, CSV and Excel files if you prefer those.

When does the webhook fire?

Every time a crawl run finishes successfully. Import.io POSTs to your URL with the extractor ID, total URL count and links to the run’s JSON, CSV and Excel outputs. You can view a sample payload in the extractor’s settings.

How do I secure the webhook?

Webhooks accept custom headers. Add a shared-secret header in the extractor’s webhook settings and reject any request that doesn’t carry it.

Do endpoint downloads count as queries?

No. The latest-run CSV, JSON and Google Sheets endpoints return data your extractor already collected, so they don’t count toward your plan. Only the Live Query API runs an extraction per request.

How fresh is the data?

As fresh as your schedule. Run the extractor hourly, daily or weekly and the endpoint always serves the latest successful run.

Related integrations.

All 50 integrations

WebhookEndpoint

Google Cloud Run functions

An HTTP function that lands each run in Cloud Storage.

Setup 15 minWebhookEndpoint

Zapier

Start a Zap every time an Import.io run succeeds.

Setup 5 minWebhook

Make

A custom webhook scenario that runs on every successful extraction.

Setup 5 minS3 / SFTPManaged

Amazon S3

Every run delivered into your bucket, on schedule.

Setup 10 min

Run AWS Lambda
on fresh data.

Start free trialTalk to sales