← All integrations
Import.io + Google Cloud Run functions

Process Import.io runs in Google Cloud

Deploy an HTTP function as the Import.io extractor’s webhook. On each run it checks a shared secret and writes the latest results to Cloud Storage, ready for BigQuery.

Connects via
WebhookEndpoint
Setup
15 min
Available on
Webhooks: every planEndpoints: every plan
Start free trialTalk to sales
Python
import os
import functions_framework, requests
from google.cloud import storage

@functions_framework.http
def importio_webhook(request):
    if request.headers.get("X-Webhook-Secret") != os.environ["WEBHOOK_SECRET"]:
        return ("", 401)
    data = requests.get(os.environ["IMPORTIO_JSON_URL"]).content
    (storage.Client().bucket(os.environ["BUCKET"])
        .blob("importio/latest.ndjson").upload_from_string(data))
    return ("", 204)

How it connects.

Source

The web

Any public page, rendered, through managed proxies

Engine

Import.io

Rendering, proxy routing, captcha handling, extraction

Path

Webhook

POST on every successful run

Destination

Google Cloud Run functions

An HTTP function that lands each run in Cloud Storage.

Also connects via data endpoints.

What teams do with it.

A landing zone in GCS

Every run as NDJSON in a bucket.

Straight into BigQuery

Load the file into a table from the same function.

Pub/Sub fan-out

Publish a message so downstream jobs start on new data.

Set it up.

  1. Deploy the function with an HTTP trigger.
  2. Set BUCKET, WEBHOOK_SECRET and IMPORTIO_JSON_URL.
  3. Add the function URL as the extractor’s webhook with an X-Webhook-Secret header.

Questions.

Can Import.io deliver into GCS without a function?

Yes, with Managed Services. They deliver directly into Google Cloud Storage or BigQuery on your schedule.

When does the webhook fire?

Every time a crawl run finishes successfully. Import.io POSTs to your URL with the extractor ID, total URL count and links to the run’s JSON, CSV and Excel outputs. You can view a sample payload in the extractor’s settings.

How do I secure the webhook?

Webhooks accept custom headers. Add a shared-secret header in the extractor’s webhook settings and reject any request that doesn’t carry it.

Do endpoint downloads count as queries?

No. The latest-run CSV, JSON and Google Sheets endpoints return data your extractor already collected, so they don’t count toward your plan. Only the Live Query API runs an extraction per request.

How fresh is the data?

As fresh as your schedule. Run the extractor hourly, daily or weekly and the endpoint always serves the latest successful run.

Related integrations.

All 50 integrations

WebhookEndpoint

AWS Lambda

A function URL that lands every run in S3 or DynamoDB.

Setup 15 minWebhookEndpoint

Zapier

Start a Zap every time an Import.io run succeeds.

Setup 5 minWebhook

Make

A custom webhook scenario that runs on every successful extraction.

Setup 5 minManagedWebhook

Google Cloud Storage

Managed delivery into your GCS bucket, ready for BigQuery.

Setup 10 min

Run Google Cloud Run functions
on fresh data.

Start free trialTalk to sales