← All integrations
Import.io + PostgreSQL

Web data in PostgreSQL

Read the Import.io extractor’s latest run, stamp it and append it to a Postgres table. The same pattern works on Supabase, Neon, RDS and Cloud SQL.

Connects via
EndpointREST API
Setup
10 min
Available on
Endpoints: every planAPI: every plan
Start free trialTalk to sales
Python
import io, os, requests
import pandas as pd

# "CSV from latest successful run" on the extractor's Integrate tab
csv = requests.get(os.environ["IMPORTIO_CSV_URL"]).content.decode("utf-8-sig")
df = pd.read_csv(io.StringIO(csv))
from sqlalchemy import create_engine

engine = create_engine(os.environ["DATABASE_URL"])  # postgresql+psycopg://user:pass@host/db
(df.assign(fetched_at=pd.Timestamp.now(tz="UTC"))
   .to_sql("competitor_prices", engine, if_exists="append", index=False))

How it connects.

Source

The web

Any public page, rendered, through managed proxies

Engine

Import.io

Rendering, proxy routing, captcha handling, extraction

Path

Data endpoint

Latest successful run, CSV or JSON

Destination

PostgreSQL

Append each run into Postgres with pandas and SQLAlchemy.

Also connects via the REST API.

What teams do with it.

An app database with market data

Serve competitor prices or listings from your own API.

Change history

Append every run with a timestamp and query what moved.

pgvector search

Embed extracted text and search it next to your relational data.

Set it up.

  1. pip install requests pandas sqlalchemy "psycopg[binary]".
  2. Set IMPORTIO_CSV_URL and DATABASE_URL.
  3. Run the script after each extraction, from cron or a webhook.

Questions.

How do I trigger it on each run?

Point the extractor’s webhook at the function or job that runs this script.

Do endpoint downloads count as queries?

No. The latest-run CSV, JSON and Google Sheets endpoints return data your extractor already collected, so they don’t count toward your plan. Only the Live Query API runs an extraction per request.

How fresh is the data?

As fresh as your schedule. Run the extractor hourly, daily or weekly and the endpoint always serves the latest successful run.

How do I authenticate?

Pass your API key as the _apikey query parameter. Copy it from account settings at app.import.io.

Is the API on every plan?

Yes. The trial and Standard include rate-limited API access. Professional and Advanced include full API access and webhooks.

Related integrations.

All 50 integrations

EndpointREST API

MySQL

The documented pandas and SQLAlchemy loader.

Setup 10 minEndpointREST API

SQL Server

Load extractor results into SQL Server or Azure SQL.

Setup 10 minEndpointREST API

MongoDB

NDJSON rows straight into a collection.

Setup 5 minMCPWebhook

n8n

Import.io as an AI Agent tool and as a workflow trigger.

Setup 5 min

Fresh web data
in PostgreSQL.

Start free trialTalk to sales