Notebooks on fresh data
Schedule the extractor and every notebook run starts from the latest successful crawl.
← All integrations
Import.io + Python & pandas
Import.io data in Python and pandas
Point requests at the latest-run endpoint and your extractor’s rows land in a DataFrame. Use the Import.io Data Extraction REST API when you need a specific run.
import io, os, requests
import pandas as pd
# "CSV from latest successful run" on the extractor's Integrate tab
csv = requests.get(os.environ["IMPORTIO_CSV_URL"]).content.decode("utf-8-sig")
df = pd.read_csv(io.StringIO(csv))
print(df.head())import json, os, requests
run_id = "…" # from GET https://api.import.io/crawlruns/
r = requests.get(f"https://api.import.io/crawlruns/{run_id}/json",
params={"_apikey": os.environ["IMPORTIO_API_KEY"]})
rows = [json.loads(line) for line in r.text.splitlines() if line.strip()]
print(len(rows), rows[0])How it connects.
Source
The web
Any public page, rendered, through managed proxies
Engine
Import.io
Rendering, proxy routing, captcha handling, extraction
Path
Data endpoint
Latest successful run, CSV or JSON
Destination
Python & pandas
Latest-run data into a DataFrame in three lines.
Also connects via the REST API.
What teams do with it.
Analysis pipelines
Clean, join and aggregate web data with the rest of your Python stack.
Historical backfills
List past runs through the API and load each one to build a time series.
Set it up.
pip install requests pandas.- Copy “CSV from latest successful run” from the extractor’s Integrate tab into
IMPORTIO_CSV_URL. - Read it into pandas, or fetch a specific run’s NDJSON through the API.
Questions.
Why decode with utf-8-sig?
The CSV starts with a byte-order mark so Excel opens it correctly. utf-8-sig strips it so your first column name comes through clean.
Do endpoint downloads count as queries?
No. The latest-run CSV, JSON and Google Sheets endpoints return data your extractor already collected, so they don’t count toward your plan. Only the Live Query API runs an extraction per request.
How fresh is the data?
As fresh as your schedule. Run the extractor hourly, daily or weekly and the endpoint always serves the latest successful run.
How do I authenticate?
Pass your API key as the _apikey query parameter. Copy it from account settings at app.import.io.
Is the API on every plan?
Yes. The trial and Standard include rate-limited API access. Professional and Advanced include full API access and webhooks.
Related integrations.
REST APIEndpoint
REST API
Start extractors, list runs and download results from api.import.io.
Setup 15 minEndpointREST APINode.js
Fetch extractor results in Node with the built-in fetch.
Setup 5 minEndpointREST APIR
Extractor results into a tibble with httr2 and readr.
Setup 5 minEndpointREST APIPostgreSQL
Append each run into Postgres with pandas and SQLAlchemy.
Setup 10 min