← All integrations
Import.io + MongoDB

Web data in MongoDB

Import.io’s extractor JSON output is one object per line, so every row becomes a document with no mapping step.

Connects via
EndpointREST API
Setup
5 min
Available on
Endpoints: every planAPI: every plan
Start free trialTalk to sales
Python
import json, os, requests
from pymongo import MongoClient

lines = requests.get(os.environ["IMPORTIO_JSON_URL"]).text.splitlines()
docs = [json.loads(line) for line in lines if line.strip()]
MongoClient(os.environ["MONGODB_URI"]).web_data.listings.insert_many(docs)

How it connects.

Source

The web

Any public page, rendered, through managed proxies

Engine

Import.io

Rendering, proxy routing, captcha handling, extraction

Path

Data endpoint

Latest successful run, CSV or JSON

Destination

MongoDB

NDJSON rows straight into a collection.

Also connects via the REST API.

What teams do with it.

Flexible product records

Keep varying attributes across retailers without a fixed schema.

Atlas Search on web data

Full-text and vector search over extracted listings.

Event-driven apps

Change streams react to each new batch of rows.

Set it up.

  1. pip install requests pymongo.
  2. Set IMPORTIO_JSON_URL and MONGODB_URI.
  3. Parse each line and insert_many.

Questions.

How do I avoid duplicates?

Use a unique index on the URL or product ID field and upsert with bulk_write instead of insert_many.

Do endpoint downloads count as queries?

No. The latest-run CSV, JSON and Google Sheets endpoints return data your extractor already collected, so they don’t count toward your plan. Only the Live Query API runs an extraction per request.

How fresh is the data?

As fresh as your schedule. Run the extractor hourly, daily or weekly and the endpoint always serves the latest successful run.

How do I authenticate?

Pass your API key as the _apikey query parameter. Copy it from account settings at app.import.io.

Is the API on every plan?

Yes. The trial and Standard include rate-limited API access. Professional and Advanced include full API access and webhooks.

Related integrations.

All 50 integrations

EndpointREST API

PostgreSQL

Append each run into Postgres with pandas and SQLAlchemy.

Setup 10 minEndpointREST API

MySQL

The documented pandas and SQLAlchemy loader.

Setup 10 minEndpointREST API

SQL Server

Load extractor results into SQL Server or Azure SQL.

Setup 10 minEndpointREST API

Node.js

Fetch extractor results in Node with the built-in fetch.

Setup 5 min

Fresh web data
in MongoDB.

Start free trialTalk to sales