← All integrations
Import.io + Databricks

Web data in Databricks

Import.io Managed Services delivers into S3, Azure Blob or Google Cloud Storage. Auto Loader picks up each new file into a Delta table.

Connects via
S3 / SFTPManaged
Setup
15 min
Available on
S3 / SFTP: Professional and AdvancedManaged: Managed Services and Aperture
Talk to salesSee Managed Services
PySpark
(spark.readStream.format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("cloudFiles.schemaLocation", "/Volumes/web_data/raw/_schemas/prices")
    .load("s3://your-bucket/importio/competitor-prices/")
    .writeStream
    .option("checkpointLocation", "/Volumes/web_data/raw/_checkpoints/prices")
    .trigger(availableNow=True)
    .toTable("web_data.raw.competitor_prices"))

How it connects.

Source

The web

Any public page, rendered, through managed proxies

Engine

Import.io

Rendering, proxy routing, captcha handling, extraction

Path

File delivery

Each run into S3 or SFTP

Destination

Databricks

Auto Loader picks up every delivery into a Delta table.

Also connects via managed delivery.

What teams do with it.

Bronze tables of web data

Raw deliveries in Delta, with schema evolution handled.

Feature pipelines

Competitor prices and availability as features for pricing and demand models.

Training corpora

Extracted text and records staged for fine-tuning and evaluation.

Set it up.

  1. Take delivery into S3, Azure Blob or Google Cloud Storage.
  2. Point Auto Loader at the delivery prefix with cloudFiles.format json.
  3. Run it as a scheduled job with availableNow, or continuously.

Questions.

Which storage should we use?

The one your workspace already reads. Managed Services deliver into S3, Azure Blob and Google Cloud Storage; self-service plans deliver into S3.

Which plans include S3 and SFTP delivery?

Professional and Advanced. Managed Services deliver into S3, SFTP and every other supported destination on any contract.

Who sets up managed delivery?

Your Managed Services team. They agree destination, format, schedule and schema with you, then build, run, monitor and QA the feed under an SLA.

Which destinations can Managed Services deliver into?

Amazon S3, Google Cloud Storage, Azure Blob Storage, SFTP, Snowflake, BigQuery, API, email and Slack alerts. Aperture delivers into the same destinations.

Related integrations.

All 50 integrations

ManagedEndpoint

Snowflake

Delivered into Snowflake by our team, or loaded with write_pandas.

Setup 15 minManagedEndpoint

BigQuery

NDJSON loads natively; managed delivery lands it for you.

Setup 10 minS3 / SFTPManaged

Amazon Redshift

Files land in S3; COPY loads them into Redshift.

Setup 15 minS3 / SFTPManaged

Amazon S3

Every run delivered into your bucket, on schedule.

Setup 10 min

Fresh web data
in Databricks.

Talk to salesSee Managed Services