Comet vs REST Query APIs

When we talk about getting data out of import·io using our APIs, there are two main methods that we describe. These are known as REST queries and Comet queries. I want to quickly explain the key differences between them and what this means for developers.

 

REST Queries

REST queries are the simplest way to query using import·io – although they do come with some caveats. They are essentially a single HTTP request, and the connection is held open until the data or an error is returned. This means the HTTP request can be open for up to 30 seconds.

The main disadvantage with this approach is that in most cases you cannot retrieve multiple pages of data. So if you have trained a Connector to get multiple pages of data, you will only be able to get the first page of data back with a REST query – unless your Connector meets specific criteria. There is a tool for testing whether your Connector is compatible with REST pagination on our docs.

There are some other issues that you may have to watch out for as well. For example, it’s not possible to query more than one data source at a time with a REST query – unlike the Comet query API. Also, bear in mind that the connections could be held open for a very long time, and you would quickly run into a limit if you are making the connections from a web browser, which only permits 6 open connections per domain at a time.

We have detailed developer documentation about our REST query API.

 

Comet Queries

Comet querying is more complex than REST querying, but the tradeoff is that you can obtain much better results.

Our Comet querying API is based on the CometD messaging protocol, using the long-polling transport. This means that connecting can be complex – which is why we strongly recommend the use of one of our many client libraries.

As mentioned in the above section on REST, Comet querying supports several more features than the REST query API.

First of all, you are able to retrieve multiple pages of data. If your Connector is trained with pagination and there are enough pages of results, you will receive multiple messages for your query, each with the next page of result data.

Additionally, it is possible to query multiple sources with the same set of inputs (federation). You issue one query, and get multiple messages back from each source you queried with the pages of data from those sources (it also combines with pagination, so you can get multiple pages for each of the sources you query).

The last point to mention here is that because the Comet query API is asynchronous, you can issue multiple queries down the same channel concurrently without the risk of running out of resources, as with REST queries. Of course there are some practical limits to this, but you get much more throughput with the Comet query API.

The best place to get started with our Comet query API is on our integrate page, which you can use to setup one of our client libraries for exactly which sources you would like to work with.

 

Summary

If you are looking to quickly get a bit of data from an import·io data source for a few quick queries, then the REST query API is the way to go – it is quick to make an HTTP request and get the data you want back again.

However, you will almost certainly quickly outgrow this technique, and where you find yourself querying multiple sources, issuing multiple queries simultaneously, or increasing your query throughput; then moving to a client library and therefore the Comet query API is strongly recommended.

Extract data from almost any website


INSTANT FREE ACCESS