Import.io User Guide

The Import.io JSON API Response


The Import.io API returns API responses to most API requests in a JSON format that is slightly different than typical JSON.

The following screen capture shows a typical JSON response:

 

The following screen capture shows a newline-delimited JSON response:

 

It is a bit different structure, as in the first example we are sending only one JSON object, while in the second we send multiple objects. The second syntax is not recognized by browsers as valid JSON file, and cannot be automatically parsed as JSON object in JavaScript code.

We are aware that this kind of syntax can be confusing for developers, believe me we are using this syntax as well, and we have to add some additional processing around it, to support this format in our frontend applications.

But this format brings us also a few advantages, for which it is worth it to add some additional processing. The second format is really good for fast processing, list data. If we have stuff like list of URLs, or list of results from the execution done over time, it is much better to save those results, in the second format. It is much faster to save data in this format, because it is a simple append only operation for the file where the results are stored. It is also much faster to read, if you are not interested in the end of the list, before you start displaying the beginning of it. In other words, you don’t need to download the whole file, to start seeing meaningful results. This is really important with big data dumps, like our results. If your extractor returned 400k rows, believe me you don’t really want to open this file in a usual text editor. Even Google Docs might have a problem with processing such a file. But if the file is saved in the second format, you can easily have a look at the first 100 results, and later simply paginate through the rest, making it possible to easily display such results even on mobile devices.

So in case you are wondering how to programmatically process the newline-delimited format of the JSON result from our extractor run, hopefully the code snippet below will be able to help. It is a simple code written in JavaScript, that downloads the result file from our API, and displays each line in the file.

 

// This is a simple approach to downloading the data from our API
 // and processing it as JSON Object. This method is only sufficient for
 // small API responses (should work for responses below 100k results).
 // If your response contains more results, try having a look at this Stack Overflow QA:
 // http://stackoverflow.com/questions/15121584/how-to-parse-a-large-newline-delimited-json-file-by-jsonstream-module-in-node-j
 // We start by getting data from our API:
 let ExtractorGuid =0000000-0000-0000-0000-000000000000 // You can find this in the URL bar on the dashboard following https://dash.import.io/ 0000000-0000-0000-0000-000000000000

let apiKey ="YOUR_API_KEY" // You can get it here https://import.io/data/account/
 letAPI_ENDPOINT=`https://data.import.io/extractor/${ExtractorGuid}/json/latest?_apikey=${apiKey}`
 let response;

// We are using the Fetch API, that is only available in newer browsers (details: https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API). Try this code in Newest Chrome Console.
 fetch(API_ENDPOINT)
 .then(resp=> {
 response = resp;

return response.text();
 })
 .then(responseBody=> {
 if(!response.ok) {
 throw responseBody;
 }
 return responseBody;
 })
 .then(txt=> {
 constjsonArray=JSON.parse(`[${txt}]`);
 console.log('First ten or less results: '+JSON.stringify(jsonArray.slice(0, 10)));
 console.log('Total number of results: '+jsonArray.length);
 })
 .catch(error=>console.error(error));