Last updated: 2pm PST, Wednesday 9th March 2016
The new Query API has been in production for 24 hours now at 99.9% availability.
We will continue to monitor this situation but you can assume that normal service has been resumed. Any questions please reach out to firstname.lastname@example.org
First published: 9am PST, Tuesday 8th March 2016
The Import.io Query API experienced issues over the first 7 days of March such that about 10% of users’ queries were being rejected by our servers.
We believe that we have identified the problem that was causing this issue and we have developed a fix that is now deployed.
Massively increased query volume
Making sure that Import.io is always available to you is one of the most important things that we do here at Import.io. In 2015 our users extracted data from over 5.5 billion web pages and during that time we are proud to say that we maintained a 99.5% service availability on our Query API.
Since the beginning of this year we have seen a dramatic increase in the number of users extracting data from web pages using Import.io. Indeed, in the past 7 days the query volumes passing through our platform have been up to 37 times greater than the 2015 daily average.
This massively increased query volume and the spiking nature of these queries has been having an impact on our ability to service requests on our Query API. This has resulted in about 10% of queries being rejected by our servers (an effective 90% service availability on our Query API). This means that you may have received error messages while using Import.io in the past 7 days or so.
We have implemented a new version of our Query API that is designed to spread query load more evenly across users by enforcing a fair-usage, query-rate limit of 3 queries per second. If you exceed the query-rate limit, then the Import.io platform will return HTTP 429 (Too Many Requests) status codes to you. If you see these codes then simply slow down your query rate a little and try again.
Part of the problem that we were experiencing was caused by the fact that some users were querying Import.io at a much greater rate than other users. Spreading our query resources more evenly across users should result in fewer query failures for everyone, with immediate effect.
Since 2015, we have been accepting select users onto the Import.io platform as paying customers. If you are already a paying customer, you were not affected by the service disruptions discussed here. Reach out to your Customer Success manager if you have any further questions.
We are currently working on a new version of Import.io that will include a new back end architecture that scales faster in response to increased query demand and will thus improve our ability to handle peaks in query volumes.