API Publish Failure
Looks simple right? But actually, underneath, this page has a whole lot of bad happening. Here’s that same page with JS turned off:
Even if you turn JS back on (so you can map your data), processing it takes a long time. And if it takes too long, your API will timeout causing a publish failure.
Unfortunately, there’s not much you (or even we) could do about it….until now that is.
We’ve developed a way to re-render pages with JS and transform them into HTML sites. If you put render.import.io/?url= before your site URL like so…
And then use that URL when training data, we’ll try to re-render the site and create a new HTML site which we host temporarily on our servers (just long enough for you to get your data). Here’s that same site from before, once it’s been passed through render.import:
Looks just the same as the JS on version, but underneath it’s all HTML. That means we can map the data we want and publish the API – we can even run it through Magic…
Note: Make sure you include the http or https as part of your URL when using render.import
Combating Infinite Scroll
The other bad thing about this site is that it has infinite scroll. And not the nice kind of infinite scroll where there’s a button we can click and find out the URL pattern for more pages. The kind of infinite scroll that just loads more results as you scroll.
This will re-render your site in HTML and programmatically scroll through the number of pages you asked for (in this case 10). At the end of if you get one reaaallyy long page which you can put into Magic.
There’s no official limit on the number of pages you can scroll though, but for each page you add a load time of about 1 second. So if you ask for 50 pages, it’s going to take a little while to render.
Learn more about JS render and infinite scroll here.
Tell us what you think
Want to help us test new features? Sign up to get a first look at upcoming features before they’re released!
Turn the web into data for free
Create your own datasets in minutes, no coding required
Powerful data extraction platform
Point and click interface
Export your data in any format
Unlimited queries and APIs