Last Friday, the whole import.io team had a mini hackday – we’ve been working on our upcoming Data Summit event and wanted some cool things to show you all! You’re going to have to come along or watch our event videos to see the finished products, but I want to show you how we built out the data sources we needed to power our apps and solve some really cool problems.
First a little data project of my own!
I don’t know if you’ve noticed, but I’m a huge football fan! The World Cup is starting on Thursday, so I thought I’d use data to help me predict the winner (as well as win my £10 bet). A lot of the talk recently has been around whether or not England will be able to compete in the high humidity of Brazil. To find out, I used import.io to get data on the humidity level for each country that has it hosted the World Cup, and compared that to the humidity level in the countries of the winner and runner-up. Based on this data, my prediction is for Brazil to take away the famous World Cup trophy. While we’ve got a cool little spreadsheet with that data in it, I wonder if any of you guys can come up with a cool visualisation or infographic using this data?
Now let’s take a look at the apps that we had built during the hackathon…
Dan’s Team – The Fruit Machine of Data
For their project, Dan’s team decided to build a data fruit machine. The idea here is to challenge you to build a mashup using two random import.io data sources! Dan and his team built out some great data sources which resulted in some hilarious results from spinning the wheel, including correlating dulux paint colours with crime rates!
One of the sources they used was Horse and Hound – check out the video to see how you can get data from that site, or view the data here.
Nick’s Team – Find Your Perfect Job
Have you ever had trouble finding your dream job? Then this app may be able to help! By asking you a series of (sometimes apparently irrelevant) questions, the app will search a big database of jobs to provide you with your ideal position. But to make this work, you need a lot of detailed information on a lot of jobs! This is where import.io comes in.
One example of a source they used is indeed.com – you can see how I built an Extractor for this site in our webinar recording. In addition to that, I walked you through building a Crawler and then using its advanced settings in order to make sure that the Crawler will get you all of the data you need.
Here’s the Indeed.com Extractor and the Asos Crawler.
When you are looking at building out a huge dataset from many sources, it is important to check you are using a consistent schema – i.e. input and output names and types. Combining lots of jobs websites together is significantly easier when you have the same schema, as adding new sources to your app is as trivial as making a source and adding its GUID to your queries.
My Team – The Data Jukebox
Our (my) idea was an app to help you discover new music. To use it you put in a search term (like your name for instance) and the app would search our lyrics database, which we made using a Crawler, to find all the songs with that word in it. We then passed those songs through a Connector to YouTube, so you could listen to the song, and another Connector to the All Music site which would show you information about the artist.
When we were creating our Connector to All Music, we used a bit more functionality than just a basic recording. Firstly the source requires JavaScript to be enabled in order to get data, so the “Not Working” button came in to play. In addition to that, I recorded not only typing in a search box, but also clicking on a number of links to get to the final data I wanted.
Finally, if you get an error message saying your Connector couldn’t be published as an API, you can still get the static data that you trained in a static Dataset. If this happens, you can contact support and we’ll do our best to sort it for you. In the meantime though you can always hit the “Edit” button, and record another query if you need more data quickly.
Here’s the All Music Connector.
Tell us what you think!
What do you think of our hack apps? Do you have any other ideas, or have you built something cool with import.io data? Then get in touch with us at hello@import.io and we may feature you on our blog!
2 Comments. Leave new