The Big Data Phenomenon
You’ve heard the word bandied around; at conferences, in trade journals, and probably around your office – Big Data. It’s out there, and everyone says you should be doing something with it.
“Advances in data gathering, computing power and connectivity mean that we have more information than ever before at our fingertips. IBM estimates that by 2020 there will be 300 times more information in the world than there was in 2005.”
– John Hsu, Guardian Journalist
Clearly there’s a lot of data out there on the web and the potential of this data to solve business problems and provide insight is huge.
“The promised land of new data-driven businesses, greater transparency into how operations actually work, better predictions, and faster testing is alluring indeed.”
– Stefan Biesdorf, David Court, and Paul Willmott, McKinsey
Using Data for Business Intelligence
The uses of data in a business setting are many and varied. It’s generally best to start by thinking of the key questions you want to be able to answer and then work backwards to figure out what information you will need in order to answer them. Here are just a few to get you thinking:
- What is the true 360º view of customer?
- What are the most important market trends?
- What is the competitor landscape?
- What is the optimal pricing model?
- Where can we find new business leads and opportunities?
In answering these questions, access to data is critical. Much of the information you will need resides outside the organization on the web and often in unstructured forms such as social-network conversations. Being able to access this data programatically from the web is key to harnessing the true power of big data.
Getting Data from the Web
It used to be the case that you needed a developer, about a day per unique data source, and a healthy supply of caffeine to bribe them with in order to get data one of two ways:
- Accessing a site’s existing public API – of which there are relatively few
- Writing custom web scraping code – difficult to write and often unreliable
These days there are lots of services which can help you get data from the web a lot quicker and with minimal need for developers. If you are going to take advantage of the web data revolution, you will need the right tool or service for the job. When choosing a tool, make sure you consider the following:
- Can the software get data from the sites you need?
- Not all sites are created equal, some require JavaScript to be enabled or for you to interact with the site in some way (ex. filling out a form) to access the information you need and not all data data services will be able to fulfil those requirements
- Will the company be able to scale to meet your data needs?
- This data will soon become vital to your organization and you need to choose a service that will be able to keep up with your increasing demand for data
- How easily can you integrate the data into your desired output?
- Data quality and accessibility is key when choosing a data service. Make sure you know what output you need your data to be in and select a service that can provide you with it
- Is it affordable?
- Some services charge per month, others per data query – make sure you asses how much data you will need and how often you want it refreshed to work out the expected cost
- Are they able to offer you adequate support?
- If you’re new to the concept of APIs, make sure you choose a service with a good level of support who will be on hand to help you get your data and integrate it into your internal system
Have a Plan
The real key to using Big Data in your organization is to have a strategic plan. Know the questions you want to answer and make sure that your plan highlights the critical decisions, or trade-offs, a company must make. You must also define the initiatives in order to prioritize: for example, which businesses will get the most capital, whether to emphasize higher margins or faster growth, and which capabilities are needed to ensure strong performance.
To get a free acount and start accessing data: Create Free Account