“The world seems to suffer from a “data not invented here syndrome”, a bias toward using internal data that was created inside the four walls of a business. This syndrome affects both companies that use data and publishers that create products using data.” — Dan Woods, Forbes Contributor, Why External Web Data Is Getting Vastly More Valuable
Data is everywhere. That fact is not necessarily new or interesting—great minds of past generations have long harnessed and utilized data in order to inform their decisions, create and test hypotheses, and attempt change the world.
But “data is everywhere” doesn’t have the same meaning for our generation as it did for past generations. For us, it’s a nod to the fact that we are more connected to each other, to the businesses we buy from, and to the world than ever before. Today, thanks to the widespread adoption and use of the Internet, we have instant and real-time access to an incomprehensibly large amount of data. As each day passes, more and more new data is being made available to us at incredible speeds and volumes.
To put things in perspective, consider the following:
- 2.7 zetabytes of data existed in the digital universe as of 2012, according to the GovLab Index (1 zetabyte = 1 billion terabytes)
- According to IBM, 2.5 exabytes – that’s 2.5 billion gigabytes (GB) – of data was generated every day in 2012
- As of April 2011, the U.S. Library of Congress collected 235 terabytes of data (1 terabyte = 1,000 gigabytes)
- As of 2012, Walmart collected 2.5 petabytes of data every hour (1 petabyte = 1,000 terabytes)
It’s safe to say that these numbers have dramatically risen over the past five years, as more and more companies, organizations, and people have recognized the power and value of what we now refer to as “Big Data,” “Business Intelligence,” and “Data Science.”
When most people think about these terms—Big Data and Business Intelligence—they think about data from an internal perspective (i.e. data from inside an organization or operation that can be collected, interpreted, and acted upon).
But here’s the problem with internal data as it relates to Business Intelligence:
It only paints part of the picture.
So what’s missing?
The answer, as you may have guessed, is external data (i.e. relevant, useful, and available data from outside an organization or operation).
A Brief Rundown on External Data
As mentioned above, external data refers to data generated from outside an organization. External data can come from a variety of places and serve nearly every industry in the business world.
In recent years, open data initiatives have really made it easier than ever to find and access external data from a number of sources. For example, the U.S. government makes available 131,000+ datasets on the federally-run website data.gov.
You can also find and access for free a handy Excel document on the site that compiles a list of open data sites for other nations around the world.
Tech giants like Google and Amazon have also paved the way in terms of collecting, storing, and making available external data.
There are also a number of tools that make accessing and viewing external data really easy, but we’ll touch on that a bit further down in this article.
Despite all of this data being made available 24/7 to basically anyone who has an internet connection, a lot of businesses have yet to put aside the time and resources needed to explore and learn from external data.
But more and more business leaders and data professionals are starting to recognize the value of external data when it comes to making business decisions.
The Value of External Data For Businesses
There are a number of reasons why more and more businesses and data professionals are incorporating external data analytics into their decision making processes. Here are just a few worth mentioning that really highlight why now is the perfect time to go all in on external data:
- External data can provide you with a bigger picture. As a business owner or data professional, you need to be collecting, evaluating, and acting on internal data. But as mentioned, that really only gives you part of the picture. In order to get the full view, you have to look to external data (user-generated data, public data, competitor data, partner data, etc).
- Accessing external data is not costly. Thanks to initiatives by governments and businesses around the world, accessing external data doesn’t cost a lot of money. In fact, a lot of databases can be accessed for free.
“Open government data is important because the more accessible, discoverable, and usable data is the more impact it can have. These impacts include, but are not limited to: cost savings, efficiency, fuel for business, improved civic services, informed policy, performance planning, research and scientific discoveries, transparency and accountability, and increased public participation in the democratic dialogue.” — Statement found on the Data.gov Impact page
Where the cost does come into play, however, is organizing, evaluating, and applying external after the external data to specific business needs (that’s where experienced data scientists and analysts come into the picture!).
- Technology and tools have made accessing external data easier and more convenient than ever. It’s never been easier or more convenient to access external data. As the world continues to become more and more connected, and as technology continues to advance, it’s becoming a lot easier to find, collect, and interpret external data. You don’t need a computer science degree or a Masters in Data Science in order to benefit from external data. You definitely want someone who does have those degrees on your team in order to dive deeper into the internal and external data you ultimately collect, but you don’t necessarily need them in order to access or collect the data itself. A lot of the external tools that are available today are incredibly easy to use.
- External data can give you real-time, minute-by-minute updates on industry, consumer, and product trends. This is the biggest value for business and it’s why external data is so important. External data analytics can have a major impact when it comes to making decisions about the future of a business, learning more about the health of an industry, determining which new products to release and where to release them, and many, many other areas. In a lot of cases, the tools and sites that collect and present external data are updating information in real-time—which is invaluable during times when an informed decision needs to be made fast.
- External data can give you a leg up on competition. The other major benefit of external data is that it creates an opportunity to get a leg up on competition. There are a lot of tools out there that make it easier than ever to keep an eye on your competition in order to stay ahead of the game. With competition for the attention of online consumers at an all time high, the ability to quickly, easily, and regularly check up on competition in invaluable and can mean the difference between growing your business or closing your doors for good.
- More and more data is being uploaded to the web everyday. Consider the following statement from Rose Business Technologies:
“IDC estimates the volume of digital data will grow 40% to 50% per year. By 2020, IDC predicts the number will have reached 40,000 EB, or 40 Zettabytes (ZB). The world’s information is doubling every two years. By 2020 the world will generate 50 times the amount of information and 75 times the number of “information containers” while IT staff to manage it will grow less than 1.5 times.”
Image Source: http://www.rosebt.com/blog/total-enterprise-data-growth-2005-2015
What You Need To Know Before Diving In Head First
The value of external data for business is clear. So what do you need to know before going all in on external data? Here are three points:
- External data will be more unfamiliar than internal data. It might take you a bit longer to get a grasp on external data, simply because it’s more complex and more unfamiliar than the internal data that you’re used to compiling and evaluating.
- It’s important that you have someone on your team who can help you interpret external data. Collecting external data is relatively easy, but you’ll want someone on your team who can help you package it, interpret it, and apply it to your business and the questions you’re trying to answer.
- It’s essential that you take external data into consideration along with internal data when making business decisions. Remember: your goal is to paint a complete picture. The only way to do that is to collect and interpret both external and internal data.
Tools & Sources
Now that you’re ready to start investing in external data, you might be wondering where to start. There are a lot of tools and resources out there you can use, and they will vary depending on your needs. Here are three examples of tools/sources you can use and why you might use them:
1. Data.gov – This site includes 131,000+ datasets on a wide variet of topics. it also offers tools and resources to conduct research, develop web and mobile applications, design data visualizations, and more. You might use this site to learn more about your target demographic, the health of your industry, to predict future trends, or to analyze your surrounding geographic location.
2. Import.io – The import.io platform collects over 10 Million records from the web every day. They’ve collected 100 Billion data records from more than 300,000 websites. The web is a great data source for real-time competitive analysis of competitor products (pricing and reviews). This post gives a good overview of the options for getting data from the web.
3. SEMrush – This tool allows you to get insights into your competitors’ strategies in display advertising, organic and paid search, and link building. You might use it to improve your online presence and to connect with more prospects online.
Additional Resources
Want to learn more about external data? Check out these links:
- http://www.irmac.ca/0809/Techniques%20for%20Collecting%20and%20Harmonizing%20External%20Data%20v3.ppt
- http://www.forbes.com/sites/danwoods/2012/11/29/how-external-big-data-opens-a-disruptive-frontier-for-business-intelligence/
- http://a16z.com/2015/01/22/machine-learning-big-data/
- https://www.import.io/wp-content/uploads/2015/06/External-Data-in-Insurance_USA_Executive-Briefing.pdf
- https://www.capgemini.com/resources/the-deciding-factor-big-data-decision-making
- http://www.rosebt.com/blog/total-enterprise-data-growth-2005-2015
How valuable do you think external data is to business? Leave a comment for us below!