What is grey literature?
Typically when policy-makers and practitioners need to understand how something works, for example a painkiller or a particular method of conserving an endangered species, they would commission a piece of research to investigate and provide an answer. This research can be very time-consuming and expensive, and would likely just be one case study. I work in an area of research called secondary synthesis. Essentially this means getting the most out of research we already have, rather than commissioning new research. By looking at what has already been done and published we can combine lots of smaller studies into one large experiment, using data that exists in research articles.
When most people think of scientific research they think of the final, polished articles that are published in academic journals: glossy magazines full of words, tables and figures. Whilst this is true, a lot of research exists outside of these journals, what we call grey literature. This is scattered across the internet in unpublished PhD theses, government papers and organisational reports. This research may not have been subject to the rigorous review processes that are involved in publication of academic articles, but it may hold really useful data. For example, if we want to know whether creating national parks has caused benefit or harm to people living around them we might want to know how many schools, clinics and roads have been built by park authorities, and this information may be held on the park’s website in annual reports.
How import.io is helping
In order to carry out a really rigorous secondary synthesis, or systematic review, we need to consider all of the data (evidence) both in academic journals and grey literature. Whilst academic literature is relatively simple to search and the records easy to download, searching for grey literature usually takes weeks of hunting, and once it’s complete it’s very hard to document exactly what was done and which evidence was considered to be useful or not. Recording this information is really important to ensure that the reviews are transparent, objective and repeatable (we follow really strict methodology to undertake systematic reviews – learn more about that here.
This is where import.io comes in. Not only can we use import.io connectors to combine searching of multiple websites into one great dataset, but once we’ve finished searching loads of sites, we can share these APIs with our colleagues who are also researching similar topics and need to look for grey literature from the same sites. Furthermore, what was once a totally non-transparent activity is now fully recorded in detail. We can show which sites were searched when and using which terms. We can show which search records were excluded and which were included in the review. We can update our searches at the click of a button. import.io will save us time and resources but also significantly improve the transparency of grey literature searching in systematic reviews in the future.
Putting theory into practice
Here is an example to illustrate just how useful the software is. In a systematic review of the impacts of protected areas on human wellbeing the websites of 28 organisations, for example the World Bank, were searched for relevant information. Up to 23 search terms were used in each website and the first 50 to 60 results were checked for relevance by eye. Any relevant articles were downloaded and read in full later. This took between about 45 minutes and 2 hours per site – around 40 hours in total. Using import.io, I was able to set up connectors for these sites in minutes. Although the search results would need to be checked for relevance, there would be a significant time saving. But what’s more, the searches and their results are saved in a database that lists precisely how each record was found. This substantially increases the transparency of this process, making future updates of the review (a necessary task as new information becomes available) possible in a fraction of the time it would otherwise have taken.
As well as using import.io in my research, I’m working with Nick Scott at import.io to write up methodology articles to help other researchers learn how to make the most of this in their own work: increasing resource efficiency and making everyday tasks more transparent. So often as a researcher I wish there was an application that could help save me time (I’m one of those people who would rather spend weeks working out how to save myself hours of menial work). We’re using import.io to make our systematic review funding stretch a little bit further and produce more reliable research. I’m looking forward to finding out other ways the software can help me, too.
Dr. Neal Haddaway
About the author
Dr Neal Haddaway is a Project Manager at MISTRA EviEM where he works with researchers, policy-makers, and stakeholders to produce evidence reviews and evidence-based guidelines in environmental management. His current projects include a map of research on the impacts of farmland management on soil organic carbon.
Follow him on Twitter: @nealhaddaway