How we learnt to stop worrying and love web scraping - Nature

For Nicholas DeVito, Georgia Richards and Peter Inglesby, custom webscrapers have driven their research — and their collaborations.

Such a tool is called a ‘web scraper’, and our group employs them regularly. We use them to collect information from clinical-trial registries, and to enrich our OpenPrescribing.net data set, which tracks primary-care prescribing in England — tasks that would range from annoying to impossible without the help of some relatively simple code.

In the case of our coroner-reports project, we could manually screen and save about 25 case reports every hour. Now, our program can save more than 1,000 cases per hour while we work on other things, a 40-fold time saving. It also opens opportunities for collaboration, because we can share the resulting database....

For Nicholas DeVito, Georgia Richards and Peter Inglesby, custom webscrapers have driven their research — and their collaborations.

Such a tool is called a ‘web scraper’, and our group employs them regularly. We use them to collect information from clinical-trial registries, and to enrich our OpenPrescribing.net data set, which tracks primary-care prescribing in England — tasks that would range from annoying to impossible without the help of some relatively simple code.

In the case of our coroner-reports project, we could manually screen and save about 25 case reports every hour. Now, our program can save more than 1,000 cases per hour while we work on other things, a 40-fold time saving. It also opens opportunities for collaboration, because we can share the resulting database. And we can keep that database up to date by re-running our program as new PDFs are posted.

Here, we offer some basics about web scraping and how you can start using it in your research projects...