Bird Watchers and Computers: A Love Story Named Visipedia

Computer scientists built Visipedia, an interactive field guide app for birders.
A White Breasted Nuthatches, in profile. (Photo: Universal Images Group/Getty Images)
Nov 30, 2012· 3 MIN READ
Rachel is a science journalist writing for venues such as The New York Times and Smithsonian.

Binocular-wielding birders excel at spotting elusive Brown Creepers pecking about the tree trunks, or picking out the nasal yammerings of White-breasted Nuthatches flitting amidst the hickories. As it turns out, these skills translate well from the field to the cyber lab. Birders are lending computer scientists their enthusiasm and expertise to help train complex machine learning algorithms to recognize and identify avian images.

If a storm blows birds off the New England coast into Northeast backyards, for example, scientists can monitor how those species fare afterwards by following citizen science sightings.

“This is not just a tech thing, but a community thing,” said Serge Belongie, a computer scientist at the University of California, San Diego, and visiting professor at Cornell Tech in New York City. “We’d like to create an infrastructure that could provide interactive field guides centered around citizen science.”

To achieve this, Belongie and his colleagues created Visipedia, a machine learning tool that helps identify images on a fine grain scale. Visipedia works as a sort of reverse Wikipedia; instead of starting with a search word, users start with a search image—a bird, a handbag, an architectural landmark—and work with the computer to identify it. Users upload photos onto the Visipedia, which then suggests a list of possible matches. If none of these matches stand out, however, the user and computer engage in a game resembling 20 questions, with the person providing input such as a bird’s beak shape or breast patterns while the computer gains smarts and gives better suggestions. “On the back end, there’s constantly more and more data coming in, which improves the system’s performance the more you use it,” Belongie explained.

MORE: California Birds Fly in the Face of Climate Change Prediction

Getting this program off the ground required a great deal of initial data, however, and this is where the birders swooped in. “This is a large community that has self-organized around their passion for birds, and they’re interested in helping efforts that will allow them to sort their data and make it more searchable,” Belongie said.

At first, Belongie and his colleagues gathered their data using Amazon’s Mechanical Turk program, which pays users to perform tasks. They uploaded around 6,000 bird images taken from Flickr and asked participants to identify features such as beak shape and plumage coloration in a series of images.

Users responded unusually well, writing in with unsolicited positive feedback and demanding for more birds when the researchers removed the program to analyze their data.

Though the group attained promising results with this initial trial, Belongie acknowledged that the system was inherently biased. Computer scientists had selected the bird images, meaning the photos were not rooted in reality but rather in the scientists’ interpretation of reality. They had no way of knowing how the software would perform if real birders uploaded actual query photos taken in the field. “None of us were birders at heart,” Belongie said. “It was a step in the right direction, but it was kind of awkward.”

Salvation arrived on the wings of Cornell University, whose ornithology lab is generally regarded as a mecca for all things flighty and feathery. Cornell’s ornithologists were in the midst of launching and refining Merlin, a smart online bird identification tool that relies on community-sourced data. When the ornithologists caught wind of Belongie’s project, they approached him and proposed a collaboration. “We found that we were both using a few of the same techniques and that we also shared common goals,” said Jessie Barry, the Merlin project leader at the Cornell Lab of Ornithology. With Belongie’s team onboard, she said, “we were able to add a whole new feature into Merlin to allow people to upload their photos and identify those birds.”

Barry and her colleagues provided around 70,000 images of over 750 North American species. What’s more, the University presented the computer scientists’ work to their community. The birders took to the new Visipedia-driven task like ducks to water, with around 45,000 people participating within a day after Cornell announced the project on its Facebook page.

Belongie, too, came around to his feathery subjects appeal—to a point. “I think I get what the deal is with birding, but the whole wake up at 4:30 a.m. thing sort of clashes with the computer science part of me,” he admitted.

Belongie thinks the Visipedia bird app will be up and running with the new Cornell dataset by mid-April, and Barry plans to incorporate profiles for most North American species within the next two years.

Once Visipedia becomes widely available, the estimated 46 million birders in the U.S. may serve as data sources all over the country. If a storm blows birds off the New England coast into Northeast backyards, for example, scientists can monitor how those species fare afterwards by following citizen science sightings.

While Visipedia does have commercial potential—for identifying trending clothing or furniture, for example—its creators are more interested in refining the tool’s community-oriented application. Once the platform is in place, Belongie thinks it can serve other interest groups equally well, whether their interests lie in beetles, butterflies or fungi. Birds are only the beginning.