Finding good ideas at the nexus of computers and sociology


by Chris Emery  

Catholic Relief Services faced a daunting challenge: getting advice from several thousand people at once.

To improve its hiring and training efforts, the Baltimore-based charity, which assists poor and vulnerable people around the world, wanted to ask its employees what makes someone an effective relief services worker. But surveying the organization’s more than 4,000 employees, who work out of 150 offices worldwide and speak a range of languages, was no easy task.

“We were looking for a tool for mass collaboration,” said Stephen Moles, a manager in the charity’s human resources department. “Most of what we found allowed people to post comments and edit documents together, but neither of those functions exactly suited our needs.”

Fortunately, an acquaintance referred Moles to Matthew Salganik, an assistant professor of sociology at Princeton who has teamed up with Princeton computer scientists to develop a new way for organizations to solicit ideas from large groups of people and simultaneously have them vote on the merit of the ideas generated by the group.


Called All Our Ideas, the survey tool melds concepts from sociology and computer science to allow an organization to quickly set up a free website where large numbers of people can contribute and rank ideas.

“This is a hybrid method that combines the quantifiability of surveys with the openness to new ideas that comes from interviews and focus groups,” Salganik said. “Recent advances in web-based computing make it possible. It’s something that we probably couldn’t have done just five years ago.”

Salganik teamed up with two computer science graduate students, Nadia Heninger and Bill Zeller, to develop All Our Ideas. He connected with them through the Center for Information Technology Policy (CITP), a joint venture of the School Engineering and Applied Science and the Woodrow Wilson School of Public and International Affairs. The center seeks to inform public discussion of digital technologies and promote the use of technology in helping societies function.

In creating the research-based survey program, Salganik drew inspiration from a decidedly non-academic source: Kitten Wars, a website that allows site visitors to upload photographs of kittens and vote for the cats they think are the cutest. Visitors to the Kitten Wars site are presented with two photos at a time and asked to choose which kitten is cuter. In this manner, big-eyed fur balls rise to the top rank, while hairless breeds sink to the bottom.

Similarly, All Our Ideas allows survey creators such as Moles to present respondents with a question and let them choose which of two answers they find more appropriate. Also, respondents can submit a new idea, which will join the pool of ideas that are voted on by others.

In 2008, soon after the first version of All Our Ideas was released, the Princeton Undergraduate Student Government used the application to canvas students about priorities to guide the efforts of the student government officials. Despite the fact that student officials had seeded the survey with about 200 potential priorities, two of the entries voted most important by the students were new submissions, ideas the officials hadn’t considered.

“No matter how hard we try, we have blind spots,” Salganik said. “Organizations want to know what they don’t know. In the past though, that’s been difficult. If you ask for suggestions, especially online, you may get hundreds or thousands of ideas, but that can take months to sift through. On the other hand, if you use a survey with preset questions and answers, you can get lots of data but few new ideas.”

In preparation for launching its survey, Catholic Relief Services ran a pilot study earlier this spring, soliciting a subset of its 4,000 employees. Four of the top five ranked ideas were seed entries from the organization, but one was new, contributed by an employee. A user had suggested that an exemplary relief services worker “works with a clear priority of assisting the poor.”

“It’s something that is clearly a part of our mission, but it wasn’t articulated in our seeded ideas,” said Moles. “One thing that I appreciate about this tool is that it allows people to put things in their own words. Part of what we are trying to accomplish is to find out not only what is effective but what is meaningful.”

Edward Felten, director of CITP, said the application could provide a valuable tool for governments to understand public opinion.

“The process by which governments get comment from the public is really coming under strain,” he said. “Agencies solicit comments, but the Internet has meant that they get far more responses than they used to. They are required by law to sort through them, but often don’t have time and resources for such a task.”

After receiving initial support through CITP, the main source of financial support for the project has been through Google’s Faculty Research Awards program.

More recently, Google provided funds for Salganik to hire two computer science students through its Summer of Code program that supports open-source software development. Google recently provided All Our Ideas with more than $100,000 in credits to be used for advertising through Google’s Adwords program.

Other organizations that have launched their own idea marketplaces using All Our Ideas include Columbia University, Bowie State University in Maryland and Empower Our Neighborhoods, a New Brunswick, New Jersey, community development organization. A number of other organizations are considering using the application, including large city governments, federal agencies and multi-national firms.

Heninger, the computer science graduate student who worked on the theoretical underpinnings of All Our Ideas, said the algorithms behind the application continue to evolve. “As a theorist, it’s a great source of problems, because it’s something real,” she said. “The hope is that we can adopt more sophisticated techniques, allowing us to generate a lot more information than you would think you could.”

In addition to providing organizations with a tool for communicating with their constituents, Salganik said that All Our Ideas and similar online platforms could revolutionize social science.

“In the past, if we wanted to conduct a sociology experiment, we were pretty much limited to the students we had on campus,” he said. Online social networking could allow research at a vastly different scale, involving thousands or even millions of people. “The Web has changed the way we live, but not the way we do sociology—yet.”