What’s the best way to solve a wicked problem — by working in a large group sharing ideas via the web or as individuals? That’s the question George S. Davidson (1400) and his research team attempted to resolve this summer in an experiment funded by Sandia’s internal Laboratory Directed Research and Development (LDRD) program.
The research, conducted by George, Courtney Dornburg (12335), Susan Stevens (12335), Stacey Hendrickson (6761), Travis Bauer (6341), and Chris Forsythe (6341), yielded some surprising results.
“In this day and age of email and the Internet, our expectations were that computer-mediated group brainstorming, i.e., across the web with no face-to face contact, was going to have the best results,” George says. “What we found, however, was that people working as individuals were at least as effective and possibly more so than those brainstorming in a group over the web when trying to solve wicked, tangled problems, both in terms of quality and quantity.”
George assembled a team that consisted of himself, three psychologists (two of them PhD students), and two cognition researchers to investigate tools and methods for bringing large numbers of people together to solve difficult problems via the web. The team anticipated that the group brainstorming could result in a huge pool of ideas that might lead to solutions. The research team decided to pursue an electronic brainstorming experiment built around the common face-to-face technique used at Sandia where people submit ideas written on yellow sticky Post-It® notes.
As the team designed the experiment, the initial issue was coming up with a difficult question that had no right or wrong answer, Courtney says. In June Courtney, a psychologist who has worked at Sandia for two years, attended a new-hire breakfast hosted by Labs Director Tom Hunter. During the breakfast, Tom posed a challenge to the new hires, asking them to think about the implications of two popular models in management theory. In one view workers are just another natural resource to be used. In contrast, the second model sees employees as assets, which can be made more valuable by investing in their development.
Courtney shared the question with the team, and they unanimously decided to make it the wicked problem for the experiment.
They recruited 120 Sandia employees and contractors and 26 student interns through Sandia Daily News and word of mouth to participate in the experiment, which lasted four days in August. The participants were broken into two groups, those who worked alone and did not see the ideas of the other participants and those who worked in a group and were able to see and build on the ideas of the other members in the group via the Labs’ intranet. Of the 120 employees and contractors, 69 contributed ideas.
During the experiment, participants logged onto the website anonymously and saw the question displayed at the top of the screen. They were asked to input their ideas — the more the better, but no name calling or abusive language.
The research was unusual because, while studies have been conducted with large brainstorming groups, most have been performed in academic settings with college students as participants. This experiment was, to the research team’s knowledge, the first done in a laboratory/ industrial setting over an extended period of time.
In addition, most previous studies looked only at quantity — number of responses — while only a small number examined the quality of the ideas. The Sandia experiment analyzed both. Responses to Tom’s question were scored by team members according to originality, feasibility, and effectiveness. Responses were also analyzed by STANLEY, a Sandia-developed text analysis tool that looked at common noun phrases and key words.
“We were amazed at the length and quality of the responses, both from the people working as a group and those working individually,” Courtney says. “People were very engaged, often writing long, detailed responses.”
She adds that what was most interesting is that the quality of ideas from the people responding as individuals was “significantly better across all three quality ratings.”
Courtney says the finding that individuals are more successful than groups in computer-mediated brainstorming suggests a time- and cost-saving potential for companies. Generally, when electronic group brainstorming is compared to verbal brainstorming, it is touted as having the advantages of shorter meetings, increased participation by remote team members, better documentation via electronic recording, and cash savings. But the Sandia research suggests that people working to solve problems on their own might involve less time and, thus less expense, than electronic group brainstorming.
While individuals working alone fared nominally better in this study, George says, the research also indicates that group on line brainstorming can be effective when ideas are needed from large numbers of people.
“I expect that in coming years better software, including threaded discussions with moderators to focus the work and prediction markets to evaluate quality, will become a tool that large organizations will use to solve wicked problems,” he says.
Themes emerge from ‘wicked problem’ experiment
Group members and individuals working alone provided more than 200 concepts and ideas to answer Labs Director Tom Hunter’s question dealing with two models in management theory. In one view workers are a natural resource and in the other they are assets. Each concept was explored and analyzed.
Among themes that emerged from the responses were:
- Most participants balked at being managed under a resource-driven model and preferred the asset-driven model.
- The participants noted the need to maintain and develop new skills, which conflicts with the constant pressure to secure funding and with being the low-cost provider in Work for Others projects.
- Good leaders and empowered employees ask great questions and approach problems with creativity, which builds on employee development. In this vein, one participant suggested the metaphor of “manager as a gardener,” nurturing, developing, but also organizing and weeding.
- The need for risk tolerance and institutional courage was a recurring theme.