People Under Glass: Focus Groups Good and Bad
One of my favorite things I used to do in my old job was research. Not sitting in a library reading, a different kind of research—public opinion research—specifically, focus groups. That is where you put a dozen people in a room and talk to them about…things. You can show them advertisements, TV shows, logos, or just lead a discussion with them on a specific topic. There is a moderator in the room who has a script and carefully leads the discussion, making sure everyone participates and nobody intimidates people into not expressing their true thoughts. There is always a mirror in the room, and behind that mirror sit analysts, cameras, and clients watching everything. And eating. There is a lot of eating that goes on behind the glass. But that is another story.
The thing I find most enjoyable about the focus groups (besides the food, did I mention the food?) is that I am utterly fascinated by how people think. I am enthralled by learning what people think and why they think it. (Upon reflection this represents an interesting dichotomy
in my personality: I love learning about people, but I hate talking to them. I love learning why people think what they think, but I usually hate them for thinking it. Gosh I’m complex. If only I could study myself from behind glass.)
Back to the point, there are two ways to conduct focus groups: the intellectually honest or curious way, and The Other Way. I have absolutely no use for the latter, and frankly, if you hear about any research results and aren’t working in the marketing division of a company, you are probably hearing about results derived from The Other Way. That is why I immediately tune out when the media or politicians start talking about “75% of Americans believe this”, or “one in three no longer support that,” or my personal favorite, “fully two-thirds of registered somethings support the other”. (Seriously, what is a “full two-thirds”? As opposed to a partial two-thirds. Wouldn’t a partial two-thirds be something like maybe…a half.) Anytime you hear the results of a poll, you need to keep your salt nearby. I am confident I could design a series of questions that would show a plurality of Americans believe up is down. (While a full two-thirds of them don’t know what “plurality” means.)
But then there is the intellectually honest polling. It’s a study designed to learn what people actually think and why; it is not designed to show that what people think fits the expectations of what the client thinks or wants people to think.
I’ve read dozens of research reports that were just that: telling the client what the client wanted to hear. Whether the client was a powerful politician, a Fortune 100 CEO, or a grassroots organization that really should not have been wasting members’ money by dabbling in the brutal world of public opinion research; the reports were all essentially the same. They were all totally useless.
I should say I do appreciate the kind of polling that helps you achieve a political end. Let’s say you are trying to move a certain piece of legislation through Congress and you think Democrats will not support it because they believe their constituents are not supportive of the legislation. It is helpful to have an ad that informs the politicians from that side of the aisle that “84% of registered Democrats support blah blah blah.” Of course, you need the research to support the ad, and I always found it helpful for the research to actually show that 84% of registered Democrats do support whatever it is you are claiming they support AND that you just asked the questions straight up. I made certain of that last point. I would insist we share the questions and the raw data with anyone who asked, sometimes even including it in the ad itself. If you can’t share the research, you are probably manipulating the data and you shouldn’t make the claim. You will get caught. (Talk about a fundamental truth of politics. Of life. You will get caught. You will. Accept it, because it is going to happen.)
I know one notorious pollster whose methods were always suspect; he rarely, if ever, shared his questions and raw data with us, just results. A prime indication that the data was worthless. We used to joke that he would show people an ad and then ask them, “would you say this was THE MOST effective ad you have ever seen on this topic, or just ONE OF THE MOST effective ads you have ever seen on this topic?” His clients were always so happy with their ads. Until they got them out in the real world and found they couldn’t replicate the research results.
But honest research? Delicious. I love it. And I learned so much about people. One of the main things I learned was that people don’t really need to know much about something to have an opinion of it. A strong opinion. And that is scary. It sounds somewhat innocuous—and it is when you are talking about new toppings for pizza or new scents for room freshener. But it is frightening when you are talking about public policy decisions that will affect millions of people. It makes you realize the founding fathers were on to something with the Electoral College. And that things like low voter turnout is probably not that bad after all.
Unfortunately my focus group days seem to be behind me for the foreseeable future—at least being behind the glass. I’m always willing to participate in a focus group as a subject, although I would find it hard to resist the temptation to mess with the people behind the glass in ways that only someone who has spent many hours behind the glass in the dark…eating…and watching…and eating, could know how to do.
Now I have to feed my lust for insight into people in other ways. Like sitting in a coffee shop with headphones on, pretending to listen to music, while actually listening to everything the women behind me are saying. (In case you didn’t guess, I’m doing that right now, and I’ll have something to report on the seemingly fragile mental health of new mothers soon.)
This entry started as an intro into my upcoming series I referenced in an earlier post and then took on a life of its own. The future pieces are from a collection entitled, “The We’re Trying to Have a Society Here Essays.” In them I’ll look at the little things people do that make me think we as a society are in serious trouble. Thinking about how to intro the essays got me thinking about focus groups, and that got me here. So, thanks for reading, stay tuned, and just let me know on a scale from 1 to 7 how good you think this entry was, where “1” means you think it was really good, and “7” means you think it was one of the best things you’ve ever read.
One of my favorite things I used to do in my old job was research. Not sitting in a library reading, a different kind of research—public opinion research—specifically, focus groups. That is where you put a dozen people in a room and talk to them about…things. You can show them advertisements, TV shows, logos, or just lead a discussion with them on a specific topic. There is a moderator in the room who has a script and carefully leads the discussion, making sure everyone participates and nobody intimidates people into not expressing their true thoughts. There is always a mirror in the room, and behind that mirror sit analysts, cameras, and clients watching everything. And eating. There is a lot of eating that goes on behind the glass. But that is another story.
The thing I find most enjoyable about the focus groups (besides the food, did I mention the food?) is that I am utterly fascinated by how people think. I am enthralled by learning what people think and why they think it. (Upon reflection this represents an interesting dichotomy
in my personality: I love learning about people, but I hate talking to them. I love learning why people think what they think, but I usually hate them for thinking it. Gosh I’m complex. If only I could study myself from behind glass.)
Back to the point, there are two ways to conduct focus groups: the intellectually honest or curious way, and The Other Way. I have absolutely no use for the latter, and frankly, if you hear about any research results and aren’t working in the marketing division of a company, you are probably hearing about results derived from The Other Way. That is why I immediately tune out when the media or politicians start talking about “75% of Americans believe this”, or “one in three no longer support that,” or my personal favorite, “fully two-thirds of registered somethings support the other”. (Seriously, what is a “full two-thirds”? As opposed to a partial two-thirds. Wouldn’t a partial two-thirds be something like maybe…a half.) Anytime you hear the results of a poll, you need to keep your salt nearby. I am confident I could design a series of questions that would show a plurality of Americans believe up is down. (While a full two-thirds of them don’t know what “plurality” means.)
But then there is the intellectually honest polling. It’s a study designed to learn what people actually think and why; it is not designed to show that what people think fits the expectations of what the client thinks or wants people to think.
I’ve read dozens of research reports that were just that: telling the client what the client wanted to hear. Whether the client was a powerful politician, a Fortune 100 CEO, or a grassroots organization that really should not have been wasting members’ money by dabbling in the brutal world of public opinion research; the reports were all essentially the same. They were all totally useless.
I should say I do appreciate the kind of polling that helps you achieve a political end. Let’s say you are trying to move a certain piece of legislation through Congress and you think Democrats will not support it because they believe their constituents are not supportive of the legislation. It is helpful to have an ad that informs the politicians from that side of the aisle that “84% of registered Democrats support blah blah blah.” Of course, you need the research to support the ad, and I always found it helpful for the research to actually show that 84% of registered Democrats do support whatever it is you are claiming they support AND that you just asked the questions straight up. I made certain of that last point. I would insist we share the questions and the raw data with anyone who asked, sometimes even including it in the ad itself. If you can’t share the research, you are probably manipulating the data and you shouldn’t make the claim. You will get caught. (Talk about a fundamental truth of politics. Of life. You will get caught. You will. Accept it, because it is going to happen.)
I know one notorious pollster whose methods were always suspect; he rarely, if ever, shared his questions and raw data with us, just results. A prime indication that the data was worthless. We used to joke that he would show people an ad and then ask them, “would you say this was THE MOST effective ad you have ever seen on this topic, or just ONE OF THE MOST effective ads you have ever seen on this topic?” His clients were always so happy with their ads. Until they got them out in the real world and found they couldn’t replicate the research results.
But honest research? Delicious. I love it. And I learned so much about people. One of the main things I learned was that people don’t really need to know much about something to have an opinion of it. A strong opinion. And that is scary. It sounds somewhat innocuous—and it is when you are talking about new toppings for pizza or new scents for room freshener. But it is frightening when you are talking about public policy decisions that will affect millions of people. It makes you realize the founding fathers were on to something with the Electoral College. And that things like low voter turnout is probably not that bad after all.
Unfortunately my focus group days seem to be behind me for the foreseeable future—at least being behind the glass. I’m always willing to participate in a focus group as a subject, although I would find it hard to resist the temptation to mess with the people behind the glass in ways that only someone who has spent many hours behind the glass in the dark…eating…and watching…and eating, could know how to do.
Now I have to feed my lust for insight into people in other ways. Like sitting in a coffee shop with headphones on, pretending to listen to music, while actually listening to everything the women behind me are saying. (In case you didn’t guess, I’m doing that right now, and I’ll have something to report on the seemingly fragile mental health of new mothers soon.)
This entry started as an intro into my upcoming series I referenced in an earlier post and then took on a life of its own. The future pieces are from a collection entitled, “The We’re Trying to Have a Society Here Essays.” In them I’ll look at the little things people do that make me think we as a society are in serious trouble. Thinking about how to intro the essays got me thinking about focus groups, and that got me here. So, thanks for reading, stay tuned, and just let me know on a scale from 1 to 7 how good you think this entry was, where “1” means you think it was really good, and “7” means you think it was one of the best things you’ve ever read.
1186 words.
No comments:
Post a Comment