A few months ago, I wrote about the USDA’s horrible design for the new food pyramid.

Personally, I don’t understand how people are supposed to get the message about balancing an active lifestyle with a well-balanced diet by seeing Xiao Xiao on a staircase next to a vertical-striped triangle that begs the question “Why is this a pyramid again?”.

For those of you wondering how the hell they settled on this design, the USDA has published a report (PDF) on the decision-making process that led to the infamous “MyPyramid”. To be fair, the report does point out some important problems with the original design :

Many participants indicated that they were familiar with the Food Guide Pyramid…In addition, many participants mentioned some of the nutrition messages the Pyramid was designed to convey, including the importance of eating a variety of foods (foods from every level of the Pyramid), moderation, and proportionality (eat more from the base and less from the tip).

Despite this familiarity with the Pyramid and recognition of some of its nutrition messages, participants had difficulty recalling more specific information about the tool and did not always understand this information correctly. Less than one-fifth of participants were able to place all food groups in the Pyramid on their correct tiers,2 and one-in-eight could not place a single food group on its correct tier. More than half of the participants, however, placed the fats, oils, and sweets group correctly atop the Pyramid.

Participants also expressed a great deal of confusion regarding serving sizes. They were not able to effectively recall the serving size recommendations for each food group, nor were they able to compare how their typical meals correspond to the recommendations.

I’ll leave it up to you guys to decide whether you think the new design actually solves any of those problems.

The rest of the report is standard focus group crap, but I did like this bit :

Participants also compared the staircases in each concept. They noted that they preferred the stairs in Concept J5 to those in Concept D because there are fewer of them and thus, “would be easier to climb.” A few other participants commented that the lighter shading of the stairs (in Concept J5) makes it seem easier to reach the top, and this factor, coupled with the boldness of the figure places the emphasis on the person, not on the climb.

Yes, they chose the design that made it look “easier” to be active. I suppose it’s a good thing that they didn’t have a design that showed the figure sitting on the couch playing Mario Kart.

The nitpicking over the staircase, however, was near the end of the focus group testing. Before they had settled on the idea of a stick figure climbing a triangle, they tried out some pretty abysmal designs :

Yawn. I don’t really have much more to add than “Focus groups suck”. More important than their general suckiness, focus groups also aren’t very reliable for three important reasons :

The people who hold focus groups are rarely agnostic about the results they’d like to see. As a result, they’ll often do everything they can to get the results they desire, often without even realizing it. Reading through the USDA report, it seemed pretty obvious that the researchers were in love with one or two designs and did what they could to make those designs the favorites.

Focus groups are a horrible way to determine how something will be received in the “real world”. Looking over these designs, the focus group respondents were likely led to their answers with questions about which design best suggested exercise and nutrition was the key to a healthy lifestyle. Those of us seeing these designs without being observed behind a one-way mirror would probably have a first reaction more like “Why is that stick figure on a staircase?”

Consumer research is really, really hard work. Just ask Ross, who worked at a research company for a few years, and he can tell you all about how difficult it is to get accurate results. I worked at the same place for a week and it was the most miserable job I’ve ever had. In order to get a handful of people to participate in a survey, you have to approach hundreds of people who would rather punch you in the face than answer a brief questionnaire. The people who actually do agree to join in the group rarely give you useful information. The dirty little secret of the survey business is that in the end, the results that are sent back to the customer are often embellished to fit the demographic requirements, exaggerate the number of people questioned, or reach conclusions that are deemed productive. Shhhh…don’t tell anyone.

Of course, the researchers behind the food pyramid survey were well aware of how unreliable their testing was, which is why they inserted this little caveat in the opening section of the report :

Recruiting techniques and small sample size, however, mean that results are not statistically representative of a larger population. Consequently, all findings were considered descriptive and directional.

If we could only convince movie studios and record companies to be this honest…..