When you love the tools more than the things you build

As my friend Dan Hon would say, a thought fell out during my time attending the AAPOR conference in May.

They care a lot more about the methods of understanding than what it is they're using the methods to understand.

I don't mean this to be uncharitable – it's just that many people who work in political science and polling are extremely interested in how precise, accurate, scalable, repeatable and affordable their tools are. And they spend a lot of time evaluating and improving the tools – arguably more time than they spend thinking about the implications of what the tools help them learn.

In other words: the instruments are the objects of their focus, while the objects they measure, are instrumental.

This is not exactly new to the social sciences. For a few hundred years, we've been developing new ways to collect data about each other and our nations, and new ways to analyze and interpret that data. Along the way, we've also criticized those methods. But as a general matter, we never really question the underlying assumptions that cause us to create these tools in the first place. Or when somebody does question those assumptions, the practitioners interpret the criticisms as not so much about the purpose of the tools, but about how finely tuned the tools are.

"Once we perfect this tool for measuring the average person," they seem to believe, "then you'll see there's nothing wrong with the idea of the average person, and you'll stop worrying about the ethical and social implications of trying to identify the average person."

The average person is only one concept that could be slotted into that formulation. It's an idea that informs how we study publics. Which means it is an idea that informs our politics, and what we invest public and private resources into, and also how we see ourselves in comparison or community with others. It may be a useful fiction, but like a lot of concepts in the social sciences, it is a fiction people think is in some sense true.

And we should probably spend just as much time interrogating concepts like these as we do perfecting the instruments for illustrating those concepts.

But for some reason, we don't.

Maybe it's because we have other experiences that inform this habit: in the natural sciences we have so many examples of decades or centuries of observing a phenomenon and testing it not to understand how it works, but to prove that it works. Later we figure out the mechanism of action; we discover which parts of our description of a phenomenon were right and which were wrong.

Opinions are less mechanistic than biological processes, even though they in some sense arise out of them. It seems that more of the people doing the describing and measuring are only incidentally interested in understanding the mechanisms of action. We can say that people tell us they have less trust in institutions than similar people in earlier times said they had. We can perhaps say what people tell us are the reasons for that loss of trust. But it's much harder to say how people lose trust – certainly it's much harder to do this with a quantitative instrument (I'd argue it's effectively impossible to do that). And in any case, you'd have to ask that question first and then develop methods that are designed to get you an answer to that question.

Thing is – those methods are expensive, time consuming, in some sense subjective; they are longitudinal, and deeply psychological, and extremely personal. It can be difficult for people who like tidy numerical representations of human behavior and emotion and opinion to accept that while the details of an individual's experience can seem so specific and so infinite, the shape of the stories are relatively few and extremely familiar. Once you've heard a handful of stories, you have, in some sense, heard them all. But you have to get awfully close to an awful lot of people and sit with the inevitable discomfort of that, before you can recognize the shapes.

There is, lurking beneath all this effort at describing the relationship between people's thoughts, feelings and actions, a discomfort with an essential truth: people are messy. They don't file themselves neatly away into rows and columns. No one characteristic is entirely determinative of any outcome. When you rely on heuristics, human behavior is hard to predict – even though a lot of human behavior is a result of humans relying on heuristics.

Have you ever tried to describe yourself? Like, your complete self, not just how you look (which is hard enough), but also how you feel inside, and what makes you happy or sad or fearful, and whether the voices that encourage or discourage you are your own voice or that of someone from your childhood, your job, the infinite scroll? If you answered yes, thank you for going to therapy or adopting a mindfulness practice. Also: you especially know how hard it is.

Here's my theory: Every theorist and innovator and critic and practitioner of the social sciences is often engaged in this work as much to understand themselves better as to understand other people. And this may be why it seems like a lot of pollsters prefer measuring people at a distance, or in sterile environments, or through randomized control trials. They see the messiness as a bug, not a feature. They are, of course, wrong.

Put another way, many social scientists do this work to soothe themselves.

Adolphe Quetelet is regarded as the father of the average man. He was trained as an astronomer and used averages to predict the position of various celestial bodies in the night sky. What a thing of beauty, to know just where to look to see a star, any night of the year. How close to perfection.

Why couldn't we use the same concepts to describe – and to predict – humans?

Quetelet was a Belgian man who came of age in a revolutionary time – and frankly did not care for it. It was too much tumult, too little predictability. He wanted to feel more stable, more secure, and one way to do this was to first, calculate the average man, and then, try to shape all men into this beautiful average – average being, to him, perfection, precisely because it was stable.

In other words, he was working out some stuff.

And most of the great social scientists before and after him were working out some stuff. Some were eugenicists who wanted to purify their preferred race. Some were refugees from fascism who wanted to prevent another genocide. Some wanted to understand the human mind better to help people live happy and fulfilled lives; others wanted to use it to manipulate and persuade (Often, these were the same people).

Description – whether written or numerical – is never neutral. And that's because the people who do the describing aren't neutral either. Nobody is.

This week we learned that a kid with enough cash to register a website could inject entirely fabricated opinion data into the political bloodstream and have a major impact on the discourse – not only about polling accuracy but about the popularity of various movements and ideologies. He says he saw a big polling miss in the Michigan Senate primary and wondered just how credulous the media and campaigns and the public might be (both very and not, it turned out). Not a neutral experiment, nor a neutral experimenter.

The ethical implications of his experiment – for pollsters, aggregators, journalists, political campaigns, and voters – are deep. But so too are the ethical implications of the median voter theory, probability-based sampling, regression analysis, the focused interview, and (god help us) synthetic sample.

We miss a lot when we ignore the people who invent and perfect these methods, and their motivations, and the ethical and social and political aftereffects of their experiments. We should look at them all more closely.

As we head into the fall, I'm going to share some of the stories of the innovators and inventors who created the tools and methods for opinion research. A lot of people working some stuff out. Maybe it'll be a nice break from the inevitable horserace coverage we'll face coming out of Labor Day.

If you have a particular tool, method or theory you'd like to know the provenance of, send me an email at farrah@crosstabspodcast.com. I like getting email, and I like learning about these absolute weirdos. You can be my assignment editor. Fun, no?