What the Data Will Never Reveal

pie chart hell

Few books have been recommended to me so frequently and gushingly as Dubner and Levitt’s Freakonomics. After tracking down a used copy in a musty Brisbane book exchange, I devoured it before lunch the next day.

It really is a compelling book. Its premise is that conventional wisdom is often wrong, because society’s experts use their informational advantage to serve their own interests, rather than to give us the truth. This isn’t because they’re particularly selfish, but simply because they are human beings, and like the rest of us, they operate out of personal incentives.

Armed with this insight, the authors, journalist Stephen Dubner and economist Steven Levitt aim to overturn conventional wisdom by looking at what the data actually indicates. Does capital punishment actually deter criminals? Does going to a good school really increase your kid’s chances of career success? They also scoured public records to answer other burning questions: Why do drug dealers live with their moms? Why are so many babies named Madison these days? (Hint: it has to do with a 1984 Tom Hanks movie.)

How much do parents really matter?

One of the most interesting chapters is on parenting. For decades, child development experts have been giving parents hard-worded do’s and don’ts of child-rearing, often in complete defiance of other experts, or even themselves. Should an infant sleep on his front or back? Is a pacifier an indispensable tool, or a dangerous vice? Is spanking essential for well-disciplined children, or does it teach them to be violent?

When it comes to parenting, the conventional wisdom is all over the map, but it does agree on one very broad point: that parenting technique is the crucial determining factor in the child’s success as an adult. The right parenting choices are what make the difference between bright and bleak futures for the child.

This, at least, seems self-evident, but Dubner and Levitt still questioned it. They analyzed a vast set of data about the parents of children in Chicago’s massive public school system to determine which factors matter and which don’t, as far as the child’s success was concerned. They frame this study as the answer to a provocative question: How much do parents really matter?

They identified sixteen contributing factors to be tested, including the parents’ income, whether the child is spanked or not, whether the parent takes them to the museum regularly, and whether the child’s family is intact. They correlated these factors with children’s academic test scores, adjusting for biases from the other factors, and separated them into two groups: the factors that make a significant difference, and the ones that don’t.

Among the factors that did matter:

  • The parents are highly educated
  • The parents have high socioeconomic class
  • The child’s mother had her first child at 30 or older
  • The child had a low birthweight
  • The child’s parents speak English in the home
  • The child is adopted

And among the factors that didn’t:

  • The child’s family is intact
  • The child’s mother didn’t work between its birth and kindergarten
  • The child is regularly spanked
  • The parents read to the child regularly
  • The parents regularly take the child to museums
  • The child frequently watches television

There is a general but undeniable pattern here, as noted by the authors:

…the first list describes things that parents are; the second list describes things parents do.

In other words, the greater proportion of the parents’ effect on the child’s development is determined by circumstances that exist before they ever have children. Clearly the actions of parents do matter, but most of the result can be forecast before the question of “how to parent” even comes up.

But hold on a minute…

Some of you may have noticed the gaping flaw in this argument. The discussion began as a challenge to the notion that parenting style matters a great deal, yet all that’s been tested is the effect of sixteen hand-picked factors on standardized test scores!

Are test scores an adequate measurement of a child’s success in life? What is success anyway? Maybe my Dad’s reading me Around the World in 80 Days as a kid never improved my geography grades, but does that mean it didn’t matter? I won’t try and argue that it did, only that I think equating test grades to success is quite a leap.

Is “How much do parents really matter?” really the question this data is answering? Yet that is how they characterize the results.

Never mind that the entire study is confined to families from Chicago.

The Hazards of Scientific Testing

Don’t get me wrong, I loved Freakonomics, and Levitt’s discoveries about parenting are fascinating and helpful in understanding human behavior. The book makes economics interesting for perhaps the first time ever, and dismantles many other standing dogmas.

It’s such a good book, I hate to use it as a bad example, but the chapter on parenting does illustrate three weaknesses of relying only on empirical testing to give you the truth:

Results are always subject to our interpretation, even if the data is flawless. Because we have to stack up all incoming information against what we already think, our biases are always part of the picture, even if the tests themselves are bias-free. We can’t escape personal biases, no matter what we do. If a majority of others share your conclusion, that might convince you that your biases aren’t skewing your opinion, but maybe they have the same biases as you, or, maybe the fact that others agree with you is a bias itself. Maybe it’s not even a maybe.

This means even airtight data can lead us to inappropriate conclusions. I was generally a very good student, test-scores-wise, but I wasn’t always happy, or successful at anything but getting good marks.

And am I successful today? In my opinion yes, but an economist might disagree: I grossed about $16000 in the last twelve months, and at the moment I’m unemployed and technically homeless. Yet it’s been the best year of my life, and I would say I’m more successful than I’ve ever been — because I’m the happiest I’ve ever been.

I think happiness is the best indicator of success, because it’s what we all want, but how do you measure it? You can’t, and that reveals another weakness of empirical testing:

Science is predisposed toward that which lends itself to measurement and testing. To conduct a proper study, you need variables that can be measured and tested. Grades. Income levels. Percentages. Yes/No answers. But how do you measure happiness? The amount of time you spend smiling? Whether you say you’re happy?

Happiness is complex, personal and subjective, so there will never be a good way to test for it. Often when researchers want to test for something slippery, they find a related quantity that they can test for, to hopefully reflect what they’re actually trying to find — such as measuring children’s test scores to gauge parental success.

Scientific analysis has done a wonderful job of measuring the measurable aspects of the universe, but there are bound to be enormous realms of knowledge for which it is ill-suited. Why should everything be readily testable?

How would science tell you, for example, what enlightenment is, whether it exists at all, and whether it means the same thing to two different people. Science’s methods can’t do much for us here, but that doesn’t mean it isn’t worth investigating, or that our understanding of it isn’t extremely beneficial to humanity.

There is no standing definition for enlightenment because by all accounts it eludes description, cannot be understood conceptually and isn’t really anything to begin with. Where would science even begin to tackle this one?

Science and religion have long been at odds, for reasons political as much as spiritual. But clearly spiritual experiences (whatever that even means to you) are wholly subjective, and you’re never going to be able to test them, measure them or even explain them in ways that another person could really get.

There is a reason Zen masters teach their students with nonsensical parables called koans. They can’t simply pass on the information to another person, and that’s because the teaching isn’t information. So they must use abstract poetry to try and disengage the student’s thinking from its normal empirical approach in order for them to comprehend something unmeasurable and indescribable.

And thirdly:

No matter how convincing the data is, that’s all it can do: convince you of something. No book or study can actually bestow the inarguable, end-all-be-all Truth on you. We tend to think of learning as acquiring more and more of the truth. But we’re really just collecting more and more evidence, more reasons to believe X or Y is true, and Z is not.

That’s all knowledge is: belief. Everything you know is just a belief you don’t believe is worth questioning anymore. That’s fine. It may make perfect sense to believe it. The evidence may be overwhelming. But we can forget that education is only ever the push and pull of opinion, by means of conveying evidence. A study is evidence. The endorsement of an expert is evidence. Photos are evidence. Personal experience is evidence.

Science often gives us the most reliable evidence, and there is probably no better tool for generating solid reasons to believe a particular thing. But it can only give us a better, safer position from which to make what is ultimately a leap of faith — adopting a new belief.

Dubner and Levitt convinced me of all sorts of things, but ultimately I’m taking their word for it. Even if I had done all the legwork myself, it would only just make me more sturdily convinced of those particular conclusions. Or maybe I would have written a book arguing something completely different.

In any case, the intriguing conclusions in Freakonomics would never have reached me had they not already been so wildly popular. I bought the book only because it was so highly recommended. This is another powerful mechanism at work: the more attractive a point of view is to us, the more convincing it is.

What makes it attractive? Well, that it makes you feel good to agree with it — either because you are fond of the source, because you feel you’ve been let in on some powerful secret, or because you already feel the same way.

It works the other way too. If you are a staunch anti-abortionist, you might not find the book very convincing, because you’ll find one of Levitt’s theories quite unpalatable, no matter what your left brain makes of the data:

Perhaps the primary reason Freakonomics is so popular is that it contains Levitt’s famous finding that the rash of abortions following the 1973 Roe v. Wade ruling caused crime to drop dramatically twenty-one years later — the inevitable result of millions of unwanted, crime-prone children never having been born. If your emotions don’t like that one, they’ll tell you to look elsewhere for answers, and naturally you’ll find different ones.

The data they bring to the table is convincing, in part because it is so explicit, but mostly because Freakonomics is such a charismatic book. “Dazzling” is a word that appears in many blurbs about it. It does dazzle. It’s so dazzling and user-friendly that I think the reader is in danger of falling in love with its conclusions because they’re just so curious and irresistible. And because everyone else already has.

In fact, I am still quite infatuated with it, enough to recommend that you go out and buy this book. If you made it to the end of this post, it’s a safe bet that you’ll enjoy it.

…if you’re willing to take my word for it.

R

Photo by GDS Digital

Learn to live in the present

Everyday mindfulness has transformed my life, and has for countless others. You can use it to reduce stress, deal calmly with trouble, and experience joy and peace throughout each day. Making it a habit is easier than you probably think. Learn how.


{ 20 Comments }

DiscoveredJoys June 17, 2010 at 9:06 am

First, a joke:

Those that can, do
Those that can’t do, teach
Those that can neither do not teach, work as executives in

The point of the joke? I was told by an instructor in a financial course run by my old employer that economists fell into two categories. The really good ones, who nobody ever heard of, who work privately for the really really mega rich, making them richer. All the other economists are not that good and they work as lecturers, newspaper columnists, or TV economic pundits.

Explains a lot really.

David June 17, 2010 at 6:59 pm

Makes sense. In the intro of the book Steven Levitt describes himself as rather uninterested and unskilled in conventional economics. He doesn’t really care about the stock market or trade. He’s more interested in using economics as a tool to answer questions about human behavior. So I guess he falls in the second category.

Trish Scott June 17, 2010 at 10:32 am

Bravo!

Reminds me of the old saying, “If you can’t dazzle them with your brilliance, baffle them with your bullshit.”

You have shown that it doesn’t really matter which you do :D.

David June 17, 2010 at 7:01 pm

Haha.. the book is quite good though! Because it is so upfront with its data, I felt much less of a BS factor than other books by experts.

Joy June 17, 2010 at 12:24 pm

I’ve come to rely on my intuition and gut sense about things rather than external facts and figures. The “experts” can talk amongst themselves for all I care. I only just sometimes enjoy the external evidence as something interesting to which to compare my direct experience.

David June 17, 2010 at 7:01 pm

Intuition is one of those things that is difficult to measure and therefore easy to dismiss.

Brenda (betaphi) June 17, 2010 at 1:53 pm

I’ve always felt that happy children grow up to be happy adults and that happiness, as you say, is the best indicator of success. My best advice for raising happy children is “do less, not more.” Children are naturally happy until adults start interfering, imposing, indoctrinating, all with the intention of being “good” parents and teachers. Leave those kids alone!

David June 17, 2010 at 7:03 pm

That was one of the other conclusions in the book, that “helicopter parenting” tends to backfire, or at least be a waste of time.

Char (PSI Tutor:Mentor) June 17, 2010 at 4:05 pm

As one who facilitates other to learn research design I agree with a lot of what you say at a general level; except that there is always bias, even in the most standardised instrument~ which is why we use stats to factor out the probability of the result occurring by chance.

You also mentioned that the factors for the parent study were handpicked~ did they use a Factor Analysis? That let’s the data organise itself (depending on test type). I would need to see their method: sample size, sampling method, the reliability and validity values of the tests they used prior to administration etc

The correlation study with the presidents~ well, we know that a relationship, even a strong one that is significant (beyond chance) does not mean cause and effect; those with large feet may have a strong positive relationship to the purchase of caramel popcorn at movies; doesn’t mean that the larger footed person is more likely to order it.

Not all empirical studies use standardised scores, qualitative studies can still be empirical. The point of the systematic method is to decrease bias (cause it is always there).

The biggest issues with empirical science is that many can cling to it as the only way to investigate the world (many non-Western nations use other ways); and that we don’t include it in the education system for general learning, so that when one reads the stats one knows how to critique what one is reading.

David June 17, 2010 at 7:10 pm

They used the phrase “regression analysis.” The sample size was very big (Chicago’s school system is huge) but they didn’t seem to think it mattered that it was confined to one city.

They are also quite adamant about explaining the difference between correlation and causation, and they explain how they determined a causal effect each time they find one.

The biggest issues with empirical science is that many can cling to it as the only way to investigate the world (many non-Western nations use other ways); and that we don’t include it in the education system for general learning, so that when one reads the stats one knows how to critique what one is reading.

Ah, this is what I was getting at. :)

Char (PSI Tutor:Mentor) June 17, 2010 at 7:49 pm

Regression is used for making predictions, but if the data does not meet the assumptions (normal distribution etc) “it’s crap in, crap out”.

Too large a sample can make even a slight difference significant, e.g., some gender difference studies

They seem confident that they have met the assumptions for the test (I am assuming they pre-tested to be sure) and as they used standardised scores could use a test which allows results to be generalised to a wider population.

Evidently, their population is not meant to be cross-cultural ~:-), but even within North America there are dramatic differences between individuals, schools, living environments and other contextual variables, so exactly who the results could be generalised to would depend on whom they identified as the population.

michi June 17, 2010 at 6:43 pm

Hi! It is so great these past several posts to read about how our own perceptions and biases colour or thoughts and experiences — I agree! (Hahaha, I guess this is why I have particularly enjoyed these posts, haha!)

You are touching on some issues taken up by Derrick Jensen, I think particularly in one of his earlier books, A Language Older than Words. (http://www.derrickjensen.org/) Ha, he’s also an EXTREME left-wing eco-anarchist, so his later stuff is really out there…

Anyway hahaha excellent posts, in general but especially lately! :D

David June 17, 2010 at 7:19 pm

Thanks Michi. I didn’t notice I was running a theme this last few weeks, but it looks like I am. I guess everything we do hinges on perception. I’ll check out derrickjensen.org

JoyChristin June 17, 2010 at 7:53 pm

‘K. Since there is another “joy”…I’ll tweak my name a bit…I do love that there is so much joy….
As far as books on any topic..they are ultimately a composite of what the author wishes to share…plus, when I read I am imposing my own expections/experiences on what I am reading..so you are right in that it is all a jumble of perception.
I used to want a book for every single thing, to guide me in all..show me this example, explain to me how to do this..I find the information most valuable, however I also find my own truth to be that I may devour information on anything, yet in *living* I find true knowledge. For example, you may explain to me the proper technique for sailing in heavy weather, but when I’ve actually been in heavy weather (even the term is subjective) I’ve found that 1-the book didn’t cover my exact situation, and 2–I’ve learned to thrive in heavy weather saiing by going far beyond my compact textbook…And parenting..shoot there isn’t a parenting book that exists to cover my experiences :) Same with love…
We want reassurance, knowledge, respite, encouragement, inspiration..we turn to books..when it’s all right here at our fingertips in the living of life…
Just *do* it..parent..sail..love..live…all of it :)

Brad June 17, 2010 at 10:27 pm

As I recall, didn’t Levitt and/or Dubner themselves quibble with the conclusion on abortion? They made the proposition that you can equate each saved fetus to a fraction of a real human being. I think they used something small, like one-hundredth of a real person. And then they showed that the amount by which each abortion saves lives later down the road (by preventing murder) is less than the the “partial human” expunged in the operation. At least I recall something like that.

Science is about empirical results, by definition. But those empirical results can be good indicators of internal states. A happy person will say they are happy more than a sad person. It’s no surprise that the citizens of countries like Denmark say they are happier overall (in surveys” than countries like Somalia. And then there are neurological correlates, which are great indicators of subjective experience.

You are a fast reader. I think the book took me a couple weeks to get through.

David June 18, 2010 at 11:49 pm

Yeah I thought those “cost of a human life” calculations were pretty arbitrary and meaningless. I thought it was weird that they even left that in the book.

Dolley May 3, 2011 at 11:27 pm

You’re a real deep thinker. Thanks for sahring.

Merily September 27, 2011 at 7:57 pm

Hahah, I was already considering buying it before I got to the end of the post. I like that you can question the conclusions. It will make me think more deeply when I read it but I will still try to keep an open mind.

fun and interesting bachelorette party ideas June 21, 2013 at 10:03 pm

Attractive section of content. I just stumbled upon your website and in accession capital to assert that I get actually enjoyed account your blog posts.
Anyway I will be subscribing to your augment and even I achievement you access consistently fast.

J Dilla July 18, 2013 at 7:10 am

Great article David. I too was caught in the strange position of both loving Freakonomics for its bold and convincing style, and having misgivings about some of its rather dubious assertions and conclusions.

Comments on this entry are closed.

{ 2 Trackbacks }


Raptitude is an independent blog by . Some links on this page may be affiliate links, which means I might earn a commission if you buy certain things I link to. In such cases the cost to the visitor remains the same.