Monday, 8 July 2013

Beware of magic numbers

Every so often social media erupts with a new, numeric meme. This magic number purports to summarise in one easy, often round integer, some universal truth. The classic example is the “You only use 10% of your brain, we have all this untapped potential” hokum that you often hear regurgitated. My most recent numeric meme experience was “you have 35,000 thoughts a day”.
My Facebook and Twitter feeds started trending wit this claim and because I am a scientist, a data nerd and a profound killjoy, I decided I would investigate this little snippet. There are a couple of problems with any claim like this one;

1. How is a thought operationalized? In sciences like psychology, there aren't universal definitions the way there are in other sciences. For example, if I made the claim that on average people are 9 ft tall, you could whip out your ruler, gather a random sample of people and get measuring. You and I would have the same definition of one foot (i.e. 30.48 cm) and you would easily be able to demonstrate that my number is bogus. Since we all have the same rulers a claim like this is easier to assess objectively.

A thought however, is not an standardised unit. There aren't even very good definitions of a thought, much less guides on how to determine one discrete thought from another.
Wikipedia gives us this helpfully circular definition:
“Thought can refer to the ideas or arrangements of ideas that result from thinking, the act of producing thoughts, or the process of producing thoughts. In spite of the fact that thought is a fundamental human activity familiar to everyone, there is no generally accepted agreement as to what thought is or how it is created.”
What Wikipedia has generally correct, is that there is no generally accepted agreement on what a thought is. Certainly in cognitive psychology we wouldn't use the term ‘thought’ as an operational term, because it is too vague and unspecified.
Additionally, thoughts aren’t really discrete units like centimetres or feet. Human minds work using a kind chaotic spread of activation. What this means is that the activation of one idea spreads to semantically linked thoughts and concepts. This is why you can start talking about work, remember you had a coffee at work, think that you like coffee a lot more than tea, and start talking about tea in the next breath. Consider that chain of thoughts for a minute. How many thoughts is that? Is that one big thought about coffee, your experience of it today and your rating of it relative to tea? Or is it three thoughts, one for each concept? What about the sub-threshold thoughts that helped you link all those ideas together?

2. What is the source of the claim? If I am going to go around repeating an empirical claim, I want to make sure that the claim actually has some evidence to back it up. So the next step in my over-analysing of this popular meme was to look for the source of the claim. A fairly rudimentary search on Google Scholar confirmed my suspicions. There was definitely no published empirical work asserting that we have 35,000 thoughts a day, or any other specific number either. The closest I could come to was a nutrition study that aimed to measure how many diet related thoughts we have a day. Even this study, with its relatively limited scope had great difficulty in defining what a single food related thought was, and they essentially just ended up asking participants to mark down in a diary every time they thought about food. This approach would not really work for measuring how many thoughts of any kind we have in a day.

Imagine me asking you to write down in a journal every time you had a thought. The very act of thinking "I better write down all my thoughts today" is a thought. Even if you sat basically catatonic for the whole day, if you remembered the task at all, you would sit there tallying the whole day. Apart from being immensely impractical that kind of a study wouldn't tell us very much about the act of thinking in a natural setting.

Having established that there was no published empirical source, I kept digging. At this point, serious alarm bells started ringing. Not only could I not find any empirical source, I couldn't find any primary source of any kind. It seemed like this meme had just sprung into life of its own accord, infecting social media like a virus. Finally, after a few hours of digging, I found THE source. It was a promotional info-graphic which made a number of empirical claims, amongst them our 35,000 thought cap. Here it is for your viewing pleasure (and a link for those who are truly curious).

If you look bottom, you will see, in tiny pink writing, a handful of references. At this point I got excited and thought that maybe I had misjudged the magic number. Maybe there was some science behind the hype. Or not.

The first reference was, a self-help website article on the ways to help defeat decision fatigue. It outlined a list of fairly sensible time saving techniques like making lists and setting time limits on tasks. All very practical but sadly lacking in magical numbers. Certainly nothing worthy of meme-ification.

The second reference was, a blog post on an investment website which featured some very interesting and totally unsubstantiated claims like this gem “Ninety-five percent of the choices we make every day are managed autonomously by the filters we have developed though our interaction with the world and society. The other five percent of the decisions we make are made by our active self: our ego.” This article was full of magic numbers and certainly some of them might be meme-ifiable but sadly, my magic number was nowhere to be seen.

The next reference was This was a link to a page describing the anatomy and function of the amygdala (it is involved in the processing of emotions, amongst other things). This was certainly a very science heavy link, giving some neuroanatomical authority to the claim, but was unfortunately also totally unrelated to the magic number.

Just when I starting to lose all hope I found this I saw the number 35,000. I got very excited. My data-philic heart started to pound. The reference was a blog post by basketball coach, trainer, author and husband, Cornell Thomas and the key line, the source of all this meme excitement of social media had been found.

“A survey I read said the average adult makes about 35,000 decisions per day.”

And that was it. No reference. No link. No nothing, just this one sentence. Now it might be true that Cornell Thomas read just such a survey. Unfortunately without the source, this claim has all the empirical strength of a Cosmopolitan personality test.

To make matters worse, the info-graphic itself was part of a press release promoting….you guessed it, a decision making app. (For those who are interested: Hours of my time, thousands of Twitter and Facebook mentions, and this empirical claim is reducible to a marketing ploy. 

There is a moral in all of this of course, other than the fact that I probably need to get a hobby. When you hear any kind of empirical claim, don’t take it at face value. Don’t assume that it is correct, that is has been researched, or that the speaker even knows whether it is true or not. This is especially important now as Australia rolls into election time and politicians start flinging around “facts” like these ones.

“Most asylum seekers arriving by boat are economic migrants and on some boats 100% of the asylum seekers are in fact economic migrants and not genuine refugees.” - Bob Carr June 2013 (One of many sources)

This despite the fact that government records demonstrate that more than 90% of asylum seekers arriving by boat are found to be genuine refugees (Asylum seeker facts). Five minutes of research, and I mean easy Google research, can disprove many claims like this, but most people will never even take the trouble to fact check for themselves. This willful ignorance only allows politicians and other people to get away with all kinds of unsubstantiated claims.

We have the capacity for so many thoughts a day (although not necessarily 35,000), let’s use some of them critically.

 P.S At the time of writing this a new thought meme is circulating. Apparently 70,000 is the new 35,000.

No comments:

Post a comment