Four Things You can do to Sort Luck from Skill and Avoid Mistakes in Determining Outcomes

Share

Luck
The business literature is filled with many instances in which lunch and skill are lumped together. Learning to sort luck from skill will not only help you make better bets, it will also help you analyze you work and that of your team from a fresh perspective.

In Think Twice, Michael J. Mauboussin says:

we have difficulty sorting skill and luck in lots of fields, including business and investing. As a result, we make a host of predictable and natural mistakes, such as failing to appreciate the team’s and the individual’s inevitable reversion to the mean.”

[…]

the idea is that for many types of systems, an outcome that is not average will be followed by an outcome that has an expected value closer to average.

Daniel Kanehman captured the idea that any system that combines skill and luck will revert to the mean over time. In measuring outcomes, we don’t consider that often activities are a combination of luck and skill — the amount of each depends on the activity.

Mauboussin provides examples for investing, but this is a principle that holds true for business overall. He says that when we ignore the concept of reversion to the mean, we make three types of mistakes:

  1. we think we’re special
  2. we misinterpret what the data says
  3. we don’t focus our feedback on the part we can control

Feedback based only on outcomes is nearly useless if it fails to distinguish between skill and luck.

We compound these mistakes by falling for the halo effect.

The halo effect is the human proclivity to make specific inferences based on general impressions.

Performance is relative and in The Halo Effect Phil Rosenzweig showed this mistake is pervasive in the business world. There is a tendency to look at a company’s overall performance and to make attributions about its culture, leadership, values, and more. Our thinking is prejudiced by financial performance. So in good times, companies are praised and their success is attributed to a variety of internal factors. While in bad times, companies are criticized and these same factors may be attributed for the failures. The reality is more complicated and dependent upon uncertain and unpredictable factors. Says Mauboussin:

the media often perpetuates the halo effect. Successful individuals and companies adorn magazine covers, along with glowing stories explaining the secrets to their success. The halo effect also works in reverse, as the press points out the shortcomings in poor-performing companies. The press’s tendency to focus on extreme performance is so predictable that it has become a reliable counter-indicator.

 Mauboussin suggests a four-point checklist to avoid the mistakes associated with reversion to the mean:

1. Evaluate the mix of skill and luck in the system you are analyzing

A simple test to tell if an activity involves skill is to ask if you can lose on purpose. Table 8-1 in the book illustrates some examples.

What determines the outcome

we should be careful when we draw conclusions about outcomes in activities that involve luck — especially conclusions about short-term results. We’re not very good at deciding how much weight to give to skill and to luck in any given situation. When something good happens, we tend to think that it’s because of skill. When something bad happens, we write it off to chance. So forget about the outcome and concentrate instead on the process.

2. Carefully consider the sample size

Daniel Kahneman and Amos Tversky established that people extrapolate from small sample sizes.

The more that luck contributes to the outcomes you observe, the larger the sample you will need to distinguish between skill and luck.

[…]

Jerker Denrell, a professor of organizational behavior at Stanford Business School, has shown the link between the sample size and learning. In his paper, Why Most People Disapprove of Me: Experience Sampling and Impression Formation, Denrell argues that the first impression you have of a person or organization can determine your future degree of interaction.

3. Watch for change within the system or of the system

One obvious example is individual changes in skill level.

While our first inclination is to think in terms of acquiring new skills, we should also consider how we lose some skills over time. Think athletes, for example.

Further, the system itself may change.

4. Watch out for the Halo Effect

A whole cottage industry, including business school professors and consultants, is working hard to offer business people tidy solutions for their problems.

[…]

But any time you see an approach offering secrets, formulas, rules, or attributes to achieve success, you can be sure that someone is selling you a nostrum.”

Tidy stories, while appealing, do not take into consideration specific circumstances and context.

“spotting the halo effect requires discipline

 

[image above CC0 Public Domain]


Conversation Agent – Valeria Maltoni

Share

Determining Consumer Sentiment: Trends and Common Mistakes

Share

Most marketers (89%) say they still rely heavily on manual analysis to determine consumer sentiment and do not yet feel comfortable leaving the task completely to software, according to recent research from Snapsify.

The report was based on data from an online survey of 70 social media community managers and marketing analysts who are regularly tasked with determining consumer opinion based on written content, such as Web comments, social media posts, tweets, product reviews, and emails.

Despite the availability analytics tools to process such data, most of the professionals surveyed report that they still conduct manual research: 33% say they always do so, 30% more often than not, and 26% sometimes.

More than half of respondents (59%) say they still do some manual analysis because they believe the sentiment word clouds used by software programs are incomplete, 21% do so because they do not have enough budget for technology tools, and 11% because they don’t trust the accuracy of their current tools.

Below, additional key findings from the report.

Difficulties 

Respondents say their biggest challenges in determining consumer sentiment are separating useful signals from noise and having too much data to sort through.

Common Mistakes

  • 38% of respondents say a common mistake made in the analysis of social content is trusting automated sentiment results without verifying.
  • 24% say people often do not analyze enough data/do not understand the concept of sample size.
  • 23% say marketers sometimes rely too much on analytics programs that are not sophisticated enough to determine correct sentiment and themes.

Tips for Success

  • 63% of respondents say researchers must understand the business need driving the study if a sentiment report is to be successful.
  • 19% say success lies in accurately assessing the qualitative aspects of the data.

About the research: The report was based on data from an online survey of 70 social media community managers and marketing analysts who are regularly tasked with determining consumer opinion based on written content, such as Web comments, social media posts, tweets, product reviews, and emails. Respondents came from a mix of B2B and B2C-focused companies.

Ayaz Nanji is a digital strategy and content consultant. He is also a research writer for MarketingProfs. His experience includes working as a strategist and producer of digital content for Google/YouTube, the Travel Channel, and AOL.

LinkedIn: Ayaz Nanji

Twitter: @ayaznanji

MarketingProfs All In One

Share