Four Things You can do to Sort Luck from Skill and Avoid Mistakes in Determining Outcomes


The business literature is filled with many instances in which lunch and skill are lumped together. Learning to sort luck from skill will not only help you make better bets, it will also help you analyze you work and that of your team from a fresh perspective.

In Think Twice, Michael J. Mauboussin says:

we have difficulty sorting skill and luck in lots of fields, including business and investing. As a result, we make a host of predictable and natural mistakes, such as failing to appreciate the team’s and the individual’s inevitable reversion to the mean.”


the idea is that for many types of systems, an outcome that is not average will be followed by an outcome that has an expected value closer to average.

Daniel Kanehman captured the idea that any system that combines skill and luck will revert to the mean over time. In measuring outcomes, we don’t consider that often activities are a combination of luck and skill — the amount of each depends on the activity.

Mauboussin provides examples for investing, but this is a principle that holds true for business overall. He says that when we ignore the concept of reversion to the mean, we make three types of mistakes:

  1. we think we’re special
  2. we misinterpret what the data says
  3. we don’t focus our feedback on the part we can control

Feedback based only on outcomes is nearly useless if it fails to distinguish between skill and luck.

We compound these mistakes by falling for the halo effect.

The halo effect is the human proclivity to make specific inferences based on general impressions.

Performance is relative and in The Halo Effect Phil Rosenzweig showed this mistake is pervasive in the business world. There is a tendency to look at a company’s overall performance and to make attributions about its culture, leadership, values, and more. Our thinking is prejudiced by financial performance. So in good times, companies are praised and their success is attributed to a variety of internal factors. While in bad times, companies are criticized and these same factors may be attributed for the failures. The reality is more complicated and dependent upon uncertain and unpredictable factors. Says Mauboussin:

the media often perpetuates the halo effect. Successful individuals and companies adorn magazine covers, along with glowing stories explaining the secrets to their success. The halo effect also works in reverse, as the press points out the shortcomings in poor-performing companies. The press’s tendency to focus on extreme performance is so predictable that it has become a reliable counter-indicator.

 Mauboussin suggests a four-point checklist to avoid the mistakes associated with reversion to the mean:

1. Evaluate the mix of skill and luck in the system you are analyzing

A simple test to tell if an activity involves skill is to ask if you can lose on purpose. Table 8-1 in the book illustrates some examples.

What determines the outcome

we should be careful when we draw conclusions about outcomes in activities that involve luck — especially conclusions about short-term results. We’re not very good at deciding how much weight to give to skill and to luck in any given situation. When something good happens, we tend to think that it’s because of skill. When something bad happens, we write it off to chance. So forget about the outcome and concentrate instead on the process.

2. Carefully consider the sample size

Daniel Kahneman and Amos Tversky established that people extrapolate from small sample sizes.

The more that luck contributes to the outcomes you observe, the larger the sample you will need to distinguish between skill and luck.


Jerker Denrell, a professor of organizational behavior at Stanford Business School, has shown the link between the sample size and learning. In his paper, Why Most People Disapprove of Me: Experience Sampling and Impression Formation, Denrell argues that the first impression you have of a person or organization can determine your future degree of interaction.

3. Watch for change within the system or of the system

One obvious example is individual changes in skill level.

While our first inclination is to think in terms of acquiring new skills, we should also consider how we lose some skills over time. Think athletes, for example.

Further, the system itself may change.

4. Watch out for the Halo Effect

A whole cottage industry, including business school professors and consultants, is working hard to offer business people tidy solutions for their problems.


But any time you see an approach offering secrets, formulas, rules, or attributes to achieve success, you can be sure that someone is selling you a nostrum.”

Tidy stories, while appealing, do not take into consideration specific circumstances and context.

“spotting the halo effect requires discipline


[image above CC0 Public Domain]

Conversation Agent – Valeria Maltoni


SEO is no longer a skill you can just pick up: Training an SEO

85 Flares Twitter 46 Facebook 0 Google+ 3 LinkedIn 29 Buffer 7 Email StumbleUpon 0 Pin It Share 0 Reddit 0 85 Flares ×

Back in 2011, when I was first starting out in my SEO career; bright eyed, bushy tailed and spinning content like there was no tomorrow, I wrote an article on ‘How to Learn to do SEO’ and one of my very first sentences was:

Throw yourself in at the Deep End

SEO (or digital marketing, online PR etc) is not an industry where people sit down for 3 months, hold your hand and train you. And nor should it be.

I am now going to dedicate the rest of this article talking about how I was wrong.

Within my (admittedly completely subjective) experience, SEO managers are not always the best at training. We get our juniors in, give them some Moz resources, a small client and after a few short weeks expect results to start coming in. There’s a couple of reasons for this, firstly, because that’s how we learnt; scrappy SEOers armed only with Scrapebox, a reliable content writer on oDesk and an insatiable desire to achieve top ranking positions! We learnt through trying, testing, tweaking, failing and sharing our knowledge with others (hence the eruption of SEO being an industry where people will openly share and contribute their learnings to the wider community).

Secondly, because the fundamental basics of SEO really aren’t all the difficult to grasp (yet to prove this on my parents, however).

Thirdly, because it’s hard to train something so fluid and changeable. Tactics can alter monthly so any training programme that’s set up is likely to be out of date before the year is out. SEO is also largely dependant on personal preference as to how you go about it, there’s the basic things you need to do (have a site that is accessible, content that is relevant and pages that are authoritative) though the means of going about this differ hugely. Utter the word infographic at a search conference and walk out to “are shit!”, “great for small budgets!”, “spam signal!” and so forth. There is no mutually agreed consensus as to what good SEO tactics are so training them seems redundant and we often leave people to work things out alone.

However there’s a problem in this, in that SEO is no longer a single discipline, it’s not as easy anymore as tweaking some content and buying some links. While the fundamental basics of SEO really aren’t all the difficult to grasp, juggling multiple clients or projects, prioritising tasks and making the most of your time really can be.

For scalable, long term SEO there has to be a strategy, there has to be a process, there has to be a start, middle and desired end and when you’re new to the channel, this will require guidance.

Start with an assessment

When I was training SEO beginners, I found myself accidentally making assumptions about their knowledge of algorithms, or what a keyword was, or what a link was. When you live work in an industry in the way SEOs often do, going back to the absolute basics can be really tough and I’d end up wasting time pitching training at the wrong level. A potential solution would be to start out by asking juniors to fill out an assessment form to gauge a better understanding of their knowledge with questions ranging from the very basic to the slightly more complex and based on this assessment, you can provide staff with the right training.

Move on to Analytics

Knowing where to start, and what to prioritise, is one of SEOs main challenges. Where there’s a hundred different problems to fix and a hundred different opportunities to exploit taking all of this stuff and sitting down at your desk and just picking something small to do is a challenge.

One of the most invaluable skills to develop within your SEO weaponry (I regret typing that) is the ability to find something tactical and actionable that will have an impact on performance and just do it.

Analytics; be it GA and GWT, Omniture, WebTrends etc needs to be trained but not simply ‘you can see traffic here’ and ‘this is where you find conversions’ but more, ‘this is how you find pages that need content improving’. There’s a nice starter guide on Google Analytics Help.

The Landing Pages report shows the URLs to your website that have generated the most impressions in Google Web search results. With this report, you can identify landing pages on your site that have good click through rates (CTR), but have poor average positions in search results. These could be pages that people want to see, but have trouble finding.

Excel at Excel!

Invest in proper excel training for your staff. Having an advanced knowledge of Excel will save reams and reams of time and is an investment that will more than pay off in the long run. I was fantastically bad at Excel for a very long time (mainly because I was always listening to Slipknot in GCSE IT) and I lost considerable time manually doing things that had I known, Excel would have just done for me. It’s important to remember that Excel is nigh on impossible learn alone, as without knowing it’s capabilities, you easily miss out on faster ways to do things.

Anybody working in an SEO team (be that within outreach or technical) will significantly benefit from being proficient in pivots, IF functions, LOOKUPS, RAND and loads more. There’s a nice article on every Excel for SEO here.

Save tools for later

Tools are great; they save us time, help us define processes and scale, however they are not a great start point for a total newcomer to SEO. Simply because, you need to understand the manual process first. For example, a friend was training one of his team on outreach and he taught her to manually build lists, find influencers using Google advanced queries and how to write a good, tailored email.

It was only after a few weeks of her doing this that he said, “okay, so here’s Scrapebox where you can harvest search results, here’s how to mail merge, here’s how to find influencers using Followerwonk/Buzzsumo/Linkdex/etc”

She was annoyed he’d let her do all of that work manually, but he thought it was important she understood the actual process initially, before getting to grips with how to speed it up.

Get expert training from other channels

You know who’s really good at link building? PRs.

Invest in training from PR or creative experts to instill good outreach habits from the start. Depending on your particular flavour of SEO, establishing the mindset of relationships first and links second is likely to get better results longterm.

Create test sites

Learning on the job is still the fastest way to learn, however it can be difficult to do this on a client or brand site. When I first started out I was so afraid of messing up or doing something wrong that I struggled to anything. Create some test sites (and crucially some time!) for your juniors to play on. Better still, create some budget for your juniors to create some test sites themselves, walk them through setting up hosting, integrating WP etc. This is an excellent resource for absolute beginners.

You can also use these as test grounds to run some research to create blog posts or whitepapers on.

Test and Assess

Find sites that have a lot wrong with them (send me a DM and I’ll give you some ;)) and use these to continually assess the level of the team – what did they find? What did they miss? Where are the gaps in the knowledge?

Using the same site means they can also track their own progress and acknowledge how much they’ve progressed.

Ultimately, the main thing you need to do when training an SEO is make time. It’s easy to get bogged down in client work, pitches or other demands but setting significant time to train someone effectively is never time wasted.

My last comment in that blog post I wrote those years ago however still remains very true, and that is:

Nobody knows everything

There is not a single person in the SEO industry who knows absolutely everything, so don’t be afraid to admit your faults and weaknesses.

85 Flares Twitter 46 Facebook 0 Google+ 3 LinkedIn 29 Buffer 7 Email StumbleUpon 0 Pin It Share 0 Reddit 0 85 Flares ×

Kirsty Hulse is Head of SEO at Found and has six year’s experience defining search strategies for some of the world’s biggest brands, as well as small E-commerce start ups.

State of Digital