Finding a Valid Hypothesis for “What is the Meaning of Life?”

Share

Meaning_of_life_1763245

“No one can live happily who has regard to himself alone and transforms everything into a question of his own utility; you must live for your neighbor, if you would live for yourself.”

[Seneca]

This bit of wisdom is from a member of a group of ancient philosophers — Plato, Cicero, Seneca. The Stoics, as they called themselves because they used to discuss their subjects on the porch (Stoa in Greek), or the public square, debated the question of what constitutes a good person.

A philosopher, statesman, and dramatist, and tutor and then adviser to the emperor Nero, Seneca wrote it as part of the 124 Epistles in c. 65 AD. Their legacy and influence stood the test of time. That is because, as award-winning psychologist Jonathan Haidt says in The Happiness Hypothesis: Finding Modern Truth in Ancient Wisdom:

The ancient philosophers were often good psychologists […] but when modern philosophy began to devote itself to the study of logic and rationality, it gradually lost interest in psychology and lost touch with the passionate, contextualized nature of human life.

It is impossible to analyze “the meaning of life” in the abstract, or in general, or for some mythical and perfectly rational being. Only by knowing the kind of beings are actually are, with the complex mental and emotional architecture that we happen to posses, can anyone ever begin to ask about what would count as a meaningful life.

He goes on to say that philosophy has become more psychological and passionate in recent years. 

In a supremely readable and enjoyable book, Haidt takes us on a tour of select ancient wisdom to help us arrive at the conclusion that we can now answer what might be called the Holy Question “What is the meaning of life?” though we have known the answer for a hundred years. That’s because many of the remaining pieces to that puzzle have fallen into place over the last ten.

Books and movies that try to answer the question, so so only in jest:

In The Hitchhiker’s Guide to the Galaxy, a gigantic computer built to answer the Holy Question spits out its solution after 7.5 million years of computation: “forty-two.”

In the closing scene of the movie Monty Python’s The Meaning of Life, the answer to the Holy Question is handed to the actor Michael Palin (in drag), who read it aloud: “Try to be nice to people, avoid eating fat, read a good book every now and then, get some walking in, and try to live in harmony with people from all creeds and nations.”

These answers are funny precisely because they take the form of good answers, yet their content is empty and mundane. These parodies invite us to laugh at ourselves and ask: What was I expecting? What kind of answer could have satisfied me?

Philosophy does teach we should analyze the question, “to clarify what is being asked before giving an answer,” says Haidt. Some possible methods to find it out meaning:

  • definition — distinguish the meaning of the word “life” from life itself
  • symbolism or substitution — life does not stand for anything; it is life itself we seek to understand
  • plea for help in making sense of something (ref. to people’s intentions and beliefs) — what do I need to know in order to understand something an act where I did not catch its cause

We don’t know what we don’t know, which makes it hard as we try to frame the question well. However, when we ask the question of the meaning of life, what we hope for is some enlightenment:

Once the Holy Question has been re-framed to mean “tell me something enlightening about life,” the answer must involve the kind of revelations human beings find enlightening.

There appear to be two specific sub-questions to which people want answers, and for which they find answers enlightening:

1. the question of the purpose of life “Why are we here?”

2. the question of purpose within life “What should I do to have a good, happy, fulfilling, and meaningful life?”

We hope for a set of principles or goals to guide our actions and give our choices meaning and value. The two questions are distinct, says Haidt, because:

The first asks about life from the outside, it looks at people, the Earth, and the stars as objects“Why do they all exist?” — and is properly addressed by theologians, physicists, and biologists.

The second question is about life from the inside, as a subject“How can I find a sense of meaning and purpose?” — and is properly addressed by theologians, philosophers, and psychologists.

The second question can also be examined by scientific means. The Happiness Hypothesis Haidt forms is an examination of this second question. He reaches this point after explaining why we have too much wisdom, and drawing on:

… ten ancient ideas and a great variety of modern research findings to tell the best story I can about the causes of human flourishing, and the obstacles to well being that we place in our paths.

Each chapter includes an account of how the human mind works — by way of ancient truths like the fact that the mind is divided into parts that sometimes conflict (the rider and the elephant), and that life is the creation of our mind — and then an account of our social lives.

The two truths for our social lives are that of the Golden Rule or Reciprocity as the most important tool to get along with people, and that we are all, by nature, hypocrites, which makes it hard to follow the Golden Rule well.

These tools — and the ancient stories contained in the book to illustrate their power — are at our disposal to help us engage with the question(s) as they arise, improve them, and connect them to our lives.

 


Conversation Agent – Valeria Maltoni

Share

How to Know if Your Split Test is Valid (Hint: Statistics Can Lie)

Share

Did you know that conversion optimization can actually decrease your profit?

When you run tests, it’s easier than you might think to misinterpret the results and make faulty conclusions.

The “winner” that you pick might end up being the long-term loser if you’re not careful.

Equally as bad, if the lessons you take from that test are incorrect, you will multiply your losses as you implement tactics based on those bad conclusions elsewhere in your business.

If you’ve never taken an introductory statistics course, don’t worry — I have you covered. We’ll also go over the ways statistics in your split tests can be deceiving.

Stats 101: A Basic Crash Course

If you never had to take Stats 101 at college or university, you missed out on some exciting stuff… um, not really.

However, there are still a few things from that course you need to know before split testing will make any real sense.

I’m going to go over these concepts quickly now so you’re not confused in the future. If you are already a stats pro, just skip down to the next section.

What in the World Is a Confidence Interval?

Whether you use a conversion tool like Optimizely, or a simple web app like IsValid, you’ll notice that conversion rates always come with a range.

For example, check out this screen shot of a sample test:

ISValid split test confidence interval

The conversion rate is currently 4.3%, but there’s a range underneath it from 0.6 to 8.0%. This means that given a large enough sample, the conversion rate could fall anywhere in that range.

Now this doesn’t mean that the far ends (0.6 or 8.0) are likely, it just means that they are possible.

How Important is Significance?

Statistically significant — have you heard that term before?

The significance of a test tells us how confident we should be that we have the right result when we are picking from 2 or more options.

When you run a basic A/B test, you’ll have a confidence interval for each option. In many cases, these 2 confidence intervals will overlap.

See the example below, for example. The original could have a conversion rate of up to 5.6%, while the variation (the current winner) could have a conversion rate as low as 0.6%.

split test interval comparison

Does this mean the current results are useless? No, not at all.

But it means that we need to calculate the significance of the test in order to determine how confident we can be when we choose the variation as the winner.

According to the tool, the significance is currently 91.1%. This means that 91.1% of the time, the variation is the best performing option. However, that leaves 8.9% of the time where the original is actually the best.

ISValid split test significance

In reality, tests are typically run until a 95% or higher significance is achieved. Even at 95%, 1 in 20 tests will end up with you picking the worst option. While it would be ideal to test everything to a 99%+ significance level, it’s not always possible due to traffic or time limitations.

A note on significance: If you can only get to 95% significance in most tests, that’s not ideal, but it’s okay. Just understand that not every lesson you learn is going to be correct, and that you should expect a conflicting result once in a while.

A Critical Variable: Sample Size

Flip a coin 10 times, and you’re fairly likely to get lopsided results, like 3 heads (30%) and 7 tails (70%), even though in theory, they should be split 50/50.

Flip that coin 100 times, and you’ll get closer to the real probability, something like 48 heads and 52 tails.

See where I’m going with this?

The larger the sample size you have when running a test, the more accurate the results are.
Your sample size is one of the most important factors in determining the significance of a test.

There are plenty of simple sample-size calculators out there that you can use for free. Just about every conversion optimization tool has a calculator built-in as well.

Here’s a look at Optimizely’s free web calculator:

sample-size-calculator

In this case, you’d need to run the test until you had 10,170 samples (views) for each option.

So that’s Stats 101 in about 5 minutes. Let’s move on to determining if your split test results are actually valid.

Sample Size is Not Always Accurate — Here’s Why…

Here’s what most business owners do when split testing:

  1. Calculate required sample size
  2. Run test for that long
  3. Pick a winner from the results

That doesn’t seem crazy, does it?

But there are some serious flaws that could have negative effects on your bottom line.

You MUST Segment Your Traffic

Segmenting simply means to divide something up.

In the case of web traffic, you can segment in three main ways:

  1. By source: Traffic comes from different places. Google, Bing, social media, email links and more. Visitors from different sources of traffic tend to behave and convert differently.
  2. By behavior: Did they come to the test page from a certain page on your site? Do some of your visitors read 3+ pages on your site on their first visit or visit at least 5 times a month?
  3. By outcome (conversion): Which visitors convert the best? If applicable, which of those later buy your more expensive service or product?

There are occasions where you can segment by 2 or 3 of the above types all at once. It just depends how detailed you’d like to go.

Getting back to testing validity, the point is that your results can be invalid if you do not pay attention to segments.

Example time…

Your sample size calculator says you need 10,000 visitors for each variation. You do that, and see that one side is the clear winner. However, after digging a bit deeper, you see that the winner had an extra 2,000 visitors from search engines (because of variance, just like flipping a coin). You find that search-engine visitors convert really well on your site, thus skewing the results.

After running the test until the amount of traffic from each traffic source levels out, you see that the original is actually the best — mistake avoided.

You need to consider variation in the most important segments for your business and test before declaring a winner. It may be the sources of traffic, certain behaviors, country, or more. Learn how to segment your visitors with Google Analytics.

Sample Size Does Not Always Reflect a Business Cycle

Check out just about anyone’s analytics reports and what do you see? Massive variation in traffic numbers based on the day. Usually a peak during the week, followed by a massive dip on the the weekend.

However, it’s not always just numbers. You get different visitors based on the day. If you dig a bit deeper, you’ll likely notice that the traffic you get from different sources also changes a bit from day to day.

google analytics dashboard

A “business cycle” typically refers to one week for most businesses, although it may be different for yours. Whatever time period encompasses most types of your typical visitors is a business cycle.

What happens when your sample size calculator says to run a test for 10,000 impressions, and you have 20,000 visitors in one day?

You finish the test in one day. But this doesn’t take your business cycle into account. You might have a valid result for visitors on a Monday, but not necessarily for visitors overall.

This particular testing problem isn’t usually a problem unless you have great traffic numbers. Nonetheless, be aware of it.

Takeaway:
Always run a test for at least 1 business cycle.
You can always test more variables (multivariate testing) if you have excess traffic.

Assessing the Validity of a Test

If you’re feeling a bit overwhelmed, don’t worry. We can simplify this process into 3 main steps:

1. Calculate Your Minimum Sample Size

Determine what level of confidence (significance) you’d like in your test’s results, and calculate a sample size based off of it. This will be the minimum number of impressions/visitors that your variations need.

2. Check for Discrepancies in Segments

Before the test is complete, you should already know how to segment the visitors to your website. Once the minimum sample size is complete, dig deeper to determine if there are any major discrepancies. If so, keep the test running.

3. Assess Your Business Cycle

Your tests need to run for a whole number interval of business cycles. If the minimum sample size is up after half a cycle, or 1.75 cycles, keep it running until the next whole number (1 or 2 cycles respectively in this example).

That’s all there is to it, 3 fairly easy steps. Statistics are your friends, as long as you understand them.

Do you have any questions? Leave them below and I’ll answer them the best I can.

Read other Crazy Egg posts by Dale Cudmore


The Daily Egg

Share