Sunday, April 16, 2023

You can be too clever with visualizations

Front page of the paper

I saw this visualization on Friday, April 14th 2023, on the front page of the New York Times. It was an illustration for an article about increasing the use of electricity to tackle climate change in the United States. Unfortunately, the visualization is at best confusing. You can read the full article here: https://www.nytimes.com/interactive/2023/04/14/climate/electric-car-heater-everything.html

(New York Times 14th April, 2023)

The message

The message the article was trying to convey was a simple one: it was a modeling exercise for a more electrified US with lower energy consumption. The animations in the article made clear where electricity use would need to grow and roughly by how much.

Why the presentation is bad

The visualization is a sort of pie chart. In most cases, pie charts are very bad data visualizations, and this article compounds the problem by using a non-standard form. 

Just looking at the charts, can you tell me what the percentages are for Transportation, Industrial, etc. now and in the "electrified future"? Can you tell me what's growing?

The article makes plain the modeling work is for reduced energy consumption. Looking at the two charts, can you tell me what the reduction is and over what timescale it occurs?

I could go on, but it's easy to see for yourself what's wrong. Look at the charts and tell me what you take away from them.

The article contains animations that make the message clearer, but even so, it took me a lot of work to figure out what was going on. This takes us to the major visualization sin here: the level of effort to understand what's going on is too high.

What's the takeaway?

You can get too clever with visualizations. Just because you can, doesn't mean you should. Keep things simple and easy to understand.

Sunday, April 2, 2023

I'm curious if you want to buy from me

Over the last few months, I’ve been getting sales prospecting emails that all have the same keyword: “curious”. Here are a few (anonymized) examples:

…and was curious to learn more about your…

…but curious what time to action looks like…

Just curious to check on my previous email …

…but curious if you have any thoughts on the above…

I'm curious if you run into challenges maintaining…

I was curious to learn about your current…

I am curious to see if you are looking…

(A curious cat - Dimitri Houtteman, CC0, via Wikimedia Commons)

I ignore all these emails. Why? Because they’re poor prospecting.

Here’s the reality. These emails are being sent by BDRs or SDRs and they’re prospecting for business. This is a fine, even noble thing to do. But the sender isn’t curious and they’re not interested in my views; the sender wants to know if I’m a prospect. 

These are poor prospecting emails for several reasons. First off, the word “curious” is now a tip-off that the email is prospecting and so destined for quick deletion (I rarely get passed the word curious). Secondly, my email provider is getting smarter and it’s now diverting “curious” emails to my spam folder, I found half of these examples there. Unfortunately for the senders, “curious” is a relatively uncommon word so spam filters easily learn to filter it out.

I’m very busy. I don’t have time to read through a long prospecting email. If the first word I come across is a trigger (e.g., “curious”) I delete the email. The prospecting emails I read are ones that address a real business issue and get to the point quickly. Some really good prospecting emails have mentioned good quality case studies I’m interested in and made it easy for me to read them (I’m not filling in a long registration form and I’m not giving you my phone number).

(There seems to be a broken selling model here. I don't have time to meet with every salesperson who contacts me. You need to establish value first. So provide me with a low-effort way to see if I'm interested then let's escalate commitment. It's a bit like a romantic relationship; very few people get married after a first date.)

I’m sure there must be a BDR or SDR course out there that tells people to use words like “curious” or “interested”. Perhaps in the past, it may even have worked. I’m not convinced it does now. It’s time to move on.

Friday, March 31, 2023

How to improve your business writing

The need for better business writing

Let's face it, a lot of business writing is just bad; you wouldn't read it if you didn't have to. You're left wondering whether the person who wrote it hated writing it as much as you hated reading it.

The problems are well-known: turgid prose, needless complexity, cryptic meanings, pseudo-academic obscurity, and so on.

Annoyingly, books on business writing aren't much help for improving business writing. They explain the difference between an abstract and an introduction and they talk about the need for structure, but that's about it. Style guides are of little use too; they give you advice on how to abbreviate American states but don't help you make your reports readable.


A better type of writing

But business writing doesn't have to be like this. What if I told you there was a style of writing that people:

  • voluntarily read 
  • described as "sparkling", "insightful", or "game-changing" 
  • won Nobel prizes for.

Wouldn't you want to know what it was?

Of course, I'm talking about fiction writing.

Before you object, let me tell you something: fiction writers do a lot of things that business writers should do. They do it so well, they make a living from it. 

Fiction writing basics

Let's look at the basics of what fiction writers do.

Write for their audience. Most, if not all writers know who their target audience is and what they want. Thriller writers, literary fiction writers, and romance writers tailor their writing for the people who buy their books. If they don't, they won't get published and won't get paid. The first lesson for business writers: know your audience and what they want to hear.

Have a good story to tell. No one likes a boring story. Some fiction writers can take a dull situation and turn it into something amazing through insight and storytelling; they can make the ordinary exciting. The second lesson for business writers is: tell your audience an engaging story.

Organize your story. Lots of business writing books tell you about paragraphs and structure, but fiction writing books go much further. There are all kinds of narrative structures available to you; you can tell a story in reverse chronological order, from different character perspectives, with different-length chapters, and so on. Of course, there's also the choice of point of view. Stories don't have to follow a linear time narrative and your business report doesn't have to be linear either. The third lesson for business writers is: use the appropriate organization to tell your story.

Use appropriate language. This is the part I'm saddest about. Fiction writers use all kinds of ways to make their stories engaging:

  • using the active voice rather than the passive voice
  • sentence length
    • use shorter sentences for action and longer sentences for thought
    • vary sentence length for variety
  • avoid using adverbs (Stephen King!)
  • use simple words rather than complex ones (unless the complex word makes a particular point)
  • remove unnecessary words
  • use alliteration, anaphora, tricolon, and other rhetorical techniques
  • ...and so on.

What was the last report that made you think? What was the last report that made you feel anything at all? What was the last report that used alliteration, anaphora, and tricolon crescens to persuade you? The fourth lesson for business writers is: be artful about the language you use to tell your story. 

What to read next

There are a lot of great books on writing fiction. Here are just a few I recommend.

  • Stephen King, On writing.
  • Ursula K. Le Guin. Steering the craft.

Ian Fleming wrote about writing. What he said has aged badly in many places, but his underlying message is clear: he wrote for his audience and he thought deeply about what they wanted. Here's the piece.

Sunday, March 19, 2023

Is it OK to lie in business for the right reason?

Can lying in business be OK?

A long time ago I was in a very difficult ethical position. The best thing to do for the business was to lie and mislead technical executive management, but it didn't sit right with me. My boss was more flexible, he did the misleading for me, and the result was the executives made the right decision. Listen to my story and tell me what you think was the right thing to do.


The situation

My company has previously bought exclusively from supplier X, but supplier X was having financial difficulties and had stopped innovating; their products were extremely expensive relative to the market. My company had just spent a large amount of money with X and our technical executive management was very committed to them, it was very apparent we were an "X shop". At the same time, the technology had changed and new entrants had come to the market offering faster and cheaper products. 

My team needed new hardware to generate network traffic. We tried the solution from X, but it just didn't give us anything like the performance we needed and was extremely expensive. We then tried a product from one of the new entrants which gave us the performance we needed (and more) and was much cheaper. In fact, performance was 10x faster and we had the test results to prove it. So my team wanted to buy from the new entrant.

The dilemma

My manager told me that if we told the truth that the new entrant was 10x faster than X and much cheaper, technical executive management wouldn't believe us and we would lose credibility, in fact, it's likely we would be told to go with technology from X even though it wasn't good enough. 

I wanted to educate technical executive management and show them what we'd found. My boss said that was a bad idea and we should spin a story technical executive management could accept.

What should we do?

What happened

My boss took the results and did some massaging. He told the technical executive team that while the new entrant wasn't as fast as company X, it was a lot cheaper and was a better fit for this particular project - he implied it would be a sacrifice for the team to go with the new entrant, but we would do it to save money. He reassured executive management that their prior decision to go with X was sound because the other projects were different. He presented a version of the results that hinted we needed more equipment from the new entrant than we would need from X, but it would still be cheaper overall.

We got permission to go with the new entrant. My boss told me that technical executive management had commented that the new entrant had really come a long way and that maybe in five years they would be as good as X.

Subsequent events

Within a year, supplier X hit bigger financial problems and was taken over. It stopped producing networking equipment completely. My employer moved off their hardware within two years and exclusively bought equipment from new market entrants. 

The story in the specialized press was that X had offered inferior and over-priced products for some time. When new entrants came into the market with faster and cheaper technology, X couldn't compete. X had been reliant on inertia and existing business relationships for sales, but of course, that came to an end eventually.

Technical executive management talked about how their decision to go with the new entrant for my project showed they were on top of things. However, company C-level leadership had changed and they wanted better technical decision-making, so the entirety of the technical executive management team changed. However, I was long gone by this point.

Sunk cost fallacy

This is an example of the sunk cost fallacy where people remain committed to something because of the amount of effort they've previously put into it, even though going with something new would be better. There are numerous examples in business and politics. 

In this case, technical executive management had invested a lot in supplier X, including their own credibility. Because of that investment, they weren't going to change suppliers easily, even though they "should".

Unfortunately, I've seen other examples of the sunk cost fallacy over the years, but never anything as bad as this. Organizational inertia is a real thing and it can be gut-wrenching to make changes. Sadly, the heralds of change are often punished and end up leaving because their message is uncomfortable; the nail that sticks up is the one that's hammered down.

What's the right thing to do?

Over the years, I've come to the conclusion that my boss made the right call in the circumstances. Yes, technical executive management was wrong, but they were deep into the sunk cost fallacy and weren't thinking rationally. There was no way they would have accepted our results, even though they were true. They needed a rationale that would enable them to cling to X while giving us permission to do something new, and my boss gave it to them. The best possible solution for the company was for technical executive management to realize how the market had shifted and change their buying behavior, but they just weren't ready to do so and it would have been career suicide for us to try.

Ultimately, it's not about doing what's right, it's about making the change you can.

What do you think?

Monday, January 9, 2023

The longest journeys in the world

How far can I go by boat, or by car, or by plane?

What are the longest journeys you could make by boat without crossing land, by land without crossing water, and by plane? The answers are a little surprising as we'll see. For comparison, the circumference of the earth is about 24,901 miles, so you'll get a sense of how far around the world you can go.

What's the longest distance I could sail in a straight line without crossing land?

Here's an image of the world using the typical Mercator projection. What would you say is the longest journey you could make over water in a straight line without crossing land?

(Strebe, CC BY-SA 3.0, via Wikimedia Commons)

What if I told you this was the longest straight line journey.

(Source: Reddit)

The journey is 19,940 miles, so nearly all the way around the world. It starts in Sonmiani, Pakistan and ends in the Karaginsky District, Russia.

The reason why this path looks so 'kinked' is because of distortions introduced by mapping a 3d object onto a 2d surface and of course by the Mercator projection. If you look at the path on a globe, it becomes more obvious it's a straight line. Here's a YouTube video that shows an animation of the path.

This path was originally found by a Reddit user (Patrick Anderson, AKA kepleronlyknows) and then proved by Rohan Chabukswar and Kushal Mukherjee. It's a great example of ctizen science.

What's the longest distance I could walk in a straight line without crossing water?

Rohan Chabukswar and Kushal Mukherjee found the answer in their paper, here it is.

This isn't that surprising and the curve should be no surprise either, this is just a grear circle that's entirely on land. It's about 7,000 miles long, so much shorter than the longest sailing. It goes from Sagres, Portugal to Jinjiang, Quanzhou, China

What's the longest plane journey?

The earth is round, so the furthest journey you could do is to the antipodes which is halfway round the world (12,450 miles). In reality, planes don't exactly follow great circles all the time, so we might expect even a journey to the antipodes to be a bit longer.

Here are the world's longest scheduled flights according to Upgraded Points:

RouteAirlineDistanceScheduled Flight Time
1. Singapore (SIN) to New York (JFK)
Singapore Airlines9,537 miles18 hours, 50 minutes
2. Singapore (SIN) to Newark (EWR)
Singapore Airlines9,523 miles18 hours, 45 minutes
3. Perth (PER) to London (LHR)Qantas9,010 miles17 hours, 15 minutes
4. Melbourne (MEL) to Dallas (DFW)Qantas8,992 miles17 hours 35 minutes
5. Auckland (AKL) to New York (JFK)
Air New Zealand8,828 miles17 hours, 50 minutes

At 9,537 miles, flights are not yet getting half way around the world.

What's the world's longest bus journey?

Would you believe London to Kolkata, a distance of 10,000 miles, or even London to Sydney? Wikipedia has an entry all about it. 

What's the world's longest train journey?

This one is easy, it seems like almost every pub quiz asks it. The longest single journey is of course the Trans-Siberian railway at 5,753 miles. The longest connecting journey goes from southern Portugal to Singapore and is around 11,653 miles.

What's the world's longest subway ride?

This was surprisingly difficult to find the answer to. It's line 6 of the Chongqing Rail Transit, which come in at 53 miles. As you might expect, it's a relatvely new subway line (2012) and is rather sleek and modern.

What's an isochrone map?

An isochrone map shows travel time from a point, for example, how long it takes to get from London to various other places. The screenshot below shows how far from Marble Arch you could go in an hour. You can play around with isochrone maps on this great website: https://app.traveltime.com/


How far might I travel in a year?

Depending on the state, the average American drives 14,263 miles per year (40 miles a day), which is over halfway round the world. If you wanted to go all the way around the world in a year, that would be about 70 miles a day.


Sunday, January 1, 2023

My advice to students looking for a job

I can't talk to you all

Every week, I get several contacts on LinkedIn from students looking for their first job in Data Science.  I'm very sympathetic, but I just don't have the time to speak to people individually. I do want to help though, which is why I put together this blog post. I'm going to tell you what hiring managers are looking for, the why behind the questions you might be asked, and what you can do to stand out as a candidate.

(Chiplanay, CC BY-SA 4.0, via Wikimedia Commons)

Baseline expectations

I don't expect new graduates to have industry knowledge and context, but I do expect them to have a basic toolkit of methods and to know when to apply them and when not to. In my view, the purpose of completing a postgraduate degree is to give someone that toolkit; I'm not going to hire a candidate who needs basic training when they've just completed a couple of years of study in the field.  

Having said that, I don't expect perfect answers, I've had people who've forgotten the word 'median' but could tell me what they mean and that's OK, everyone forgets things under pressure. I'm happy if people can tell me the principles even if they've forgotten the name of the method.

The goal of my interview process is to find candidates who'll be productive quickly. Your goal as a candidate is to convince me that you can get things done quickly, right, and well.

Classroom vs. the real world

In business, our problems are often poorly framed and ambiguous, it's very rare that problems look like classroom problems. I ask questions that are typical of the problems I see in the real world.

I usually start with simple questions about how you analyze a small data set with lots of outliers. It's a gift of a question; anyone who's done analysis themselves will have come across the problem and will know the answer, it's easy. Many candidates give the wrong answers, with some giving crazily complex answers. Of course, sales and other business data often have small sample sizes and large outliers. 

My follow-up is usually something more complex. I sometimes ask about detecting bias in a data set. This requires some knowledge of probability distributions, confidence intervals, and statistical testing. Bear in mind, I'm not asking people to do calculations, I'm asking them how they would approach the problem and the methods they would use (again, it's OK if they forget the names of the methods). This problem is relevant because we sometimes need to detect whether there's an effect or not, for example, can we really increase sales through sales training? Once again, some candidates flounder and have no idea how to approach the problem.

Testing knowledge and insight

I sometimes ask how you would tell if two large, normally distributed data sets are different. This could be an internet A/B test or something similar. Some candidates ask how large the data set is, which is a good question; the really good candidates tell me why their question is important and the basis of it. Most candidates tell me very confidently that they would use the Student-t test, which isn't right but is a sort-of OK answer. I've happily accepted an answer along the lines of "I can't remember the exact name of the test, but it's not the Student-t. The test is for larger data sets and involves the calculation of confidence intervals". I don't expect candidates to have word-perfect answers, but I do expect them to tell me something about the methods they would use.

Often in business, we have problems where the data isn't normally distributed so I expect candidates to have some knowledge of non-normal distributions. I've asked candidates about non-normal distributions and what kind of processes give them. Sadly, a number of candidates just look blank. I've even had candidates who've just finished Master's degrees in statistics tell me they've only ever studied the normal distribution.

I'm looking for people who have a method toolkit but also have an understanding of the why underneath, for example, knowing why the Student-t test isn't a great answer for a large data set.

Data science != machine learning

I've found most candidates are happy to talk about the machine learning methods they've used, but when I've asked candidates why they used the methods they did, there's often no good answer. XGBoost and random forest are quite different and a candidate really should have some understanding of when and why one method might be better than another.

I like to ask candidates under what circumstances machine learning is the wrong solution and there are lots of good answers to this question. Sadly, many candidates can't give any answer, and some candidates can't even tell me when machine learning is a good approach.

Machine learning solves real-world problems but has real-world issues too. I expect candidates with postgraduate degrees, especially those with PhDs, to know something about real-world social issues. There have been terrible cases where machine learning-based systems have given racist results. An OK candidate should know that things like this have happened, a good candidate should know why, and a great candidate should know what that implies for business. 

Of course, not every data science problem is a machine learning problem and I expect all candidates at all levels to appreciate that and to know what that means.

Do you really have the skills you claim?

This is an issue for candidates with several years of business experience, but recent graduates have to watch out for it too. I've found that many people really don't have the skills they claim to have or have wildly overstated their skill level. I've had candidates claim extensive experience in running and analyzing A/B tests but have no idea about statistical power and very limited idea of statistical testing. I've also had candidates massively overstate their Python and SQL skills. Don't apply for technical positions claiming technical skills you really don't have; you should have a realistic assessment of your skill level.

Confidence != ability

This is more of a problem with experienced candidates but I've seen it with soon-to-be graduates too. I've interviewed people who come across as very slick and very confident but who've given me wildly wrong answers with total confidence. This is a dangerous combination for the workplace. It's one thing to be wrong, but it's far far worse to be wrong and be convincing.

To be fair, sometimes overconfidence comes from a lack of experience and a lack of exposure to real business problems. I've spoken to some recent (first-degree) graduates who seemed to believe they had all the skills needed to work on advanced machine learning problems, but they had woefully inadequate analysis skills and a total lack of experience. It's a forgivable reason for over-confidence, but I don't want to be the person paying for their naïveté.

Don't Google the answer

Don't tell me you would Google how to solve the problem. That approach won't work. A Google search won't tell you that not all count data is Poisson distributed or why variable importance is a useful concept. Even worse, don't try and Google the answer during the interview.

Don't be rude, naive, or stupid

A receptionist once told me that an interview candidate had been rude to her. It was an instant no for that candidate at that point. If you think other people in the workplace are beneath you, I don't want you on my team.

I remember sitting in lecture theaters at the start of the semester where students would ask question after question about the exams and what would be in them. I've seen that same behavior at job interviews where some candidates have asked an endless stream of questions about the process and what they'll be asked at what stage. They haven't realized that the world of work is not the world of education. None of the people who asked questions like this made it passed the first round or two, they just weren't ready for the work place. It's fine to ask one or two questions about the process, but any more than that shows naivete. 

I was recruiting data scientists fresh out of Ph.D. programs and one candidate, who had zero industrial experience, kept asking how quickly they could be promoted to a management position. They didn't ask what we were looking for in a manager, or the size of the team, or who was on it, but they were very keen on telling me they expected to be managing people within a few months. Guess who we didn't hire?

Be prepared

Be prepared for technical questions at any stage of the interview process. You might even get technical questions during HR screening. If someone non-technical asks you a technical question, be careful to answer in a way they can understand. 

Don't be surprised if questions are ambiguously worded. Most business problems are unclear and you'll be expected to deal with poorly defined problems. It's fine to ask for more details or for clarification.

If you forget the name of a method, don't worry about it. Talk through the principle and show you understand the underlying ideas. It's important to show you understand what needs to be done.

If you can, talk about what you've done and why you did it. For example, if you've done a machine learning project, use that as an example, making sure you explain why you made the project choices you did. Work your experience into your answers and give examples as much as you can.

Be prepared to talk about any technical skills you claim or any area of expertise. Almost all the people interviewing you will be more senior than you and will have done some of the same things you have. They may well be experts.

If you haven't interviewed in many places, or if you're out of practice, role-playing interviews should help. Don't choose people who'll go easy on you, instead chose people who'll give you a tough interview, ideally tougher than real-world interviews.

Most interviews are now done over Zoom. Make sure your camera is on and you're in a good place to do the interview.

Stand out in a good way

When you answer technical questions, be sure to explain why. If you can't answer the question, tell me what you do know. Don't just tell me you would use the Wilson score interval, tell me why you would use it. Don't just tell me you don't know how to calculate sample sizes, tell me how you would go about finding out and what you know about the process.

I love it when a candidate has a GitHub page showing their work. It's fine if it's mostly class projects, in fact, that's what I would expect. But if you do have a GitHub page, make sure there's something substantive there and that your work is high quality. I always check GitHub pages and I've turned down candidates if what I found was poor.

Very, very few candidates have done this, but if you have a writing sample to show, that would be great. A course essay would do, but of course, a great one. You could have your sample on your GitHub page. A writing sample shows you can communicate clearly in written English, which is an essential skill.

If you're a good speaker, try and get a video of you speaking up on YouTube and link it from your GitHub page (don't do this if you're bad or just average). Of course, this shows your ability to speak well.

For heaven's sake, ask good questions. Every interviewer should leave you time to ask questions and you should have great questions to ask. Remember, asking good questions is expected

What hiring managers want

We need people who:

  • have the necessary skills
  • will be productive quickly
  • need a minimal amount of hand-holding
  • know the limitations of their knowledge and who we can trust not to make stupid mistakes
  • can thrive in an environment where problems are poorly defined
  • can work with the rest of the team.

Good luck.

Monday, November 28, 2022

Is this coin biased?

Tossing and turning

A few months ago, someone commented on one of my blog posts and asked how you work out if a coin is biased or not. I've been thinking about the problem since then. It's not a difficult one, but it does bring up some core notions in probability theory and statistics which are very relevant to understanding how A/B testing works, or indeed any kind of statistical test. I'm going to talk you through how you figure out if a coin is biased, including an explanation of some of the basic ideas of statistical tests.

The trial

A single coin toss is an example of something called a Bernoulli trial, which is any kind of binary decision you can express as a success or failure (e.g. heads or tails). For some reason, most probability texts refer to heads as a success. 

We can work out what the probability is of getting different numbers of heads from a number of tosses, or more formally, what's the probability \(P(k)\) of getting \(k\) heads from \(n\) tosses, where \(0 < k ≤ n\)? By hand, we can do it for a few tosses:

 Number of heads (k) Combinations (n)  Count  Probability 
 0 TTT 1 1/8
 1 HTT
 THT
 TTH
 3 3/8 
 2 THH
 HTH
 HHT
 3  3/8
 4 HHH 1 1/8

But what about 1,000 or 1,000,000 tosses - we can't do this many by hand, so what can we do? As you might expect, there's a formula you can use: 
\[P(k) = \frac{n!} {k!(n-k)!} p^k (1-p)^{n-k}\]
\(p\) is the probability of success in any trial, for example, getting a head. For an unbiased coin \(p=0.5\); for a coin that's biased 70% heads \(p=0.7\). 

If we plot this function for an unbiased coin (\(p=0.5\)), where \(n=100\), and \(0 < k ≤ n\), we see this probability distribution:

This is called a binomial distribution and it looks a lot like the normal distribution for large (\(> 30\)) values of \(n\). 

I'm going to re-label the x-axis as a score equal to the fraction of heads: 0 means all tails, 0.5 means \(\frac{1}{2}\) heads, and 1 means all heads. With this slight change, we can more easily compare the shape of the distribution for different values of \(n\). 

I've created two charts below for an unbiased coin (\(p=0.5\)), one with \(n=20\) and one with \(n=40\). Obviously, the \(n=40\) chart is narrower, which is easier to see using the score as the x-axis. 

As an illustration of what these charts mean, I've colored all scores 0.7 and higher as red. You can see the red area is bigger for \(n=20\) than \(n=40\). Bear in mind, the red area represents the probability of a score of 0.7 or higher. In other words, if you toss a fair coin 20 times, you have a 0.058 chance of seeing a score of 0.7 or more, but if you toss a fair coin 40 times, the probability of seeing a 0.7 score drops to 0.008.


These charts tell us something useful: as we increase the number of tosses, the curve gets narrower, meaning the probability of getting results further away from \(0.5\) gets smaller. If we saw a score of 0.7 for 20 tosses, we might not be able to say the coin was biased, but if we got a score of 0.7 after 40 tosses, we know this score is very unlikely so the coin is more likely to be biased.

Thresholds

Let me re-state some facts:

  • For any coin (biased or unbiased) any score from 0 to 1 is possible for any number of tosses.
  • Some results are less likely than others; e.g. for an unbiased coin and 40 tosses, there's only a 0.008 chance of seeing a score of 0.7.

We can use probability thresholds to decide between biased and non-biased coins.  We're going to use a threshold (mostly called confidence) of 95% to decide if the coin is biased or not. In the chart below, the red areas represent 5% probability, and the blue areas 95% probability.

Here's the idea to work out if the coin is biased. Set a confidence value, usually at 0.05. Throw the coin \(n\) times, record the number of heads and work out a score. Draw the theoretical probability chart for the number of throws (like the one I've drawn above) and color in 95% of the probabilities blue and 5% red. If the experimental score lands in the red zones, we'll consider the coin to be biased, if it lands in the blue zone, we'll consider it unbiased.

This is probabilistic decision-making. Using a confidence of 0.05 means we'll wrongly say a coin is biased 5% of the time. Can we make the threshold higher, could we use 0.01 for instance? Yes, we could, but the cost is increasing the number of trials.

As you might expect, there are shortcuts and we don't actually have to draw out the chart. In Python, you can use the binom_test function in the stats package. 

To simplify, binom_test has three arguments:

  • x - the number of successes
  • n - the number of samples
  • p - the hypothesized probability of success
It returns a p-value which we can use to make a decision.

Let's see how this works with a confidence of 0.05. Let's take the case where we have 200 coin tosses and 140 (70%) of them come up heads. We're hypothesizing that the coin is fair, so \(p=0.5\).

from scipy import stats
print(stats.binom_test(x=140, n=200, p=0.5))

the p-value we get is 1.5070615573524992e-08 which is way less than our confidence threshold of 0.05 (we're in the red area of the chart above). We would then reject the idea the coin is fair.

What if we got 112 heads instead?

from scipy import stats
print(stats.binom_test(x=115, n=200, p=0.5))

This time, the p-value is 0.10363903843786755, which is greater than our confidence threshold of 0.05 (we're in the blue area of the chart), so the result is consistent with a fair coin (we fail to reject the null).

What if my results are not significant? How many tosses?

Let's imagine you have reason to believe the coin is biased. You throw it 200 times and you see 115 heads. binom_test tells you you can't conclude the coin is biased. So what do you do next?

The answer is simple, toss the coin more times.

The formulae for the sample size, \(n\), is:

\[n = \frac{p(1-p)} {\sigma^2}\]

where \(\sigma\) is the standard error. 

Here's how this works in practice. Let's assume we think our coin is just a little biased, to 0.55, and we want the standard error to be \(\pm 0.04\). Here's how many tosses we would need: 154. What if we want more certainty, say \(\pm 0.005\), then the number of tosses goes up to 9,900. In general, the bigger the bias, the fewer tosses we need, and the more certainty we want the more tosses we need. 

If I think my coin is biased, what's my best estimate of the bias?

Let's imagine I toss the coin 1,000 times and see 550 heads. binom_test tells me the result is significant and it's likely my coin is biased, but what's my estimate of the bias? This is simple, it's actually just the mean, so 0.55. Using the statistics of proportions, I can actually put a 95% confidence interval around my estimate of the bias of the coin. Through math I won't show here, using the data we have, I can estimate the coin is biased 0.55 ± 0.03.

Is my coin biased?

This is a nice theoretical discussion, but how might you go about deciding if a coin is biased? Here's a step-by-step process.

  1. Decide on the level of certainty you want in your results. 95% is a good measure.
  2. Decide the minimum level of bias you want to detect. If the coin should return heads 50% of the time, what level of bias can you live with? If it's biased to 60%, is this OK? What about biased to 55% or 50.5%?
  3. Calculate the number of tosses you need.
  4. Toss your coin.
  5. Use binom_test to figure out if the coin deviates significantly from 0.5.

Other posts like this