How to be creative like David Bowie, R.E.M. and Coldplay

Technical people are creative too

It's obvious that songwriters are creative, but technologists have to be creative problem solvers too. Many business technical problems are poorly defined at best and require a great deal of imagination just to get started.

(It's not just artists who have to be creative. Michelangelo, Public domain, via Wikimedia Commons.)

One of the big stumbling blocks for truly solving technology problems is local optimization. Imagine you're trying to reduce the processing time for a machine learning algorithm. You can focus on optimizing calculations, but would you be better off using a different algorithm altogether? Graphically, it looks something like this.

(Local and global optimization. Adapted from Wikimedia Commons. Author: Martynas Patasius. License: Creative Commons)

You end up focusing a great deal of effort for little gain when a more original approach might yield dividends.

This begs the question, how can you be more creative, how can you find more original approaches? How do songwriters and other creatives jump-start the creative process?

There's a technique I've found very useful: Oblique Strategies, but you have to be creative about how you use it.

Oblique Strategies

The painter Peter Schmidt and the musician Brian Eno first met in the late 1960s. They had a lot in common, including an interest in the creative process. Schmidt created a list of 55 quotes to overcome artistic blocks and Eno had been working on something similar, so in 1974 they combined their efforts to produce "Oblique Strategies".

Oblique Strategies was originally a set of phrases printed on cards. The idea was simple. If you're stuck in the creative process, view one of the random cards. Try to relate what's on the card to the problem at hand. Obviously, you'll have to jump through some creative hoops to do so, which is the whole point of the exercise.

Here are some example phrases from the cards:

• Lowest common denominator
• Turn it upside down
• Always give yourself credit for having more than personality

(A card from the Oblique Strategies deck. Author: Bastiaan Terhorst, Source: Flickr, License: Attribution-NonCommercial-NoDerivs 2.0 Generic)

Who used it and what were the results?

In the 1970s, David Bowie moved to Berlin and, working with Brian Eno, recorded three albums, Low, “Heroes” and Lodger. You might be more familiar with some of the songs: Sound and Vision, Heroes, and Boys Keep Swinging. To keep the songwriting process going, the team used the Oblique Strategies deck to find new ways of thinking about music and lyrics.

(David Bowie, CBS Television, Public domain, via Wikimedia Commons)

R.E.M. even went so far as quoting some of the phrases on the cards in their music, in their song "What's the frequency, Kenneth?" they quote the card "withdrawal in disgust is not the same as apathy".

Moving to more recent times, Coldplay used the cards when recording their album Viva la Vida or Death and All His Friends, but that may be because Brian Eno was their producer. 

The best descriptions I've found for how musicians use the cards are blog bosts by Rosie Cass and Dave Dyment.

My use of oblique strategies

Oblique Strategies are not a daily tool for me. I only use the cards when I'm stuck and brainstorming with colleagues doesn't work. Because of commercial confidentiality, I'm not going to give you a detailed work problem example, instead, I'm going to give you a similar problem and how I might use the cards.

Imagine I'm trying to find missing data in a data set. I've used a statistical approach to detect missing data, but I'm stuck finding more, I know 30% of the data is missing, and I can characterize 45% of that 30%, but I don't know how to find the remaining 65%. What can I do? This is where I would turn to Oblique Strategies. The art is to use the words on the card as a starting point for your thinking.

My starting point is one of the online card generators, let's start with this one. I pull a card online:

• It is quite possible (after all)

This makes me think about what the desired end state might be. Can I identify 100% of the missing data? What 70% be enough? Maybe 45% is the best I can hope for? Do the success criteria change over time? Are there different categories of missing data? Maybe I can only get a few categories?

You can see how the train of thought continues. Let's pull another card.  

• In total darkness, or in a very large room, very quietly

This is harder. Darkness makes me think of the lack of data here. Maybe there are hints about the missing data in the data I have? The large room makes me think of the data collection process. How efficient is it and how does it work?

This card probably isn't as helpful, so let's pull in another card.

• State the problem in words as clearly as possible

Hmmm. This is more direct, but it's probably good advice to write down the problem clearly. So clearly that someone else could work on the problem. Maybe by clarifying the problem I could see a solution.

I could go on, but I think you can see the point. The cards don't provide a solution, but they're a kind of shock to the mind to think about the problem in different ways. It breaks me out of the rut of thinking-as-usual. 

I've used the cards to solve several business-technical problems over the last few years. They've enabled me to tame some very difficult problems. Once you get the idea of how to use them, the process is straightforward - it can even turn a frustrating problem into an enjoyable thinking session.

Where to get the cards

There are several online card generators, chose the one you like best:

• Stoney: http://stoney.sb.org/eno/oblique.html
• Josh Harrison: https://www.joshharrison.net/oblique-strategies/
• https://www.oblique-strategies.com/ - despite the name, this is not an "officially sanctioned" site

If you prefer the tactile feel of real cards, you can buy cards from these vendors, but the prices are high:

• https://www.enoshop.co.uk/product/oblique-strategies.html
• https://www.amazon.com/Oblique-strategies-hundred-worthwhile-dilemmas/dp/B0000EEZG9

Creativity is a must

If you're quite early in your career, then turning to your manager and saying "I'm stuck" is acceptable and even expected behavior. But as the senior person, there might be no one else to turn to, and even worse, you have to solve the "I'm stuck, help me" requests from others. Obviously, experience is a great tutor, but even experience lets us down from time to time. We all need an occasional creativity boost and Oblique Strategies is one of the methods I use.

A/B testing basics: ways of being right and wrong (frequentist version)

What are we trying to achieve?

In a typical A/B test, we're trying to find out if a change has a (positive) effect. For example, does changing the page layout increase the clickthrough rate? Despite what you've been told, we can't answer these types of questions with absolute certainty: the best we can do is provide a probable answer.  We use statistical best practices to map a probability to a pass/fail answer. 

In this blog post, I'm going to lay out some fundamentals to help you understand the process a statistician follows to translate a probabilistic result into a pass/fail result. 

A typical A/B test

To provide some focus for discussion, let's imagine we're testing to see if a discount on a website increases the rate of purchase. We'll have a control branch that doesn't have the discount and a treatment branch that has the discount. We'll measure conversion for both branches: \(c_T\) for the conversion for the treatment branch and \(c_C\) for the conversion for the control branch.

This kind of test is called a null hypothesis test. The null hypothesis here is that there is no difference, the alternate hypothesis is that there is a difference. We can write this as:
\[H_0: c_T  - c_C = 0\]
\[H_1: c_T - c_C \neq 0\]
There's something subtle here you need to know. The conversion rate we measure is an average conversion rate over many visitors, probably several thousand. Because of this, some very important mathematics kicks in, specifically something called the Central Limit Theorem. This theorem tells us our results will be normally distributed, in other words, \(c_T - c_C\) will be normally distributed, which is important as we'll see in a minute.

Types of error

I've blogged about null hypothesis tests before, so I'm only going to summarize things here. We can assume there's some underlying truth: either \(H_0\) or \(H_1\) is true. We don't know which is true and we're making an educated true/false guess. This gives us two ways of being right and two ways of being wrong. I've shown this in the table below.

		Null Hypothesis is
		True	False
Decision about null hypothesis	Fail to reject	True negative Correct inference Probability threshold= 1 - \( \alpha \)	False negative Type II error Probability threshold= \( \beta \)
	Reject	False positive Type I error Probability threshold = \( \alpha \)	True positive Correct inference Probability threshold = Power = 1 - \( \beta \)

We can't know for certain what the truth is, but we can define limits on our uncertainty. We can also define thresholds that will enable us to make reasonable pass/fail estimates. I'll show you how this works.

Assuming the null is true

The first step is to assume the null hypothesis is true, which means \( c_T  - c_C = 0\). As I explained earlier, the quantity \(c_T - c_C\) is normally distributed (this is a probability distribution, which I've blogged about before). We can compare our actual measurement of  \( c_T  - c_C\) to the theoretical distribution and ask how likely it is that the underlying value really is zero (in other words, what's the probability of the null being true?). 

Let me take a second to explain this some more. Imagine I'm trying to find out if a coin is biased. I throw it ten times and see six heads. Does this prove the coin is biased? No. It could be biased, but I don't have enough throws to say. Now imagine I've thrown the coin 100,000 times and I see 60,000 heads, does this prove bias? It's not absolutely sure, but it's highly likely the coin is biased. With statistics, we quantify this kind of analysis and set ground rules for what we consider evidence.

We can take our hypothetical A/B test and map the expected result to a standard normal distribution (very easy to do). Let's look at the standard normal distribution below, which plots a probability vs. a measurement value \(z\). Although it's true that all values are possible, the likelihood of some of them occurring is very low. For example, the probability of measuring a \(z\) value in the range \(-1  \leq z \leq 1\) is 0.68, but the probability of measuring a \(z\) value in the range \(1  \leq z \leq 3\) is only 0.16.

Certainty is impossible, but what we want to do is say whether a measurement means the null hypothesis is true or the alternate is. Put it another way, for a given measurement, how likely is it that the null is true or not? What's our threshold for acceptance/rejection? The standard procedure is to compare our measurement to the chart above. If our measurement falls in the blue zone on the chart we'll consider it means the null hypothesis is true. Anything that falls in the red zone, we'll consider the alternate is true. But we might be wrong - we can never have certainty. The size of the red area gives us the limits on our certainty. By convention, the red zones are 5% of the probability.

The standard limits we use are that we have to be in the 95% probability (blue) zone around zero to accept the null, and in the red 5% area to accept the alternate. This 5% threshold is usually called significance level and is given the symbol \(\alpha\). 

Using a threshold of 5% crudely speaking means we'll be wrong 5% of the time. Let's imagine a company running 100 tests in a year, this threshold means they'll be wrong in about 5 cases.

Surely this is enough? Surely we can now do this calculation and use \(\alpha\) to say pass/fail? No. We have assumed the null is true. But we also need to do the opposite and assume the alternate is true. 

Assuming the alternate is true

Now, we assume the alternate is true, that \( c_T - c_C \neq 0\). We can plot this out as a normal distribution too, but there's a difference. When we considered the null hypothesis to be true, we considered both sides of the normal curve, but here we only care about one side of the distribution. Remember, we're looking at the difference \( c_T - c_C \), so one side of the curve 'points' towards zero (no difference), and the other side points towards a bigger difference. We only care about the side that 'points' towards zero.

If there really is a difference, we expect a probability distribution like this below. We'll consider the alternate hypothesis to be true if our measurement lands in the blue zone, if it lands in the red zone, we'll reject the alternate. As before, the alternate could be true, and by chance, we could land in the red zone. The threshold value we'll use here is called \(\beta\). 

For reasons I won't go into, the threshold value is called the power of a test and is given by \(1-\beta\). Typical values of power range from 80% to 95%, but 80% is considered a minimum threshold. I'll have a lot more to say about power in another blog post.

Putting it together

Usually, the two charts I've shown you are shown looking like this. The sample sizes are chosen so that \(\alpha\) and \(\beta\) line up.

For our A/B test, here are the simplified steps in the process.

1. Note the number of samples in each branch, in this case, the number of samples is the number of website visitors.
2. Work out the conversion rate for the two branches and work out \( c_T - c_C \).
3. Work out the probability of observing \( c_T - c_C \) if the null is true. (This is a simplification, we work out a p-value, which is the probability of observing a measurement greater than or equal to the measurement we're seeing).
4. Compare the p-value to \(\alpha\). If \(p < \alpha\) then we reject the null hypothesis (we believe the treatment had an effect). If \(p > \alpha\) we accept the null hypothesis (we believe the treatment had no effect).
5. Work out the probability of observing \( c_T - c_C \) if the alternate is true. This is the observed power. The observed power should be greater than about 80%. An observed power lower than about 80% means the test is unreliable.

How to fail

When people new to statistics get involved in A/B testing, they sometimes make the mistake of focusing only on confidence (and p-values). This gives them insight into false positives, but it says nothing at all about false negatives. To put it bluntly, this incorrect process puts all the emphasis on the risk of doing something, but none at all on the risk of doing nothing. This kind of focus also leads to tests that are too short to be reliable.

Let me put this another way. Significance is about protecting you from buying something that doesn't work. Power is about protecting you from not buying something that works.

Why not just set the thresholds higher?

The widths of the normal distributions I've shown depend on the number of samples. The more samples there are, the narrower the curve. The thresholds depend on the narrowness of the curve. To put it simply, increasing confidence and power mean increasing the number of samples in the test, which means a longer test. So all we need to do is increase the length of the test? Not so fast, the relationship isn't a linear one. Increasing power or significance by a few percentage points could double the length of the test depending on what the power and significance levels are.

Where do these thresholds come from?

The choice of a confidence value of 95% is arbitrary and comes from statistical standard practice. There's a fierce ongoing debate in the social sciences about whether this threshold is appropriate; an emerging view is that it's too lax a standard. In a recent paper in Nature, Benjamin et al [Benjamin] argued passionately that 99.5% is a better threshold. 

Something similar applies to power. The 'industry standard' is 80%, a figure with a far murkier background [Cohen]. In my view, using this figure of 80% is wrong in almost all cases. 80% is a minimum. I'll have a lot more to say about power in another blog post.

Eye of newt and toe of frog...

I've talked glibly about accepting and rejecting hypothesis. This is a deliberate simplification on my part. The true statistical language is "fail to reject the null hypothesis" and "reject the null hypothesis". There are good fundamental reasons for using this language, but if you're not a statistical person, it's very confusing. I've chosen a simplified version to make my point.

The process for deciding an A/B test reads like a witches' brew recipe rather than a scientific process. It's reliant on arbitrary thresholds, some difficult concepts, and confusing language. The null hypothesis test itself is a shot-gun marriage of techniques. Unsurprisingly, p-values are widely misinterpreted and misunderstood [Amrhein]. 

Fundamentally, the whole process is a witches' brew; it works, but it's not satisfying. 

Fortunately, there is an alternative view using a Bayesian approach which is simpler, and more enlightening. I'll talk about the Bayesian approach in another blog post. If the Bayesian approach is more satisfying, why did I show this (frequentist) approach here? Because this approach is what people are taught.

References

[Amrhein] Valentin Amrhein, Sander Greenland, Blake McShane, Scientists rise up against statistical significance, Nature 567, 305-307 (2019)

[Benjamin] Benjamin, D.J., Berger, J.O., Johannesson, M. et al. Redefine statistical significance. Nat Hum Behav 2, 6–10 (2018). https://doi.org/10.1038/s41562-017-0189-z

[Cohen] Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillside, NJ: Lawrence Erlbaum Associates.

Goodies and baddies: how a poor model of history lets us down

The wrong model to understand history

Some of my history teachers taught me the wrong model to understand history. To be fair to them, they were simplifying complex events for children. But as an adult, I've seen journalists use the same simple model to stoke outrage and twist the meaning of historical discussions.

I'm going to tell you what the broken model of history is, show you how a simple local legend blasts it apart, and how seductive and damaging the model can be.

The heroes or villains model

There's a lot of English history, a good deal of it is bloody, complex, and hard to understand. To simplify subtle stories for schoolchildren, books and teachers often boil down stories to a few core elements, reducing historical figures to stereotypical heroes or villains. Sometimes, this works well, for example, Hitler and Mussolini fall neatly into the villain category, but in most cases, it doesn't work at all as we'll see. Even worse, the heroes or villains model lends itself to a kind of uncritical patriotism: "Britons good, foreigners bad". 

A good example of this simplification is Winston Churchill, who's often uncritically portrayed by the British press as a hero. Some writers consider any criticism of Churchill as unpatriotic and an attempt to portray him as a villain (hero or villain - no space for something else). My contention is, the hero or villain model is the wrong model to understand Churchill, or indeed any other historical figure.

Because the Churchill story is so charged, I'm going to use another historical example and try and apply the heroes or villains model to it. My story involves the English Civil War, regicide, death squads, and a local Massachusetts legend.

(The English Civil War - who were the heroes and who were the villains? Unknown author, Public domain, via Wikimedia Commons)

The backstory

The regicides of Charles I of England

By the late 1630s, Charles I and Parliament were at loggerheads over who governed the country with money, power, and religion as the key issues. The disagreement broke into armed conflict starting in 1642 and the country fought a long and bloodthirsty civil war that the Parliamentary forces eventually won in 1648. 

(Three views of Charles I, Royal Collection, Public domain, via Wikimedia Commons)

The victorious Parliamentarians put Charles I on trial for his life. Of course, the guilty verdict was obvious. In January 1649, 59 commissioners (judges) signed the death warrant for Charles I, who was executed soon after. These 59 commissioners became known as the regicides.

Oliver Cromwell

After Charles I's execution, the country became a republic, run by Oliver Cromwell who took the title "Lord Protector". Cromwell was a transformative figure in British history, his rule was effective but bloody. In particular, his Irish military campaign was brutal, even for the time, and involved multiple massacres. He violently suppressed Catholicism in both England and Ireland. Even to this day, Cromwell's name is cursed in Ireland for what he did.

(Oliver Cromwell, After Samuel Cooper, Public domain, via Wikimedia Commons)

The Restoration

Cromwell died of natural causes in 1658. Soon after, Charles' I son, Charles II, swept back into power and England's experiment with republicanism was at an end. Expediency meant that Charles II pardoned many Parliamentarians, but there was no forgiveness for the 59 commissioners who signed Charles I's death warrant. Understandably, Charles II wanted vengeance on those who'd killed his father. The fate that awaited the regicides was worse than torture and execution; they were to be hung, drawn, and quartered. Knowing this, many of them fled, some to Europe, but some fled to the new colonies in America.

(Charles II, Peter Lely, Public domain, via Wikimedia Commons)

One of the regicides, William Goffe, escaped to the New England colonies, starting off in Boston, then moving to Connecticut and then to central Massachusetts. Charles II sent secret agents to track all the regicides down, in effect, they had a license to kill. If Goffe was caught, he might be killed on the spot if he was lucky, if he was unlucky, he could expect unbearable torture before he was executed. To avoid Charles's agents, Goffe went into hiding in the town of Hadley, Massachusetts.

The angel of Hadley

In 1675, Hadley, Massachusetts was a border town. English settlers were displacing Native American tribes which led to an armed conflict called King Philip's war. On one side were the English settlers, and on the other were Native American tribes.

Legend has it that on September 1st, 1675, Hadley was attacked by Native American forces. The villagers responded to fight off the attack, but having no military experience, their defense was ineffective. It looked as if the village would be lost and destroyed.

Out of the confusion of battle, an old white-haired man appeared and took charge - none of the villagers had seen him before. He rallied and organized the villagers into an effective battle formation, and together, they managed to fend off the attack. The white-haired angel saved the village, but he vanished as soon as the battle was won. He became known as the 'angel of Hadley' for saving the town.

(The angel of Hadley, Frederick Chapman (1818-1891), Public domain, via Wikimedia Commons)

After the battle, Goffe went back into hiding with good reason. Charles' agents were still looking for him. He's rumored to have died in New Haven in 1680, though there are other accounts of him living and dying elsewhere in New England.

Goodies and baddies

Let's apply the heroes or villains model to the actors in this story and see how it holds up.

Goffe saved Hadley from destruction, therefore he's a hero. But only from the point of view of the white settlers in Hadley. If you were a Native American opponent, Goffe stopped your forces from retaking land that was rightfully yours, so he's a villain. If you were a Royalist, then Goffe was a villain because he signed Charles I's death warrant, but if you were a Parliamentarian, he was a hero.

What about Cromwell? To many in England, he's a hero for his strong leadership and military prowess, but to the Irish, he's a villain; a bloodthirsty tyrant who massacred the population and violently suppressed Catholicism.

The heroes or villains model doesn't work at all for this story. In fact, it doesn't work for almost all of history. In most cases, it's a reductio ad absurdum, suitable only for young children. Heroes or villains might be too serious a name - I should really call it the goodies and baddies model of history.

History and patriotism

All countries have their national myths and national heroes. The goodies or baddies model is often uncritically applied to historical figures, with parts of the press bolstering the goodies or baddies model, shouting down those who disagree and accusing them of a lack of patriotism.

Let's take another English example. On June 7th, 2020, protestors in the English city of Bristol pulled down the statue of Edward Colston (1636-1721) and threw it into the harbor. They were objecting to Colston's involvement in the slave trade where he made most of his money. Later in life, Colston became a philanthropist who donated large sums of money to support schools, hospitals, and almshouses, especially in the Bristol area. Because of his philanthropy, the people of Bristol erected a statue of him in 1895. 

(Edward Colston statue, Simon Cobb, CC0, via Wikimedia Commons)

Is Colston a goodie or a baddie? If you look at his involvement in the slave trade, he's definitely a baddie. If you look only at his philanthropy, he's a goodie. But as you can tell, I think the goodies or baddies model doesn't work at all. Edward Colston was both and neither - reducing him to wholly goodie or baddie is absurd.

Similarly, the model breaks down for historical greats like Churchill, who did both good things and questionable things. By unthinkingly applying a goodie and baddie model, we're preventing ourselves from reaching a deeper and richer understanding of historical people and events.

The goodies and baddies model has a use though. It generates outrage which in turn helps sell newspapers and increase ratings. It's a handy culture war tool to beat your opposition with. Most of the press coverage of the Colston statue saga focused on politicians condemning the protestors (goodies and baddies again). But what about discussing whether the statue should be there at all in the 21st century? Any criticism of Churchill is often met with a fierce response but was Churchill always and in every decision and action a goodie? Is anyone? Outrage displaces critical thinking, which may be the point.

A better model

Rather than label people goodies or baddies, it's better to ask what and why. What caused the English Civil War? Why did the victorious Parliamentarians execute Charles I? Why did Goffe believe what he believed? Why was the Colston statue still in place in 2021 and why was it erected in the first place when his role in slavery was known? 

These are deeper and less emotional topics, but if we want to truly understand history, we need to move away from a simplistic goodies and baddies model to an understanding of the times people lived in and the rich complexities of their actions. No one is wholly good or bad, and we shouldn't expect them to be.

Wicked problems in data science

Why some projects are harder than others

Over my career, I've had the experience of working on projects that have gone wonderfully well and I've worked on projects that just ran into the sand and went nowhere. I've come to recognize the red flashing warning signs for a certain type of project that's pathologically bad: they tend to be projects involving wicked problems or have the characteristics of wicked problems. Interestingly, I've come across more wicked problems in data science than elsewhere. 

(Wicked problems can be real devils to work on - they can damage your career if not handled correctly.  Elcom.stadler, CC BY-SA 4.0, via Wikimedia Commons)

Wicked problems

The term 'wicked problem' comes from the planning and policy world [Rittel and Webber] and refers to problems that are difficult or impossible to fix inside the current social, political, and economic system. A good example is solving poverty; there are many, many stakeholders, each with fiercely different views, and no clear measure of success (how is poverty measured, is the goal reduction or eliminations, etc.). Poverty is also linked to other factors too, like level of education, health, housing, etc. If you were a politician, do you think you could solve poverty?

(Properties of wicked problems. Image source: Wikimedia Commons, License: Creative Commons, Author: Christian Sarkar)

In the five decades since Rittel and Weber first discussed wicked problems, researchers have identified some of their key characteristics:

• Wicked problems are not fully understood until after the creation of a solution.
• Wicked problems have no stopping rule, there's nothing to tell you that you've reached an optimal solution.
• Solutions to wicked problems are not right or wrong: they are better or worse, or good-enough or not-good-enough.
• Every wicked problem is new: you can't apply prior learning to it.
• Wicked problems have no alternative solutions to choose from.

Rittel and Weber's seminal paper points out a key feature of these types of problems: they're not amenable to traditional project management using a phased approach (usually something like "gather data", "synthesize data", "create plan", "execute on plan", etc.).  This is crucial to understanding why projects solving wicked problems go wrong.

Wicked problems in software

If you think wicked problems sound a lot like some software development projects, you're not alone. In 1990, DeGrace and Stahl published "Wicked problems, righteous solutions" which laid out the comparison and compared the utility of different software development methodologies to solve wicked problems. To state the obvious, the killers for software project predictability are understanding the problem and applying prior learning. 

Readers who know agile software development methods are probably jumping up right now and saying 'that's why agile was developed!' and they're partly right. Agile is a huge improvement on the waterfall approach, but it's not a complete solution. Even with agile, wicked problems can be extremely hard to solve. I've had the experience of working on a project where we found a new critical requirement right towards the end, and no amount of agile would have changed that. 

Wicked problems in data science

Data science has its own wicked problems, which I'll put into two buckets.

The first is the ethical implications of technology. Facial recognition obviously has profound implications for society, but there are well-known issues of racial bias in other data science-based systems too (see for example, Obermeyer). Resolving these issues isn't only a data science problem, in fact, I would say it can't only be a data science problem. This makes these projects wicked in the original sense of the term. 

The other bucket is operational. Although some data science problems are well-defined, many are not. In several projects, I've had the experience of finding out something new and fundamental late in the project. To understand the problem, you have to solve it. For example, you may be tasked with reducing the RMSE for a model below some threshold, but as your model becomes more sophisticated, you might find irreducible randomness or as your understanding of the problem increases by solving it, you may find there are key missing features.

Here are some signposts for wicked problems in data science:

• Any algorithm involved in offering goods or services to the public. Racial, gender or other biases may become big issues and these risks are rarely outlined in the project documentation - in fact, they may only be discovered very, very late into the project. Even worse, there's often no resource allocation to manage them.
• No one in your organization has attempted to solve a problem like this before and none of the people on the project have prior experience working on similar projects.
• The underlying problem is not fully understood and/or not fully studied. 
• No clear numerical targets for project quality. Good targets might be thresholds for false error rates, RMSE, F1 scores, and so on.

What's to be done?

Outline the risks and manage them

It's always good practice to have requirements specifications and similar documents. These project documents should lay out project risks and steps to counter them. For example, facial recognition projects might include sections on bias or ethics and the steps necessary to counter them. Managing these risks takes effort, which includes effort spent on looking for risks and estimating their impact.

Expect the unexpected

If wicked problems can't be fully understood until they're solved, this is a huge project risk. If a new requirement is found late in the project, it can add substantial project time. Project plans should allow for finding something new late into the project, in fact, if we're solving a wicked problem, we should expect to find something new late in the project.

Set expectations

All of the stakeholders (technical and non-technical) should know the risks before the project begins and should know the consequences of finding something late in the project. Everyone needs to understand this is a wicked project with all the attendant risks.

Communications

Stakeholders need to know about new issues and project progress. They must understand the project risks.

Overall

If a lot of this sounds like good project management, that's because it is. Data science projects are often riskier than other projects and require more robust project management. A good understanding of the dynamics of wicked problems is a great start.