Showing posts with label corporate failures. Show all posts
Showing posts with label corporate failures. Show all posts

Monday, December 28, 2020

I won an award! How to lose by winning

Company work anniversary awards

Sometimes, companies try and do a good thing but go about it so poorly, they end up doing something bad. 

A few years ago, I worked for a large company. I got to a work anniversary which triggered an award; a plastic slab I was supposed to display on my desk. How it was delivered was eye-opening.

(Winning a trophy like this would be meaningful. Image source: Wikimedia Commons. License: Public Domain.)

I was working at a different office from my manager, so the award was sent directly to me, including the written instructions to my manager on how to give me the award

How to do it wrong

The award was a tombstone-shaped piece of transparent plastic with some vaguely encouraging words embossed on it. Other than the company logo, there was no customization of any kind (not even the employee's name), it was completely generic. The instructions gave a formal pattern for how the plastic was to be awarded. They went something like this:

  • Allocate about 20 minutes for the award ceremony.
  • Gather the employee's colleagues together.
  • Thank the employee by name for their service to the company. Mention any noticeable successes. Be warm and encouraging. Use their name. Look them in the eye.
  • Hand over the award, being sure to note that it's a recognition of their service. Use their name.
  • State that you're looking forward to working with them in the future.
  • Start a round of applause.

I told my manager that this had happened and we both laughed. I told him I was going to have an award ceremony for myself and hand myself the award using the instructions in the box. He chuckled and told me to go for it. In other words, the whole thing meant nothing to either of us.

Obviously, the company's intention was to thank employees for not leaving. They'd thought it through sufficiently well enough to have a trophy that would be displayed on desks and that wouldn't cost very much. Of course, the goal of the ceremony was to celebrate the individual and make them feel special.

Unfortunately, the trophy wasn't meaningful to anyone - it didn't even look good. The instructions left a bad taste in my mouth. My guess is, the leadership was trying to reach managers who wouldn't normally celebrate individuals' contributions. By mandating the form of the ceremony, they were trying to introduce consistency and enforce meaning, but by describing the ceremony in detail, they undermined managers - this was a form of micro-managing and hinted at bigger issues with managers' people skills.

How to do it right

By contrast, I worked for another large organization that made a very big deal of work anniversaries. People who reached a significant anniversary were called into a big meeting and personally thanked by the CEO. There were meaningful gifts for reaching multiples of 5 years. Looking back on that experience, I believe the company, and the CEO were sincere - they put a lot of effort into thanking and recognizing people. The fact that the recognition was led by the CEO made a huge difference.

Don't fake it

Employee recognition is a fraught topic and work anniversaries can be tricky. Do you celebrate or not and why? If you do celebrate, then it needs to be meaningful and focused on the person; you can't fake or mandate sincerity. If you're going to do it, do it well.

Monday, December 21, 2020

The $10 screwdriver: a cautionary management tale

Managers gone mild

I've told this story to friends several times. It's a simple story, but the lessons are complex and it touches on many different areas. See what you think.

I was a software developer for a large organization working on network-related software. For various reasons I won't go into, we had to frequently change network cards in our test computers and re-install drivers. My bosses' boss put a rule in place that we had to use IT Support to change cards and re-install drivers - we weren't to change the cards ourselves. No other team had a similar rule and there had been no incidents or injuries. Despite asking many times, he wouldn't explain why he put the rule in place.

At first, IT Support was OK with it. But as time wore on, we wanted to change cards twice a day or more. IT Support had a lot of demands on their time and got irritated with the constant requests. They wanted to know why we couldn't do it ourselves. One of the IT guys burned us a CD with the drivers on it and told us to get our own screwdrivers and change the cards ourselves. They started to de-prioritize our help requests because, quite rightly, they had other things to do and we could swap the cards ourselves. It got to the stage where we had to wait over two hours for someone to come, unscrew two screws, swap the card, and screw the two screws back in.

We were very sympathetic to IT Support, but the situation was becoming intolerable. My software development team complained to our management about the whole thing. My bosses' boss still wouldn't budge and insisted we call IT Support to change cards, he wouldn't explain why and he wouldn't escalate the de-prioritization of tickets. 

Excalibur the screwdriver

I got so fed up with the whole thing, I went out one lunchtime and bought a £7 ($10) screwdriver. It was a very nice screwdriver, it had multiple interchangeable heads, a ratchet action, and it was red. I gave it to the team. We used the screwdriver and stopped calling IT Support - much to their relief.

The blessed screwdriver

(This isn't the actual screwdriver I bought, but it looks a lot like it. Image source: Wikimedia Commons, Author: Klara Krieg, License: Creative Commons.)

The consequences

I then made a big mistake. I put in an expense claim for the screwdriver.

It went to my boss, who didn't have the authority to sign it off. It then went to his boss, who wasn't sure if he could sign it off. It then went to his boss, who did have the authority but wanted to know more. He called a meeting (my boss, my bosses' boss, my bosses' bosses' boss) to discuss my expenses claim. I heard they talked about whether it was necessary or not and whether I had bought a screwdriver that was too expensive when a cheaper one would have done. They decided to allow my expenses claim this one time.

I was called into a meeting with my bosses' bosses' boss and told not to put in a claim like that again. I was called into a meeting with my bosses' boss who told me not to put in an expense claim like that again and that I should have used IT Support every single time and if I were to do it again to buy a cheaper screwdriver. I was then called into a meeting with my boss who told me it was all ridiculous but next time I should just eat the cost. Despite asking, no one ever explained why there had been a 'rule'. Once the screwdriver existed, we were expected to use it and not call IT Support.

Of course, the team all knew what was going on and there was incredulity about the company's behavior. The team lost a lot of respect for our leadership. The screwdriver was considered a holy relic to be treasured and kept safe.

What happened next

Subsequent to these events, I left and got another job. In my new job, I ended up buying thousands of pounds worth of equipment with no one blinking an eye (my new boss told me not to bother him with pre-approval for anything under £1,000). 

All the other technical people in my old group left not long after me. 

A competitor had been making headway in the market while I was there and really started to break through by the time I left. To respond to the competitive challenge, new leadership came in to make the company more dynamic and they replaced my entire management chain.

What I learned

Here's what I learned from all this. I should have eaten the cost of the screwdriver and avoided a conflict with my management chain, at the same time, I should have been looking for another job. The issue was a mismatch of goals: I wanted to build good things quickly but my management team didn't want to rock the boat. Ultimately, you can't bridge a gap this big. Buying the screwdriver was a subversion of the system and not a good thing to do unless there was a payoff, which there wasn't. 

I promised myself I would never behave like the management I experienced, and I never have. With my teams now, I'm careful to explain the why behind rules; it feels more respectful and brings people on side more. I listen to people and I've reversed course if they can make a good case. I've told people to be wise about expenses, to minimize what they spend, but when something needs to be bought, they need to buy it.

What do you think?

If you liked this post, you might also like these ones...

Monday, August 24, 2020

Everything stops for tea

I was told this story years ago by those who were supposedly involved. I worked for the company concerned, but I'm not sure if it makes the story truer or not. In any case, it's a fun story with a subtle moral.

There was an IT department in a big company who were installing servers in an older building that didn't have dedicated server rooms or closets. Because there was nowhere else to put it, they installed a server in an office. The server was a typical innocuous beige box.

After a few weeks, there were reports of trouble in the building. Fairly regularly, e-mail and other services would go down for five minutes at about the same time of day. The IT department investigated. They checked the server configurations, but the configurations weren't to blame. They checked the cabling, but that was just fine. They checked network cards and routers, but everything seemed to be working as expected. During the whole investigation, the network kept on going down at around 10:30am for about 5 minutes, but there seemed to be no hardware or software cause.

In desperation, the IT department posted someone to sit by the server all day to watch what happened.

At about 10:30am, a secretary filled an electric kettle with water. She walked into the office, unplugged the server, and plugged in her kettle. She made herself and her boss a nice pot of tea. When the tea was brewing, she unplugged the kettle and plugged the server back in. She then went to enjoy her break and have her cup of tea.

(Image source: Wikimedia Commons Artist: Ian Smith License: Creative Commons)

So the mystery was solved. The IT department put a notice on the server plug not to unplug it and identified the server as a server. The secretary found another, less convenient place to plug in her kettle, and the world moved on.

I was told this story by the IT department. In their telling, the villain of the piece was the secretary, who they thought should have known better. At the time, I accepted this and laughed with them. Now, I disagree. In my view, the villain was the IT department and the innocent party was the secretary.

No one told the secretary that the server was important; it wasn't marked in any way. She wasn't a technical person and she had no way of knowing what the box was or what it did. Because of the age of the building, the server was in an office instead of in a server closet, so there were lots of non-technical people in the area near the server. The IT department did know what the server was, and they knew that there were non-technical people around, but they chose not to mark the server or communicate to anyone what it was or how important it was to keep it plugged in.

Bottom line: don't blame people for not being psychic - it's your responsibility to communicate.

Friday, June 26, 2020

Drunk and disorderly: not funny at company events

Loutishness isn't funny

Maybe I'm getting more puritanical as I get older, but I'm finding myself less and less tolerant of drunken loutishness at corporate events. I've seen too many instances where people have got away with poor behavior to the detriment of others and to the detriment of the organization as a whole. Let me tell you my stories; I've disguised some of the details to protect the guilty.


(Great when enjoyed responsibly, not so great when it leads to bad behavior on company time. Image credit: Wikimedia Commons (klimkin). License: Creative Commons Zero)

Ruining the lawn

One fine summer evening, the company I worked for held a large-scale open-air event. The aim was to play summer outdoor games and enjoy a drink or two in the company of our colleagues. The venue was extremely proud of its grounds and had obviously spent a lot of money creating and maintaining immaculate grass playing surfaces. Some of my colleagues realized there was no limit at the bar and set out to get drunk on company money; a task they succeeded at admirably. One of them decided it would be really funny to plant his glass in the middle of the grass. He turned his glass upside down and stomped it into the lawn. His friends thought this was funny, so they did it too. The next day, the groundskeepers discovered the stomped glasses and had to fix the damage, which meant the venue was unavailable for others. Net result? The company paid a penalty to repair the lawn and was banned from the venue (the only venue like it for miles). Everyone knew who the stompers were, but they got away with it.

The Christmas party

Christmas time seems to bring out the worst in some people at corporate events. I was at a large-scale company Christmas lunch where a lot of alcohol had been consumed. One of the employees decided it would be fun to start throwing food at other tables. The food fight escalated until someone soaked a small tablecloth in water, screw it up into a ball, and threw it at another table. It hit the other table like a bomb, exploding a carafe of red wine, splattering everyone in red wine and glass fragments. This was plainly very upsetting for the people on the table and brought the event to an end. Obviously, the company was banned from the venue, but this time I believe someone did have a word with one of the perpetrators and there was managerial discussion of curtailing bad behavior.

The overseas trip

Corporate events held away from home can be a lot of fun, but there can be problems too. I was at a corporate event held in a resort venue and I witnessed some unfortunate things. There was a mix of company people, who were getting everything for free, and vacationers, who had paid to stay at the resort. It was not a happy combination. The company people were away from family, off the leash, and with unlimited free alcohol. On the whole, their behavior towards paying guests was disrespectful at best. At one of the resort bars, I got chatting to a vacationing couple who complained to me about poor treatment from my work colleagues. The vacationers were very unimpressed by the rude and drunken behavior they saw. This was not a good company image to project.

Company culture and drunkenness

I'm happy to say, I haven't seen drunken behavior at my current employer and I think I know why. The company has taken great pains with recruitment and boorish behavior just isn't part of company culture. As a manager, I would be on top of drunken misbehavior immediately, and I believe my fellow managers would do the same too.

Advice to managers: stop it dead

Along with most managers, I've read my fair share of legal documents, case studies, and guidance. Pulling all this together, here are my thoughts:

  • You are never off-the-clock if you're at a company event or with company people. Even if a few work colleagues go out for a drink, it's still a company event.
  • Loutish behavior is not tolerable and there have to be consequences. Just as the workplace must be safe, it must be safe to be with work colleagues. It must be safe for non-employees to be around employees too. Managers need to jump on instances of bad behavior immediately.
  • Everyone must know what the rules are and there must be reminders. Managers must make it clear to new employees what's acceptable and what isn't. It's only fair that everyone knows what's expected of them.

I'm not against people having a drink at company events, I'm not even against people getting drunk at events, but I am against harmful behavior toward others. I want everyone to feel safe and to be safe.

Saturday, May 30, 2020

Inventory: your job may depend on how it's managed

Why should you care about inventory?

For your own job security, you need to understand the financial position of your employer; their true financial position will govern your pay and promotion prospects. Long before they affect your job, you can spot looming signs of trouble in company financial statements. Learning a little of accounting will also help you understand news stories as we'll see. In this blog post, I'm going to talk about one of the easiest signs of trouble to spot, inventory problems. Because it's always fun, I'm going to include some cases of fraud. Bear in mind, innocent people lose their jobs because of inventory issues; I hope you won't be one of them. 


Inventory
(Is inventory good or bad? It depends. Image credit: Wikimedia Commons. License: public domain.)

Inventory concepts

Inventories are items held for sale or items that will be used to manufacture products. Good examples are the items retailers hold for sale (e.g. clothes, food, books) and the parts manufacturers hold (e.g. parts for use on a car assembly line). On a balance sheet, inventory is listed as a current asset, which means it's something that can be turned into cash 'quickly'. There are different types of accounting inventory, but I won't go into what they are.

Inventory changes can be benign but can be a sign of trouble. Let's imagine a bookseller whose inventory is increasing. Is this good or bad?

  • If the bookseller is expanding (more sales, more shops), then increasing inventory is a sign of success.
  • If the bookseller is not expanding, then increasing inventory is deeply concerning. The bookseller is buying books it can't sell.

There are two ways of valuing inventory, which opens the door to shenanigans. Let's imagine you're a coal-burning power station and you have a stockpile of coal. The price of coal fluctuates. Do you value your stockpile of coal at current market prices or the price that you paid for it? There are two ways of evaluating inventory: FIFO and LIFO.

  • FIFO is first-in, first-out - the first items purchased are the first items sold. Inventory is valued at current market prices.
  • LIFO is last-in, first-out - the last items purchased are the first items sold. Inventory is valued at historic market prices.

If prices are going up, then LIFO increases the cost of goods sold and reduces profitability, conversely, FIFO reduces the cost of goods sold and increases profitability. There are also tax implications for the different inventory evaluation methods. 

Obviously, things are more complex than I've described here but we have enough of the basic ideas, so let's get to the fraud stories.

Inventory shenanigans

OM Group produced specialty chemicals from raw materials, including cobalt. In the early 2000s, cobalt was mostly sourced as a by-product from mines in the Democratic Republic of the Congo, a very unstable part of the world. The price of cobalt was going down and OM Group saw a way of making that work to their advantage. Their first step was to use the LIFO method of valuing their cobalt inventory. The next step was to buy up cheap cobalt and keep buying as the price dropped. Here's what that meant; because they used LIFO, for accounting purposes, the cobalt they used in production was valued at the new (low) market price, so the cost of goods sold went down, so profitability went up! The older (and more expensive) cobalt was kept in inventory. To keep the business profits increasing, they needed the price of cobalt to go down and they needed to buy more of it, regardless of their manufacturing needs. The minute prices went up, or they started eating into inventory, or they stopped buying more cobalt, profitability would fall. To put it simply, the boost to profits was an accounting shell game.

OM Group logo at the time. Image credit: Wikimedia Commons. License: Public Domain.)

As you might expect, the music eventually stopped. The SEC charged some of the executives with fraud and reached a settlement with the company, and there was a class-action lawsuit from some company investors. Unsurprisingly, the company later changed its name when the dust settled. If you want to understand how you could spot something like this, there's a very readable description of the accounting fraud by Gary Mishuris, an analyst who spotted it.  

Manufacturing plants run best when producing at a constant rate, but market demands fluctuate. If demand reduces, then inventory will increase, something that will be obvious in a company's financial statements. How can a company disguise a drop in demand? One illegal way is through something called 'channel stuffing', which is forcing your distributors and resellers to take your unsold inventory so you can record it as sales. 

Semiconductor companies are manufacturers and typically have large distribution networks through resellers covering different geographies or markets. For example, a semiconductor company may sell directly to large electronics companies but may serve smaller electronics companies through resellers, who may, in turn, resell to other distributors and so on.

Between 1995 and 2006, Vitesse Semiconductor used channel stuffing extensively to manage its earnings. It had an arrangement with its distributors that they could unconditionally sell back any chips they had bought and not sold. Here's how channel stuffing worked; if Vitesse needed to increase profits in a quarter, they would require their distributors to buy Vitesse' inventory. This would show up as an increase in Vitesse's sales for that quarter. At some point in the future, the resellers could sell the chips back to Vitesse. The chips themselves might never leave the warehouse, but might have been 'owned' by several different companies. In other words, short-term increases in profitability were driven by an accounting scam.

(Vitesse Semiconductor chip. Image credit: Raimond Spekking via Wikimedia Commons. License: Creative Commons)

Of course, this is all illegal and the SEC took action against the company; the executives were indicted for fraud. Vitesse had to restate their earnings substantially downwards, which in turn triggered a class action lawsuit. This fraud has even made it into fraud textbooks.

I want to stop for a minute and ask you to think. These are entertaining stories, but what if you were an (innocent) employee of OM Group or Vitesse Semiconductor? When the SEC arrests the leadership, what are the implications for employees? When the accounts are restated and profitability takes a nose dive, what do you think the pay and job prospects are like for the rank-and-file workers?

Inventory and politics - Brexit

A while back, I was chatting to a Brexit supporter, when a news report came on the TV; UK economic output had increased, but the increase had gone into inventory, not sales. Manufacturers and others were assuming Brexit would disrupt their supply chain, so they'd increased output to give them a buffer. I was horrified, but the Brexit supporter thought this was great news. After chatting some more, I realized they had no understanding of how inventory worked. Let's talk through some scenarios to understand why the news was bad.

  • Scenario 1: no Brexit supply chain disruption.  UK firms have an excess of inventory. They can either keep the inventory indefinitely (and pay higher costs than their overseas competitors) or they can run down inventory which means fewer hours for their workers.
  • Scenario 2: Brexit supply chain disruption.  UK firms can't get parts, so they run down inventory until supply chain issues are fixed. Selling inventory means fewer hours worked by the workers.

In both scenarios, UK firms have incurred production costs earlier than their overseas competitors, which reduces their cash flow.


(Image credit: Wikimedia Commons - Tim Reckmann. License: Creative Commons.)

This is obviously highly simplified, but you get the point. None of these scenarios are good for firms or for workers.

If increasing inventory without sales is so good, why don't firms do it all the time? In fact, why bother with the pesky business of selling at all when you can just produce for inventory? The question seems silly, but answering it leads you to consider what an optimum level of inventory is. The logic pushes you towards just in time, which leads to an understanding of why supply chain interruptions are bad.

Closing thoughts

Your job security depends on the financial stability of your employer. If you work for a company that produces public accounts, you have an opportunity to make your own risk assessment. Inventory is one factor among many you should watch. Here are some things you should look out for:

  • Changes to inventory evaluation methods (LIFO, FIFO).
  • Increases in inventory not matched to growth.
  • Increasing sales to distributors not matched to underlying market demand (especially when the inventory never leaves the company).

Yes, some companies do produce fraudulent accounts, and yes, some do hide poor performance, but you can still take steps to protect your career based on the cold hard reality of financial statements, not on hype.

Saturday, May 16, 2020

The Emperor's new objects: a two-year failed project

It’s a sad thing when people you admire turn out not to be what you thought they were andwhen you find out your heroes have feet of clay. I had an experience like that a few years ago and it taught me a lot.

(Image credit: Old Book Illustrations)

I worked for a company that decided to go for object-oriented technology in a big way. They hired a team of people to design and build components that the lesser people (e.g. me and my colleagues) would snap together to build systems more quickly. Let’s call this team Team X and the team leader Mr. Y. Team X was to be a team of superstars leading the company forward.

I admired Team X and Mr. Y greatly. They had all the skills I wanted and every time I walked past them, they were diligently designing systems; I was impressed by their energy. I used to chat to them once a week and they used to like it because I was so clearly in awe of what they were doing. Mr. Y was at great pains to show me their work, though it was plain I didn’t understand it.

Some time passed and I had the opportunity to learn object-oriented programming. I did a design course on UML, and I learned C++ and became familiar with all the key tools. I put a huge amount of effort in and went from knowing nothing to knowing a lot in a short space of time.

It had been a while since I dropped in on Team X and Mr. Y, so I said hello. As before, Mr. Y showed me their designs and work. This time, I knew about object-oriented design. Unfortunately, I immediately spotted several mistakes in their design and several places where their approach would have been very inefficient. I was foolish enough to point them out. At first, Mr. Y conceded I was right, but as I pointed out more problems, he got irritated. Eventually, I realized I wasn’t welcome anymore and left. I didn’t stop by and chat again.

After this, Team X locked down their designs and prevented anyone from viewing them, they even angled their screens away from people passing by so no-one could see their work. They stopped communicating. In the meantime, after chatting with my boss, I realized Team X didn’t know what they were doing. My boss counseled me to keep quiet and say nothing, which I did after some moaning. He told me my chances of getting real change were zero and that I would just damage my reputation by speaking out.

About six months later, Team X was all laid off. The company gossip was, they’d been employed for two years and produced absolutely nothing of value, not a single piece of executable code. The company got nothing for several people’s efforts over the entire time.

Ironically, the only group that actually produced an object-oriented system was the group I was in. We used C++ to create useful systems that actually did something, and we produced the systems in a few months with just a handful of people, none of whom claimed to be a superstar. We didn't do the huge analysis Team X did, we just built systems to meet our customers' needs.

What were my takeaways from all of this?

  • Results are what counts.
  • If you have a team working for you using a technology, make sure you understand it well enough to measure their progress.
  • Deliverables are important and you should manage projects to have some results at checkpoints throughout the project. This is the soundest way of measuring progress. Never, ever leave long gaps between deliverables.
  • A team that hides its work is a major red flag.
  • Choose your battles and only choose ones you can win or that are important. Companies do silly things all the time, but eventually, they get corrected, whether you intervene or not.

Saturday, May 2, 2020

It's a mugs' game: corporate failures, ceramics, and t-shirts

Mike's law of business

One day, I'm going to write some laws of business. One of them's going to be: "any corporate initiative where mugs or other swag is given to employees is doomed to failure." Let me tell you some sorry tales that led me to my law.

A failed campaign

I worked for a company where employee engagement was becoming an issue. At the time, the management fad was harnessing employee ideas to move companies forward. The company held an all-hands meeting where we were told about a new corporate initiative; we (the staff) were all going to have ideas about how the company could make more money. To show how serious the executive team was, they gave us all mugs. The mugs said "I'm Making A Difference" (MAD), had a pterodactyl on them, and the company name at the bottom; though what the MAD logo and the pterodactyl had to do with anything was never explained.  In the few sessions we ever had to discuss ideas, the staff focused on better management, which the executive team didn't like very much.  The mugs were the only part of the scheme that lasted.

(Enron mug available from Amazon. I never worked for Enron!)

Let's kill the competition!

At another company, there was severe competitive pressure, so the company created a development team whose mission was to create a competition killer. Close to the killer's release date, the people on the team were given mugs (which were actually really cool) and posters. Unfortunately, the killer failed miserably in the market. Much later, I found out that there were serious problems with the killer project prior to release, which led to poor team morale. The mugs and posters were a failed attempt to turn things around.

(Topologically speaking, mugs are the same as donuts. Image credit: Lucas Vieira License:Public Domain. Image source.)

Let's boost morale

As I've written elsewhere, I was at a company where a quality project imploded massively. When it became obvious the initiative was failing, the leadership floated ideas to revive the project, which, as you might have guessed, included giving staff mugs with the quality standard's name on them.

By this stage in my career, I was starting to see the link; only failed projects and initiatives hand out mugs.

(A mug from 3,700 years ago - maybe the ancient Greeks had failed projects too. Image Credit: British Museum. License: Creative Commons.)

Fleeces are the new mugs

But, as it turned out, mugs=failure wasn't entirely true. I've seen an exciting trend over the last few years. Instead of giving out mugs, companies have started to give out t-shirts or occasionally, fleeces or backpacks. In some cases, I've seen t-shirts with project logos, but more often than not, they're just corporate t-shirts. When it comes time for a quick morale boost, nowadays it's generic corporate swag all the way.  Maybe I should change my law to "giving out corporate swag might indicate impending failure".

(Not a corporate fleece. Photographer: Mike Nass License: Creative Commons Source: Wikimedia)

I'm guessing that when the COVID-19 pandemic abates, there will be a large increase in companies giving their staff mugs, or t-shirts, or fleeces, or backpacks. This is swag as heroin for morale.

The wrong cure

I know I'm being unfair to mugs and swag in general here, they're a symptom of diseases, but not the cause. The diseases are mismanagement, poor communication, and failed projects. Mugs and corporate swag are often an attempt at boosting morale when things are going too badly wrong to ignore. Unfortunately, swag isn't enough, and sometimes, it's the wrong thing to do - which is why I've come to associate mugs with failure.

Instead of marking a new beginning, mugs often mark the end; they become tombstones, not birthstones.

Saturday, April 25, 2020

The worst technical debt ever

Over the last few years, I've heard engineering teams rightly talk about technical debt and its consequences. Even non-technical executives are starting to understand its importance and the need to invest to avoid it. The other day as I was setting up a computer, I was reminded of the worst case I've ever seen of technical debt. I thought the story was worth telling here, but with a few details obscured to protect the guilty.

A few years ago, I visited one of company X's data centers. The data center was located in an older building in a slightly run-down part of town. The data center was a little hard to find because it wasn't marked in any way - there was nothing at all that made the building stand out. Outside the building, there was some trash on the sidewalk, including remnants of last night's take-outs that people had dropped on the street as they partied.

Once inside, things were different. Security at the entrance was shabby, but efficient and effective and we got through quickly. The interior was clean, but it was obvious the building hadn't been decorated in several years. Even the coffee machines had seen better days, but they worked.

We were given a detailed tour of the data center and built a good relationship with our guide. The data center had been one of the company's first and had been on the same site for several years. As you might expect, there were racks and racks of computers with technicians walking around fixing things and installing cables to connect new computers to the network. The air conditioning was loud and strong, which meant you had to be close to one another to talk - which also meant it was impossible to overhear conversations.

Late in the tour, I tripped on a loose floor tile that was a centimeter or two raised above the floor. Our guide apologized and told us we needed to be careful as we walked along. We asked why. This is where we discovered the technical debt.

Connecting computers in a data center means installing a physical cable from one computer (or router etc.) to another. You can either route the cable under the floor or on overhead trackways. Most data centers use some form of color-coded cables so you have some indication of what kind of data a cable's carrying (red cables mean one sort of data, blue another, yellow another, and so on). Some even go further and give unique labels or identifiers to cables, so you can identify a cable's pathway from end to end. Routing cables is something of an art form, and in fact, there's a sub-Reddit devoted to it: https://www.reddit.com/r/cableporn/ - from time to time I look at the pictures when I need an ordered view of the world. As you might expect, there's a sub-Reddit that focuses on the reverse: https://www.reddit.com/r/cablegore/.

Our guide told us that right from the start, the management at the data center wanted to save money and do things quickly. From time to time, routers and servers were moved or removed. Instead of removing the old cable, they just left it under the false floor and added the new cable on top of it. New cable was laid on top of old cable in any order or in any fashion, so long as the job was done cheaply and quickly, it was fine. Over time, the layers of cabling built up and up, like the strata in the rock you see at the Grand Canyon. You could even see when the company changed its cable supplier because the cable shade changed a little. Unfortunately, they always chose the same color cable (which happened to be the cheapest).

After a few years, management realized that leaving the old cable in place was a bad idea, so they instructed staff to try and remove the old cables. Unfortunately, there was so much cabling present, and it had been laid so haphazardly, it was physically impossible because the cables were so intertwined. In a few cases, they'd tried to pull up old cables by physical force, but this caused the insulation to be stripped off cables and connections failed. Obviously, leaving old cable connections just hanging around is a bad idea, so the management team told the technicians to cut off the ends of old cables as far along as they could. This meant that the old dead cable was left in place under the floor, but it all looked fine on the surface. Because the cabling ran under the floor, a superficial inspection would show that everything was working fine, especially because they'd cut the old cables as far back as they could.

Sweeping things under the rug went on for a while longer, but there was only so much false floor. By the time of my tour, there was no more space, in fact, the situation was so bad, the floor tiles wouldn't sit properly in their supports anymore. That's why we were tripping over tiles. When no one was looking, our tour guide removed one of the floor tiles to show us the cabling underneath. I was horrified by what I saw.

(Not the actual cables - but gives you a flavor of what I saw. Image source: https://commons.wikimedia.org/wiki/File:Pougny,_electric_cables_(4).jpg. License: Creative Commons. Photographer: Jean-Pierre)

Cables were packed together with no room at all between them. They had obviously been laid across each other with no organization. It was as if a demented person had been knitting with cables leaving no gaps. There was no give in the cables and it was plain it was more or less a solid mass down to the real floor.  By my estimate, the cabling went to a depth of 30cm or more. I could clearly see why it was impossible to pull out old cables: cables had no markings, so you couldn't tell them apart; they were so intertwined you couldn't unpick them, and there were so many cables, they were too heavy to lift. In fact, there was no room under the floor to do any kind of maintenance.

There were some gaps in the cables though. Our guide told us that the data center was starting to have a vermin problem. Of course, there was a ready supply of food outside, and rats and mice had found sufficiently large gaps in the cabling to set up home.

I asked what happened when they needed to connect up computers now there wasn't any room under the floor to lay anything. Our guide showed us some men working round the corner. They had stepladders and were installing overhead cable ducting. This time, the cables were properly color-coded and properly installed. It was a thing of beauty to see the ordered way they were working and how they'd laid out the cables. The cables were also individually labeled, making the removal of old cables much easier.

The next obvious question was, what about the old cable under the floor? The plan seemed to be to sweep everything under the rug. Create new overhead connections until all of the old connections were unnecessary and then leave the old cables and forget about it.

To his credit, our guide seemed ashamed of the whole thing. He seemed like a decent man who had been forced into doing bad things by poor management decisions. Notably, we never saw senior management on our tour.

A while later, I heard the data center was temporarily closed for improvements. These improvements went on for many months and I never heard exactly what they were. I suspect the executive team was embarrassed by the whole thing once they found out the extent of the problem and ordered a proper cleanup. At the time of my tour, I wondered about the fire risk, and obviously having a vermin problem is never a good thing for any business, so maybe something bad happened that made the problem impossible to ignore.

I heard a rumor sometime later that the data center had passed an external quality inspection and received some form of quality certification. I can see how this might have happened; their new processes actually seemed decent, and if they could make the floor tiles sit flat, they could hide the horror under the floor. Most quality inspections focus on paperwork trails and the inspectors I've met didn't seem like the kind of people who would want to get their hands dirty by lifting floor tiles.

So what did I learn from all of this?

  • Technical debt is real. You eventually have to pay for short-term time and money-saving decisions. There's never a good time to pay and the longer you leave it, the bigger and more expensive the fix becomes.
  • Just because something's been done a certain way for a long time, doesn't mean it's good. It might just mean the problems haven't surfaced yet.
  • If you're inspecting something, always get your hands dirty and always talk to the people doing the work. Things may look good on the outside, but might be rotten underneath. If we hadn't established a good rapport with our guide and I hadn't tripped on the floor tile, we would never have discovered the cable issue.
  • If something looks bad, look carefully for the cause. It would have been easy to blame the technicians for the cable nightmare, but it wasn't their fault. They were responding to the demands placed on them by their management. Ultimately, management is the cause of most failures.

Wednesday, April 1, 2020

A sorry tale of software quality: what went wrong and some lessons

I’m going to tell you a story about a software quality drive that went horribly wrong and what we can learn from it. To protect the guilty, I’ve disguised some details, but the main points are all true.

(The Antares launch failure. Image credit: Nasa. Source: Wikimedia Commons. Public domain image.)

I worked for an organization that got the quality bug. They decided that the cause of the software failures the company had experienced was poor quality processes, so in response, we were going to become a world leader in software quality. To do this, we (the development team) were all going to be certified to software quality standard ABC (not the standard’s name). The project was going to be completed in a year and we were going to see astounding benefits.

The company hired a number of highly paid contractors to advise us, most of whom were well-qualified. Unfortunately, many of the contractors showed two major personality traits: arrogance and condescension. Instead of creating a cooperative approach, they went for a command and control style that was disastrous. I heard there were multiple complaints about how they treated my colleagues.

Not all the contractors were well qualified. In one case, the person hired to create and manage software processes had never worked in software before and had never written any software. As you might expect, they ended up proposing weird metrics to measure quality, for example, counting the number of version control updates as a metric of quality (presumably, more updates meaning better software? It was never defined).

Everyone was trained multiple times, no expense was spared on training, and the sky was the limit for spending time on processes. We went to very, very long training sessions about once a month. These often included long descriptions of the benefits of the process and the fantastic results we might see at the end of it. In one notable case, the presenter (an external contractor) was caught out showing 'expected' results as real results - but our management team was true believers so the presenter got away with it.

As we started to put together processes, we were encouraged to meet once a week to discuss quality, which everyone did. We faithfully started to implement the processes proposed by the consultants and as customized by our teams. Most of these processes were a bit silly, didn’t really add any quality, and took time, but we implemented them anyway because that's what the leadership team wanted. The weekly meetings started to get longer and ended up taking a whole morning.

A few brave people suggested that we should have metrics that measured project deliverables, deadlines, and outages, but the ABC quality standard people wanted us all to focus on measuring the process, not the outcome, so that’s what we did.

Despite all these efforts, the failures kept on coming. The one thing that did change noticeably was that deliverables were now taking twice as long. Executive leadership was unhappy.

As the quality project started to lose support and fail, the leadership tried to change their approach from stick to carrot. One of the things they did was hold a contest for people to suggest 'fun' alternative words for the project acronym, if the project was ABC, an alternative meaning might be 'A Big Cheese'. Unfortunately, a lot of the entries were obscene or negative about the project.

Eventually, the inevitable happened. Executive leadership saw the project was going nowhere, they did a management reshuffle and effectively killed the quality drive. The contractors were shown the door. Everything went back to normal, except the promoters of ABC were no longer in senior positions. The standard ABC became a dirty word and people stopped talking about quality improvements; sadly, the whole ABC debacle meant that discussions of more sensible process improvements were taboo.

The core problem was, the process became about getting certified in ABC and not about reducing defects. We had the wrong goal.

I’ve had some time to think about how this all played out and what I would have done to make it a success.

  • Start small-scale and learn. I would have started with one team and measured the results very carefully. People talk to each other, so success with one team would have been communicated virally. To oversimplify: If it can’t work with one team, it can’t work with many.
  • Scale-up slowly.
  • Bring in contractors, but on a one-off basis and clearly as advisors. Any sign of arrogance and they would be removed. Contractors should be available on a regular basis to ensure change is permanent.
  • Focus on end results metrics, not processes. If the goal was to reduce defects, then that’s the metric that should have been measured and made public. 
  • Flexibility to change processes in response to increased knowledge.
  • No certification until the target is achieved - if certification is an option, then inevitably certification becomes the goal. Certification is a reasonable goal, but only once the targets have been reached.

I’m not against quality standards, I think they have an important role. But I do think they need to be implemented with great care and a focus on people. If you don’t do it right, you’ll get what you asked for, but not what you need.

Saturday, March 14, 2020

Niche knowledge and power - knowledge hoarding

A couple of times in my career, I’ve come across people using a strategy to gain short-term power: keeping knowledge and skills to themselves, otherwise known as knowledge hoarding. Unfortunately for them, it doesn’t work in the long term anymore. I'm going to start with some examples, then suggest how you might differentiate between an area that's genuinely hard and when someone's knowledge hoarding, before finally giving you some suggestions on what to do if you find it on your team.

(Keeping knowledge to yourself is like caging kittens. Image credit: Chameleon,  source: Wikimedia Commons, License: GNU Free Documentation)

I worked with someone who had developed some in-depth knowledge of a particular technology. I needed his help with a project and I needed his in-depth knowledge. He wouldn’t share what he knew, claiming that the technology was highly complex and difficult to understand. He insisted that he had to do the work if it was done at all, and that I needed to tell his manager how valuable he was. I later heard from others in the organization that he’d taken the same approach and that they’d caved in to him. Some managers started to believe that the technology really was as complex as he said. Fortunately, I knew enough to get started without him. After some diligent Internet searching, I found what I needed and completed my project without assistance. Unfortunately for my colleague, not too long after this, there were a couple of books published on the technology, which turned out to be much more straightforward than he claimed. His unique knowledge disappeared and his boast of enhanced value to the company evaporated within a few months. His career subsequently stalled; he was relying on his ring-fenced knowledge to give him an advantage and his prior behavior came back to haunt him.

Much later in my career, when I was older and wiser, I came across something similar. Someone two years out of college was working on a commercial tool we’ll call X. X had a sort-of programming language that enabled customization. The recent graduate claimed that only he could understand the language and only he could make the changes the company needed. This time, I didn’t even bother pursuing it. I had my team completely bypass him using a different technology. Had the recent graduate been more open, I would have gladly included them on my project and they would have been cross-trained in other technologies, instead, they ended up leaving to go to a company that used X. Problem is, X has a very small market share (< 5%) and it’s shrinking.

Over the years, I’ve seen the same story play out a number of times. Someone has knowledge of a system (e.g. Salesforce, CRMs, cloud systems, BI, sales analytics, network cards) and claims the area is too complex or difficult for others to understand and that they need to be the point person. But it always turns out not to be true. The situation resolves itself by the person leaving, or a reorganization, or a technology change or something else - it always turns out that the person was never as vital as they claimed. Ring-fencing knowledge to protect your position seems to work in the short term, but it fails spectacularly in the long run.

Of course, there are skills that are difficult to acquire and do provide a barrier to entry into an area. Good examples are statistical analysis, machine learning, and real-time system design. What’s noticeable about all of these areas is the large amount of training content freely and easily available. If you want to learn statistics, there are hundreds of online courses and books you can use. The only impediment is your ability to understand and apply theory and practice.

As a manager, how do you know if someone is ring-fencing knowledge to protect their position versus the area actually being hard? Here are the signs they might be ring-fencing:

  • Claims that only they can understand the technology.
  • Knowledge hoarding and refusal/reluctance to share. 
  • Refusing/reluctance to brief or train others - or doing it very badly.
  • No formal qualification in the area (not all areas have formal qualifications though) and no formal background (e.g. degree in software engineering). 
  • Other groups in the company or outside the company not reporting the area is hard.

Here are signs the area is actually hard:

  • Other, similar groups in the company or elsewhere reporting the area is hard.
  • Online commentary that the technology is hard.
  • Lots of online content to help people learn.
  • Definite technical requirements, like the ability to understand number theory.
  • Obvious qualifications, e.g. network engineering certification.

From a management perspective, the best thing to do is stop knowledge-hoarding behavior before it starts. Ideally, there shouldn’t be a single point of failure on your team (a bus factor of more than 1). This means consciously focusing on cross-training (something the military does very well).  If you inherit someone showing this behavior, you need to make cross-training a priority and personally intervene to make sure it’s done properly. Cross-training will involve some loss of status for the person which you need to be sensitive to and manage well.

For some people, keeping skills and knowledge to themselves makes perfect sense, it’s a great way to enhance their value to their employer. For their colleagues, it’s not good behavior, and for their manager, it can be disastrous. For (almost) everyone’s sake, you should deal with it if it’s happening in your organization.

Saturday, February 1, 2020

Company culture and the hall of mirrors

Company culture: real and distorted

When I was a child, my parents took me to a funfair and we walked through the hall of mirrors. The mirrors distorted reality, making us look taller or smaller, fatter or thinner than we really were. I've read a number of popular business books that extoll the virtues of company culture, but I couldn't help feeling they were a hall of mirrors; distorting what's really there to achieve higher book sales. 

(Reading about company culture in popular business books is like entering the house of mirrors. Image credit: Wikimedia Commons, License: Creative Commons, Author: SJu)

I was fortunate enough to spend time researching the academic literature on company culture to understand what's really there, and the results surprised me. I took a course at the Harvard Extension School on Organizational Behavior which was taught by the excellent Carmine Gibaldi. Part of this course was a research project and I chose to look at the relationship between company culture and financial performance.

What books claim vs. evidence

Given the long list of popular business books making very definitive claims about company culture, you might expect the research literature to show some clear results. The reality is, the literature doesn't. In fact, when I investigated the support for claims made in books, I couldn't find much in the way of corroborating evidence. Going back to the popular books, I found no references in many cases, in other cases I found evidence based on the author's work alone, and in other cases, the evidence was just unverified anecdotes. What I did find in the research literature was a more complex story than I expected.

Company culture and financial performance

Despite what the popular business books said, it wasn't true that a strong company culture of itself led to financial success, in fact, many companies with strong cultures went bankrupt. It wasn't true that there were a magic set of values that led to company success either, different successful companies had different values and different cultures. It wasn't true that company values are immutable, even a large company like IBM could and did change its values to survive.

What I found is that there are mechanisms by which a strong and unified company culture can lead to better performance; coordination and control, goal alignment, and enhanced employee effort. For these reasons, under stable market conditions, a strong culture does appear to offer competitive advantages. However, under changing market conditions, a strong culture may be a disadvantage because strong cultures don't have the variability necessary to adapt to change. There also seemed to be a link between culture and industry, with some cultures better suited to some industries.

I read about DEC, Arthur Anderson, and Enron. All companies with strong cultures and all companies that had been held up at the time as examples of the benefits of a strong culture. DEC's culture had been studied, extensively written about, and emulated by others. Enron had several glowing case studies written about them by adoring business schools. In all these cases, company culture may well have been a factor in their demise. Would you take DEC or Enron as your role model now? Are the companies you're viewing as your role models now going to be around in five or ten years?

Where's the beef?

Am I falling into the trap of not providing you with references for my assertions? No, because I'm going to go one stage further. I'm going to give you a link to what I wrote that includes all my references and all my arguments in more detail. Here's the report I wrote which lays out my findings - you can judge for yourself.

Popular books are mostly wrong

During my research, I read some detailed critiques of popular business books. I must admit, I was shocked at the extent to which the Emperor really has no clothes and the extent of the post hoc fallacy and survivor bias in the popular books. Many of the great companies touted in these books have fallen from grace, disproving many of the assertions the books made. If you make statements that company culture leads to long-lasting success, and your champions fail while maintaining their values, it kind of looks like your arguments aren't great. Given the lack of research support for many business books' claims, this should come as no surprise.

What to do?

Here's my advice: any popular business book that purports to reveal a new underlying truth about business success is probably wrong. Before you start believing the claims, or extolling the book's virtues, or implementing policy changes based on what you've read, check that the book stands up to scrutiny and ideally, that there's independent corroboration.