“Code Red” is one thing you never want to hear. Normally it is only on television. Unfortunately it does happen in real life.
There is another meaning that I want to give “Code Red” based on my experience in the software industry. Software that looks good on the outside but isn’t healthy on the inside would qualify for an eventual “Code Red”. The faulty code is often invisible and perhaps not run often. Perhaps it is not well suited to a new environments. Maybe it has not been updated for 15 years. Perhaps the code management hasn’t allowed for code re-design or evolution.
Regardless of the cause, the results are usually the same. Eventually there is catastrophic failure. This could range from blue screens to lost data. In some ways computer failures are worse than fires. At least fire fighters have a chance at stopping a fire. An IT employee has little hope of correcting fundamental flaws in the software product.
The flurry of security updates reveals another angle of “Code Red”. If you don’t cover yourself well and do not have a design that can handle full assault (or otherwise), then you have failed from a customer point of view. God help you if you pretend like it is not a problem or that the security risk is minimum when it isn’t.
“Code Red” usually means it is too late. In software circles, it is time for support to pick up the ball and hand it off to a programmer than can hopefully diagnose and fix the problem. Sometimes the fix actually makes things worse for other customers. Hence, the hotfix program was started where only certain customers get certain fixes until it is fully tested under an umbrella service pack.
The problem with this model is that not only does the customer have a fix that might never be fully supported, but that the programmer is under extreme pressure to find a solution to a problem that might require redesign. It is a tight spot and most programmers would react by doing the best they can in the shortest time. This equates to what some would call a “hack”. These changes over time help to de-stabilize the design and eventually the code becomes a wobbly mess.
This comes to the conclusion that fire fighting is not always the best way to handle customer problems. It helps to be a bit more pro-active with the design and development to anticipate customer issues. The advice to do things up front is so commonly heard in universities but so rarely applied in business. Experience says that thinking (designing) before doing (programming) is always better.
So, what can be done to improve this situation? I’ll give you an impromptu list of ideas that might help.
- Don’t stand idly by when the fire is about to start. Let people know before the flames take hold.
- Invest in the code base besides just new features. Old code needs upgrading and even re-designs sometimes.
- Treat the software product as a living thing. It needs many things to survive and grow. Don’t starve it when it first begins and then over feed it at the end. It doesn’t matter how much you feed it at the end if it cannot be properly fixed.
- Programmers are not silly. They have lots of ideas of making the code better. Don’t stand in their way and encourage them to help make things better. Most programmers/engineers like to see things improve.
- Glory does not come from fighting a fire that you should have prevented. Keep it in mind that every fire is bad and shouldn’t have happened in the first place. Try to figure out how to stop the fires before they start.
- As time goes by and you begin to understand the need for constant improvement, keep in mind that everything else works that way too. Who ever heard of a computer or language that lasted forever and did well in the current times? It just doesn’t happen. So, the message is to just evolve. If you don’t you will not survive.
In summary, try to prevent your software “Code Red”s. If you don’t try, you are bound to get stuck in the loop of constantly maintaining something that just keeps on costing more and more to support. On top of that, the customers will eventually get wise and move on.
Live near Brisbane, Australia. Software developer currently focused on iOS and Android. Avid Google Local Guide
Red Circle Blog 
Unfortunately in many organizations, management gives glory to those who valiantly fight fires, even if those fires were caused by the very same people. In another company one of my colleagues was lauded and rewarded for firefighting, but if we’d removed them from the team entirely there would have been far fewer fires and the remaining team – without that person – would have been more productive than with that person.
If you want to stop firefighting, you have to stop giving rewards (whether monetary or recognition or praise or whatever) for firefighting. In other words, as I often say:
Be careful what you measure (or reward), because you’ll get it.
Interestingly, the corollary is:
Beware what you don’t measure, because people will abuse it to meet other goals
This is one way that firefighting often starts – you don’t measure the amount of Code Red potential (“combustion load”?) involved in the shortcuts you are taking (to meet some measured goal such as time to market or keeping a big customer happy or …). Usually this happens over several release cycles building up an increasing amount of combustion potential, then something happens to trigger the fire.
It can then get a lot worse. If you have a significant firefighting effort it takes resources away from the next release, which leads to the temptation to cut corners, which increases the combustible load which leads to more fires in the future. It can reach the stage where you can’t do anything BUT fight fires.
So one of the first steps is to start measuring how much combustible load you have generated. This lets you make informed tradeoffs at each decision point (do we get this to market sooner for now and budget for fires later, or spend a little more time now and a lot less on fires later?) rather than burying the problem until it flares up when you can least afford it.