I'm by no means the first person to talk about this problem within our industry. But in the past year I've seen it happen more and more.
Imagine this scenario, A company is working on releasing some software, and upon release the user finds a bug. The manager comes back and says:
"We can't have any bugs in our software, what can we do to avoid this happening again?"
So the team gets together and says:
"We can reduce the number of bugs in the software by making sure all our changes are QA'd before going into production."
Now, before any code gets goes live the functionality needs to be checked. Seems fair. But still bugs appear in the software.
This time a director complains:
"We can't have customers complaining about buggy software, what can we do to stop this?"
This time the development team realise that the bug got introduced because it somebody bypassed the QA check:
"To avoid this we best make sure we document every change and get a release form signed before it goes live"
But still bugs get into production. And still the company tries to reduce the number by introducing more and more checks:
"Lets make sure it goes into staging before going live."
"Lets introduce a code freeze so after a point we can be sure we aren't introducing more bugs as we won't be changing the code."
"Lets make sure we have gated check-ins before pushing to the main branch."
"Lets make sure our devs are required to request access to the branch before pushing so we know they aren't introducing a bug."
"All SQL should be checked by an SQL team."
"All tickets should be in the correct state before the code relating is released so we can be sure all the checks are completed."
And on and on.
The problem with this, is that it doesn't matter how many checks, how slow you deploy, how much you test. Bugs are always going to get into production. And what you are actually doing by introducing all these checks and process is just slowing down how quickly you can fix a bug when it's found.
Perhaps a better idea is to work on how quickly you can fix a bug once its found. Rather than introducing more checks, make the checks you have quicker. Remove checks that slow the process down and improve the feedback cycle.
I'm not saying there should be no checks, just make sure that these checks don't interfer with the speed you can release.
It may seem counter intuitive but once you realise you can't stop all bugs getting into production then its probably your best option.
The tricky part is convincing management of this :).