I already told this story on my blog before I got hacked and lost everything, and I wanted to bring it back to life, as I find it extremely entertaining and funny.
Years back, I was working as a “promising” junior software engineer for a company that specializes in enterprise solutions and services for large telecom operators. When I came to the company they assigned me to the team who’s one of many responsibilities was to maintain and offer support in running the data through rating algorithms for a single largest operator in the country.
There were several types of these scheduled batch jobs, and they were running, hourly, daily, weekly, or monthly. One day, we received a call from their IT department, they were very upset as all batch jobs were failing, and they were failing without any error message anywhere. We sent our team on the site and they started investigating.
First thing they discovered is that every single job was failing on the same CDR (Call Data Record). Of course, first thing everyone would think about in that situation is check if CDR itself is syntactically correct or if it somehow got corrupt, or incomplete… However, CDR was totally fine.
There’s another technical detail I should mention before I let you know what happened in the end… All those jobs were written in a form of PL/SQL procedures and they were being run by a shell script, which would listen to their output, and I mean they were listening to everything, not just exit codes, but line by line output of every SQL command. In case some keywords were detected, script would terminate the job. Those keywords were something like “Error“, “Problem“, “Fail“, etc
Like this was not bad enough, scripts were actually checking if output contained any of the strings above, they did not have to match it.
After nerve wrecking hours spent on analysis, they realized that all of this was being caused by subscriber’s name in the “problematic” CDR. The woman had a quite common female name among middle age population of Bosnia and Herzegovina. Her name was…
It does not mean anything bad in Bosnian, and it does not have any relation with failure in English, but on that day it kind of DID. It was all her parent’s fault. I wonder if she has a sibling named Admin (another regular name in Bosnia and Herzegovina). LOL.