Question:

How do you deal with the discovery of a critical and undocumented component that nobody wants to acknowledge, but keeps the company running with makeshift solutions? This component often fails at the worst possible times, causing panic and chaos among the senior executives who demand an immediate fix. The supposed domain expert is unreachable, and the previous IT staff who maintained this component are no longer with the company. I recently encountered such a component.

This component was part of a software product that we acquired from another company, which gave us a competitive edge in our industry. The software product was a web app that stored its entire database and code in a single Dropbox account, which was accessed by a server instance in Heroku. The Dropbox account had almost 500 TB of data, and the web app made numerous API calls to read and write data from hardcoded database files. The Dropbox account also served as their version control system. The development team of this web app requested 40 laptops each with 4 TB of storage, without explaining why they needed so much local storage. They were very evasive and secretive about their work, and we assumed they had passed the audit before the acquisition.

Three months later, on a Friday afternoon before a major holiday, the web app started to experience intermittent outages, slow responses, and network congestion. The help desk escalated the issue to me, and I called the development team to investigate. They told me that the Dropbox account was running out of space, and they needed me to call Dropbox support to increase the storage. They also revealed that the entire web app was hosted on Dropbox, and they had no backup or monitoring in place. I was shocked and horrified by this revelation, and I contacted Dropbox support to gain access to the account. I also alerted the CISO, who informed the CFO and the general counsel about the potential privacy breach and the legal risks. The general counsel was furious and demanded an immediate solution. I worked all night to migrate the data and to check for any unauthorized access to the Dropbox account. I also developed a stomach ulcer and an abscess in my mouth due to the stress and anxiety.

There are many lessons to learn from this incident, but I am still amazed by the fragility and complexity of this component. I wonder how many other companies have similar components that are hidden and ignored, but essential for their operations.

P.S. My son suggested that I ask my fellow IT veterans to drink one for me this weekend.

Answer:

How to Handle a Nightmare Component That Nobody Knows About

Have you ever stumbled upon a critical and undocumented component that nobody wants to talk about and is secretly holding the company together with shoe string, bubble gum, and paper clips? This component often fails at the worst possible times, causing panic and chaos among the senior executives who demand an immediate fix. The supposed domain expert is unreachable, and the previous IT staff who maintained this component are no longer with the company. This is a nightmare scenario for any IT professional, and it can happen to anyone, anytime.

In this article, I will share my personal story of how I dealt with such a component, and what lessons I learned from this ordeal. This component was part of a software product that we acquired from another company, which gave us a competitive edge in our industry. The software product was a web app that stored its entire database and code in a single Dropbox account, which was accessed by a server instance in Heroku. The Dropbox account had almost 500 TB of data, and the web app made numerous API calls to read and write data from hardcoded database files. The Dropbox account also served as their version control system. The development team of this web app requested 40 laptops each with 4 TB of storage, without explaining why they needed so much local storage. They were very evasive and secretive about their work, and we assumed they had passed the audit before the acquisition.

Three months later, on a Friday afternoon before a major holiday, the web app started to experience intermittent outages, slow responses, and network congestion. The help desk escalated the issue to me, and I called the development team to investigate. They told me that the Dropbox account was running out of space, and they needed me to call Dropbox support to increase the storage. They also revealed that the entire web app was hosted on Dropbox, and they had no backup or monitoring in place. I was shocked and horrified by this revelation, and I contacted Dropbox support to gain access to the account. I also alerted the CISO, who informed the CFO and the general counsel about the potential privacy breach and the legal risks. The general counsel was furious and demanded an immediate solution. I worked all night to migrate the data and to check for any unauthorized access to the Dropbox account. I also developed a stomach ulcer and an abscess in my mouth due to the stress and anxiety.

There are many lessons to learn from this incident, but I am still amazed by the fragility and complexity of this component. I wonder how many other companies have similar components that are hidden and ignored, but essential for their operations. Here are some of the key takeaways that I hope will help you avoid or handle such a situation in the future:

• Do not assume that everything is fine just because it passed the audit. Audits are not foolproof, and they may miss some important details or risks. Always do your own due diligence and verification when acquiring or integrating a new software product or component. Ask questions, request documentation, and test the functionality and performance of the component. Do not take anything for granted or at face value.
• Do not rely on a single point of failure or a single source of truth. Having a single Dropbox account as the sole repository and host of the entire web app and database was a disaster waiting to happen. Not only was it prone to running out of space, but it also exposed the data to potential theft, loss, or corruption. There was no backup, no monitoring, no security, and no version control. Always have multiple copies, backups, and sources of your data and code, and use proper tools and platforms to manage them. Use cloud services, databases, servers, and repositories that are reliable, scalable, secure, and compliant with your industry standards and regulations.
• Do not be afraid to speak up or escalate the issue. The development team of the web app was very secretive and evasive about their work, and they did not want to admit or acknowledge the problem. They tried to hide it and fix it themselves, without informing or involving anyone else. This was a huge mistake, as it delayed the resolution and increased the risk. Always be transparent and honest about the status and issues of your component, and communicate them to the relevant stakeholders and authorities. Seek help and support from your colleagues, managers, and experts, and do not try to handle everything alone. You are not alone, and you are not the only one responsible for the component.
• Do not let the stress and anxiety get to you. This incident was one of the most stressful and anxiety-inducing experiences of my life, and it took a toll on my physical and mental health. I developed a stomach ulcer and an abscess in my mouth, and I had nightmares about the component. I felt overwhelmed and helpless, and I blamed myself for not detecting or preventing the problem earlier. This was not healthy or productive, and it did not help me solve the problem. Always take care of yourself and your well-being, and do not let the pressure or the blame affect you negatively. Seek professional or personal help if you need it, and do not be ashamed or afraid to ask for it. You are not a machine, and you are not perfect. You are a human, and you deserve respect and compassion.

I hope that this article has given you some insight and guidance on how to deal with the discovery of a critical and undocumented component that nobody knows about. This is a rare and extreme case, but it can happen to anyone, anytime. Be prepared, be vigilant, and be resilient. And remember, you are not alone.

P.S.

My son suggested that I ask my fellow IT veterans to drink one for me this weekend. Cheers! 🍻