Tales from the Trenches – S01E01

The Value of Checking Your Work

Cast your mind back to your school days. Do you remember being told to check your work? I do and the story I’ll share with you here is a great illustration of the value of checking your work or alternatively, having your work checked. We’re humans and not infallible. Mistakes will be made. The important thing is to learn from the mistakes and not repeat them.

For many years I was a security instructor. I would typically fly to an island and teach class for a week and then fly home. Sounds glamorous right? It was rewarding and the personal and professional connections I made are still with me to this day. It was also quite tiring and the funny stories of traveling and working while exhausted will have to wait for another time. Anyway, the classes were run on labs using local virtual systems running on student supplied hardware. Many times, I taught using course ware and labs of my own design but often I needed to use the official curriculum. In order provide a relevant context to the lessons, I encouraged students to follow along using their own systems. I would also provide a complimentary health-check of their environments. That is, I would take a quick look at their security management dashboards and reports and see if I could identify any issues or make recommendations for improvement.

This story takes place during one of these classes, when a student, let’s call him S, asked me for a health check. For reference this class covered the installation and use of the central management and endpoint security applications. S shows me his management console and the first thing I notice is there are zero threat events. Not a low number of events but none! For 200+ systems, even a perfectly tuned environment is going to show something, unless they’re discarding data. S proudly says to me that they don’t have any security issues or events, just look at the dashboards! I challenge him on this asking if he allows direct Internet access, email, email attachments, and removable device usage in the organization. He does so I challenge him that there just has to be something happening. I even went so far as to say that I would give him $100 out of my own pocket if the information was truly accurate!

I roll up my proverbial sleeves and dive in. The first thing I noticed was that the Windows system on which the central management server was installed did not have any endpoint security software installed. He questioned why that mattered. The management software was just for management and the system still needed the security software. So I manually installed the endpoint security software as per their standards when we notice some stability and performance issues. We, of course, had backups of their policies and other key files. A bit of digging revealed that they were using an old build of the endpoint security software. By old, I mean a version that released well before the version of Windows server in use. We applied the necessary patch and rebooted the system. I pointed out that they needed to keep their security software up to date along with everything else in the organization.

Guess what the dashboards looked like after the reboot? If you guessed that it lit up like a Christmas tree, you would be right! It turns out that the management software had lost connectivity with the SQL database over three months earlier. Rebooting the server enabled the re-connection and all endpoints began reporting their events. Thankfully, the deployed endpoint security software was resilient enough to detect and mitigate the various threats as well as update signature files in the absence of a functional central management system. You’re probably wondering what kinds of threats they were seeing. There was plenty of spyware and adware as well as removable media-based Trojans. After the understandable surprise, we worked together to ensure that they were deploying the latest appropriate version of the endpoint security software and demonstrated how to test the systems to prove that the systems were operational.

Lessons learned:

• Check your own work. If you manage to reach a point with policy tuning where things are very quiet, run some test files or activities through your systems. Ensure that you receive the alerts and that the events show up in your scheduled reports. You are scheduling reports, aren’t you?
• Have your worked checked by someone knowledgeable in the technology in question, not just an auditor who asks you to show your work.
• Your security tools need to be updated just like your operating systems and business applications.
• Don’t be afraid to ask for help. No one has dealt with everything or seen every possible situation. Develop a peer group and collaborate. As a group, you’ll be better off than trying to go it alone.

About the Author: