rcegan

Don't blow up Prod with Process Events

Had a slightly eye-opening experience recently around logging process creation events, Event ID 4688. In particular, the Detailed Tracking command line component (This is mostly a vent and I'm not offering anything new or particularly intelligent).

It's really easy in 2023 to assume an environment with a modern(ish) fleet of servers and performance that enabling 4688 and detailed tracking is an exercise in checking boxes for easy endpoint visibility in your SIEM. But, turns out, if you have:

  • A larger-by-default event log size (A good thing!)
  • You're keeping exhausted event logs on disk for an extended period of time (also a good thing!)
  • You turn on PowerShell transcription and Script Block Logging at the same time (a very good thing!)

You have a non-zero chance of creating some real headaches for the IT Operations team. Disks started running out of space due to errant RMM tools and patch management services spamming process creation events, which had a rather heinous impact on performance. It wasn't until some of the event log retention policies were cleaned up and a more careful eye was put in place on the server fleet before things normalised.

It is always worth considering the factors around your change, and that there are other teams who exist outside of our SIEM bubble. While these changes were all good and highly recommended, the combination of them occurring at once and a lack of communication and preparedness from both parties resulted in an avoidable headache for everyone. So, I guess the moral of the story is, be mindful of the details, and tip your sysadmins.