This blog is based on a talk Salem Cyber did on 3/11/25 to the H-ISAC community.
The Critical Importance of Process
Having robust alert triage processes in place for continuous learning, evolving threat landscapes, and documentation are crucial for the success of your whole team.
However, we oftentimes see market challenges that inhibit a company’s ability to adopt robust and effective processes. Some of these include:
· Global work hours limits real time peer review: The reality of cybersecurity operations in a globalized world means that teams are often distributed across multiple time zones. This geographical dispersion inherently limits the opportunity for real-time interaction across various roles and individuals. Immediate collaboration, spontaneous problem-solving, and quick consultations become difficult, hindering the ability to address critical issues with the necessary speed and agility. This can lead to delays in decision-making and potentially increase the risk of overlooking crucial information.
· A variety of ticketing and collaboration tools: In many organizations, analysts are confronted with a fragmented landscape of ticketing and collaboration tools. This lack of standardization raises several concerns. How easy is it for an analyst to seamlessly hand off duties or transfer critical information to a colleague during shift changes or when collaborating on an incident? Where is knowledge about different tools stored, and how easily can analysts locate the information they need to utilize these tools effectively? Furthermore, how easy is it for a person to work across these disparate tools, often requiring them to switch between multiple interfaces and potentially leading to errors or omissions? This complexity can severely hamper workflow efficiency and create obstacles to effective teamwork.
· Training, awareness, & documentation stale: The cybersecurity field is characterized by rapid evolution, with new threats and vulnerabilities emerging constantly. Consequently, training programs, awareness initiatives, and documentation can quickly become outdated. Stale training materials fail to equip analysts with the knowledge and skills necessary to address the latest threats. Similarly, outdated documentation may provide inaccurate or incomplete guidance, leading to missteps in incident response. This can result in analysts relying on outdated information or a lack of readily available resources to guide their actions.
· Alerts are delayed, lack context, poor fidelity: SOC teams often struggle with alerts that are not delivered in a timely manner, lack the necessary contextual information, or suffer from poor fidelity, meaning they may be inaccurate or unreliable. This poses a significant challenge: How are you empowering your team to make informed and effective decisions based on alerts that are potentially late, lack crucial details, or may even be misleading? Analysts need timely, accurate, and context-rich alerts to effectively prioritize and respond to potential security incidents.
However, despite these challenges, there is a significant opportunity to move forward and improve SOC operations:
· Remove isolation + improve team collaboration: Cybersecurity is fundamentally a shared responsibility. The more systems we collectively secure and the more effectively we collaborate, the more secure we all are as a community and as individual organizations. Breaking down silos within security teams and fostering a culture of collaboration and information sharing is crucial.
· Elevate context + encourage knowledge share: Providing analysts with comprehensive context surrounding potential security events is essential for effective decision-making. This includes enriching alerts with relevant data, background information, and threat intelligence. Encouraging a culture of knowledge sharing within teams and across the broader cybersecurity community can significantly enhance overall situational awareness and improve response capabilities.
· Trust the process + empower decision making: Establishing well-defined, documented, and consistently followed processes builds trust and empowers analysts to make sound decisions with confidence. Clear processes provide a framework for action, reduce uncertainty, and ensure that critical steps are not overlooked during incident response.
· Accelerate performance + reduce burnout: By streamlining processes, providing access to relevant information, and fostering a supportive environment, organizations can significantly accelerate analyst performance. This increased efficiency not only improves security outcomes but also reduces the burden on individual analysts, mitigating burnout and promoting a more sustainable work environment.
Operationalizing Alert Triage
Challenges organizations face in implementing and maintaining efficient alert triage processes
A alert fatigue is a symptom of exhaustion caused by several factors. Foundationally, we would put these into two groups 1) decision fatigue and 2) repetition.
Let’s look at Decision Fatigue first. It's easy to see an analyst's role as making a choice on whether an alert is a threat or a false positive. If any individual analyst gets 10 or 20 alerts a day, you’d think there making 10 or 20 decisions. But in fact, for every alert an analyst is having to make a dozen or more decisions:
- They interpret the alert use case to understand what threat is being reported
- They interrogate the details of the alert, and decide what information could be missing
- They choose what systems they’ll need to use to gather more information
- They perhaps will have to write new queries to recall the specific logs they need
- They’ll then have to interpret the data and propose a few hypotheses about why the reported activity may be occurring.
- They will then need to try and further validate or reject these hypothesis.
- They will need to draw a final conclusion
- And finally decide on what if any response and reporting actions need to be taken.
For people with analytical minds, you can get a lot of energy from this process. Especially when you’re figuring something out for the first time. On the other hand, truly new and novel alerts can be anxiety inducing. You’re always a little concerned when you’re not sure what you’re looking at or what to do next.
Documentation is very helpful, especially for processing or actions you having built strong muscle memory around. When you’re new to an organization or perhaps stepping into large responsibility, documentation can be a lifesaver. Both in reducing the number of decisions you need to make and reducing the anxiety associated with “what if I’m wrong”.
Complexities of managing and updating documentation in a constantly changing environment’’
However, documentation can be difficult and there are a few drawbacks:
1. Environmental drift: Changing factors make the documentation you have written outdated. It’s already hard to invest time in writing it in the first place. It’s even harder to keep it up to date. Often, you write documentation to solve a knowledge problem amongst a cohort of people. Eventually, those people develop muscle memory around the process and rely less on the documentation, reducing the demand signal for new documentation and updates to existing documentation. Then one day that demand signal returns, either by new people joining or other foundational changes occurring, and you find that the documentation you had may be very out of date.
2. Decision Tree explosion: There are so many possible paths to walk. In that, the number of outlier cases expand quickly beyond the point at which you can practically cover what to do. And so you don’t. Most SOC documentation defines a common starting point, perhaps a structured set of 10 to 15 things to do and check. But it won’t say anything about what to do with the information you’ve uncovered. It would be like a choose your own adventure story, but when you flip to the prescribed section, the page is blank. On one hand, that’s their job to make those decisions. On the other hand, by not having more resources for analysts, you leave open more room for misses and other costly errors.
3. Repetition: There are effectively automatic tasks analysts have to perform that require no thought. As an analyst, it can feel mind-numbing to go all the way back though your investigation to document what you did and what you found. It’s like doing the investigation twice. And not to call anyone out, but many analysts actually have a note pad of prewritten investigation notes that they will paste into a ticket. Most of the time no one will ever read an investigator's notes, which makes it easy to slack off a bit. However, auditors, executives or lawyers might identify a need to review specific cases or classes of activity that have occurred. Imagine there is an incident and now the organization wants to look back at 6 months of USB thumb drive activity. And if the analyst slacked off, those investigation notes aren’t helpful.
One of the biggest missed opportunities to create efficiencies is not leveraging the insights of individual investigations in their detection and alerting capabilities. Notes won’t solve this problem exactly, but if you think of documentation more broadly as a record of meaningful information for the purpose of sharing knowledge, then you can see the beginnings of opportunity.
A Path Forward
1. Document only what’s unique to you
Data residency | Communication protocols | Incident classification matrixes
When it comes to documentation for your Security Operations Center (SOC), the most effective approach is often to focus on what makes your organization truly unique.
A crucial area to prioritize is data residency. This refers to where your organization's data is stored and the specific regulations or policies governing that data. For many organizations, pinpointing the exact location of all data can be a complex challenge. Therefore, documenting whereto find specific types of data is crucial. This ensures that analysts know where to look for information during investigations, especially when dealing with compliance or regulatory requirements.
Another key area is communication protocols. Clear communication protocols are essential for effective incident response. This involves documenting what triggers necessitate including specific individuals or teams in the communication chain. For example, what types of security events require immediate notification of legal, public relations, or executive leadership? By outlining these protocols, you ensure that the right people are informed at the right time, facilitating a coordinated and efficient response.
Finally, prioritize documenting your incident classification matrixes. An incident classification matrix defines how your organization categorizes and prioritizes security incidents. This is highly specific to each organization, as it reflects your unique risk tolerance, business priorities, and operational context. Documenting your classification metrics ensures consistency in how incidents are assessed and handled. It also helps analysts understand the severity of an incident and the appropriate level of response required.
2. Link to 3rd Party knowledge wherever possible
Studio is a community resource to support and document your cyber investigations. It leverages the same award-winning Salem AI engine to help you determine if your activity is a real threat.
Use Virus Total to analyze suspicious files, domains, IPs and URLs to detect malware and other breaches, automatically share them with the security community.
Check out MITRE’s knowledge graph of cybersecurity countermeasures to determine realistic response remediation options based on the type of threat activity.
Finding the resources that are helpful to your company and make sure that the people in your organizations know these resources exist, they are for there for them to use, and how they should be used is super important.
3. Structure investigation notes
Let's be honest: documentation is often a pain point for security analysts. After spending hours investigating a complex cyber event, the last thing they want to do is write a lengthy narrative about it. It's a common scenario: analysts stare at a blank 500-word comment box, struggling to articulate everything they just did. This unstructured approach to documentation is inefficient and ultimately hinders the effectiveness of your security operations.
Imagine a different approach. Instead of a free-form textbox, what if analysts had a structured form to fill out? Think about it:
- "Here are the five searches I ran..."
- "Information involved in this account..."
- "Details about the assets involved..."
This kind of structure transforms documentation from a chore into a checklist. It might feel a bit robotic at first, but it offers several advantages:
- Ease of Completion: It's much easier for analysts to quickly populate pre-defined fields than to compose a lengthy essay. This saves time and reduces the mental burden of documentation.
- Consistency: Structured notes ensure that key information is captured consistently across all investigations. This eliminates variability and improves the quality of your data.
- Data Mining Potential: This is where the real power comes in. When you have highly structured notes on the back end, you can easily mine them for relevant insights. You can identify trends, track attacker behavior, and proactively improve your security posture.
