Enterprise Security Transformation: SOW to Managed Splunk (MSP)

‍

Under the historical project-based Statement of Work (SOW), which originally kicked off on October 26, 2022, Conducive successfully deployed Splunk Cloud and Splunk Enterprise Security (ES) across Alamo's core systems (including Active Directory, Windows Event Logs, openldap, Fortigate traffic, Malwarebytes, and Rapid7 scans). However, following project handoff, the unmanaged, out-of-the-box Splunk ES correlation rules began firing blindly, creating severe operational bottlenecks:

• Severe Alert Fatigue: The security console was overwhelmed by false-positive alert storms. A single rule, ESCU - Detect Computer Changed with Anonymous Account, fired over 19,000 times in 24 hours due to field alignment errors (e.g., mismatching LogonType vs Logon_Type).
• Noisy Service Accounts: The Detect New Local Admin Account rule fired 864 times in a single day, driven entirely by automated background processes of the legitimate service account .
• System and ML Errors: The Activity from Expired User Identity search failed continuously because the 'endDate' attribute was missing from the identities lookup, while the ML-based DNS Query Length Outliers rule failed because its underlying machine learning model did not exist.
• Maintenance Deficits: Unmonitored environmental drift led to truncated user identities, LDAP line-breaking bugs, and orphaned threat intelligence searches, which degraded institutional trust.
• Minimal Adoption: Although each of Alamo's five colleges had designated security leads, the system's high noise-to-signal ratio discouraged day-to-day active utilization.

• To overcome these challenges, Alamo Colleges' security leadership transitioned their engagement model to Conducive's continuous Managed Splunk Services (MSP) in January 2026. Conducive deployed a multi-stage operational tuning and infrastructure optimization strategy:
• Conducive's team systematically audited all noisy correlation searches, aligning them with Common Information Model (CIM) standards and local network realities. Broken or low-fidelity alerts were either disabled or converted to quiet Risk-Based Alerting (RBA) modifiers that only escalate when a specific asset or identity exceeds a dangerous cumulative threat score.
• To curb licensing costs in Splunk Cloud, Conducive integrated a high-performance Cribl pipeline. On April 22, 2026, Conducive engineers configured custom HTTP Event Collector (HEC) tokens for the Cribl Fortigate pipeline, successfully dropping high-volume, low-value DNS events and routing only critical security logs into Splunk.
• The MSP model provided daily proactive health support. When a massive syslog spike of tens of gigabytes occurred in February 2026, Conducive engineers immediately investigated and isolated the offending device, preventing a licensing breach. They also resolved identity truncation issues and integrated the student log tenant within Office 365.
• Conducive successfully upgraded all distributed heavy forwarder systems (including ) to Splunk version 9.3.13, closing outstanding vulnerability scans and ensuring alignment with modern Splunk Cloud security requirements.

‍

‍

Within less than six months of launching the Managed Splunk (MSP) program, Alamo Colleges transformed their security console into a clean, actionable command center:

• Eliminated Alert Storms: By modifying broken searches, implementing service account exclusions, and turning down low-fidelity alerts, daily noise plummeted from tens of thousands of daily events to a handful of highly actionable security notables.
• Optimized Data Budgets: Dynamic filtering on the Cribl Fortigate pipeline dropped low-value data at the edge, optimizing licensing spend and conserving valuable cloud indexer storage.
• Empowered Campus Leads: With clean, reliable executive dashboards and tailored reports, security leads across all five college campuses are now actively using Splunk daily to monitor their respective networks.
• Hardened Infrastructure: All heavy forwarders are fully upgraded and compliant with modern security standards, resolving historical vulnerabilities.

‍