Lead Observability Engineer (Splunk & Cribl)

We are seeking a highly motivated Lead Observability Engineer to join our Performance Engineering team. The ideal candidate will have experience with Splunk, Cribl, Infrastructure as Code (IaC), CI/CD pipelines, cloud platforms. This individual will help manage enterprise monitoring capabilities, automate platform deployments, onboard new data sources, and build scalable solutions that provide operational visibility across AWS, Azure, Google and other enterprise technologies.
Primary Responsibilities
Splunk & Cribl Platform Engineering

• Design, deploy, administer, and optimize enterprise Splunk and Cribl environments.
• Manage platform upgrades, lifecycle maintenance, performance tuning, and capacity planning.
• Implement scalable, highly available observability architectures
• Develop standards and best practices for platform operations and governance.
• Troubleshoot complex platforms, ingestion, and search performance issues.

Cloud Automation & Infrastructure as Code

• Build and maintain Infrastructure as Code solutions using Terraform
• Develop automated deployment pipelines using GitHub and IaC.
• Standardize platform provisioning, configuration management, and operational processes.
• Implement automated monitoring, alerting, and compliance validation for infrastructure services.
• Support modernization efforts utilizing AWS/Azure cloud-native services.

Software Development & Automation

• Develop and support automation solutions using Python, PowerShell, and Bash.
• Refactor legacy operational processes into modern cloud-native architectures.
• Create reusable frameworks and libraries for platform automation.
• Implement source control, peer review, and release management standards.
• Support operational reporting, dashboards, and service delivery automation.

Technical Leadership

• Serve as a technical advisor for observability, monitoring, and logging initiatives.
• Establish engineering standards and architectural patterns.
• Collaborate with Cloud Engineering, Security, Networking, Architecture, and DevOps teams.
• Evaluate emerging technologies and recommend platform improvements.

Required Qualifications

• 7+ years of experience administering enterprise observability or monitoring platforms.
• 5+ years of hands-on Splunk administration and engineering experience.
• Experience with Cribl Stream, Cribl Edge, or similar data pipeline technologies.
• Terraform and Infrastructure as Code experience.
• Experience building IaC pipelines using GitHub.
• Strong Python scripting and automation experience.
• Experience integrating systems using REST APIs and cloud-native services.
• Knowledge of AWS services including EC2, S3, IAM, Lambda, ECS/Fargate, EventBridge, and CloudWatch.
• Knowledge of Azure services including Storage Blobs, Event Grid, Functions, Event Hub.
• Experience with Linux administration and enterprise application deployments.
• Excellent communication and documentation skills.

Preferred Qualifications

• Splunk Enterprise, Security (ES) IT Service Intelligence (ITSI) experience.
• Experience with FedRAMP, NIST 800-53, CMMC, or regulated environments.
• Experience with AWS, Azure and Google Cloud Platform integrations.
• Knowledge of security logging, SIEM architectures, and enterprise monitoring strategies.
• AWS Certifications, Splunk Certifications, or Terraform Certifications.
• Experience supporting large-scale cloud environments and enterprise data onboarding initiatives.

#DICE