Lead Observability Engineer

Splunk Cloud Administration & Strategy Design, implement, and manage Splunk Cloud environments to support scalable log ingestion, indexing, and search performance. Ensure high availability, data retention, and disaster recovery strategies are in place. Automation & Integration Automate configurations and deployments using tools like Terraform, Ansible, Puppet or custom scripts. Build and manage integrations with CI/CD pipelines, cloud-native services (GCP, Azure), ITIL platforms for real-time observability. Develop and maintain custom Splunk apps and modular inputs to extend platform capabilities. Monitoring & Optimization Monitor Splunk performance metrics and proactively optimize indexing, search performance, and storage usage. Tune data inputs, parsing rules, and indexing strategies to ensure efficient resource utilization. Implement best practices for data onboarding, including source types, field extractions, and event normalization. Security & Compliance Manage user roles, access controls, and authentication mechanisms (SSO, LDAP, SAML). Ensure compliance with data governance policies and audit requirements through secure logging practices. Collaboration & Leadership Partner with application teams, SREs, and DevOps to onboard new data sources and build meaningful dashboards. Conduct knowledge-sharing sessions and mentor junior team members on best practices. Participate in architectural reviews and provide guidance on observability strategies across teams. Continuous Improvement Stay current with latest features, releases, and ecosystem tools. Contribute to the evolution of logging and monitoring standards across the organization. Identify opportunities to improve operational efficiency through enhanced visibility and automation. Requirements Splunk Certified Admin / Architect Proficient in scripting languages (Python, Bash, etc.) and automation tools (Puppet/Ansible/Terraform/Jenkins) Experience working within a SAFe environment, including participation in PI (Program Increment) Planning, Agile Release Trains (ARTs), and cross-functional collaboration across teams. Proficient in managing data inputs and universal forwarders Proficient in setting up users, roles, and authentication protocols to ensure secure access control Extensive knowledge of Splunk configuration files and their role in customizing deployments Experience in installation and management of apps within Splunk Cloud to extend functionality Skilled in monitoring, problem isolation, and system health checks to maintain performance Deep understanding of cloud platforms like Azure and GCP Demonstrated expertise in sizing, planning, and deploying distributed Splunk environments Familiarity with indexer and search head clustering Demonstrated expertise in onboarding diverse data sources and optimizing parsing for performance and scalability Experience integrating Splunk with other enterprise systems (CMDB, ticket tools, etc.) Creating intuitive, actionable dashboards according to best practices for Splunk performance

6-8 years
Required Skills & Qualifications 10 year’s experience in an infrastructure engineering or DevOps role 4 years hands-on experience with Azure, GCP, or AWS 2 years hands on experience engineering Kubernetes solutions and associated ecosystems Strong background in automation with modern automation tools like Terraform, Packer, and Puppet. Demonstrated abilities managing code and other project files with Git CI/CD experience with modern tooling like Jenkins or Azure DevOps Strong understanding of cloud networking concepts (Virtual Private Networks, Software Defined Networking, Network and Application Load Balancers, DNS, API Gateways, Routing). Solid grasp of cloud security principles and compliance standards. Proficiency in Linux and Windows system administration. Understanding of GitOps Excellent problem-solving and communication skills. Bachelor’s degree in computer science, engineering, or related field. Preferred Qualifications Certified Kubernetes Administrator (CKA) or Certified Kubernetes Application Developer (CKAD) certification Certifications in cloud technologies (e.g., AWS Certified Solutions Architect, Azure Administrator, etc). Experience with monitoring tools (e.g., Prometheus, Grafana). Familiarity with scripting languages (e.g., Bash, Python, PowerShell). Hands on experience with modern GitOps tools such as FluxCD or ArgoCD