Role: DataOps Engineer
Location: Pittsburgh, PA (Day 1 Complete Onsite)
Duration: Long Term
Skills:
- DataOps builds and manages the automated systems, pipelines, and tools that move and test data inside a company.
- Operate, automate, monitor, and continuously improve core data platforms to reduce waste, improve data flow, and ensure maximum uptime for structured datasets and analytics.
- Design and optimize robust data pipeline automation for ETL/ELT workloads, including orchestration, scheduling, and CI/CD to streamline data extraction and processing.
- Implement deep observability, manage logs, automate health checks, and rigorously track SLAs/SLOs to increase data reliability.
- Lead rapid incident response, stakeholder communications, and Root Cause Analysis (RCA) to continuously identify process gaps and correct them.
- Manage the end-to-end release lifecycle, executing automated testing (unit, performance, and end-to-end tests), smooth deployments, rollbacks, and performance tuning.
- Partner with managed service providers to test and adopt new solutions that adhere to DataOps best practices and ensure contract SLAs are met.
- Translate technical metrics into clear executive reports and facilitate collaboration with data and BI teams to enhance the quality of data products.
- Strong hands-on experience with AWS, Medallion architecture, and modern DataOps practices. Deep understanding of complex XML handling.
- Deep expertise in AWS Infrastructure (EC2, EKS, S3, Glue, Lambda), IAM, and ensuring stringent security standards are applied across all data pipelines.
- Proficient in Infrastructure as Code using Terraform and AWS CloudFormation.
- Design data engineering assets, develop reusable frameworks and patterns, enforce standards, and leverage AI-assisted coding tools (Codex).
- Strong FinOps awareness to monitor, control, and optimize cloud costs across large data environments.
- Highly proficient in Python, PySpark, and SQL.