Job Title: Kafka/Spark Developer
Location: Pittsburgh, PA | Onsite
Duration: Full-time
Job Description:
Our client is looking for mid level Kafka and Spark Software Developers to join Applications Development and Maintenance team, supporting our client which is a large Client, working in an advanced technology environment.
Duties and Responsibilities:
- As Kafka Spark Software developers, you will be responsible for developing and maintaining scalable big data solutions using Hadoop, Spark, Kafka, and Impala to support enterprise data processing and analytics initiatives.
- Design, build, and optimize batch and real-time data pipelines for ingesting, processing, transforming, and delivering large volumes of structured and unstructured data.
- Develop Spark applications using PySpark, Scala, or Java for data transformation, aggregation, cleansing, and analytical processing.
- Build and maintain Kafka producers, consumers, topics, and streaming workflows to enable reliable real-time data ingestion and event-driven architectures.
- Design and implement logical and physical data models to support data warehousing, reporting, analytics, and business intelligence requirements.
- Monitor, troubleshoot, and tune Kafka and Spark streaming jobs to improve performance, scalability, and operational reliability.
- Optimize Hadoop ecosystem components, Spark jobs, Kafka configurations, and Impala queries to improve system performance and resource utilization.
- Collaborate with architects, data engineers, DevOps teams, and business stakeholders to design and implement modern streaming and event-driven data platforms.
- Analyzing user requirements, and defines technical project scope and assumptions for assigned tasks.
- Creating technical designs for new systems, and/or modifications to existing systems.
- Translating detailed requirements into functional system designs.
- Prioritizing work, meeting deadline and also establishing and maintaining effective working relationships with clients, project team members, supervisors, and employees from other departments.
- Partner with business leaders, enterprise architects, and product owners to identify new graph-based use cases, evaluate emerging technologies, and align Neo4j initiatives with digital transformation goals.
Requirements:
- At least 5+ years of experience in Big Data development, data engineering, or distributed data processing environments.
- Strong hands-on experience with Apache Kafka, topic configuration, producer/consumer development, Kafka Connect, and Schema Registry.
- Extensive experience developing real-time data processing applications using Apache Spark Streaming and/or Spark Structured Streaming.
- Proficiency in Java, Scala, or Python (PySpark) with strong object-oriented programming and software development skills.
- Proficiency in writing and optimizing complex SQL queries using Impala, Hive, or similar distributed query engines.
- Hands-on experience with Hadoop ecosystem components including HDFS, Hive
- Experience integrating Kafka and Spark with relational databases, NoSQL databases, cloud storage platforms, and enterprise applications.
- Strong analytical, troubleshooting, and performance tuning skills in distributed streaming environments.
- Excellent communication, collaboration, and stakeholder management skills, with the ability to work effectively in Agile/Scrum teams.
- Experience working in Agile development environments with strong collaboration, technical leadership, problem-solving, and stakeholder communication skills.