Job Title: Kafka/Spark Developer
Location: Pittsburgh, Pennsylvania
Type: Permanent Full Time
Work Model: Hybrid – onsite and remote
Responsibilities
- Develop and maintain scalable big data solutions using Hadoop, Spark, Kafka, and Impala to support enterprise data processing and analytics initiatives.
- Design, build, and optimize batch and real-time data pipelines for ingesting, processing, transforming, and delivering large volumes of structured and unstructured data.
- Develop Spark applications using PySpark, Scala, or Java for data transformation, aggregation, cleansing, and analytical processing.
- Build and maintain Kafka producers, consumers, topics, and streaming workflows to enable reliable real-time data ingestion and event-driven architectures.
- Design and implement logical and physical data models to support data warehousing, reporting, analytics, and business intelligence requirements.
- Monitor, troubleshoot, and tune Kafka and Spark streaming jobs to improve performance, scalability, and operational reliability.
- Optimize Hadoop ecosystem components, Spark jobs, Kafka configurations, and Impala queries to improve system performance and resource utilization.
- Collaborate with architects, data engineers, DevOps teams, and business stakeholders to design and implement modern streaming and event-driven data platforms.
- Analyze user requirements, and define technical project scope and assumptions for assigned tasks.
- Create technical designs for new systems, and/or modifications to existing systems.
- Translate detailed requirements into functional system designs.
- Prioritize work, meet deadlines, and establish and maintain effective working relationships with clients, project team members, supervisors, and employees from other departments.
- Partner with business leaders, enterprise architects, and product owners to identify new graph-based use cases, evaluate emerging technologies, and align Neo4j initiatives with digital transformation goals.
Requirements
- At least 5+ years of experience in Big Data development, data engineering, or distributed data processing environments.
- Strong hands-on experience with Apache Kafka, topic configuration, producer/consumer development, Kafka Connect, and Schema Registry.
- Extensive experience developing real-time data processing applications using Apache Spark Streaming and/or Spark Structured Streaming.
- Proficiency in Java, Scala, or Python (PySpark) with strong object-oriented programming and software development skills.
- Proficiency in writing and optimizing complex SQL queries using Impala, Hive, or similar distributed query engines.
- Hands-on experience with Hadoop ecosystem components including HDFS, Hive.
- Experience integrating Kafka and Spark with relational databases, NoSQL databases, cloud storage platforms, and enterprise applications.
- Strong analytical, troubleshooting, and performance tuning skills in distributed streaming environments.
- Excellent communication, collaboration, and stakeholder management skills, with the ability to work effectively in Agile/Scrum teams.
- Experience working in Agile development environments with strong collaboration, technical leadership, problem-solving, and stakeholder communication skills.
#M-
#LI-
Ref: #404-IT Pittsburgh