Big Data Engineer

Pay Rate: DOEWe are seeking an experienced Senior Data Engineer to design, develop, and optimize large-scale data processing pipelines in a cloud-based environment. This role requires strong software engineering fundamentals, hands-on experience with Big Data technologies, and the ability to solve complex technical challenges involving high-volume data processing. The ideal candidate is a highly analytical problem solver who can independently investigate issues, develop scalable solutions, and collaborate with cross-functional teams to deliver reliable, high-performance data systems.

Key Responsibilities: Design, develop, and maintain scalable data processing pipelines using Big Data technologies and AWS cloud services. Collaborate with engineering teams and business stakeholders to translate requirements into effective technical solutions. Optimize existing data pipelines to improve performance, scalability, efficiency, and reliability. Develop and maintain automated unit, integration, and end-to-end tests to ensure data quality and system stability. Troubleshoot complex data processing challenges, including data skew, resource constraints, job failures, and data discrepancies. Monitor production data pipelines, proactively identify issues, and implement sustainable solutions. Apply software engineering best practices, including code quality, test automation, and continuous integration. Evaluate technical approaches and recommend improvements to data architecture and processing capabilities. Required

Qualifications: Bachelor's degree in Computer Science, Engineering, or a related technical field with 8+ years of relevant experience, or an advanced degree with 5+ years of experience. Strong hands-on experience with Big Data technologies, particularly Apache Spark and Hadoop. Experience developing and optimizing large-scale data processing pipelines, including working with high-volume or petabyte-scale datasets. Proficiency in Python, SQL, or another programming language used for data engineering. Hands-on experience with AWS services such as S3, EMR, Glue, Lambda, Athena, or Kubernetes. Strong understanding of distributed data processing, performance optimization, data skew, and resource management. Experience diagnosing and resolving complex pipeline failures, scalability limitations, and data quality issues. Knowledge of object-oriented programming, database technologies, and enterprise software development practices. Experience with automated testing, build automation, and configuration management. Demonstrated ability to independently solve complex technical problems and develop practical, scalable solutions. Strong analytical, troubleshooting, and communication skills, with the ability to explain technical concepts to diverse audiences. Preferred

Qualifications: Advanced degree in Computer Science, Engineering, or a related discipline. Experience working with large-scale enterprise data platforms in cloud environments. Familiarity with financial services, regulatory reporting, or U.S. financial market structure. Understanding of evolving industry regulations and their impact on data processing and reporting. Contributions to open-source projects, technical publications, or engineering communities. Relevant AWS certifications.

Ideal Candidate: The successful candidate will bring a combination of strong coding ability, deep technical expertise, and exceptional problem-solving skills. We are particularly interested in engineers who can demonstrate ownership of challenging technical problems, explain their approach to troubleshooting and optimization, and deliver scalable solutions in complex data environments. This position is well suited for someone who enjoys hands-on engineering, working with large datasets, and solving difficult problems that require both technical depth and creative thinking. Welcome to Consult Net, a premier national provider of technology talent and solutions. Our expertise spans across project services, contract-to-hire, direct search, and managed services onshore, nearshore, and hybrid. For over 25 years, we have connected thousands of consultants with meaningful roles through a personal, communication-driven approach, partnering with a diverse client base to build high-performing teams and create lasting impact. Our comprehensive service offerings cover a wide range of technology and engineering positions across key markets nationwide. xhyhwjd Learn more at We champion equality and inclusivity, proudly supporting an Equal Opportunity Employer policy. We welcome applicants regardless of Race, Color, Religion, Sex, Sexual Orientation, Gender Identity, National Origin, Age, Genetic Information, Disability, Protected Veteran Status, or any other status protected by law.