Lead Data Engineer

Type: Fulltime

Location: Richmond, VA (Hybrid)

Qualifications: • 8-10 years of experience as a Lead Data Engineer with a strong focus on Databricks & Azure Data Factory.• Strong experience leading the Data Engineering teams working on ADF & Databricks. Work with the teams providing technical approaches and solutions.• Proven experience with data warehousing concepts and best practices.• Strong SQL skills, ability to perform effective querying involving multiple tables and subqueries.• Strong understanding of data modeling and data quality principles.• Strong communication & collaboration skills, should be able to work closely with customer driving technical conversations.• Experience with Agile development methodologies.• Ability to work independently and as part of a team.• Lead team and drive through technical challenges and solutions. Drive technical calls with customer• Focus on continuous value additions and innovations• Experience with any other ETL tool (Talend, Informatica or similar)• Good to have Insurance domain knowledge• Good to have exposure to Gen AI concepts Databricks• Hands-on experience with Databricks features such as delta lake, clusters, notebooks, jobs, and workspaces.• Design, develop, and deploy data pipelines using Databricks, including data ingestion, transformation, and loading (ETL) processes.• Develop and maintain high-quality, scalable, and maintainable Databricks notebooks using Python.• Work with Delta Lake and other advanced features.• Leverage Unity Catalog for data governance, access control, and data discovery.• Experience working with Parquet files & Delta files for data storage and processing.• Integrate with various data sources, including but not limited to databases and cloud storage (Azure Blob Storage, ADLS, Synapse), and APIs.• Perform data quality checks and validation to ensure data accuracy and integrity.