Data Team Lead
incedoSan Rafael, CA
About the role
Databricks Data Lead
location: San Rafael, CA - Bay Area, California (Local candidates only)hybrid: 3 days/week onsite at client office inSan Rafael, CAExperience Level 8-12 years in data engineering, analytics engineering, or distributed data systems
Role Overview:
We are seeking aDatabricks Data Leadto support the design, implementation, and optimization of cloud-native data platforms built on the Databricks Lakehouse Architecture. This is a hands‑on, engineering‑driven role requiring deep experience with Apache Spark, Delta Lake, and scalable data pipeline development, combined with early-stage architectural responsibilities. The role involves close onsite collaboration with client stakeholders, translating analytical and operational requirements intorobust, high‑performance data architectures, while adhering to best practices for data modeling, governance, reliability, and cost efficiency.
Key Responsibilities:
- Design, develop, and maintainbatch and near‑real‑time data pipelinesusing
- Databricks, PySpark, and Spark SQLImplementmedallion (Bronze/Silver/Gold) lakehouse architectures, ensuring proper data quality, lineage, and transformation logic across layers
- Build and manage
- Delta Lake tables, including schema evolution, ACID transactions, time travel, and optimized data layouts
- Apply performance optimization techniques such as partitioning strategies, Z‑Ordering, caching, broadcast joins, and Spark execution tuning
- Support dimensional and analytical data modeling for downstream consumption by BI tools and analytics applications
- Assist in defining data ingestion patterns (batch, incremental loads, CDC, and streaming where applicable)
- Troubleshoot and resolve pipeline failures, data quality issues, and Spark job performance bottlenecks
- Collaborate onsite with client data engineers, analysts, and business stakeholders to gather technical requirements, validate implementation approaches, and maintain technical documentation covering data flows, transformation logic, table designs, and architectural decisions
- Contribute to code reviews, CI/CD practices, and version control workflows to ensure maintainable and production‑grade solutions
- Required Skills & Qualifications
- Strong hands‑on experience withDatabricks Lakehouse Platform
- Deep working knowledge ofApache Spark internals, including:
Spark SQLDataFrames/Datasets Shuffle behavior and execution plans AdvancedPython (PySpark)andSQLdevelopment skills Solid understanding ofdata warehousing concepts, including:
Star and snowflake schemas Analytical vs operational workloads Experience working withcloud data platformsonAWS, Azure, or GCPPractical experience with Delta Lake, including:
Schema enforcement and evolution Data compaction and optimization Proficiency with Git‑based version controland collaborative development workflows Strong verbal and written communication skills for client‑facing technical discussions Ability and willingness to work onsite 3 days/week in San Rafael, CANice‑to‑Have Skills Exposure toDatabricks Unity Catalog, data governance, and access control models Experience withDatabricks Workflows , Apache Airflow, orAzure Data Factoryfor orchestration Familiarity with streaming frameworks (Spark Structured Streaming, Kafka) and/or CDC patterns Understanding of data quality frameworks, validation checks, and observability concepts Experience integrating Databricks with BI tools such as Power BI, Tableau, or Looker Awareness of cost optimization strategies in cloud‑based data platforms EducationBachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience) Why This RoleHands‑on ownership of Databricks Lakehouse implementations in a real‑world enterprise environment Direct client‑facing exposure with a leading Bay Area organization Opportunity to evolve from senior data engineering into formal data architecture responsibilities Strong growth path toward Senior Databricks Architect / Lead Data Platform Engineer Seniority LevelMid‑Senior level Employment TypeFull‑time Job Function ConsultingIndustriesIT Services and IT Consulting Hospitals and Health Care Biotechnology Research
Benefits: Medical insurance Vision insurance 401(k)
Matching similar jobs
JOB OVERVIEW
Experience level
Lead
Location
San Rafael, CA
Occupation
Data Warehousing Specialists
Industry
Computer Systems Design Services
Posted
yesterday
Tired of running searches?
Rank the roles you'd take once, and matches like these arrive on their own.
CREATE PROFILE