Python backend engineer with PySpark/Spark

Rate: $57/hrRole Summary- Senior backend engineer building the core integrations between Unity and the enterprise Spark engine and SparkFlow framework- Designs distributed, high-throughput systems that power data processing and integration workflows at enterprise scale- Operates with significant autonomy, setting technical direction within a regulated, large-scale engineering environmentKey ResponsibilitiesPlatform & Integration Engineering- Design, build, and operate integrations between Unity and the Spark engine and SparkFlow framework- Develop robust APIs, SDKs, and connectors enabling reliable data movement and workflow orchestration- Implement abstractions that simplify SparkFlow adoption for downstream platform consumersDistributed Systems Design- Architect scalable, fault-tolerant systems handling large data volumes and concurrent workloads- Apply best practices for performance tuning, caching, partitioning, backpressure, and resilience- Drive design reviews, RFCs, and architectural decisions for core platform componentsReliability & Operations- Build systems with first-class observability (metrics, logs, traces) and SLOs- Establish operational excellence: runbooks, alerting, capacity planning, and incident response- Champion automated testing, and progressive delivery practicesRequired Qualifications- 6 years of backend engineering experience, with deep expertise in distributed systems- Expert-level Python; strong API design (REST, gRPC) and asynchronous programming- Proven track record delivering large-scale, production-grade systems- Strong fundamentals in concurrency, data structures, and systems performance experiencePreferred Qualifications- Hands-on experience with Apache Spark (PySpark, Spark SQL, Structured Streaming)- Experience building or integrating with workflow/data orchestration frameworks (Airflow, Autosys)- Exposure to greenfield platform builds in regulated financial enterprises- Experience with event-driven architectures (Kafka) and lakehouse formats (Iceberg, Delta, Hudi)

Technical Skills: Languages: Python (expert), Scala or Java (plus), SQL- Distributed Compute: Apache Spark- Storage & Data: S3/ADLS/GCS, Iceberg, Delta Lake, Parquet, Postgres- Streaming & Messaging: Kafka, Pub/Sub- APIs & Services: REST, gRPC, GraphQL, OpenAPI- Containers & Cloud: Docker, Kubernetes, Azure/GCP, Terraform- CI/CD: GitHub ActionsSoft Skills & Leadership Expectations - Strong technical judgment and ability to operate