Logo
SEARCH JOBSPARTNERSHIPSIGN INCREATE PROFILE

Director of Software Engineering (Node.js & Web Scraping Expert)

roman health pharmacyLos Angeles, CA

2 days ago

Occupations

Software DevelopersComputer Systems Engineers/ArchitectsWeb Developers

Industries

Computer Systems Design ServicesSoftware PublishersCustom Computer Programming Services
APPLY NOW

About the role

We are seeking aDirector of Software Engineeringwith deep expertise inNode.js developmentandlarge‑scale web scraping. This role will lead theengineering team, designing and optimizinghigh‑performance, distributed web scraping systems. The ideal candidate hasextensive experience in handling anti‑bot measures, data pipeline optimization, and scalable cloud‑based architectures .
Key Responsibilities:
  • Software Engineering & Web Scraping Leadership
  • Architect, develop, and maintainscalable and distributed web scraping systemsusing
  • Node.js .Design and implementdata extraction pipelinesto process large volumes of structured and unstructured data.
Develop solutions tobypass anti‑bot mechanisms, includingCAPTCHA handling, session management, fingerprinting, and IP rotation .Optimize scraping processes forperformance, reliability, and efficiencywhile managingproxy services(residential, datacenter, rotating). Overseedata storage and processing strategies, ensuring high availability and consistency. Collaborate with Product, DevOps, and Data Science teamsto integrate extracted data into analytics and business applications. Implementbest practices for microservices, API integrations, and real‑time data streaming .
Key Responsibilities:
  • Scalability, Security & DevOpsLead the transition tocloud‑native, containerized, and serverless architecturesfor web scraping.
  • Ensure compliance withlegal and ethical standards(robots.txt, GDPR, CCPA, etc.). Optimizecloud resources(AWS, GCP, or Azure) to support high‑throughput scraping.
  • Managereal‑time monitoring and alertingsystems to detectscraping failures, IP bans, or performance bottlenecks .Work closely with
  • DevOps teamsto optimizeCI/CD pipelines, automated deployments, and system scalability .Key Resposnibilities – Engineering Team Management & Strategy
  • Lead, mentor, and grow ahigh‑performance engineering team .Define and execute thetechnology roadmap, aligning with business objectives.
  • Foster a culture ofcontinuous learning, collaboration, and innovation .Implementagile development methodologies (Scrum, Kanban)to optimize project execution.
  • Ensurecode quality, security, and best practicesacross all engineering efforts.
Qualifications & Experience: Technical Expertise10+ yearsof experience insoftware engineering, with at least5+ yearsinweb scraping and large‑scale data extraction .Strong hands‑on expertise inNode.js, Puppeteer, Playwright, Cheerio, Selenium, and headless browser automation .Extensive experience inhandling CAPTCHAs, IP rotation, session management, and anti‑bot evasion techniques .Deep knowledge ofproxy management (residential, datacenter, rotating, and VPNs). Experience with NoSQL/SQL databases (MongoDB, PostgreSQL, Redis, Elasticsearch, etc.) .Familiarity withdata processing frameworks (Kafka, RabbitMQ, Spark, Airflow, etc.)Strong experience withCI/CD, containerization (Docker, Kubernetes), and cloud deployment (AWS/GCP/Azure) .
Qualifications & Experience: Leadership & Soft SkillsProven track record ofscaling engineering teams and leading complex projects .Strongproblem‑solving and debuggingskills, especially forscraping challenges and performance bottlenecks .Excellentcommunication and stakeholder managementskills. Passion formentorship, team development, and continuous learning .
Preferred Qualifications: Experience withmachine learning for data extraction and NLP .Knowledge ofbrowser fingerprinting and bot detection mechanisms .Familiarity withenterprise‑scale web crawling frameworks(Scrapy, Colly, Apify, etc.).Prior leadership experience indata‑driven businesses or web scraping startups .

Matching similar jobs

JOB OVERVIEW

Experience level

Manager

Location

Los Angeles, CA

Occupation

Software Developers

Industry

Computer Systems Design Services

Posted

2 days ago

Tired of running searches?

Rank the roles you'd take once, and matches like these arrive on their own.

CREATE PROFILE