Build large-scale web crawling and data acquisition infrastructure for AI research and model development. You will work with research teams to create reliable, high-quality datasets from a changing and often adversarial web.
Build and operate crawlers across diverse domains, including social media, travel, and multilingual websites.
Design high-throughput, fault-tolerant systems that collect millions to billions of URLs per day.
Handle anti-bot systems, rate limits, and dynamic, JavaScript-heavy sites.
Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.
Work with research teams to align data collection and dataset quality with modeling needs.
Monitor crawl coverage, performance, and data quality, and optimize infrastructure for cost, latency, and reliability.
At least 2 years of experience building web crawlers or large-scale data pipelines, with 4 or more years preferred; candidates with PhDs may qualify with 2 or more years.
Experience operating distributed data pipelines at terabyte scale and building high-throughput, fault-tolerant data acquisition systems.
Strong proficiency in Go, Rust, Python, Java, or C++, with deep knowledge of HTTP, networking, and browser behavior.
Hands-on experience addressing anti-bot systems, rate limits, and JavaScript-heavy websites; experience with proxies, IP rotation, and request orchestration is valuable.
Familiarity with headless browsers such as Playwright or Puppeteer, and cloud or bare-metal infrastructure.
Experience with data quality evaluation, NLP pipelines, or machine learning dataset curation is a plus.
Base salary is $160,000 to $250,000 USD annually. Visa sponsorship is not available.
Remote. No primary location was specified.