Research Crawling Engineer
Posted 27 OctFound 27 OctClosed 1 Nov
Fit
The keywords the posting names, against what your resume says.Add your resume to see which of its keywords you already have, and get a resume tailored to this posting. 10 a month, free.
The Posting
ABOUT THE ROLE
Join a data infrastructure team building large-scale web crawling systems and datasets for AI research and model development. This role focuses on reliable data acquisition across diverse, changing web environments.
WHAT YOU'LL DO
- Build and maintain web crawlers across social, travel, and multilingual websites.
- Design high-throughput, fault-tolerant systems that collect millions to billions of URLs per day.
- Adapt crawling approaches to anti-bot systems, rate limits, and dynamic or JavaScript-heavy sites.
- Develop pipelines to clean, deduplicate, filter, and normalize web data at TB-to-PB scale.
- Build and maintain datasets for research and model training in collaboration with research teams.
- Monitor crawl performance, coverage, and data quality, and adjust systems as websites change.
- Optimize crawling infrastructure for cost, latency, and reliability across cloud or bare-metal environments.
WHAT WE'RE LOOKING FOR
- At least 2 years of experience building web crawlers or large-scale data pipelines; typically 4 or more years, or 2 or more years with a PhD.
- Hands-on experience building and operating web crawlers at scale, including data pipelines handling TB-scale workloads.
- Strong proficiency in Go, Rust, Python, Java, or C++, with experience in distributed systems or parallel processing.
- Deep knowledge of HTTP, networking, and browser behavior, including experience working with anti-bot systems, rate limits, and JavaScript-heavy sites.
- Experience with headless browsers, proxy systems, IP rotation, and request orchestration is valuable.
- Familiarity with data quality evaluation, large-scale data processing, and cloud or bare-metal infrastructure.
- Experience with NLP pipelines, machine learning dataset curation, or AI research data is a plus.
- A degree in computer science, engineering, or a related technical field is helpful; a PhD is a plus.
COMPENSATION & BENEFITS
Salary range: $160,000 to $250,000 annually. Visa sponsorship is not available.
LOCATION
Remote.
Facts
- Source
- Ashby
- External id
- 0780dffa-32d3-4ed3-859f-e4aab270f840
- Location
- Remote, United States
- Remote
- Remote, from the posting
- Employment
- Full time
- Pay
- $160,000 to $250,000, read from the text
- Posted
- 27 Oct 2026, 17:55
- First seen
- 27 Oct 2026, 21:26
- Closed
- 1 Nov 2026, 03:26
- Status
- Closed
- Posting id
- ec8b7dad-defb-5806-9a01-e46c840ba8fe