At a glance
Research Crawling Engineer is an open full-time job at True Talent Solutions Partners (Remote · Remote).
| Company | True Talent Solutions Partners |
|---|---|
| Location | Remote · Remote |
| Type | Full-time |
| Department | Client Roles |
| Pay | Create a free account to see the pay |
| Posted | 2026-11-08 |
| Found on | Ashby |
About this job
About the Role
Join a data infrastructure team building large-scale web data collection systems for AI research and model development. This role focuses on reliable, high-volume crawling and dataset preparation across diverse and changing web environments.
What You'll Do
- Build and maintain web crawlers across social, travel, and multilingual websites.
- Design high-throughput, fault-tolerant systems capable of collecting millions to billions of URLs per day.
- Address anti-bot systems, rate limits, and dynamic websites using browser-based and other crawling techniques.
- Develop pipelines to clean, deduplicate, filter, and normalize web data at terabyte to petabyte scale.
- Collaborate with research teams to create datasets suited to model training and research needs.
- Monitor crawl coverage, performance, and data quality, adapting systems as websites change.
- Optimize crawling infrastructure for cost, latency, and reliability across cloud or bare-metal environments.
What We're Looking For
- At least 2 years of experience building web crawlers or large-scale data pipelines; typically 4 or more years, with 2 or more years considered for candidates with a PhD.
- Experience operating data pipelines at terabyte scale or larger, including high-volume crawling or data acquisition.
- Strong programming skills in Go, Rust, Python, Java, or C++.
- Deep knowledge of HTTP, networking, browser behavior, distributed systems, and parallel processing.
- Hands-on experience navigating anti-bot systems, rate limits, and JavaScript-heavy websites.
- Experience with headless browsers such as Playwright or Puppeteer, proxy systems, and IP rotation.
- Background in data quality evaluation, web data processing, or cloud and bare-metal infrastructure.
- Experience with NLP pipelines or machine learning dataset curation is a plus.
Compensation & Benefits
Visa sponsorship is not available.
Location
Remote.
Questions about this job
Where is the Research Crawling Engineer job at True Talent Solutions Partners?
Remote. The role can be done remotely.
What does the Research Crawling Engineer job at True Talent Solutions Partners pay?
The listing states a pay range. Create a free account on this site to see it.
Is this a full-time or part-time job?
The listing describes it as Full-time.
How do I apply to True Talent Solutions Partners?
Create a free job seeker account on this site, then use the Apply button to open the employer's own posting.
Hiring for this role at True Talent Solutions Partners? Create your hiring profile with your work email, so job seekers can find and connect with you.
All jobs at True Talent Solutions Partners · More jobs near Remote