Web Crawling Engineer
We are partnering with a fast-growing AI infrastructure company building foundational technology powering the next generation of AI-native search and retrieval systems. The team is building large-scale infrastructure to crawl and index the web, train advanced embedding/retrieval models, and operate high-performance distributed systems at massive scale.
They are looking for experienced engineers who enjoy solving deep infrastructure and internet-scale engineering problems.
The Role
As a Web Crawler Engineer, you will help design and build internet-scale crawling infrastructure capable of processing hundreds of millions of webpages efficiently and reliably.
This role sits at the intersection of distributed systems, performance engineering, browser automation, and large-scale data infrastructure.
You will work on problems such as:
- Distributed web crawling at massive scale
- Intelligent crawl scheduling and prioritisation
- JavaScript-heavy and dynamic site rendering
- Anti-bot detection and evasion handling
- Crawl politeness, rate limiting, and domain-aware orchestration
- High-throughput data pipelines and infrastructure optimisation
Responsibilities
- Build and scale distributed web crawling systems processing 100M+ pages daily
- Design highly performant infrastructure for crawl orchestration and scheduling
- Improve handling of dynamic content and JavaScript-rendered websites
- Work with browser automation tooling and Chrome DevTools Protocol (CDP)
- Optimise crawling efficiency, reliability, and resource utilisation
- Develop systems for rate limiting, crawl politeness, and domain management
- Contribute to low-level performance optimisation across the stack
Requirements
- Strong experience building scalable backend or distributed systems
- Prior experience working on web crawlers, scraping infrastructure, search infrastructure, browser automation, or adjacent systems
- Experience with high-performance languages such as Rust, C++, or Go
- Familiarity with TypeScript, Playwright, Puppeteer, or browser automation tooling
- Understanding of modern web technologies and dynamic rendering
- Strong systems thinking and performance optimisation mindset
- Interest in AI infrastructure, search, and knowledge retrieval systems
Nice to Have
- Experience with Chrome DevTools Protocol (CDP)
- Experience handling anti-bot systems at scale
- Experience with distributed job orchestration systems
- Exposure to search engines, indexing systems, or retrieval infrastructure
- Kubernetes / cloud infrastructure experience