Ricardo Borges

Ricardo Borges

Personal blog

Web Scraping problems and their solutions

So recently I decided to build my own jobs agregator, I thought: If all job posts in ATSs are open and public, I could easily scrape them, right? No. At least no “easily”. I faced some scraping challenges that are, luckly, well-known challenges with equally well-known solutions, and I figured it is good topic for a post, so here we go.

TL;DR

  • Web scraping is the automated process of extracting data from websites and converting it into a structured format.
  • Request Flooding & IP Bans: Use a FIFO queue and limit how many requests run at once.
  • Bot-Revealing Request Headers: Rotate the headers, for example the User-Agent header.
  • Predictable Request Timing: Add a random delay before each request (jitter)
  • Retry Storms on Temporary Failures: Appy exponencial backoff and jitter.
  • Hung Connections: Set a timeout for the requests.

What is Web Scraping

Web scraping is the automated process of extracting data from websites and converting it into a structured format. In this post I will list problems found during the implementation of a scraper and their solutions.

Request Flooding & IP Bans

Firing every request at once floods the target site, which answers with 429 Too Many Requests or bans your IP. The solution is to put a cap on how many requests run at once, and make every extra request wait in a FIFO queue:

  1. A request asks for a slot. If one is free, it starts and the active counter goes up by 1.
  2. If every slot is taken, the request joins the back of the queue and waits.
  3. When a request finishes, whether it worked or failed, it gives its slot back and the first request in the queue starts.

Bot-Revealing Request Headers

By default, HTTP libraries announce themselves in the User-Agent header (axios/1.6.0, python-requests/2.31, undici), and firewalls block those names on sight. Sending thousands of requests with the same identity also stands out, and so do headers that contradict each other (for example, a Windows User-Agent next to macOS client hints).

The solution: User-Agent & Header Rotation.

Send the full, consistent set of headers that a real, recent browser sends, and rotate between several browser profiles:

  1. Keep a pool of up-to-date browser profiles: Chrome, Firefox, and Safari on Windows, macOS, and Linux.
  2. Pick one profile per request for stateless APIs, or keep the same profile for the whole session when crawling across several pages.
  3. Send the profile's complete header set, not just the User-Agent:
1{
2  "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
3  "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
4  "Accept-Language": "en-US,en;q=0.9",
5  "sec-ch-ua-platform": "\"macOS\"",
6  "sec-ch-ua-mobile": "?0"
7}

As you can see it’s not compicated, here’s a pro tip: drop browser versions older than 6–12 months. An outdated version is a strong sign of a bot.

Predictable Request Timing

Scrapers tend to send requests at perfectly even intervals, such as exactly every 500ms. Anti-bot systems measure the gaps between requests, and gaps that never vary are an obvious sign of a machine. Workers that finish together also fire their next requests together, which looks like a burst.

The solution is a request jitter, sounds complicated, but it just consists on adding a random delay before each request.

Retry Storms on Temporary Failures

Servers fail temporarily (502, 503, 504, dropped connections) or throttle you (429). Retrying right away, or after the same fixed delay for every worker, sends a new spike of traffic at the moment the server is trying to recover. This is called the thundering herd and we don’t want that.

Solution: Exponential Backoff with Jitter

After each failure, wait twice as long as the time before (exponential backoff), and randomize each wait so workers retry at different times (jitter).

However be aware that not all responses need a retry, here’s a table that tells when it’s worth to retry the request:

ResponseAction
2xx, 304Success. Return the data.
429, 502, 503, 504, timeoutRetry with backoff.
400, 404, 422 , 401, 403Fail right away. Retrying gives the same result.

Also, if the server sends a Retry-After header, use that value instead. Limit retries to 2–4 attempts, and cap the wait at 30–60 seconds.

Hung Connections

A server or firewall can silently stop replying. Without a timeout, the request may hang for hours. Some anti-bot systems do this on purpose ("tarpitting"), trickling out 1 byte every few seconds to tie up your workers and use up memory.

The solution is a old and good timeout. Give every request a deadline and cancel it when the deadline passes.

github iconlinkedin icon
Buy Me A Coffee