Skip to main content

Glossary

Rate Limiting

Rate limiting is a server-side control that caps the number of requests an IP address, API key, or user can make within a time window. Exceeding the limit typically returns HTTP 429 (Too Many Requests) and may trigger a temporary or permanent block.

For scrapers, working within or around rate limits is the practical difference between a sustainable pipeline and a blocked one. Well-designed scraping systems add exponential back-off, request jitter, and per-domain throttling to stay under detection thresholds. Many APIs publish their rate limits explicitly in documentation; websites rarely do, requiring empirical testing to find safe request rates.