GlossaryProtocolsBeginner

HTTP Headers

HTTP headers are small pieces of metadata sent with every web request and response, describing things like the browser, content type, and cookies. They quietly shape how servers treat your traffic.

Last updated June 8, 2026

Definition

HTTP headers are key-value pairs of metadata exchanged between a client and server with each request and response. They convey context the actual content does not - such as which browser is being used, what languages are accepted, what cookies are set, and how content should be cached.

How they work

A request includes headers like User-Agent, Accept-Language, Referer, and Cookie, while a response carries headers like Content-Type, Set-Cookie, and Cache-Control. Servers read request headers to decide how to respond and whether the visitor looks legitimate.

Why they matter for scraping

  • Bot detection - missing, inconsistent, or default headers (like a Python library's user agent) are a classic giveaway of automation.
  • Realistic fingerprints - matching the full header set and order of a real browser helps requests blend in.
  • Session control - the Cookie header carries login and session state across requests.

Crafting authentic headers is one of the most important and cost-free techniques for avoiding blocks, since servers and anti-bot systems heavily rely on them to separate humans from scrapers.

Examples

1

A User-Agent header identifying the request as Chrome on Windows

2

An Accept-Language header telling the server to return English content

3

A Set-Cookie response header establishing a login session

Common Use Cases

Setting realistic browser headers to avoid being flagged as a bot
Passing authentication and session state through the Cookie header
Sending an Accept-Language header to retrieve localized content
Debugging why a server returns different responses to your scraper versus your browser

Frequently Asked Questions

They carry metadata about a request or response - such as the browser type, accepted languages, cookies, and content type - that servers use to decide how to handle the traffic.
Anti-bot systems inspect headers to spot automation, so sending a complete, realistic, browser-like set of headers helps your scraper avoid detection and blocks.
The User-Agent header is the most common culprit, since default values from libraries like Python requests instantly reveal automated traffic.

Keep Learning

All terms

Web Scraping

Web scraping is the automated extraction of data from websites — fetching pages programmatically and parsing their content into structured data.

Read definition

HTTP Proxy

An HTTP proxy is an intermediary server that forwards web (HTTP/HTTPS) requests on your behalf, able to read, cache and filter traffic at the application layer.

Read definition

Browser Fingerprinting

Browser fingerprinting identifies and tracks a device by combining dozens of browser and system attributes — like fonts, canvas rendering and user agent — into a near-unique signature.

Read definition

User Agent

A user agent is the identifying string a browser sends with every request, telling the server which browser, version and operating system you are using.

Read definition
Back to Glossary