Signed Requests: Reverse-Engineering HMAC

When an API uses HMAC signing, every request carries a header like X-Signature: <hex> computed from the body, the URL, the timestamp, and a secret. The server recomputes the signature on its side; if they don't match, the request is rejected.

For scrapers, this is one of the harder auth patterns. The "secret" might be in the JS bundle (good, find it), or derived through a chain of steps (bad, read the bundle carefully).

How to spot it

Signs in DevTools:

• Headers like X-Signature, X-HMAC-Signature, X-Hub-Signature, Signature, X-Verification-Hash.
• A X-Timestamp or X-Date header sent alongside.
• Maybe an X-Nonce or X-Request-Id for replay protection.
• The signature changes every request (unlike a static API key).

Body changes → signature changes. URL changes → signature changes. That's the giveaway.

The classic recipe

A standard HMAC request is built like:

canonical_string = METHOD + "\n" + PATH + "\n" + TIMESTAMP + "\n" + BODY
signature = hex(hmac_sha256(secret, canonical_string))

Headers:

X-Timestamp: 1732567800
X-Signature: 3a2f1e...

Variations:

• Some include the body hash separately (X-Body-SHA256: ...) and sign the hash, not the body.
• Some include the query string in the canonical_string.
• Some include selected request headers (Host, Content-Type).
• Some use base64 instead of hex.
• Some use HMAC-SHA1 (rare in 2026).

Catalog108 example

The /challenges/api/auth/hmac-signed lab uses:

canonical = METHOD + "\n" + PATH + "\n" + TIMESTAMP + "\n" + BODY
signature = hex(hmac_sha256("catalog108-hmac-secret", canonical))

Headers:

• X-Timestamp: <unix-seconds>
• X-Signature: <hex>

The secret is embedded in the JS bundle (lesson 3.20 pattern).

Python implementation

import time, hmac, hashlib, json, requests

SECRET = b"catalog108-hmac-secret"
BASE = "https://practice.scrapingcentral.com"

def sign(method: str, path: str, body: str = "") -> dict[str, str]:
  ts = str(int(time.time()))
  canonical = f"{method}\n{path}\n{ts}\n{body}"
  sig = hmac.new(SECRET, canonical.encode(), hashlib.sha256).hexdigest()
  return {"X-Timestamp": ts, "X-Signature": sig}

def get_signed(path: str):
  headers = sign("GET", path)
  return requests.get(f"{BASE}{path}", headers=headers)

def post_signed(path: str, body: dict):
  body_str = json.dumps(body, separators=(",", ":"), sort_keys=True)
  headers = sign("POST", path, body_str)
  headers["Content-Type"] = "application/json"
  return requests.post(f"{BASE}{path}", headers=headers, data=body_str)

print(get_signed("/challenges/api/auth/hmac-signed").json())

PHP version

function sign(string $method, string $path, string $body = ''): array {
  $secret = 'catalog108-hmac-secret';
  $ts = (string) time();
  $canonical = "{$method}\n{$path}\n{$ts}\n{$body}";
  $sig = hash_hmac('sha256', $canonical, $secret);
  return ['X-Timestamp' => $ts, 'X-Signature' => $sig];
}

use GuzzleHttp\Client;
$client = new Client(['base_uri' => 'https://practice.scrapingcentral.com']);

$headers = sign('GET', '/challenges/api/auth/hmac-signed');
$res = $client->get('/challenges/api/auth/hmac-signed', ['headers' => $headers]);
echo $res->getBody()->getContents();

Body canonicalization, the silent killer

For POST/PUT, the body's exact bytes must match what the server signs. Common traps:

• Whitespace. {"a":1} and {"a": 1} produce different signatures. Always serialize identically.
• Key order. {"a":1,"b":2} vs {"b":2,"a":1}, different. Some servers require sorted keys; others require the JS-serialization order. Match exactly.
• Unicode escaping. é vs é. Pick one and stick with it.
• Trailing newline. Some serializers add \n; some don't.

The fix: use the same canonical form the client JS uses. In Python, json.dumps(obj, separators=(",", ":"), sort_keys=True) produces a compact, sorted form that's common. PHP: json_encode($obj, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE).

When in doubt, capture a real request, examine the raw body bytes, and replicate exactly.

Reverse-engineering the signing algorithm

When the algorithm isn't documented (i.e. always), the workflow:

1. Capture a real signed request in DevTools. Note the body, timestamp, and signature values.
2. Open Sources → search for the signature header name (X-Signature) in the bundle.
3. Set a breakpoint on the line that sets the header. Trigger the request. When paused, walk back through the call stack to find where the signature is computed.
4. Read the function: what's the input (canonical string)? What's the algorithm (sha256? sha1? base64?)?
5. In Python, reproduce. Set the same timestamp and body bytes. Compare your computed signature to the captured one.
6. If they match, you've cracked it. If not, you're missing a step (sort, trim, hash-first).

Patterns to recognize

Pattern	How to spot
AWS SigV4	Authorization: AWS4-HMAC-SHA256 Credential=.... Heavy canonicalization.
GitHub webhook style	X-Hub-Signature-256: sha256=.... Body is the canonical input.
Stripe webhook	Stripe-Signature: t=<ts>,v1=<sig>.
Slack webhook	X-Slack-Signature: v0=..., X-Slack-Request-Timestamp: ....
AliCloud / TencentCloud / Apple StoreKit	Each has its own canonicalization.

Some are open-source (@aws-sdk/signature-v4, cocoa-pods libraries). When you recognize one, lean on the published library rather than re-implementing.

Replay protection

Servers typically reject signatures where:

• Timestamp is more than ~5 minutes old (replay protection).
• Same signature was already used (if there's a nonce / request ID).

For scrapers: make sure your timestamp is current (Unix-seconds, server time) and don't cache signatures.

Hands-on lab

Hit /challenges/api/auth/hmac-signed in DevTools. Find the signing function in the JS bundle (Sources + Search + breakpoint). Note the canonical string format. Reproduce it in Python or PHP. Test by issuing a signed GET, confirm 200. Test by issuing one with a wrong signature, confirm 401. Test with a stale timestamp, confirm 403 or similar. You've now reverse-engineered a real signing protocol.