httpcached

A local HTTP reverse-proxy cache for file-style resources — Linux package repos (apt / yum / apk), the npm registry, PyPI, container blobs, and any other resource served over plain HTTP. It cuts load on upstream mirrors, eliminates wasted re-downloads, and dramatically speeds up repeated/CI builds.

Built on GoFiber v3 (fasthttp), zap and viper.

How it works

client ──HTTP──> GoFiber (fasthttp)  ──>  cache handler
   1. Host selects a site; longest-prefix path match selects an upstream
   2. Rules (ext / path / host / upstream) resolve a Policy (ttl, min_hits, ...)
   3. lookup:
        fresh hit         → stream from disk            X-Cache: HIT
        stale + validators → conditional GET (304/200)   X-Cache: REVALIDATED
        cold (< min_hits)  → pass-through, not cached     X-Cache: PASS
        admitted miss      → single-flight fetch + tee    X-Cache: MISS

Storage: bodies live in sharded blob files (blobs/ab/cd/<sha256>); metadata is a JSON .meta sidecar next to each blob. Writes are atomic (temp file → fsync → rename); a crash can only leave an orphan blob, which is cleaned on the next startup scan.

Quick start

go build -o httpcached .
./httpcached            # reads ./config.yaml, listens on server.port

Point a client at it by Host + path prefix. With the sample config:

# apt: /etc/apt/sources.list ->
#   deb http://mirror.local:8000/ubuntu jammy main
# (resolve mirror.local to the cache host, or use the cache IP directly)

# npm:
npm config set registry http://mirror.local:8000/npm/

# pip:
pip install --index-url http://mirror.local:8000/pypi/simple some-pkg

# raw check:
curl -H 'Host: mirror.local' http://127.0.0.1:8000/ubuntu/dists/jammy/InRelease -D-

Every response carries an X-Cache: HIT|MISS|REVALIDATED|PASS|STALE header. GET /_cache/stats returns JSON counters (hits, misses, hit ratio, disk usage…).

Configuration (config.yaml)

Send SIGHUP to hot-reload the cache section (rules / sites / upstream pool) without dropping the cache.

cache:
  storage:
    dir: "./cache_data"      # blobs/ + tmp/ live here
    max_size: "50GB"         # soft cap; LFU/LRU eviction above it (0 = unlimited)
    max_age: "720h"          # hard retention cap
    caretaker_interval: "5m"
  upstream:                  # shared origin connection pool / timeouts
    max_conns_per_host: 256
    max_idle_conns_per_host: 64
    idle_conn_timeout: "90s"
    dial_timeout: "10s"
    response_header_timeout: "30s"
    follow_redirects: true
    # user_agent: "httpcached/1.0.0"   # override UA sent to origins (empty = forward client UA)
  admission:
    window_size: 100000      # max keys tracked for popularity
    default_min_hits: 2
  defaults:                  # fall-back policy; rules inherit unset fields
    cache: true
    ttl: "24h"
    revalidate: true
    min_hits: 2
    stale_if_error: true
  rules:                     # first match wins
    - name: immutable-artifacts
      match: { ext: [deb, rpm, apk, whl, tgz, jar] }
      policy: { immutable: true, ttl: "720h", min_hits: 1 }
    - name: repo-metadata
      match: { path_regex: '(InRelease|Packages(\.\w+)?|repomd\.xml|.*\.json)$' }
      policy: { ttl: "5m", revalidate: true, min_hits: 1 }
    - name: no-cache-dynamic
      match: { path_regex: '(/token|/v2/auth)' }
      policy: { cache: false }
  sites:                     # Host -> site; path prefix -> upstream
    - hosts: ["mirror.local", "127.0.0.1"]
      routes:
        - { path: "/ubuntu", upstream: "http://archive.ubuntu.com/ubuntu" }
        - { path: "/npm",    upstream: "https://registry.npmjs.org" }
        - { path: "/pypi",   upstream: "https://files.pythonhosted.org" }
    - hosts: ["npm.local"]
      routes:
        - { path: "/", upstream: "https://registry.npmjs.org" }

Policy fields

field	meaning
cache	false ⇒ never cache (pure pass-through)
ttl	freshness lifetime
immutable	never revalidate; serve until evicted
revalidate	when stale, do a conditional GET (If-None-Match / If-Modified-Since)
min_hits	requests before an object is admitted to disk (1 = cache immediately)
respect_origin	fold the origin's Cache-Control / Expires into the TTL
stale_if_error	serve a stale copy if the origin is unreachable during revalidation
ignore_query	drop the query string from the cache key
max_object_size	objects larger than this are streamed through but not cached

Behaviour notes

• Only GET is cached; other methods and no-cache policies are proxied verbatim.
• Accept-Encoding is not forwarded, so cached bodies are identity-encoded and safe to replay to any client.
• Range requests are served from a fully-cached object (206); a range for an uncached object is passed through (the cache is never populated from a partial response).
• Hop-by-hop headers are stripped both ways. Proxy-chain headers from a front nginx/CDN (X-Forwarded-*, Forwarded, X-Real-IP, Via) are also stripped before contacting the origin — forwarding them makes some CDNs "correct" the scheme with a redirect loop.
• Origin fetch failures are logged (with the URL and, for redirect loops, the full hop chain) in addition to the 502 body returned to the client.

Limitations

• No Docker Registry v2 token-auth handshake (generic HTTP caching only — works for repos/registries that don't require a per-request token).
• Multi-range requests are served as their first range.
• Admission uses an approximate LRU frequency filter (good enough to separate cold from hot).

Development

go test ./... -race      # unit + integration (single-flight, admission, range, persistence)