Skip to main content
CLI man page: safesquid-prefetch(5)
Prefetch queue flow

Prefetch queue flow

​
Overview

The Prefetch section (safesquid-prefetch(5)) configures SafeSquid to proactively download resources into its cache before the client actually requests them.

​
Core Mechanics (C++ Source Validation)

Async queue, cache pre-check, HTML parser first-match.
  • Asynchronous Queue: Prefetching is driven by an asynchronous, background worker thread (prefetch_thread) that monitors a PREFETCH_QUEUE. It does not block the primary client connection.
  • Cache Pre-Validation: Before initiating a prefetch download, the thread checks if the object exists in the cache. If the object is already cached, the prefetch is aborted to save bandwidth.
  • Request Spoofing: The background thread creates a synthetic, unattached HTTP connection, forces it through the routing pipeline, and uses a hardcoded User-Agent to request the file from the origin.

​
Schema Fields

​
Global Fields

  • Enabled (enabled): When off, setup_callbacks does not register HTML parsers — no automatic prefetch.
  • Threads (threads): Configured worker count for prefetch queue processing. This value is not applied in the current build.
  • Queue size (queuesize): Maximum URLs waiting in the prefetch queue; full queue rejects new URLs. Already-cached URLs are not queued.
  • Host limit (hostlimit): Maximum queued prefetches per host name ( 0 disables per-host cap). Duplicate URLs are never queued twice.

​
Rule-Based Fields (Per Connection Tuning)

  • Enabled (enabled): Disabled rows are skipped. Enabled rows are tested top to bottom; the first HTML-parser match queues prefetch URLs.
  • Comment (comment): Operator notes only. Does not affect prefetch matching.
  • Profiles (profiles): Limit this entry to connections that already carry these Access Profile tags. SafeSquid tests against the connection’s active profiles: blank ignores profiles; a positive tag applies when present; prefix ! applies when absent. With multiple tags, the first list entry that resolves decides (positive match -> apply; negated match -> skip; lone unmatched ! -> apply). Prefetch HTML parser rules are walked top to bottom; the first matching row decides which tag/attribute URLs are queued for early fetch.
  • Tag name (tag): HTML element name to scan in text/html responses (case-insensitive). Example: a , img , link . Blank skips this row during parser matching.
  • Tag attribute (attribute): Attribute on the tag that holds the URL (for example href or src ). When present on the element, its value is passed to the URL regex below.
  • Attribute pattern (pattern): POSIX regex tested against the attribute URL after tag/attribute match. On match, the URL is resolved and queued (unless already cached or queue full). Blank accepts any non-empty attribute value.
  • Maximum file size (maxsize): Skip prefetch when the response Content-Length exceeds this size. 0 = no size cap.
  • Recursion level (recursion): Set the Recursion limit to 0, or a higher positive integer. Prefetched content is analyzed, to discover new prefetchable URLs. Recursion limits this iteration. Note: Setting to 0 causes links to be followed, indefinitely.
  • appcontent (appcontent): No description provided.

​
Prefetch rule fields

  • Profiles — Limit the rule to connections with matching Access Profile tags. Blank matches all.
  • Tag name — HTML element to scan (case-insensitive), for example a, img, link. Blank skips the row.
  • Tag attribute — Attribute holding the URL (for example href or src).
  • Attribute pattern — POSIX regex on the attribute value after tag/attribute match. Blank accepts any non-empty value. On match, the URL is resolved and queued unless cached or queue full.
  • Maximum file size — Skip prefetch when response Content-Length exceeds this size. 0 = no cap.
  • Recursion level — How many levels prefetched HTML is analyzed for further links. Note: setting 0 follows links indefinitely per field help.

​
Processing order

  1. Enable Prefetching and Caching.
  2. During HTML responses, enabled Prefetch rows are checked top to bottom.
  3. The first row whose profiles match and whose tag/attribute/pattern succeeds drives prefetch for that parser pass.
  4. Matched URLs enter the queue subject to queue size, host limit, and cache state.
  5. Caching Prefetch window limits duplicate prefetch writes for the same URL within N seconds.
Some prefetch hooks and the Prefetch now CGI handler require ENABLE_PREFETCH at build time. If manual prefetch fails, confirm your build includes prefetch support.

​
Examples

​
1 — Prefetch linked CSS and scripts

  • Row 1: Tag link, attribute href, pattern \.css$
  • Row 2: Tag script, attribute src, pattern blank
Result: on HTML pages, stylesheets matching .css and all script src URLs are queued for early fetch when row 1 or 2 matches first for that element scan order.

​
2 — Staff-only prefetch

  • Prefetch row Profiles: STAFF
  • Access Profiles adds STAFF for internal users only
Result: only staff connections trigger automatic prefetch; other users parse HTML without prefetch queue activity from that row.

​
3 — Manual warm-up of a portal page

  • Prefetch now: enter https://intranet.example.com/
Result: when the handler is enabled, the URL is queued and fetched into cache so the next user request may hit cache immediately.

​
How to verify

  1. Enable CACHE in LOG_LEVEL; look for cache writes with prefetch-related flags.
  2. Load an HTML page twice — second load should show cache hits for prefetched assets when Caching and rules allow.
  3. Use Manage cached objects to search for prefetched URLs.
  4. Watch queue rejection if Queue size or Host limit is exceeded (reduce rules or raise limits).

Related topics

Caching and PrefetchingPerformance AcceleratorsStartup Parameters