cache: shared-memory-backed Dir for fast restart#13328
Conversation
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in “fast restart” cache directory implementation by hosting each stripe’s in-memory directory (raw_dir) in POSIX shared memory, allowing subsequent traffic_server starts to attach the prior directory quickly instead of rebuilding from disk. It also adds operator tooling (traffic_ctl cache shm status|clear), configuration records, extensive AuTest coverage, unit tests for trust gates, and design/admin documentation.
Changes:
- Add shared-memory directory infrastructure (
CacheShm*) plus integration into cache startup, stripe directory allocation, and shutdown/clean-marking paths. - Add
traffic_ctl cache shm status|clearcommands and newproxy.config.cache.shm.*records. - Add unit + AuTest suites and documentation for the shm fast-restart feature.
Reviewed changes
Copilot reviewed 32 out of 32 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/gold_tests/logging/ts_process_handler.py | Hardens psutil cmdline handling for macOS/permission-restricted processes. |
| tests/gold_tests/cache/shm_poke.py | Adds Linux-only helper to tamper with /dev/shm segments for trust-gate testing. |
| tests/gold_tests/cache/replay/cache-shm-fast-restart.replay.yaml | Proxy Verifier replay traffic used to validate hit-after-restart behavior. |
| tests/gold_tests/cache/gold/cache_shm_state_after_shutdown.gold | Gold output for traffic_ctl cache shm status validation. |
| tests/gold_tests/cache/cache_shm_unclean_shutdown.test.py | AuTest for rejecting dirty segments after SIGKILL and rebuilding from disk. |
| tests/gold_tests/cache/cache_shm_storage_mismatch.test.py | AuTest for partial attach behavior when storage layout changes. |
| tests/gold_tests/cache/cache_shm_schema_mismatch.test.py | Linux-only AuTest that pokes schema_version and verifies drop+rebuild. |
| tests/gold_tests/cache/cache_shm_purge_on_disable.test.py | AuTest for purge-on-disabled-start behavior and traffic_ctl exit codes. |
| tests/gold_tests/cache/cache_shm_fast_restart.test.py | End-to-end AuTest for clean shutdown -> shm attach -> cache HIT without origin contact. |
| tests/gold_tests/cache/cache_shm_concurrent_attach.test.py | AuTest for concurrent-attach guard (flock vs owner_pid liveness backstop). |
| tests/gold_tests/cache/cache_shm_bad_disk_dropped.test.py | AuTest for partial attach and orphan reclaim when a disk is removed from storage. |
| src/traffic_ctl/traffic_ctl.cc | Adds `traffic_ctl cache shm status |
| src/traffic_ctl/CMakeLists.txt | Builds the new traffic_ctl command source and adds cache include path for shared headers. |
| src/traffic_ctl/CacheShmCommand.h | Declares the traffic_ctl cache shm command handler. |
| src/traffic_ctl/CacheShmCommand.cc | Implements shm status/clear via direct shm_open/mmap, shared purge primitive, and exit codes. |
| src/records/RecordsConfig.cc | Registers proxy.config.cache.shm.* configuration records. |
| src/iocore/cache/unit_tests/test_CacheShm.cc | Adds unit tests for ABI hash, storage signature, layout round-trip, prefix normalization, and liveness checks. |
| src/iocore/cache/StripeSM.cc | Adds fast-attach path that can skip disk dir read + recovery when shm directory is trusted; adjusts shutdown behavior. |
| src/iocore/cache/Stripe.h | Adds _shm_directory_is_valid() and strengthens flush API via [[nodiscard]]. |
| src/iocore/cache/Stripe.cc | Uses shm-backed directory allocation; adds shm bounds validation; detaches shm mappings safely in destructor. |
| src/iocore/cache/CMakeLists.txt | Builds new CacheShm.cc and adds test_CacheShm to the cache unit tests. |
| src/iocore/cache/CacheShmPurge.h | Introduces shared header-only purge/enumerate/unlink primitive for server startup and traffic_ctl. |
| src/iocore/cache/CacheShmLayout.h | Defines the shared control-segment layout and prefix normalization utilities. |
| src/iocore/cache/CacheShm.h | Declares the CacheShm facade for lifecycle + stripe attach/create + trust gates. |
| src/iocore/cache/CacheShm.cc | Implements shm control segment lifecycle, trust gates, stripe attach/create, orphan reclaim, and clean-shutdown marking. |
| src/iocore/cache/CacheProcessor.cc | Calls CacheShm initialization before stripe construction and finalization after cache init. |
| src/iocore/cache/CacheDir.cc | Marks shm control segment clean only after shutdown sync has quiesced writers. |
| src/iocore/cache/AggregateWriteBuffer.h | Marks flush() as [[nodiscard]] to encourage handling short-write failures. |
| src/iocore/cache/AggregateWriteBuffer.cc | Converts flush failure from an assertion to a false return for graceful handling. |
| doc/developer-guide/cache-architecture/shm-fast-restart.en.rst | Adds detailed design doc: layout, gates, attach modes, shutdown semantics, tooling, and platform notes. |
| doc/developer-guide/cache-architecture/index.en.rst | Hooks shm fast-restart document into the cache architecture index. |
| doc/admin-guide/files/records.yaml.en.rst | Documents new proxy.config.cache.shm.* records and operator-facing behavior/tooling. |
Cold-start cache initialization rebuilds each stripe's in-memory directory from disk on every restart -- multi-minute on large caches. Host the directory in POSIX shared memory so the next process start attaches the existing segment in milliseconds instead of rebuilding it. Recovery stays binary and fail-safe: when the segment cannot be trusted (crash, reboot, ABI/schema or storage mismatch, failed validation) the start drops it and rebuilds via the existing disk path, and reads still validate Doc magic + key so a stale entry is a miss, never corruption. Opt-in behind proxy.config.cache.shm.enabled (default 0), where it is a functional no-op.
masaori335
force-pushed
the
asf-master-dir-on-shm
branch
from
8358d37 to
e0ee0af
Compare
POSIX shm names permit only the leading '/', so a misconfigured name_prefix like "foo/bar" would build a name shm_open rejects with EINVAL. Strip embedded '/' during normalization instead of preserving it.
On glibc < 2.34 (e.g. CentOS 7) shm_open/shm_unlink live in librt, so traffic_ctl and inkcache fail to link. Add an optional rt::rt target that is a no-op where the library is folded into libc (modern glibc, macOS).
There was a problem hiding this comment.
Approving. This is well built and safe to land. It is opt-in and off by default, the shared-memory attach path validates the mapping size before use and bounds the stripe count, and every failing system call is logged with its name and error string and then propagated back to the caller.
One correction to my earlier note. I wrote that a fast-attach bug returning a wrong offset or length would still pass CI. That is not accurate, and I withdraw it. The fast_restart test wires the origin to return a 502 on the hit transaction, so if the shared-memory directory failed to resolve the cached object, ATS would fall through to the origin and the 502 would fail the test. The real gap is narrower than I first stated.
The items below are all non-blocking for an off-by-default feature, but worth closing out before this is relied on in production.
-
The testing table in
doc/developer-guide/cache-architecture/shm-fast-restart.en.rstlists acache_shm_data_integrityautest that is not in this PR. Please either add the test or drop the row. -
The objects the tests serve are small and single-fragment. A multi-fragment or large-object case after a fast attach would cover the one path the 502 trick does not fully exercise.
-
The rejection branches in
_shm_directory_is_valid()and the invalidate-on-shutdown paths have no direct unit coverage, as the caveats note. They are close to pure header math, so each branch is a cheap unit test.
I also agree with the open Copilot threads on this PR.
-
In
CacheShm.cc:229,unlink_all_known_segments()unlinks stripe names read from the control segment. On the path that drops an untrusted or corrupt control segment those names are untrusted, so a corrupt segment could lead toshm_unlinkon unrelated objects. Since the names are always built fromname_prefixviabuild_stripe_shm_name(), gating the unlink loop on a prefix match, and only running it when the control magic is valid, closes this with no effect on the normal path. -
The three
storage.configreferences in the design doc at lines 132, 340, and 430 should bestorage.yamlto match the tests and the rest of this PR.
Thanks for the clean, well-documented change. The approval stands.
A corrupt control segment (invalid magic) left the stripe-name bytes untrusted, yet the drop path unlinked them, so a bad segment could shm_unlink unrelated POSIX objects. Only walk the stripe table when the header magic is intact and restrict unlinks to our own name prefix, matching the guard already in purge_segments(). Also correct storage.config -> storage.yaml in the design doc and drop the testing-table row for the not-yet-added cache_shm_data_integrity autest.
|
Thank you for taking a look. I addressed 1. 4. and 5. by 0cc3b52. |
| if (addr == MAP_FAILED) { | ||
| int e = errno; | ||
| Warning("mmap(%s, %zu) failed: %s", name.c_str(), size, strerror(e)); | ||
| if (out_errno != nullptr) { | ||
| *out_errno = e; | ||
| } | ||
| return nullptr; | ||
| } |
| ++reclaimed; | ||
| } | ||
| if (reclaimed > 0) { | ||
| Note("cache shm: reclaimed %u orphaned stripe segment(s) after storage change", reclaimed); |
| // All writers are now stopped (every stripe's mutex was held and released by the | ||
| // shutdown threads above), so the directory is final -- only now is it safe to | ||
| // mark the shm control segment clean. | ||
| CacheShm::mark_clean_shutdown(); |
There was a problem hiding this comment.
I don't think "All writers are now stopped" is right here. shutdown() only holds/releases each stripe's mutex, which excludes concurrent writers, not future ones. An ET_NET thread can still complete a write after a stripe's snapshot, and this runs before shut_down_event_system().
Was the intention that a host would need to be fully drained before the cache dir can be persisted?
| if (!this->_write_buffer.is_empty()) { | ||
| Dbg(dbg_ctl_cache_dir_sync, "Dir %s: flushing agg buffer first", this->hash_text.get()); | ||
| this->flush_aggregate_write_buffer(this->fd); | ||
| if (!this->flush_aggregate_write_buffer(this->fd)) { |
There was a problem hiding this comment.
What happens if the agg buffer write is pending in AIO? Before this PR, the double write was probably harmless, but now the wrong write_pos would get picked up by the next ATS pid.
| this->directory.raw_dir = static_cast<char *>(ats_memalign(ats_pagesize(), directory_size)); | ||
| // Try a shared-memory-backed directory first; fall back to the hugepage / | ||
| // aligned-heap path when shm is disabled or the attach/create fails. | ||
| this->directory.raw_dir = CacheShm::attach_or_create_stripe(hash_text.get(), directory_size); |
There was a problem hiding this comment.
This preempts the ats_alloc_hugepage fallback, so a box relying on real MAP_HUGETLB hugepages for the dir silently drops to 4K tmpfs pages once shm is enabled — the MADV_HUGEPAGE replacement is a separate default-off knob.
| { | ||
| // The operator gives only the middle word (e.g. --prefix ats); frame it the | ||
| // same way the server does so the two agree on segment names. | ||
| std::string configured = DEFAULT_PREFIX; |
There was a problem hiding this comment.
records.yaml already has the prefix and traffic_ctl reads it for the config commands — why require the operator to pass --prefix at all?
There was a problem hiding this comment.
In the case of the prefix in record.yaml is changed, but want to remove the old shm from traffic_ctl cmd.
| be trusted, including when: | ||
|
|
||
| - the previous instance did not shut down cleanly (for example, it crashed), | ||
| - the on-disk storage layout described by :file:`storage.yaml` changed, |
There was a problem hiding this comment.
Does storage.yaml changing actually cause segments to be discarded?
The code comments say:
/// Fingerprint of the storage topology. Not a trust gate (see initialize()):
/// informational, drives the "storage changed" log wording.
static uint64_t storage_signature(const Store &store);```
| } | ||
| int oflags = O_RDWR; | ||
| if (access == ShmAccess::Create) { | ||
| oflags |= O_CREAT; |
There was a problem hiding this comment.
Please use O_EXCL for creation, to prevent the possibility of inheriting permissions from a malicious pre-existing file.
Motivation
Cold-start cache initialization rebuilds each stripe's in-memory directory from disk on every
restart — multi-minute on large caches.
Approach
Host the directory (
raw_dir) in POSIX shared memory so the next process start attaches theexisting segment in milliseconds instead of rebuilding it. Recovery is binary and fail-safe:
anything untrustworthy (crash, reboot, ABI/schema mismatch, failed validation, bad disk) falls
back to the existing disk-rebuild path, and reads still validate
Docmagic + key so a staleentry is a miss, never served corruption.
New configs & traffic_ctl commands
Opt-in behind
proxy.config.cache.shm.enabled(default0, a functional no-op). Also addsproxy.config.cache.shm.{name_prefix,use_hugepages,purge_stale_on_start}and atraffic_ctl cache shm status|clearcommand.Details
Design, recovery model, configuration, and platform notes are in
doc/developer-guide/cache-architecture/shm-fast-restart.en.rst(added in this PR) and theproxy.config.cache.shm.*entries inrecords.yaml.Testing
test_CacheShm(ABI hash, storage signature, control round-trip, name length, prefixnormalization, process liveness).
cache_shm_*suites — fast restart, unclean shutdown, schema/storage mismatch,bad-disk drop + orphan reclaim, concurrent-attach refusal, purge-on-disable.
Caveats
_shm_directory_is_valid()rejection branches or thebad-disk / flush-failure
invalidate_stripe_directory()paths (planned follow-up).flockguard is authoritative on Linux but ano-op on macOS POSIX shm, where it falls back to an owner-pid liveness check.