😉[v0.3.41] Release Note: Template-Driven MTMD, Broader Multimodal Inputs, and Smarter N-Gram Drafting

[JamePeng]

Release Note: v0.3.41

Template-Driven MTMD, Broader Multimodal Inputs, and Smarter N-Gram Drafting

This release is mainly focused on improving the multimodal architecture, making MTMD chat handling cleaner, more flexible, and easier to extend for future image, audio, and video workflows. I also continued syncing with the latest llama.cpp APIs and improved several developer-facing diagnostics around model templates, shared library loading, and Windows OpenMP runtime discovery.

Highlights

1. Cleaner MTMD Prompt Rendering Pipeline

In this release, I refactored the MTMD prompt processing path by separating prompt rendering, media placeholder normalization, and MTMD tokenization into dedicated helper stages.

The new helpers include:

• _render_mtmd_prompt() for pure chat template rendering
• _replace_media_placeholders() for normalizing rendered media tags and URLs into MTMD runtime markers
• _render_and_replace_media() for the combined render-and-normalize stage
• _mtmd_tokenize() for encapsulating the llama.cpp mtmd_tokenize binding

This removes inline render-and-replace logic from _process_mtmd_prompt() and makes the overall pipeline easier to reason about. Prompt construction, media marker validation, and C++ tokenization are now more clearly separated.

Strict media marker validation is still preserved after normalization to ensure token and bitmap alignment remains correct.

2. Better Generic MTMD Chat Template Support

GenericMTMDChatHandler has been improved to better support model-provided chat templates.

The generic handler can now:

• Accept an optional named chat template
• Load a named template from the model at call time via llama_model_chat_template()
• Fall back to the model’s default chat template
• Fall back again to the built-in MTMD CHAT_FORMAT when no model template is available

I also added extra_template_arguments support to MTMD chat handlers and pass these arguments into the Jinja chat template render call. This makes it possible for generic templates to receive render-time options such as:

• enable_thinking
• add_vision_id
• model-specific Jinja variables
• other template-defined flags

This is an important step toward making the generic MTMD path more template-driven, instead of requiring every new multimodal GGUF model to immediately have a dedicated Python chat handler.

3. Broader Multimodal Media Extraction

MTMD media extraction now supports more common multimodal content shapes used by different model chat templates.

In addition to OpenAI-style chunks such as:

• image_url
• audio_url
• video_url

the handler now also accepts typed chunks and direct media keys such as:

• {"image": "..."}
• {"audio": "..."}
• {"video": "..."}

A shared media extraction helper was added to support URLs, local paths, existing data URIs, and inline base64 payloads.

Capability checks are still preserved, including strict input_audio format validation and explicit errors for missing or ambiguous media payloads.

4. Video Input Support for MTMD

MTMDChatHandler now supports video_url input.

The implementation detects whether the loaded libmtmd build supports video helpers and rejects video input early when MTMD_VIDEO is unavailable.

Media loading and bitmap creation were updated for the newer helper wrapper API. The decoded bitmap and optional video helper context are now kept alive until mtmd_tokenize completes, then released afterward.

I also consolidated duplicated audio and video byte-loading logic into a shared _load_bytes() helper and reused it for image loading. Remote media requests now also include richer default HTTP headers.

5. Multimodal Handlers Moved to llama_multimodal

The multimodal chat handler code has been moved out of llama_chat_format.py and into a dedicated llama_multimodal.py module.

This includes:

• MTMDChatHandler
• GenericMTMDChatHandler
• model-specific multimodal chat handlers

llama_chat_format.py had become too large as multimodal support expanded beyond image-only use cases. Moving the MTMD path into its own module makes the codebase easier to maintain and prepares it for future image batch evaluation, video frame processing, and broader multimodal workflows.

Backward-compatible re-exports are kept in llama_chat_format.py, so existing imports should continue to work.

clip_model_path is also kept as a deprecated initialization alias for mmproj_path in the base MTMD handler.

6. README Updates for Generic MTMD Usage

The README now includes a new GenericMTMDChatHandler usage guide.

The old Llava-style multimodal loading example has been replaced with a template-driven generic MTMD example for modern multimodal GGUF models.

The documentation now explains:

• Loading mmproj through Llama
• Chat template resolution order
• Using extra_template_arguments
• When to prefer a dedicated multimodal chat handler
• The mmproj_path naming
• The migration to llama_multimodal
• The generic handler’s role as a flexible fallback for models without dedicated handlers

The Generic MTMD Chat Handler directory index has also been updated.

7. Smarter N-Gram Map Speculative Drafting

The speculative decoding path also received several improvements.

The n-gram map draft logic now:

• Stores accepted draft lengths per key/value and truncates future drafts accordingly
• Allows key-only mode to draft on any key match without applying min_hits
• Selects k4v continuations by frequency instead of latest occurrence
• Skips ambiguous k4v drafts when the top continuation is not dominant
• Tracks fixed-size k4v continuations so frequency statistics remain comparable

This should make draft selection more stable and less likely to over-draft in ambiguous continuation cases.

8. Safer Model Chat Template Handling

LlamaModel.model_chat_template() now returns Optional[str] and accepts name=None for the default model chat template.

This fixes a case where llama_model_chat_template() may return nullptr when no chat template is available. Instead of decoding a null pointer, the wrapper now returns None, allowing callers to apply their own fallback logic safely.

I also updated LlamaModel.vocab_type to use self.vocab and added additional None checks.

9. Better Developer Diagnostics

This release includes several improvements for debugging build and runtime issues.

Shared library loading errors now include a listing of the searched directory contents. This makes it easier to diagnose missing, misplaced, or incorrectly named library files.

A new _format_library_dir_contents() helper was added, and the extra diagnostic work is only performed on the failure path, so successful imports do not pay any additional overhead.

For Windows builds, CMake now improves LLVM OpenMP runtime discovery for libomp140.x86_64.dll. The search prefers the VS 2022 VC143 OpenMP redistributable and keeps System32 as the final fallback. Diagnostics now report the selected runtime source and path, warn when an explicit override points to a missing file, and keep a clear warning when no OpenMP DLL is found.

10. Upstream llama.cpp Sync

This release updates the bundled llama.cpp source to:

ggml-org/llama.cpp@3899b39ce2acc2e019f149b7107f24b6ca297390

It also syncs the llama.cpp llama, mtmd, and ggml API bindings as of 2026-07-07.

Community Contributions

Thanks to the contributors included in this release:

• @alcoftTAO for the initial generic multimodal chat handler prototype
• @patrikpatrik for adding the Windows command prompt scenario and installation sanity check documentation to the README

Summary

Overall, v0.3.41 is a structural release for multimodal support.

The main goal is to make MTMD handling cleaner, more generic, and more extensible. The codebase now has a clearer separation between text-only chat formatting and multimodal processing, better support for model-provided templates, broader media input compatibility, early video support, and improved diagnostics for developers building or loading the package.

More information:
12861b9...b9b5859

— JamePeng

[JamePeng]

When using GenericMTMDChatHandler, pay close attention to the input fields. If the model's built-in chat template is constructed using a Jinja template with an image type but you provide an image_url type instead, the system will fail to recognize it correctly. The Python-side logic performs a validation check for this substitution, and you will encounter a failure—such as ValueError: GenericMTMDChatHandler(_replace_media_placeholders): media marker mismatch (marker_count=1, media_count=2)—triggered by this mismatch.

[JamePeng]

The multimodal video functionality has not yet been fully implemented; currently, only the input linking is complete. The upstream llama.cpp integration is not yet fully finished, and the ffmpeg dependency may not be bundled into the wheel in the future—configuration for its invocation might instead rely on specifying a file path.