Skip to content

[codex] add long-form generation mode#313

Open
Lee-take wants to merge 4 commits into
OpenBMB:mainfrom
Lee-take:codex/long-form-generation
Open

[codex] add long-form generation mode#313
Lee-take wants to merge 4 commits into
OpenBMB:mainfrom
Lee-take:codex/long-form-generation

Conversation

@Lee-take

@Lee-take Lee-take commented May 23, 2026

Copy link
Copy Markdown

Summary

  • add VoxCPM.generate_long_form() for long scripts by splitting text into shorter segments
  • reuse the first generated segment as a fixed continuation prompt when no external reference is provided
  • keep an explicit external reference audio as the stable reference voice when the caller supplies one
  • ensure the seed prompt_text matches the spoken transcript, excluding voice-design control text that is not spoken
  • expose the mode through CLI flags: --long-form, --long-form-max-chars, and --long-form-silence-ms
  • document Python and CLI usage in the English and Chinese READMEs

Why

Very long single-pass generation can accumulate autoregressive drift. The long-form path keeps each generation pass short and re-anchors later segments with stable prompt context, which is an API/CLI-level mitigation that does not change model weights or decoder internals.

The generated seed segment is no longer duplicated as both reference audio and prompt audio. If no external reference is supplied, the long-form path uses prompt continuation only; if the caller supplies an external reference, it is preserved as reference conditioning. This avoids over-conditioning on the same generated seed audio while still keeping a stable continuation anchor.

The seed prompt transcript must match the seed prompt audio exactly. Voice-design control text guides the first generation pass, but it is not part of the spoken audio; including it in later continuation prompt text can make the continuation conditioning inconsistent and degrade long-form output.

Tests

  • python -m pytest tests/test_core_long_form.py tests/test_cli.py -q
  • python -m compileall src tests -q
  • local installed VoxCPM2 checks: python tests/test_voxcpm_long_form.py and python tests/test_voxcpm2_voice_anchor.py
  • local long-form smoke run with VoxCPM2 CUDA runtime produced a 108.14s WAV at 48kHz mono using fixed seed-prompt continuation and shorter default segments

Not run: full model-dependent upstream test suite, because this local PR environment only installed lightweight unit-test dependencies and no full torch/model runtime.

@a710128

a710128 commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Thanks a lot for this PR, @Lee-take! 🙏 The long-form direction is great, and I really appreciate that this is a pure API/CLI-level mitigation — no model weights or decoder internals touched. A few details are genuinely well done:

  • The seed prompt_text deliberately excludes the (control) prefix so it matches the spoken audio — that's an easy thing to miss, and your commit + inline comment nailed it.
  • VoxCPM2-only reference_wav_path passthrough (can_use_reference) correctly avoids leaking reference conditioning into the v1.x path.
  • Clean commit history, TemporaryDirectory cleanup, and both English/Chinese README updates. Nice.

I pulled the branch and ran the unit tests (22/22 green) and probed the text splitter on real-world inputs. A few things I'd love to see addressed before we merge 👇

🔴 Should fix before merge

1. _split_long_text_for_tts breaks on common English text. With the default max_chars, I hit these:

Input Output Problem
"Hi. Hi. Hi. Hi. Hi." (max=8) ['Hi.Hi.', 'Hi.Hi.', 'Hi.'] inter-sentence space lost → "HiHi"
"The number is 3.14 ..." splits after 3. decimal point treated as sentence-end
"Many languages, e.g. Python..." splits after e.g. abbreviation treated as sentence-end
"...web development." (max=40) [..., 'web deve', 'lopment.'] mid-word hard cut → audible artifacts

Root causes: (a) . is treated as a strong sentence terminator (too common in abbreviations/decimals/URLs for English), (b) pieces are concatenated with current += subpiece without inserting a space (fine for CJK, breaks English), and (c) the subpiece[:max_chars] hard cut is word-boundary-unaware. Since the splitter is the entry point for the whole feature, I think this is the most important thing to harden. A word-boundary-aware fallback + space-joining for non-CJK would go a long way.

2. max_chars default is inconsistent across entry points. I see 90 in _split_long_text_for_tts, 55 in generate_long_form and the CLI flag, and 120 in the README examples. Could we converge on one value (and drop or align the 90 default, which is effectively dead since generate_long_form always passes max_chars explicitly)? A one-line note on why the default was chosen would help too.

3. The seed prompt is re-fed to every later segment. seed_prompt_text = segments[0] can be up to max_chars long, so each later pass runs roughly len(seed) + len(segment) ≈ 2 × max_chars tokens. If a user raises max_chars (e.g. the README's 120) to reduce drift, each segment approaches a long single pass again, which partially defeats the purpose. Worth either documenting this clearly or capping the prompt with a separate seed_max_chars.

🟡 Worth discussing

4. (control) injection isn't sanitized. app.py strips ()() from user control text to protect the (control)text format, but generate_long_form injects it raw. control="warm (friendly)" would produce nested parens. Aligning with the app.py behavior would keep CLI/webui consistent.

5. Python API has no control + prompt_text guard. The CLI rejects this combo via validate_clone_args, but a direct generate_long_form(control=..., prompt_text=...) call would both inject (control) into the first segment and keep the user prompt — an undefined state. A small warning/ValueError at the API boundary would close that gap.

6. --long-form isn't exposed to the batch subcommand (it doesn't use _add_common_generation_args). If that's intentional (batch = one short line per call), a note in the PR description is enough; otherwise it's a tiny addition.

7. Minor: for the first segment, denoise is a no-op in pure voice-design mode (denoise only affects external prompt/reference audio). A docstring note that denoise applies only to external prompt/reference, and that later segments are never denoised, would avoid confusion.

🧪 Test suggestion

test_core_long_form currently uses ideal inputs like "First! Second! Third!". Could we add a regression case with a realistic English paragraph containing abbreviations, decimals, and words near the boundary? That's exactly where the splitter currently struggles, and it'd lock the behavior once fixed.

Really solid contribution overall — the splitter robustness (1) and the default consistency (2) are my main blockers; the rest are mostly polish. Happy to help review the follow-up. Thanks again! 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants