perf: deduplicate identical word graphs within a build

[AmitMY]

_build_graphs built a fresh graph for every word occurrence. In natural text most words repeat, so this allocates many identical subgraphs (and, during training, an independent get_merges memo for each). Dedup them: repeated words share one immutable subgraph.

def _build_graphs(self, texts):
    if self.cache_maxsize == 0:
        units = self.units                                    # dedup disabled
    else:
        units = lru_cache(maxsize=self.cache_maxsize)(self.units)
    return tuple(words(text, ..., units=units) for text in texts)

Configurable via Tokenizer(cache_maxsize=None) (plumbed through all subclasses): None (default) unbounded, a number bounds the cache for huge vocabularies, 0 disables dedup.

Why the cache is build-local (not set once in __init__): scoping it to one build matters two ways —

1. It's freed before training, so it doesn't pin the pre-merge word graphs in memory while the trainer merges past them. (Holding it for the whole run drops the memory win from −38% to ~−4%.)
2. A word graph is a NodesSequence whose memoized get_merges depends on GraphSettings; a build-local cache lives under one settings regime and can't serve a stale memo to a later run — the hazard that makes @cache chinese_character_to_graph incorrect.

Output identical — new test test_cache_maxsize_does_not_change_merges pins that 0 and 10 produce the same merges; full digests unchanged; 138 tests pass.

Measured back-to-back (50 texts / 200 merges; best of 3):

	main	this PR
BPE	0.414s / 2.14MB	0.401s / 1.33MB (−3% / −38%)
BNE n=4	0.806s / 3.63MB	0.786s / 2.63MB (−2% / −28%)
Boundless	0.494s / 2.28MB	0.482s / 1.42MB (−2% / −38%)

Shared subgraphs also share their get_merges memo (computed once per unique word instead of per occurrence) — the source of the speed bonus.

🤖 Generated with Claude Code