fix(runtime): string </sort compare by UTF-16 code unit, not code point

[proggeramlug]

Fixes #6151.

Summary

ECMAScript compares strings by UTF-16 code unit — for < > <= >= and
the default Array.prototype.sort order. Perry compared raw UTF-8 bytes
(= code-point order), which diverges once an astral character (code point

U+FFFF) is involved:

"\u{1F600}" < "｡"              // U+1F600 vs U+FF61 — was false, now true
["｡", "\u{1F600}", "b"].sort() // was [b,😀,｡], now [b,｡,😀]

An astral char's UTF-16 encoding is a surrogate pair leading with 0xD800–0xDBFF,
which sorts before BMP characters in 0xE000–0xFFFF; code-point order sorts it
after.

What it does

• Adds string::utf16_cmp_bytes(a, b) — decodes UTF-8 and compares via
  encode_utf16().cmp(), falling back to raw byte order if either side is not
  valid UTF-8 (WTF-8 lone surrogates, a known categorical gap).
• Routes the three lexicographic string-order sites through it:
  • </>/<=/>= operator (value/equality.rs)
  • js_string_compare (string/compare.rs)
  • default sort comparator, both key-comparison sites (array/sort.rs)

BMP-only strings (ASCII, Latin accents, etc.) are unchanged — byte order and
UTF-16 order agree there. localeCompare (locale-sensitive collation) is a
different operation and is untouched.

Validation

Byte-for-byte vs node --experimental-strip-types: astral </>/<=/>=,
two-astral, mixed BMP+astral, surrogate-vs-high-BMP; default sort of
astral/emoji/accents/ASCII words/numbers/mixed types/empty+dupes; comparator
sort and localeCompare unaffected.

Summary by CodeRabbit

• Bug Fixes
  • Improved string sorting and comparison to follow JavaScript UTF-16 ordering more closely.
  • Fixed edge cases involving non-BMP characters, so lists and comparisons now behave more consistently.
  • Kept existing behavior for invalid text data while updating how valid strings are ordered.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds a utf16_cmp_bytes helper implementing ECMAScript UTF-16 code-unit string comparison with UTF-8 fallback, re-exports it, and replaces raw byte comparisons in string relational operators (js_string_compare, js_jsvalue_compare) and default Array.prototype.sort key ordering with this helper.

Changes

UTF-16 Comparison Fix

Layer / File(s)	Summary
utf16_cmp_bytes helper and re-export crates/perry-runtime/src/string/compare.rs, crates/perry-runtime/src/string/mod.rs	Adds utf16_cmp_bytes comparing UTF-8 byte slices by UTF-16 code units (with raw byte fallback for invalid UTF-8), and re-exports it as pub(crate) from the string module.
String relational comparison uses UTF-16 order crates/perry-runtime/src/string/compare.rs, crates/perry-runtime/src/value/equality.rs	js_string_compare and js_jsvalue_compare now call utf16_cmp_bytes instead of comparing raw bytes directly, with an updated comment clarifying UTF-16 semantics.
Default array sort uses UTF-16 order crates/perry-runtime/src/array/sort.rs	sort_defined_values and the dense fast path in sort_array_receiver now sort key pairs using utf16_cmp_bytes instead of String::cmp.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately describes the main UTF-16 string comparison change.
Description check	✅ Passed	The description covers the summary, concrete changes, related issue, and validation, even if it doesn't mirror the template exactly.
Linked Issues check	✅ Passed	The changes implement #6151 by switching relational string comparison and default sort ordering to UTF-16 code-unit semantics.
Out of Scope Changes check	✅ Passed	The diff stays focused on the string comparison and sort fix with no obvious unrelated code changes.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/string-utf16-ordering

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

crates/perry-runtime/src/array/sort.rs (1)

162-162: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Sort keys are always valid UTF-8 — redundant from_utf8 validation on every comparison.

sort_key_string returns a Rust String (valid UTF-8 via unwrap_or("")), so utf16_cmp_bytes always takes the Ok branch here. The std::str::from_utf8 validation runs on every one of the O(n log n) comparisons but can never fail. For large arrays with long strings, this adds measurable overhead.

Two options, in order of effort:

1. Skip validation (low effort): compare encode_utf16() iterators directly since the bytes are already known-valid UTF-8.
2. Pre-encode once (medium effort): change pairs from Vec<(String, f64)> to Vec<(Vec<u16>, f64)>, encoding each key to UTF-16 once upfront, then compare &[u16] slices — eliminates both per-comparison validation and re-encoding.

Option 1: Use encode_utf16().cmp() directly

-            pairs.sort_by(|a, b| crate::string::utf16_cmp_bytes(a.0.as_bytes(), b.0.as_bytes()));
+            pairs.sort_by(|a, b| a.0.encode_utf16().cmp(b.0.encode_utf16()));

Applied at both line 162 and line 629.

This is a nice-to-have; the shared utf16_cmp_bytes helper is the right call for consistency in the direct comparison paths (js_string_compare, js_jsvalue_compare) where raw bytes may be WTF-8.

Also applies to: 629-629

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/perry-runtime/src/array/sort.rs` at line 162, The `pairs.sort_by` path
in `sort.rs` is doing redundant UTF-8 validation through `utf16_cmp_bytes` even
though `sort_key_string` always produces a valid `String`. Update the sorting
logic in the affected compare path(s) to avoid calling `std::str::from_utf8` on
every comparison, either by comparing `encode_utf16()` results directly or by
pre-encoding the keys once before sorting. Keep `utf16_cmp_bytes` for the
raw-byte comparison helpers like `js_string_compare` and `js_jsvalue_compare`,
but not for the `pairs` sort path.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/perry-runtime/src/array/sort.rs`:
- Line 162: The `pairs.sort_by` path in `sort.rs` is doing redundant UTF-8
validation through `utf16_cmp_bytes` even though `sort_key_string` always
produces a valid `String`. Update the sorting logic in the affected compare
path(s) to avoid calling `std::str::from_utf8` on every comparison, either by
comparing `encode_utf16()` results directly or by pre-encoding the keys once
before sorting. Keep `utf16_cmp_bytes` for the raw-byte comparison helpers like
`js_string_compare` and `js_jsvalue_compare`, but not for the `pairs` sort path.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: aecde187-f7c0-4dc7-8976-026cc2d68ad8

📥 Commits

Reviewing files that changed from the base of the PR and between 7159c2a and 3db01f1.

📒 Files selected for processing (4)

• crates/perry-runtime/src/array/sort.rs
• crates/perry-runtime/src/string/compare.rs
• crates/perry-runtime/src/string/mod.rs
• crates/perry-runtime/src/value/equality.rs