Skip to content

fix(isthmus): use Substrait type system when building Calcite RelBuilder#973

Merged
bestbeforetoday merged 1 commit into
substrait-io:mainfrom
nielspardon:par-decimal-roundtrip-typesystem
Jun 30, 2026
Merged

fix(isthmus): use Substrait type system when building Calcite RelBuilder#973
bestbeforetoday merged 1 commit into
substrait-io:mainfrom
nielspardon:par-decimal-roundtrip-typesystem

Conversation

@nielspardon

Copy link
Copy Markdown
Member

When converting Substrait to Calcite, the RelBuilder was created via RelBuilder.create(Frameworks...) without specifying a type system, so its internal RexBuilder used Calcite's default type system (max decimal precision 19) while the converted expressions carried SubstraitTypeSystem types (precision 38). RelBuilder.project runs RexSimplify by default; with the mismatched type system it re-derived decimal arithmetic at precision 19 and then wrapped the result in a preserving CAST(... AS DECIMAL(38,0)) to keep the declared type.

For TPC-H queries 8 and 14 this surfaced as the numerator SUM(CAST(CASE ... AS DECIMAL(19,0))) while the denominator was left uncast. In PostgreSQL, where the DECIMAL columns are arbitrary-precision NUMERIC, the cast truncated the fractional values in the numerator only, so the round-tripped SQL produced different results from the reference. This was latent until Calcite 1.42, which changed RelDataTypeSystemImpl so the max numeric precision/scale are derived from getMaxPrecision(DECIMAL)/getMaxScale(DECIMAL).

Changes

  • Build the Substrait-to-Calcite RelBuilder with the Substrait type system, exposed via a new ConverterProvider.getTypeSystem() accessor (derived from the type factory so the two never disagree).
  • SubstraitTypeSystem gains a public no-arg constructor because Calcite's Frameworks/Avatica machinery re-instantiates the type system from its class name when it is supplied to a FrameworkConfig.
  • Removed queries 8 and 14 from the integration test's expected-failures list, and dropped the now-empty expected-failures code path.

Testing

  • ./gradlew :isthmus:test — all unit tests pass.
  • ./gradlew :isthmus:integrationTest — all 22 TPC-H queries (including 8 and 14) now produce results matching the reference SQL in PostgreSQL.

Closes #954
Closes #955

🤖 Generated with AI

When converting Substrait to Calcite, the RelBuilder was created via
RelBuilder.create(Frameworks...) without specifying a type system, so its
internal RexBuilder used Calcite's default type system (max decimal precision
19) while the converted expressions carried SubstraitTypeSystem types
(precision 38). RelBuilder.project runs RexSimplify by default; with the
mismatched type system it re-derived decimal arithmetic at precision 19 and
then wrapped the result in a preserving CAST(... AS DECIMAL(38,0)) to keep the
declared type.

For TPC-H queries 8 and 14 this surfaced as the numerator
SUM(CAST(CASE ... AS DECIMAL(19,0))) while the denominator was left uncast. In
PostgreSQL, where the DECIMAL columns are arbitrary-precision NUMERIC, the cast
truncated the fractional values in the numerator only, so the round-tripped SQL
produced different results from the reference. This was latent until Calcite
1.42, which changed RelDataTypeSystemImpl so the max numeric precision/scale are
derived from getMaxPrecision(DECIMAL)/getMaxScale(DECIMAL).

Build the Substrait-to-Calcite RelBuilder with the Substrait type system,
exposed via a new ConverterProvider.getTypeSystem() accessor (derived from the
type factory so the two never disagree). SubstraitTypeSystem gains a public
no-arg constructor because Calcite's Frameworks/Avatica machinery
re-instantiates the type system from its class name when it is supplied to a
FrameworkConfig.

Removed queries 8 and 14 from the integration test's expected-failures list, and
dropped the now-empty expected-failures code path.

Closes substrait-io#954
Closes substrait-io#955

@mbwhite mbwhite left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bestbeforetoday bestbeforetoday merged commit b370df5 into substrait-io:main Jun 30, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants