Skip to content

feat: new differ v4#479

Open
bzp2010 wants to merge 5 commits into
mainfrom
bzp/refactor-differ-v4
Open

feat: new differ v4#479
bzp2010 wants to merge 5 commits into
mainfrom
bzp/refactor-differ-v4

Conversation

@bzp2010

@bzp2010 bzp2010 commented Jun 20, 2026

Copy link
Copy Markdown
Collaborator

Description

Refactor differ v4, no major changes were made, and it passed the old test cases.

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible

Summary by CodeRabbit

Release Notes

  • Bug Fixes
    • CLI dumps now remove server-assigned id fields by default (kept only when --with-id is provided).
    • Stream route exports no longer nest id under metadata.
    • Validation now treats plugins under consumer groups as optional.
  • New Features
    • Sync and validation now use the newer differ engine for more consistent, ordered change detection.
  • Refactor
    • Updated the differ implementation and migrated related tasks and tests.

@coderabbitai

coderabbitai Bot commented Jun 20, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: d91a2f95-1c45-48ba-9abd-c9798bb3c2cf

📥 Commits

Reviewing files that changed from the base of the PR and between 388a675 and 5e55045.

📒 Files selected for processing (1)
  • schema.json
✅ Files skipped from review due to trivial changes (1)
  • schema.json

📝 Walkthrough

Walkthrough

Introduces DifferV4, a schema-driven configuration differ that replaces DifferV3. The SDK gains FieldListType, Zod schema metadata annotations, and RESOURCE_DIFFER_META to drive the new differ algorithm. All CLI commands, server handlers, and backend e2e tests are updated to use the new Differ alias. The stream route transformer no longer emits a metadata field, and the dump command's cleanup helper is narrowed to remove only id fields (not metadata).

Changes

DifferV4 Implementation and Call-site Migration

Layer / File(s) Summary
Field registry and type definitions
libs/sdk/src/core/resource.ts, libs/sdk/src/core/field-registry.ts, libs/sdk/src/core/index.ts, schema.json
Adds FieldListType enum for merge strategies (MAP, OBJECT_MAP, ATOMIC, ARRAY), creates field-registry.ts with FieldMeta union type and differFieldRegistry, exports field-registry APIs from core index, makes ConsumerGroup.plugins optional, and aligns JSON schema required fields.
Zod schema metadata annotations
libs/sdk/src/core/schema.ts
Imports FieldListType and withDifferMeta; annotates six core schemas (routeSchema, streamRouteSchema, serviceBaseSchema, sslSchema, consumerSchema, consumerGroupSchema) with listType/nested/configKey/stripItemFields metadata; exports previously-internal schemas.
Differ metadata schema and resource contracts
libs/sdk/src/core/differ.ts
Refactors imports, adds CollectionKind enum (ARRAY/RECORD), implements readFieldMeta to extract field annotations from Zod schemas, defines ResourceDifferMeta interface, and populates RESOURCE_DIFFER_META with per-ResourceType metadata including name/ID generation, default-type resolution, and per-field merge strategies.
DifferV4 core algorithm
libs/differ/src/differv4.ts, libs/differ/src/index.ts
Implements DifferV4 class with extractTuples, diffResource, handleCreate/handleUpdate/handleDelete, extractSubConfig, applyAtomicStrips, postprocessSubEvent, diffPlugins, and mergeDefault; uses order table for event sorting and schema metadata from RESOURCE_DIFFER_META to drive diffing; updates index.ts to alias DifferDifferV4.
Stream route transformer and dump id-removal
libs/backend-apisix/src/transformer.ts, apps/cli/src/command/utils.ts, apps/cli/src/command/dump.command.ts
Removes metadata: { id } from ToADC.transformStreamRoute output; ensures FromADC.transformConsumerGroup always sets plugins; renames recursiveRemoveMetadataField to recursiveRemoveIdField (strips id only, not metadata); updates dump command to call the renamed helper.
CLI command integration with Differ
apps/cli/src/server/sync.ts, apps/cli/src/server/validate.ts, apps/cli/src/tasks/diff.ts
Updates sync, validate, and diff commands to import and call Differ instead of DifferV3.
All tests migrated to Differ
apps/cli/src/command/utils.spec.ts, libs/differ/src/test/basic.spec.ts, libs/differ/src/test/consumer.spec.ts, libs/differ/src/test/custom-id.spec.ts, libs/differ/src/test/service-upstream.spec.ts, libs/differ/src/test/upstream.spec.ts, libs/differ/src/test/usecase.spec.ts, libs/backend-api7/e2e/validate.e2e-spec.ts, libs/backend-apisix-standalone/e2e/cache.e2e-spec.ts, libs/backend-apisix-standalone/e2e/resources/service-inline-upstream.e2e-spec.ts, libs/backend-apisix-standalone/e2e/validate.e2e-spec.ts, libs/backend-apisix/e2e/validate.e2e-spec.ts
CLI utils test updated to assert recursiveRemoveIdField removes top-level id fields; all differ unit tests migrated from DifferV3.diff to Differ.diff; all backend e2e validation/cache/inline-upstream specs updated to call Differ.diff for event generation.

Sequence Diagram(s)

sequenceDiagram
  participant CLI as CLI (diff/sync/validate)
  participant Differ as Differ (DifferV4)
  participant RESOURCE_DIFFER_META as RESOURCE_DIFFER_META
  participant Schema as Zod Schema Metadata
  participant Backend as Backend

  CLI->>Differ: Differ.diff(local, remote, defaultValue)
  Differ->>RESOURCE_DIFFER_META: iterate resource entries
  RESOURCE_DIFFER_META->>Schema: readFieldMeta(schema)
  Schema-->>RESOURCE_DIFFER_META: FieldMeta per field
  loop each ResourceType in RESOURCE_DIFFER_META
    Differ->>Differ: extractTuples(config, meta)
    Differ->>Differ: diffResource(localTuples, remoteTuples, meta)
    note over Differ: handleCreate / handleUpdate / handleDelete
    Differ->>Differ: applyAtomicStrips, mergeDefault, diffPlugins
    Differ->>Differ: postprocessSubEvent for nested resources
  end
  Differ->>Differ: flatten ONLY_SUB_EVENTS, sort by order table
  Differ-->>CLI: sorted Event[]
  CLI->>Backend: backend.sync(diff) / backend.validate(events)
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Suggested reviewers

  • LiteSun
  • guoqqqi
  • juzhiyuan
🚥 Pre-merge checks | ✅ 5 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
E2e Test Quality Review ⚠️ Warning PR adds 49 test files but contributor unchecked the "added tests" checklist item. Tests exist and include E2E testing against real services, but: (1) They are all test migrations for DifferV4 backw... Either check the "added tests" checklist or remove the test files and explain why they weren't needed. Add specific tests for new DifferV4 features, fix misleading test names, and document what functionality each test validates.
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'feat: new differ v4' accurately describes the main change: introducing a new DifferV4 implementation while replacing DifferV3 across the codebase.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Security Check ✅ Passed Security review of DifferV4 refactoring found no critical vulnerabilities. Debug logging is controlled by ADC_DIFFER_DEBUG flag and includes full config objects only when explicitly enabled. SSL ke...

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch bzp/refactor-differ-v4

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (1)
libs/sdk/src/core/differ.ts (1)

68-73: ⚡ Quick win

Include configKey in FieldMeta for MAP fields to avoid downstream hardcoding.

The schema already carries configKey metadata (e.g., consumer credentials), but the FieldMeta type drops it. That forces manual mapping later and can drift from schema annotations.

💡 Proposed refactor
 export type FieldMeta =
-  | { listType: typeof FieldListType.MAP; listMapKey: string; nested?: boolean }
+  | {
+      listType: typeof FieldListType.MAP;
+      listMapKey: string;
+      nested?: boolean;
+      configKey?: keyof InternalConfiguration;
+    }
   | { listType: typeof FieldListType.OBJECT_MAP }
   | { listType: typeof FieldListType.ATOMIC; strip?: boolean }
   | { listType: typeof FieldListType.ARRAY; stripItemFields?: string[] };
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@libs/sdk/src/core/differ.ts` around lines 68 - 73, The FieldMeta type variant
for FieldListType.MAP is missing the configKey field that carries metadata in
the schema, which forces downstream manual mapping. Add a configKey property to
the MAP variant of FieldMeta (the one with listType: typeof FieldListType.MAP;
listMapKey: string; nested?: boolean) to capture and preserve the schema
annotation and prevent downstream hardcoding.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@libs/differ/src/differv4.ts`:
- Around line 84-93: The debug log in the differ.logger?.debug call is logging
full config and resource payloads (local, remote, defaultValue) which can expose
sensitive information like tokens, auth headers, or private keys. Replace these
full object logs with safer summaries such as counts of keys, array lengths, or
hashes instead of the complete objects. Keep the transactionId and message
fields as they are safe to log. Apply this same fix to the other similar logging
statements in the file at the locations mentioned in the comment (lines 351-363
and 445-455).
- Around line 494-505: The isObjectButNotArray function incorrectly returns true
for null values since typeof null === 'object' in JavaScript, causing the
mergeDefault method to recursively call itself with null as the defaults
parameter, which then fails when trying to iterate entries on null. Fix this by
adding a null check to the isObjectButNotArray function definition so it
explicitly excludes null from being considered as an object (add val !== null to
the condition before checking typeof val === 'object').

In `@libs/sdk/src/core/schema.ts`:
- Line 367: The plugins field in the consumer group schema is currently
required, which breaks compatibility with existing configs that omit the
consumer_group.plugins setting. Locate the plugins field definition in the
consumer group schema (the line with pluginsSchema.meta({ listType:
FieldListType.OBJECT_MAP })) and add the appropriate optional modifier (likely
.optional()) to make the plugins field optional, consistent with other resource
schemas.

---

Nitpick comments:
In `@libs/sdk/src/core/differ.ts`:
- Around line 68-73: The FieldMeta type variant for FieldListType.MAP is missing
the configKey field that carries metadata in the schema, which forces downstream
manual mapping. Add a configKey property to the MAP variant of FieldMeta (the
one with listType: typeof FieldListType.MAP; listMapKey: string; nested?:
boolean) to capture and preserve the schema annotation and prevent downstream
hardcoding.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 8767be4f-5e29-4ad6-b622-d2e3e8599729

📥 Commits

Reviewing files that changed from the base of the PR and between c4c041f and 259ca3c.

📒 Files selected for processing (24)
  • apps/cli/src/command/dump.command.ts
  • apps/cli/src/command/utils.spec.ts
  • apps/cli/src/command/utils.ts
  • apps/cli/src/server/sync.ts
  • apps/cli/src/server/validate.ts
  • apps/cli/src/tasks/diff.ts
  • libs/backend-api7/e2e/validate.e2e-spec.ts
  • libs/backend-apisix-standalone/e2e/cache.e2e-spec.ts
  • libs/backend-apisix-standalone/e2e/resources/service-inline-upstream.e2e-spec.ts
  • libs/backend-apisix-standalone/e2e/validate.e2e-spec.ts
  • libs/backend-apisix/e2e/validate.e2e-spec.ts
  • libs/backend-apisix/src/transformer.ts
  • libs/differ/src/differv3.ts
  • libs/differ/src/differv4.ts
  • libs/differ/src/index.ts
  • libs/differ/src/test/basic.spec.ts
  • libs/differ/src/test/consumer.spec.ts
  • libs/differ/src/test/custom-id.spec.ts
  • libs/differ/src/test/service-upstream.spec.ts
  • libs/differ/src/test/upstream.spec.ts
  • libs/differ/src/test/usecase.spec.ts
  • libs/sdk/src/core/differ.ts
  • libs/sdk/src/core/resource.ts
  • libs/sdk/src/core/schema.ts
💤 Files with no reviewable changes (2)
  • libs/differ/src/differv3.ts
  • libs/backend-apisix/src/transformer.ts

Comment thread libs/differ/src/differv4.ts
Comment thread libs/differ/src/differv4.ts
Comment thread libs/sdk/src/core/schema.ts Outdated
@bzp2010 bzp2010 added test/api7 Trigger the API7 test on the PR test/apisix-standalone Trigger the APISIX standalone test on the PR labels Jun 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

test/api7 Trigger the API7 test on the PR test/apisix-standalone Trigger the APISIX standalone test on the PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant