Skip to content

feat(serdes): Add filesystem serdes for large payload storage#494

Open
ayushiahjolia wants to merge 2 commits into
mainfrom
feat/filesystem-serdes
Open

feat(serdes): Add filesystem serdes for large payload storage#494
ayushiahjolia wants to merge 2 commits into
mainfrom
feat/filesystem-serdes

Conversation

@ayushiahjolia

Copy link
Copy Markdown
Contributor

Issue #, if available: #479

Description of changes:

  • Adds create_filesystem_serdes() - a SerDes implementation that stores serialized values on a durable filesystem (Amazon S3 Files or Amazon EFS mounted to Lambda). This keeps checkpoint sizes small regardless of payload
    size, with optional inline preview for observability.

Features -

  • ALWAYS mode: Every value written to file; checkpoint stores file pointer
  • OVERFLOW mode: Inline if < 256KB, overflow to file if larger
  • URI path encoding: Human-readable paths derived from ARN (<functionName>/<executionName>/<invocationId>/<operationId>.json)
  • HASH path encoding: Fixed-length SHA-256 paths for safety
  • Preview: Optional inline preview in checkpoint envelope for console/API visibility without reading the full file
  • ARN parsing: Compact directory structure from durable execution ARNs

Tests -

  • Added unit tests, e2e tests with replay and integration tests.

Next PRs -

  • Cloud integration tests (requires S3 Files or EFS mount infrastructure)
  • Documentation update in aws-durable-execution-docs repo

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@ayushiahjolia ayushiahjolia force-pushed the feat/filesystem-serdes branch from 4bd0a83 to 84e6e0c Compare June 26, 2026 23:56
@ayushiahjolia ayushiahjolia marked this pull request as ready for review June 27, 2026 00:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants