Skip to content

Add proxy integration tests and missing UTS spec coverage#462

Open
paddybyers wants to merge 8 commits intouts-integrationfrom
uts-integration-proxy
Open

Add proxy integration tests and missing UTS spec coverage#462
paddybyers wants to merge 8 commits intouts-integrationfrom
uts-integration-proxy

Conversation

@paddybyers
Copy link
Copy Markdown
Member

@paddybyers paddybyers commented May 3, 2026

Summary

  • Proxy-based integration test specs: A programmable Go test proxy sits between the SDK and the Ably sandbox, enabling fault injection (drop connections, inject errors, delay responses) for testing connection recovery, heartbeat, auth reauth, presence reentry, and REST fault handling. The proxy is controlled via a REST API on a control port — each test creates a session with declarative rules, then inspects an event log after the test completes.
  • Reorganised UTS docs: Moved guides into docs/, rewrote README as a concise entry point, added writing-derived-tests.md for translating UTS specs to language-specific tests.
  • Added missing unit/integration UTS specs covering 14 spec areas that previously had no test coverage.

Proxy approach

The test proxy (uts/proxy/) is a Go binary that:

  1. Accepts sessions via POST /sessions — each session gets its own listener port and targets a specific upstream (sandbox-realtime/sandbox-rest)
  2. Supports declarative rules that match on message type, action, channel, HTTP method/path, and fire actions (drop, delay, inject message, return HTTP error)
  3. Supports imperative actions via POST /sessions/:id/actions for ad-hoc fault injection mid-test
  4. Records all events to a queryable log via GET /sessions/:id/log
  5. Sessions auto-cleanup on DELETE or timeout

SDK test harnesses (e.g. ably-js) build and auto-launch the proxy from their test setup — no manual proxy startup needed.

Proxy integration test specs added

Spec file Spec points Description
connection_resume.md RTN15a-j, RTN16d/k/l Connection resume, recovery key, fatal resume errors
channel_faults.md RTL2f, RTL4f, RTL13b Channel detach/reattach on transport failure
heartbeat.md RTN23a/b Heartbeat timeout detection
rest_faults.md RSC15a/d, TO3l8 REST fallback host selection
auth_reauth.md RSA4b/c/d, RTN22 Auth token renewal and server-initiated reauth
presence_reentry.md RTP17a-f Presence auto re-entry after resume
connection_open_failures.md (existing, extended) Connection open failure scenarios

Missing unit/integration specs added

Spec file Spec points Description
backoff_jitter_test.md RTB1a/b Backoff coefficient and jitter range
token_expiry_non_renewable_test.md RSA4a1/a2 Non-renewable token expiry handling
auth_callback_errors_test.md RSA4c/d/e/f Auth callback error scenarios
connection_recovery_test.md RTN16d-l Recovery key format and usage
forwards_compatibility_test.md RSF1/RTF1 Unknown protocol fields/actions
network_change_test.md RTN20a-c Network change events
push_channels.md (unit + integration) RSH7a-e Push channel subscriptions
channel_subscribe.md (extended) RTL22/MFI Message filter subscriptions
channel_publish.md (extended) RTN7e Publish error matches errorReason
rest_channel_attributes.md (extended) CHD2/CHM2 All ChannelMetrics fields

Test plan

  • Verify proxy binary builds: cd uts/proxy && go build -o test-proxy .
  • Review proxy test specs match features.md requirements
  • Verify completion-status.md reflects all new specs
  • Cross-check with ably-js test implementations (separate PR in ably-js)

🤖 Generated with Claude Code

paddybyers and others added 4 commits May 3, 2026 09:55
@paddybyers @claude
Fix mislabeled spec points in proxy integration test specs
9031eb7 
- channel_faults.md: Remove incorrect RTL4h reference; the test that
  replaces ATTACHED with ERROR is RTL14, not RTL4h. Update test name,
  channel name prefix, and rule comment accordingly.

- rest_faults.md: Fix spec point references from RSC15a to RSC15m/REC2c2
  (the correct spec points for fallback host behaviour when the fallback
  domain set is empty). Update test title, comments, and channel names.

- Update proxy binary.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@paddybyers @claude
Add tier 1+2 proxy integration test specs and proxy writing guidance
7f85583 
New proxy test specs:
- Test 21 (RTN15j): Fatal ERROR mid-session → FAILED state
- Test 22 (RTN15g/g2): connectionStateTtl expiry prevents resume
- Test 23 (RTN19a/a2): Unacked messages resent after resume
- Test 24 (RTL12): ATTACHED with resumed=false → channel UPDATE event
- Test 25 (RTL3d): Channels reattach after connection recovery
- Test 26 (RTN22/RTC8a): Server-initiated re-authentication
- Test 27 (RTP17i/RTP17g): Presence re-enter on non-resumed reattach

Also adds "Writing Proxy Tests" guidance to integration-testing.md
covering late fault injection and two approaches for early faults.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@paddybyers @claude
Reorganise UTS docs: move guides to docs/, rewrite README
f4751ed 
Move completion-status.md, integration-testing.md, and the two skill
files (write-test-spec, write-derived-tests) into a new uts/docs/
directory. Strip skill frontmatter from the writing guides. Fix stale
integration-proxy/ references to match actual integration/proxy/ layout.
Rewrite README to reflect current spec counts and link to docs.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@paddybyers @claude
Add UTS specs for missing test coverage
c8f5b47 
New specs:
- RTB1: backoff and jitter for connection retries
- RSA4a: token expiry with non-renewable tokens
- RSA4c/d/f: auth callback error handling
- RTN16: connection recovery (recovery key, msgSerial, channelSerials)
- RTN20: network change events (browser-only)
- RSF1/RTF1: forwards compatibility (unknown fields/actions)
- RSH7: push channel subscriptions (unit + integration)

Extended specs:
- RTL22/MFI: message filter subscriptions (channel_subscribe)
- RTN7e: error reason on publish failure (channel_publish)
- CHD2/CHM2: all ChannelMetrics fields (rest_channel_attributes)
- RTN16d/RTN16l: proxy-based recovery tests (connection_resume)

Updates completion-status.md and README spec counts.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions Bot temporarily deployed to staging/pull/462 May 3, 2026 19:38 Inactive
@paddybyers paddybyers mentioned this pull request May 3, 2026
Open
4 tasks
@paddybyers @claude
Add RSC15f test: expired fallback not resurrected by late in-flight s…
fcfb8a5 
…uccess

Tests that a request completing successfully against a cached fallback
host after fallbackRetryTimeout has expired does not re-pin that host.
Uses the existing onRequest handler with a held PendingRequest pattern
rather than introducing a new mock primitive.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions Bot temporarily deployed to staging/pull/462 May 3, 2026 19:50 Inactive
@paddybyers @claude
Add RTN15a proxy test variant for TCP close without WebSocket close f…
22e5304 
…rame

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions Bot temporarily deployed to staging/pull/462 May 4, 2026 08:11 Inactive
@paddybyers @claude
Remove proxy source code; now lives at ably/uts-proxy
f4cebc8 
The Go test proxy has moved to https://github.com/ably/uts-proxy.
Updated references in README, writing guide, and proxy infrastructure
spec to point to the external repo.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions github-actions Bot temporarily deployed to staging/pull/462 May 4, 2026 08:39 Inactive
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@paddybyers