test(core): Add E2E coverage for Instance AI workflow setup apply actions (no-changelog)#28907
test(core): Add E2E coverage for Instance AI workflow setup apply actions (no-changelog)#28907
Conversation
…ions (no-changelog) Adds two Playwright specs under `@capability:proxy` that exercise the full workflow-setup confirmation UI end-to-end: - Full-wizard apply: user fills an empty HTTP Request URL, applies, and both the parameter edit and the pre-attached Slack credential persist to the workflow JSON. - Later / partial apply: completing one card and clicking Later on the last step partial-applies — the completed card's edits stick, the untouched node stays untouched. Supporting changes: - `InstanceAiWorkflowSetup.vue`: data-test-ids for the step counter, applying spinner, listening callout, error banner, and the applied / partial / deferred submit states. - `InstanceAiPage.ts`: page-object locators for every setup-card affordance used by the new specs. - `workflow-api-helper.ts`: `getWorkflow(id)` helper to fetch a workflow via the REST API for persistence assertions. - `fixtures.ts`: Slack `users.profile.get` success mock installed before each instance-ai test so the FE's background credential test passes during replay (existing recordings hit real Slack and returned `invalid_auth`). https://linear.app/n8n/issue/INS-113
…-changelog) Pins the contract between the confirmation payload and the saved workflow JSON so a future change to `resumeData` handling or to the `applyNodeChanges` merge logic is caught at the unit layer: - `applyNodeChanges` — asserts that merged `nodeParameters` values survive into the saved workflow JSON (HTTP URL filled in via the apply path ends up on the persisted node). - `workflows.tool` setup action — asserts that `resumeData.nodeParameters` is forwarded to `applyNodeChanges` unchanged on the apply path. - `credGroupKey` — three new cases covering same-credential-type node grouping, HTTP Request distinct-URL splits, and credential-type isolation when node types differ.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…ation assertions (no-changelog)
Performance ComparisonComparing current → latest master → 14-day baseline Memory consumption baseline with starter plan resources
docker-stats
Idle baseline with Instance AI module loaded
How to read this table
|
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
1 issue found across 14 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="packages/testing/playwright/tests/e2e/instance-ai/instance-ai-workflow-setup.spec.ts">
<violation number="1" location="packages/testing/playwright/tests/e2e/instance-ai/instance-ai-workflow-setup.spec.ts:74">
P2: According to linked Linear issue INS-113, confirm mode should be seeded with one auto-applied valid credential. This test file creates a Slack credential in both tests, but the reset endpoint only clears workflows/threads, so the first credential persists into the second test and can make confirm-mode auto-apply non-deterministic.</violation>
</file>
Architecture diagram
sequenceDiagram
participant Test as Playwright / User
participant FE as Frontend (Vue/Pinia)
participant BE as Backend (Tool Executor)
participant LLM as Orchestrator LLM
participant Mock as MockServer (Slack)
participant DB as Workflow Service / DB
Note over Test,DB: Workflow Setup Initiation
Test->>FE: Send "Set up workflow" message
FE->>BE: POST /instance-ai/messages
BE->>LLM: Process request
LLM-->>BE: tool_call: workflows(action: 'setup')
BE->>DB: getAsWorkflowJSON()
BE->>BE: analyzeWorkflow() (Identify missing params/creds)
BE-->>FE: Return Setup Card UI metadata
Note over Test,DB: Configuration Interaction
alt User Edits Parameters
Test->>FE: Fill parameter (e.g., HTTP URL)
FE->>FE: CHANGED: Debounced (200ms) sync to paramValues
else Credential Validation
FE->>BE: POST /credentials/test
BE->>Mock: NEW: Slack users.profile.get mock (E2E fixture)
Mock-->>FE: Credential valid
end
Note over Test,DB: Apply / Persistence Flow
alt Action: Apply (Full or Partial)
Test->>FE: Click "Apply"
FE->>FE: NEW: Mount applying-spinner (isApplying = true)
FE->>BE: POST /instance-ai/confirm (resumeData with nodeParameters)
Note over BE: NEW: resumeData.nodeParameters forwarded to tool
BE->>BE: CHANGED: applyNodeChanges() merges edits into JSON
BE->>DB: updateFromWorkflowJSON() (Persist to DB)
BE-->>FE: SSE Tool Result Stream
FE->>FE: NEW: Unmount spinner only after SSE result
else Action: Later (Defer)
Test->>FE: Click "Later"
FE->>BE: POST /instance-ai/confirm (action: 'defer')
BE-->>FE: SSE Tool Result Stream
Note over FE,DB: Workflow remains unchanged in DB
end
Note over Test,DB: Verification
FE->>DB: NEW: Refresh workflow via GET
DB-->>FE: Updated Workflow JSON
FE->>Test: Render success / deferred state
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review, or fix all with cubic.
…hangelog) The instance-ai reset endpoint wipes threads and workflows between tests but leaves credentials in place. Both B2 tests were creating a Slack credential with the same name, so by the second test two matching credentials existed and confirm mode's "single valid credential auto- applies" precondition was no longer guaranteed.
Instance AI Workflow Eval Results8/8 built | 5 run(s) | pass@5: 70% | pass^5: 16%
Failure detailsCreate a workflow that handles contact form submissions via a webhook. / happy-path — 4/5 passed
Create a workflow that handles contact form submissions via a webhook. / missing-fields — 4/5 passed
Create a workflow that handles contact form submissions via a webhook. / partial-action-failure — 0/5 passed
Create a workflow that handles contact form submissions via a webhook. / empty-message — 4/5 passed
Create a workflow that handles contact form submissions via a webhook. / invalid-email — 0/5 passed
Get all the Linear issues created in the last 2 weeks. Filter them for / happy-path — 0/5 passed
Get all the Linear issues created in the last 2 weeks. Filter them for / multi-team-creator — 0/5 passed
Get all the Linear issues created in the last 2 weeks. Filter them for / no-cross-team-issues — 3/5 passed
Get all the Linear issues created in the last 2 weeks. Filter them for / unknown-creator — 2/5 passed
Get all the Linear issues created in the last 2 weeks. Filter them for / api-error — 0/5 passed
Every day, get the posts made in the past day on 3 different Slack cha / happy-path — 4/5 passed
Every day, get the posts made in the past day on 3 different Slack cha / empty-channel — 3/5 passed
Every day, get the posts made in the past day on 3 different Slack cha / high-volume — 2/5 passed
Every day, get the posts made in the past day on 3 different Slack cha / channel-not-found — 0/5 passed
Every day, get the posts made in the past day on 3 different Slack cha / insufficient-permissions — 0/5 passed
Every day, fetch all open GitHub issues from repository 'acme-corp/bac / happy-path — 0/5 passed
Every day, fetch all open GitHub issues from repository 'acme-corp/bac / no-bugs — 1/5 passed
Create a workflow that receives webhook notifications with a JSON body / high-priority — 3/5 passed
Create a workflow that receives webhook notifications with a JSON body / medium-priority — 3/5 passed
Create a workflow that receives webhook notifications with a JSON body / low-priority — 3/5 passed
Fetch the latest posts from the JSONPlaceholder API (GET https://jsonp / happy-path — 4/5 passed
Fetch the latest posts from the JSONPlaceholder API (GET https://jsonp / empty-response — 4/5 passed
Fetch the latest posts from the JSONPlaceholder API (GET https://jsonp / all-filtered — 2/5 passed
Every hour, check the current weather for London, New York, and Tokyo / happy-path — 2/5 passed
Every hour, check the current weather for London, New York, and Tokyo / no-alerts — 2/5 passed
|
1 participant
Summary
Adds end-to-end Playwright coverage for the Instance AI workflow-setup confirmation UI — the confirm-mode apply, confirm-mode defer, full-wizard apply, and full-wizard Later / partial apply flows triggered from the setup card. Also pins the backend contract at the unit layer so a future regression in
resumeData→applyNodeChanges→ persisted workflow is caught outside of e2e.Both specs run in the
sqlite:e2econtainer project and stay green across 10 consecutive sequential runs (--repeat-each=5 --workers=1).Test cases
instance-ai-workflow-setup.spec.tsshould render confirm mode when a single valid credential auto-appliesinstance-ai-workflow-setup.spec.tsshould defer the whole setup when Later is clicked in confirm modeinstance-ai-workflow-setup-actions.spec.tsshould apply parameter and credential edits and persist them to the workflowurl) and the existing credential assignment to the workflow JSONinstance-ai-workflow-setup-actions.spec.tsshould partially apply completed cards when Later is clicked on the last stepparamValuesas the source of truth across step changes), and partial apply persists the completed card's edits (HTTPurl) while the untouched Slack node keeps its empty credentialsAll four specs exercise the real
workflows(setup)suspend/resume contract end-to-end — the orchestrator LLM, thePOST /instance-ai/confirmendpoint,applyNodeChanges→workflowService.update, the SSE tool-result stream back to the FE, and the post-apply workflow GET.Behind the scenes
A few things had to line up to make the apply path observable and deterministic from a Playwright harness:
InstanceAiWorkflowSetup.vueunmounts synchronously the momentisApplyingflips true, so waiting for the setup card to hide resolves before the backend save — the subsequent workflow GET would race the commit. Theinstance-ai-workflow-setup-applyingspinner stays visible for the full apply window (POST → tool-result SSE → save commit), so watching it appear and disappear guarantees persistence before we assert.ParameterInputdebounces@update:model-valueby 200 ms, so a shortwaitForTimeout(300)after.fill()lets the value propagate intoparamValuesbefore Apply fires.list-workflowstool is sensitive to same-named workflows across parallel workers during recording — unique names prevent the LLM from slipping into a disambiguation response.users.profile.getsuccess mock in the fixture. The FE'sbuildNodeCredentialsintentionally excludes any credential whose test has failed. Seed tokens in these tests are fake, so the fixture installs a success mock on mockserver before loading recordings — mockserver's FIFO matching serves it ahead of any recorded Slack response. Same shape as the existing LLM mocks.Regression tests
Two unit tests pin the tool-layer contract:
applyNodeChanges— mergednodeParametersvalues land on the saved workflow JSON.workflows(setup)resume path —resumeData.nodeParametersis forwarded toapplyNodeChangesunchanged.Plus three
credGroupKeycases covering same-credential-type node grouping, HTTP Request distinct-URL splits, and cross-type isolation.Related Linear tickets
https://linear.app/n8n/issue/INS-113
Review / Merge checklist