feat(webapp): Add test payload AI generation to the test page based on payload schemas#3188
feat(webapp): Add test payload AI generation to the test page based on payload schemas#3188
Conversation
Add a tabbed sidebar to the Test page for standard tasks, reusing the ClientTabs pattern from the Query page. - Options tab: existing sidebar content (machine, version, queue, etc.) - AI tab: AI-powered payload generation with streaming, supports JSON Schema, inferred schema from recent runs, and task source code lookup via tool calling for tasks without schemas - Schema tab: displays payload JSON Schema (from schemaTask), inferred schema (from recent runs via @jsonhero/schema-infer), or empty state with schemaTask docs and example code Data layer changes: - Surface payloadSchema and inferredPayloadSchema from TestTaskPresenter - Add payloadSchema and fileId to WorkerDeploymentWithWorkerTasks type - Decompress zlib-deflated source files for AI context New resource route for AI payload generation with SSE streaming, task source code tool calling, and prompt engineering to prevent hallucinated payloads.
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (2)
🚧 Files skipped from review as they are similar to previous changes (2)
📜 Recent review details⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (26)
WalkthroughAdds a three-tab sidebar (Options, AI, Schema) to the Test page via TestSidebarTabs, with AIPayloadTabContent for AI-assisted payload generation and SchemaTabContent for schema display. Implements on-the-fly payload schema inference in TestTaskPresenter using Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes 🚥 Pre-merge checks | ✅ 1 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (1 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ericallam
changed the title
ericallam
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
There was a problem hiding this comment.
🧹 Nitpick comments (1)
apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx (1)
298-306: Greedy regex may capture too much text.The fallback regex
(\{[\s\S]*\}|\[[\s\S]*\])is greedy—it will match from the first{to the last}in the entire text. If the LLM includes multiple JSON-like structures or explanatory text with braces, this could capture invalid content.Consider using a non-greedy pattern or attempting to parse incrementally.
Suggested fix using non-greedy matching
- const jsonMatch = text.match(/(\{[\s\S]*\}|\[[\s\S]*\])/); + // Use non-greedy match, try multiple candidates if first fails + const jsonMatches = text.matchAll(/(\{[\s\S]*?\}|\[[\s\S]*?\])/g); + for (const match of jsonMatches) { + try { + return JSON.stringify(JSON.parse(match[1]), null, 2); + } catch { + continue; + } + }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@apps/webapp/app/routes/resources.orgs`.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx around lines 298 - 306, The current greedy regex (/\{[\s\S]*\}|\[[\s\S]*\]/) used to build jsonMatch from text can span too much; update the match logic to use a non‑greedy pattern (e.g. change to /\{[\s\S]*?\}|\[[\s\S]*?\]/) or implement an incremental/balanced-brace scan that finds the first well-formed JSON object/array and then JSON.parse it; locate the code that references text and jsonMatch in the ai-generate-payload handler and replace the regex with the non‑greedy version or the incremental parser so you only attempt to parse a single balanced JSON substring.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In
`@apps/webapp/app/routes/resources.orgs`.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx:
- Around line 298-306: The current greedy regex (/\{[\s\S]*\}|\[[\s\S]*\]/) used
to build jsonMatch from text can span too much; update the match logic to use a
non‑greedy pattern (e.g. change to /\{[\s\S]*?\}|\[[\s\S]*?\]/) or implement an
incremental/balanced-brace scan that finds the first well-formed JSON
object/array and then JSON.parse it; locate the code that references text and
jsonMatch in the ai-generate-payload handler and replace the regex with the
non‑greedy version or the incremental parser so you only attempt to parse a
single balanced JSON substring.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro
Run ID: 8a0997a1-6286-4116-ad2d-bf6dfb35c576
📒 Files selected for processing (1)
apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: units / packages / 📊 Merge Reports
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (3, 8)
- GitHub Check: units / internal / 🧪 Unit Tests: Internal (2, 8)
🧰 Additional context used
📓 Path-based instructions (7)
**/*.{ts,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
**/*.{ts,tsx}: Use types over interfaces for TypeScript
Avoid using enums; prefer string unions or const objects instead
**/*.{ts,tsx}: In TypeScript SDK usage, always import from@trigger.dev/sdk, never from@trigger.dev/sdk/v3or use deprecated client.defineJob
Import from@trigger.dev/coresubpaths only, never from the root
Use the Run Engine 2.0 (@internal/run-engine) and redis-worker for all new work, not legacy V1 MarQS queue or deprecated V1 functions
Files:
apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx
{packages/core,apps/webapp}/**/*.{ts,tsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Use zod for validation in packages/core and apps/webapp
Files:
apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx
**/*.{ts,tsx,js,jsx}
📄 CodeRabbit inference engine (.github/copilot-instructions.md)
Use function declarations instead of default exports
Files:
apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx
apps/webapp/app/**/*.{ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc)
Access all environment variables through the
envexport ofenv.server.tsinstead of directly accessingprocess.envin the Trigger.dev webapp
Files:
apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx
apps/webapp/**/*.{ts,tsx}
📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc)
apps/webapp/**/*.{ts,tsx}: When importing from@trigger.dev/corein the webapp, use subpath exports from the package.json instead of importing from the root path
Follow the Remix 2.1.0 and Express server conventions when updating the main trigger.dev webapp
Files:
apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx
**/*.{js,ts,jsx,tsx,json,md,yaml,yml}
📄 CodeRabbit inference engine (AGENTS.md)
Format code using Prettier before committing
Files:
apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx
apps/**/*.{ts,tsx,js}
📄 CodeRabbit inference engine (CLAUDE.md)
When modifying only server components (apps/webapp/, apps/supervisor/, etc.) with no package changes, add a .server-changes/ file instead of a changeset
Files:
apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx
🧠 Learnings (8)
📚 Learning: 2026-03-02T12:42:56.114Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: apps/webapp/CLAUDE.md:0-0
Timestamp: 2026-03-02T12:42:56.114Z
Learning: Applies to apps/webapp/app/routes/**/*.ts : Use Remix flat-file route convention with dot-separated segments (e.g., `api.v1.tasks.$taskId.trigger.ts` for `/api/v1/tasks/:taskId/trigger`)
Applied to files:
apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx
📚 Learning: 2025-11-27T16:26:58.661Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/webapp.mdc:0-0
Timestamp: 2025-11-27T16:26:58.661Z
Learning: Applies to apps/webapp/**/*.{ts,tsx} : Follow the Remix 2.1.0 and Express server conventions when updating the main trigger.dev webapp
Applied to files:
apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Generate example payloads for tasks when possible
Applied to files:
apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Export tasks with unique IDs within the project to enable proper task discovery and execution
Applied to files:
apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx
📚 Learning: 2025-11-27T16:27:35.304Z
Learnt from: CR
Repo: triggerdotdev/trigger.dev PR: 0
File: .cursor/rules/writing-tasks.mdc:0-0
Timestamp: 2025-11-27T16:27:35.304Z
Learning: Applies to **/trigger/**/*.{ts,tsx,js,jsx} : Use `.withStreams()` to subscribe to realtime streams from task metadata in addition to run changes
Applied to files:
apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx
📚 Learning: 2025-12-08T15:19:56.823Z
Learnt from: 0ski
Repo: triggerdotdev/trigger.dev PR: 2760
File: apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.runs.$runParam/route.tsx:278-281
Timestamp: 2025-12-08T15:19:56.823Z
Learning: In apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.runs.$runParam/route.tsx, the tableState search parameter uses intentional double-encoding: the parameter value contains a URL-encoded URLSearchParams string, so decodeURIComponent(value("tableState") ?? "") is required to fully decode it before parsing with new URLSearchParams(). This pattern allows bundling multiple filter/pagination params as a single search parameter.
Applied to files:
apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx
📚 Learning: 2026-02-03T18:27:40.429Z
Learnt from: 0ski
Repo: triggerdotdev/trigger.dev PR: 2994
File: apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.environment-variables/route.tsx:553-555
Timestamp: 2026-02-03T18:27:40.429Z
Learning: In apps/webapp/app/routes/_app.orgs.$organizationSlug.projects.$projectParam.env.$envParam.environment-variables/route.tsx, the menu buttons (e.g., Edit with PencilSquareIcon) in the TableCellMenu are intentionally icon-only with no text labels as a compact UI pattern. This is a deliberate design choice for this route; preserve the icon-only behavior for consistency in this file.
Applied to files:
apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx
📚 Learning: 2026-02-11T16:37:32.429Z
Learnt from: matt-aitken
Repo: triggerdotdev/trigger.dev PR: 3019
File: apps/webapp/app/components/primitives/charts/Card.tsx:26-30
Timestamp: 2026-02-11T16:37:32.429Z
Learning: In projects using react-grid-layout, avoid relying on drag-handle class to imply draggability. Ensure drag-handle elements only affect dragging when the parent grid item is configured draggable in the layout; conditionally apply cursor styles based on the draggable prop. This improves correctness and accessibility.
Applied to files:
apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx
🔇 Additional comments (6)
apps/webapp/app/routes/resources.orgs.$organizationSlug.projects.$projectParam.env.$envParam.test.ai-generate-payload.tsx (6)
1-14: LGTM!Imports are well-organized and follow project conventions—using
envfrom~/env.server,$replicafor read operations, and appropriate SDK imports.
15-20: LGTM!Request schema uses zod as per project guidelines, with appropriate field definitions.
65-70: Stop logging user prompts.The
promptfield contains user-provided content that may include sensitive data. This was previously flagged and should be removed from the log.Suggested fix
logger.info("[AI payload] Generating payload", { taskIdentifier, hasPayloadSchema: !!payloadSchema, hasCurrentPayload: !!currentPayload, - prompt, + promptLength: prompt.length, });
88-105: Verify model name and consider lowering temperature.
gpt-5-miniis not a recognized OpenAI model name. Did you meangpt-4o-mini?Additionally,
temperature: 1maximizes randomness which may produce inconsistent JSON structures. For structured output generation, a lower temperature (0.3-0.7) typically yields more reliable results.Suggested fix
const result = streamText({ - model: openai(env.AI_RUN_FILTER_MODEL ?? "gpt-5-mini"), - temperature: 1, + model: openai(env.AI_RUN_FILTER_MODEL ?? "gpt-4o-mini"), + temperature: 0.7, system: systemPrompt,[raise_major_issue, request_verification]
What are the available OpenAI model names for the API as of 2026?
201-214: Stop logging source contents and scope the output.Two previously-flagged issues remain:
- Line 210: Logging
contents: decompressedleaks proprietary source code to logs- Line 213: Returns the entire file to the model, which can exceed token limits on larger files
Suggested fix
+ const MAX_SOURCE_CHARS = 8000; // Reasonable limit for LLM context + + const truncated = decompressed.length > MAX_SOURCE_CHARS; + const sourceSnippet = truncated + ? decompressed.slice(0, MAX_SOURCE_CHARS) + "\n\n// ... truncated ..." + : decompressed; + logger.info("[AI payload] Found task source code", { taskIdentifier, filePath: file.filePath, - contentLength: decompressed.length, - contents: decompressed, + contentLength: sourceSnippet.length, + truncated, }); - return `File: ${file.filePath}\n\n${decompressed}`; + return `File: ${file.filePath}\n\n${sourceSnippet}`;
223-283: LGTM!
getTaskFromDeploymentproperly handles the nullable worker case (matching the PR commit fix).buildSystemPromptprovides comprehensive, well-structured instructions for payload generation with appropriate handling of schema presence/absence.
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
2 participants
Add a tabbed sidebar to the Test page for standard tasks, reusing the
ClientTabs pattern from the Query page.
Schema, inferred schema from recent runs, and task source code lookup
via tool calling for tasks without schemas
schema (from recent runs via @jsonhero/schema-infer), or empty state
with schemaTask docs and example code
Data layer changes: