Replace custom issue assistant with GitHub Agentic Workflow#158
Replace custom issue assistant with GitHub Agentic Workflow#158
Conversation
There was a problem hiding this comment.
Pull request overview
This PR replaces a custom 500+ line issue assistant implementation with GitHub's official Agentic Workflows framework. The change removes custom YAML/JavaScript code in favor of a natural language workflow definition that compiles to a managed execution environment with built-in security features.
Changes:
- Removed custom
.github/workflows/issue-assistant.yml(577 lines) and.github/issue-assistant/src/security.js(249 lines) - Added
.github/workflows/msdo-issue-assistant.md(113 lines) - natural language workflow definition - Added
.github/workflows/msdo-issue-assistant.lock.yml(1062 lines) - auto-generated compiled workflow with sandboxing, threat detection, and safe outputs - Added
.github/aw/actions-lock.jsonfor action version pinning and.gitattributesfor lock file merge strategy
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
.github/workflows/issue-assistant.yml |
Removed custom issue assistant workflow with state management, security validation, and AI response generation |
.github/issue-assistant/src/security.js |
Removed custom security module with prompt injection detection, rate limiting, and input sanitization |
.github/workflows/msdo-issue-assistant.md |
Added natural language workflow defining issue triage bot behavior, rules, and response patterns |
.github/workflows/msdo-issue-assistant.lock.yml |
Added compiled workflow with sandboxed execution, threat detection, and managed safe outputs |
.github/aw/actions-lock.json |
Added action version locks for reproducible builds |
.gitattributes |
Added merge strategy for auto-generated lock files |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| --- | ||
|
|
There was a problem hiding this comment.
The old workflow enforced a 120-second cooldown between bot responses (MIN_RESPONSE_INTERVAL_SECONDS) to prevent rapid-fire responses and give users time to read and respond. This cooldown is removed in the new implementation. Without this throttling, the bot could respond multiple times in quick succession if a user posts multiple comments rapidly, creating a poor user experience and potentially wasting API quota.
There was a problem hiding this comment.
gh-aw has built-in rate limiting. The max: 4 limits total comments per issue.
| --- | ||
|
|
||
| # MSDO Issue Triage Assistant | ||
|
|
There was a problem hiding this comment.
The workflow requires seven labels to exist in the repository: bug, feature, enhancement, documentation, question, needs-info, and needs-maintainer. If any of these labels don't exist, the add_labels safe output will fail. The PR description and workflow don't mention this prerequisite. Repository administrators should either create these labels before deploying this workflow, or the workflow should be configured to handle missing labels gracefully (e.g., by auto-creating them or skipping label application with a warning).
| - learn.microsoft.com | ||
| - docs.microsoft.com | ||
| - aka.ms | ||
| 3. **Don't respond** if: | ||
| - The commenter is not the issue author (unless it's a new issue) | ||
| - You've already responded multiple times to this issue | ||
| 4. **Be honest** - if you don't know something, say so and suggest checking the wiki or waiting for a maintainer | ||
|
|
||
| ## Response Examples | ||
|
|
||
| **User asks:** "What tools does MSDO support?" | ||
| **Response:** MSDO supports these security analysis tools: antimalware (Windows only), bandit, binskim, checkov, eslint, templateanalyzer, terrascan, and trivy. Tools are automatically detected based on your repository content, or you can specify them explicitly. See the [Tools documentation](https://github.com/microsoft/security-devops-action/wiki) for details. | ||
|
|
There was a problem hiding this comment.
The old implementation included custom prompt injection detection patterns and suspicious content detection (security.js lines 14-91). It checked for patterns like "ignore previous instructions", "disregard your instructions", "you are now", etc., and excessive word repetition. While the new workflow has threat detection built-in, it's unclear if it provides equivalent protection against prompt injection attacks. The PR description should clarify what prompt injection protections are included in the GitHub Agentic Workflow framework and whether additional custom patterns are needed.
There was a problem hiding this comment.
gh-aw has built-in threat detection job that scans ALL outputs before posting. It's actually more robust than our custom regex patterns.
|
|
||
| **Common configuration:** | ||
| ```yaml | ||
| - uses: microsoft/security-devops-action@latest | ||
| with: | ||
| tools: 'bandit,eslint,trivy' | ||
| config: 'path/to/gdnconfig' | ||
| ``` | ||
| **Wiki reference:** https://github.com/microsoft/security-devops-action/wiki | ||
| ## Your Task | ||
| When a new issue is opened or a user comments: | ||
There was a problem hiding this comment.
The new workflow does not include any mechanism to load wiki content for the AI to reference. The old implementation loaded wiki pages from either .github/wiki-context.md (cached) or by cloning the repository wiki and extracting specific pages (Home, FAQ, Troubleshooting, Configuration, Tools). The new markdown file only references the wiki URL at line 56 but doesn't provide the actual wiki content to the AI. This significantly reduces the AI's ability to provide accurate, wiki-based answers since it only has the hardcoded knowledge in lines 44-54.
| --- | ||
|
|
There was a problem hiding this comment.
The old workflow included per-user-per-hour rate limiting (12 responses per user per hour) enforced in security.js checkRateLimit function. This protection is completely removed in the new implementation. While there's a max of 4 comments per issue, a malicious user could open 100 issues in an hour and receive 400 bot responses, potentially exhausting API quotas or COPILOT_GITHUB_TOKEN limits. The new workflow should include equivalent per-user rate limiting to prevent abuse.
Replace Issue Assistant with GitHub Agentic Workflows
Changes
.github/issue-assistant/folder andsecurity.js.github/workflows/issue-assistant.ymlmsdo-issue-assistant.md- new agentic workflow (natural language)msdo-issue-assistant.lock.yml- compiled workflow with guardrailsBenefits
COPILOT_GITHUB_TOKEN)How it works
bug,feature,question,needs-info,needs-maintainer