[prompt-clustering] Prompt Clustering Analysis - 2026-10-11

[github-actions[bot]]

Summary

Analysis Period: Last 30 days (2026-09-11 → 2026-10-11)
Total Tasks Analyzed: 569
Clusters Identified: 10
Overall Success Rate: 83.8% (477 merged / 569 total)

Full Analysis Report

General Insights

• Most Common Task Type: Cluster 6 — general workflow runtime fixes (117 tasks, 20.6% of total)
• Highest Success Rate: Cluster 7 — firewall/network egress logging (94.6% merged)
• Most Complex Tasks: Cluster 1 — CLI/MCP version & dependency pinning (avg. 82.9 files changed per PR)
• Outliers: 1 outlier cluster identified (Cluster 1 — CLI/MCP version & dependency pinning), with a 71.9% merge rate vs. 83.8% overall (gap: 11.9 points) across 57 PRs
• Data limitation: agenticworkflows logs was probed for turn/cost/duration metrics but a single 3-day, count-limited request already took ~18s and returned only a 1-hour slice — pulling 30 days of data across hundreds of runs is not feasible in this run's time budget. Findings below rely on PR metadata, comments, and reviews only. File/comment/review averages have 100% coverage across all 10 clusters (all 569 PRs have cached full detail).

Cluster Analysis

Cluster 6: General workflow runtime fixes

• Size: 117 tasks (20.6% of total)
• Success Rate: 82.9%
• Top Keywords: workflow, workflows, fixes, update, github, agent
• Characteristics: Broad catch-all of runtime bug fixes and small feature additions across workflow execution
• Example PRs: Fix Dependabot configuration syntax #60122, Remove cloud-hypervisor security review warning #60304, Grant PR read permission to centralized slash commands #60347

---

Cluster 0: Safe-output validation & error handling

• Size: 73 tasks (12.8% of total)
• Success Rate: 78.1%
• Top Keywords: validation, safe, issue, output, outputs, failure
• Characteristics: Validation logic and failure handling for the safe-outputs pipeline
• Example PRs: Prevent Jira smoke workflow failures when credentials are unavailable #60199, Validate external safe-output secrets before activation #60267, Add additive global footers to safe-output bodies #60275

---

Cluster 2: Cross-engine runtime & automation bugfixes

• Size: 71 tasks (12.5% of total)
• Success Rate: 87.3%
• Top Keywords: em, sub, aic, run, pr sous, chef
• Characteristics: Mixed bugfixes touching engine-specific behavior (Codex, Claude, Copilot) and PR automation (PR Sous Chef, AI Credits)
• Example PRs: Prevent Copilot CLI staging for Gemini ARC/DinD workflows #62415, Preserve step-installed skills during base-branch restore; allow Claude's Skill tool #62419, Fix Codex engine.config heredoc indentation #62420

---

Cluster 1: CLI/MCP version & dependency pinning (OUTLIER)

• Size: 57 tasks (10.0% of total)
• Success Rate: 71.9% — 11.9 points below overall
• Top Keywords: version, aw, gh aw, gh, mcp, cli
• Characteristics: Bumping/pinning gh-aw-firewall, gh-aw-mcpg, Copilot/Claude CLI versions, hardening CLI setup/download paths
• Example PRs: Require immutable gh-aw action-mode references #60227, Add aw.yml default for schedule seed #60429, Harden gh-aw CLI setup downloads #60436

---

Cluster 9: Usage artifacts & detection coverage

• Size: 61 tasks (10.7% of total)
• Success Rate: 88.5%
• Top Keywords: usage, artifact, job, detection, coverage, regression
• Characteristics: Token/usage accounting artifacts, regression-detection coverage
• Example PRs: Cache audit results in logs JSONL #60183, Preserve continue-on-error on generated agent jobs #60350, Prevent daily AIC guardrail blocks after pre-agent failures #60355

---

Cluster 5: Model routing & agentic engine support

• Size: 56 tasks (9.8% of total)
• Success Rate: 82.1%
• Top Keywords: model, copilot, routing, engine, agent, claude
• Characteristics: Model-routing logic and per-engine (Copilot/Claude/Codex) feature support
• Example PRs: Route Grok 4.5 and 4.6 through the Responses API #60139, Fix PR Sous Chef threat-detection model #60321, Add reasoning effort support to assign-to-agent #60346

---

Cluster 7: Firewall/network egress logging

• Size: 55 tasks (9.7% of total)
• Success Rate: 94.6% (highest)
• Top Keywords: blocked, network, firewall, summary, details, domain
• Characteristics: Firewall-blocked-domain reporting and network egress diagnostics — the most reliably-mergeable task type
• Example PRs: Restore Copilot usage accounting without the agent sandbox #60134, Honor bot allowlists before confused-deputy checks #60135, Fix daily AIC snapshot completeness and quota handling #60136

---

Cluster 4: GitHub Actions job/log handling

• Size: 33 tasks (5.8% of total)
• Success Rate: 81.8%
• Top Keywords: work, actions, job, check, pr, logs
• Characteristics: Fixing failing CI jobs and Actions log/check handling, often [WIP]-titled auto-triggered fixes
• Example PRs: [WIP] Fix failing GitHub Actions job 'build' #60289, [WIP] Fix failing GitHub Actions job lint-go-golangci #60316, [WIP] Fix failing GitHub Actions job #60645

---

Cluster 8: Multi-target checkout & regression coverage

• Size: 28 tasks (4.9% of total)
• Success Rate: 89.3%
• Top Keywords: target, repository, checkout, shared, regression coverage, regression
• Characteristics: Multi-repository/target checkout logic for logs downloads and regression tests
• Example PRs: Stop starting log targets after reaching the shared count limit #60266, Stop queued log targets after timeout #60273, Fix --count/--timeout wiring for multi-target logs downloads #60308

---

Cluster 3: Safe-output ledger & artifact handling

• Size: 18 tasks (3.2% of total)
• Success Rate: 88.9%
• Top Keywords: ledger, output, workflow, artifact, safe output, safe
• Characteristics: Repo-memory ledger backend and safe-output artifact plumbing — smallest cluster
• Example PRs: Add repo-memory backend for daily AIC guardrail #62958, [WIP] Add distributed ledger capability to repo-memory #64199, Move daily workflow review history to repo-memory ledgers #64337

Success Rate by Cluster

Cluster	Theme	Tasks	Success Rate	Avg Files Changed	Avg Comments	Avg Reviews
7	Firewall/network egress logging	55	94.6%	74.8	10.8	5.1
8	Multi-target checkout & regression coverage	28	89.3%	11.1	7.1	3.7
9	Usage artifacts & detection coverage	61	88.5%	80.8	6.6	3.5
3	Safe-output ledger & artifact handling	18	88.9%	34.0	9.1	4.8
2	Cross-engine runtime & automation bugfixes	71	87.3%	46.6	15.0	7.0
6	General workflow runtime fixes	117	82.9%	39.9	5.2	3.1
5	Model routing & agentic engine support	56	82.1%	70.6	6.8	4.0
4	GitHub Actions job/log handling	33	81.8%	21.5	6.2	3.8
0	Safe-output validation & error handling	73	78.1%	27.4	5.3	3.4
1	CLI/MCP version & dependency pinning	57	71.9%	82.9	5.1	2.8

Full Data Table (sample, 3 PRs per cluster)

PR #	Title	Cluster	Outcome	Comments	Reviews
60199	Prevent Jira smoke workflow failures when credentials are unavailable	0	Closed	1	0
60267	Validate external safe-output secrets before activation	0	Merged	7	1
60275	Add additive global footers to safe-output bodies	0	Merged	4	4
60227	Require immutable gh-aw action-mode references	1	Closed	0	1
60429	Add aw.yml default for schedule seed	1	Closed	0	0
60436	Harden gh-aw CLI setup downloads	1	Closed	5	2
62415	Prevent Copilot CLI staging for Gemini ARC/DinD workflows	2	Merged	3	2
62419	Preserve step-installed skills during base-branch restore	2	Merged	18	7
62420	Fix Codex engine.config heredoc indentation	2	Merged	10	7
62958	Add repo-memory backend for daily AIC guardrail	3	Merged	21	7
64199	[WIP] Add distributed ledger capability to repo-memory	3	Merged	41	18
64337	Move daily workflow review history to repo-memory ledgers	3	Merged	1	1
60289	[WIP] Fix failing GitHub Actions job 'build'	4	Merged	1	1
60316	[WIP] Fix failing GitHub Actions job lint-go-golangci	4	Merged	7	4
60645	[WIP] Fix failing GitHub Actions job	4	Closed	0	0
60139	Route Grok 4.5 and 4.6 through the Responses API	5	Merged	6	3
60321	Fix PR Sous Chef threat-detection model	5	Closed	19	2
60346	Add reasoning effort support to assign-to-agent	5	Merged	6	5
60122	Fix Dependabot configuration syntax	6	Merged	1	1
60304	Remove cloud-hypervisor security review warning	6	Merged	7	2
60347	Grant PR read permission to centralized slash commands	6	Merged	1	1
60134	Restore Copilot usage accounting without the agent sandbox	7	Merged	2	1
60135	Honor bot allowlists before confused-deputy checks	7	Merged	26	7
60136	Fix daily AIC snapshot completeness and quota handling	7	Merged	33	5
60266	Stop starting log targets after reaching the shared count limit	8	Merged	6	2
60273	Stop queued log targets after timeout	8	Merged	6	2
60308	Fix --count/--timeout wiring for multi-target logs downloads	8	Merged	8	3
60183	Cache audit results in logs JSONL	9	Merged	9	1
60350	Preserve continue-on-error on generated agent jobs	9	Merged	3	1
60355	Prevent daily AIC guardrail blocks after pre-agent failures	9	Merged	11	4

Key Findings

1. CLI/MCP version-pinning work is the least reliable task type: Cluster 1 (version bumps for gh-aw-firewall, gh-aw-mcpg, Copilot/Claude CLI pins, CLI setup hardening) merges at 71.9% vs. 83.8% overall — the only cluster meeting the outlier threshold (≥15 PRs, ≥10-point gap).
2. Firewall/network diagnostics is the most dependable task type: Cluster 7 merges at 94.6%, well-scoped and narrowly-themed (blocked-domain summaries), suggesting narrow, single-purpose prompts correlate with higher merge rates.
3. Engagement, not complexity, separates merged from unmerged PRs in the outlier cluster: within Cluster 1, merged PRs actually changed more files on average (97.3 vs 45.9) and received more reviewer attention (3.4 reviews vs 1.2) than unmerged ones — the unmerged PRs were smaller but got markedly less review engagement, not more pushback.

Lowest-Merge Cluster Root Cause

• Cluster: 1 — CLI/MCP version & dependency pinning
• Merge Rate Gap: 71.9% vs 83.8% overall (gap: 11.9 points)
• Primary Blocker: Silent/unexplained closures — the majority of unmerged PRs in this cluster were closed with little-to-no reviewer engagement rather than rejected after active review
• Confidence: Medium (direct textual evidence recovered for 6 of 16 unmerged PRs; the remaining 8 closed with zero comments/reviews, so the underlying maintainer reasoning for those cannot be confirmed from PR metadata alone)
• Evidence PRs: Disable uv Actions caching and isolate sandbox runtime paths #66958 (explicitly split into smaller PRs per maintainer request after heavy review), Refresh available container image versions and pins #66122 (closed after multiple smoke-test engines failed: Codex ❌, Playwright ❌, Web-fetch ❌), Harden gh-aw CLI setup downloads #60436 (review flagged "moderate findings" for Windows parity and retry-duration limits), Pin Copilot CLI to 1.0.80 to avoid MCP regressions #62482/Add engine-independent tool profile configuration #62951/Investigate campaign-query memory estimation gap #64583 (closed with 0 comments and 0 reviews — no recorded rationale)

Blocker Category	Count	Share	Notes
Other/unknown (silent closure, 0–minimal engagement)	8	57%	#60429, #62482, #62488, #62797, #62951, #63045, #64583, #66121 — closed with little or no comment/review trail
Review friction	3	21%	#60227 ("unresolved setup pins"), #60436 (Windows parity / retry limits), #62412 (config contract inconsistency, MCP permission regression)
Scope/complexity mismatch	2	14%	#64268 (repeated scope changes to env-var naming convention), #66958 (maintainer explicitly asked for it to be split into smaller PRs; later superseded by #67017)
CI/test failure	1	7%	#66122 (multiple smoke-test engines failed: Codex, Playwright, Web-fetch)

(2 additional PRs in the cluster — #67240, #67597 — remain open as of the analysis date and were excluded from the closed-PR blocker tally.)

Recommendations

1. Narrow version-pin PRs to single-dependency, single-purpose diffs. Cluster 1's merged PRs (e.g. "Bump gh-aw-firewall to v0.28.16 and gh-aw-mcpg to v0.4.21") succeed when scoped to one or two version bumps; broader "harden CLI setup" or multi-concern PRs in the same cluster stall more often.
2. Add an explicit close-reason comment for abandoned/superseded automation PRs. 57% of this cluster's unmerged PRs closed with zero review/comment trail, making it impossible to learn why they failed — a one-line "superseded by #X" or "deprioritized because Y" comment would materially improve future analysis and prompt tuning.
3. Model cluster 7's prompt pattern (firewall/network diagnostics) for other low-performing clusters — its narrow, single-outcome framing ("summarize blocked domains") correlates with the highest merge rate (94.6%) in the dataset.
4. For Cluster 1, route reviewer attention proactively rather than relying on self-selection. Since the data shows unmerged PRs got less review engagement rather than more rejection, consider auto-requesting a reviewer or adding a stale-PR nudge specifically for CLI/dependency-pin PRs to prevent them from going quiet before anyone weighs in.

---

Generated by Prompt Clustering Analysis (Run: 38129655546)

Generated by 📊 Copilot Agent Prompt Clustering Analysis · claude · agent · 150.7 AIC · ⊞ 6.8K · ◷

• expires on Oct 12, 2026, 2:05 AM UTC-08:00

[github-actions[bot]]

This discussion has been marked as outdated by Copilot Agent Prompt Clustering Analysis.

A newer discussion is available at Discussion #67963.