Advent of AI 2025 - Day 6: Automating GitHub Issue Triage with Goose

Advent of AI 2025 - Day 6: Automating GitHub Issue Triage with Goose

Boost on DEVToot it!Share on BlueskyTweet It!Share on LinkedIn

4 minute read

I've edited this post, but AI helped. These are meant to be quick posts related to the Advent of AI. I don't have time if I'm doing one of these each day to spend a couple hours on a post. 😅

Advent of AI 2025 - Day 6: Automating GitHub Issue Triage with Goose

I built an automated GitHub issue triage system using Goose for Advent of AI 2025 Day 6. The system categorizes festival feedback, assigns priorities, and welcomes first-time contributors. It runs in GitHub Actions and handles everything from label creation to posting helpful comments.

The Challenge

Day 6 of Advent of AI asked us to build a feedback management system for a fictional winter festival. The festival coordinator was drowning in feedback, ranging from "the heaters are broken" to "where's the lost and found?"

The goal was to automatically categorize issues into urgent, bug, feature, or question, and assign priority levels. I went with Goose running in GitHub Actions because it felt like the right fit for this kind of automation.

What I Built

The system automatically:

  • Categorizes issues (urgent, bug, feature, question)
  • Assigns priority levels (high, medium, low)
  • Detects sentiment (positive, neutral, negative)
  • Creates required labels if they don't exist
  • Posts helpful triage comments explaining the categorization
  • Welcomes first-time contributors with a friendly message

All of this happens within seconds of someone creating or editing an issue.

first time contributor flow

How It Works

The Workflow

The GitHub Actions workflow triggers on issue creation or edit. Here's the flow:

  1. Installs and configures Goose CLI with OpenRouter
  2. Checks if this is the user's first issue (no prior issues or PRs)
  3. Posts a welcome message if it's their first contribution
  4. Auto-creates any missing labels
  5. Runs Goose to analyze the issue
  6. Applies labels based on Goose's analysis
  7. Posts a triage comment with reasoning

The Triage Prompt

Instead of hardcoding the triage logic in the workflow, I put it in a separate file at .github/triage_prompt.txt. This makes it easy to tweak the criteria without touching the workflow.

The prompt defines:

  • What makes something urgent vs a bug vs a feature
  • How to assign priority levels
  • When to mark sentiment
  • The expected JSON output format

Goose reads this template, I substitute the issue title and body, and it returns structured JSON with labels and a comment.

Bonus: First-Time Contributor Welcome

I added detection for first-time contributors. If someone has never created an issue or PR in the repo, they get a warm welcome message before the automated triage comment runs.

The detection script checks their issue count and PR count via GitHub CLI. If it's their first issue and they have no PRs, welcome message. Otherwise, skip straight to triage.

Real Examples

I created three test issues to verify everything worked:

Issue #1: Heating System Failing


Title: Heating system failing in the Main Tent
Body: The heaters stopped working tonight. Very cold, several booths reported no heat. Needs immediate attention.

Labels: urgent, priority:high, sentiment:negative

Issue #2: Photo Booth Feature Request


Title: Photo booth should print festival-themed frames  
Body: Suggestion: Add printed frames with festival graphics so people can take physical souvenirs.

Labels: feature, priority:low, sentiment:positive

Issue #3: Lost and Found Question


Title: Where is the lost & found located?
Body: Visitor found a jacket and wants to return it. Where should they go?

Labels: question, priority:low, sentiment:neutral

You can see the actual issues and workflow runs in the GitHub repo. The Actions tab shows each triage run in detail, including what Goose returned and which labels got applied.

Customization

Everything's externalized for easy editing:

Triage logic: Edit .github/triage_prompt.txt to change how issues get categorized

Welcome message: Edit .github/WELCOME_MESSAGE.md to customize the first-timer greeting

First-time detection: Edit .github/check_first_time_contributor.sh to change what counts as "first-time" (currently: first issue AND no PRs)

No YAML knowledge required to customize any of these.

Keeping It Simple

I could have gone wild with bonus features but stuck to the MVP plus one extra: the first-time contributor welcome. That felt like it added real value without overcomplicating things.

The system is production-ready. It handles errors, creates missing labels automatically, and logs everything for debugging. The only manual step is adding your OPENROUTER_API_KEY as a GitHub secret.

Win for Maintainers

As a maintainer, something like this is great. It triages and auto labels. Sure it might not get the labeling right all the time, but this removes a chunk of burden off of open source maintainers.

The Results

The system triages issues in under 2 minutes. It's running live on the repo and you can create test issues to see it in action. The documentation is thorough (maybe too thorough) and includes setup instructions, customization guides, and troubleshooting tips.

Check out the repo if you want to see the full implementation or steal the setup for your own projects.

OpenGraph image for https://github.com/nickytonline/advent-of-ai-2025-day-6-triager

Try submitting an issue and let me know how it goes!

Wrapping Up

That's a wrap! Goose for AI analysis piece reliably, GitHub Actions for automation.

If you want to stay in touch, all my socials are on nickyt.online.

Until the next one!

Photo by Amin Alizadeh on Unsplash