Appendix G: The Tool Evaluation Scorecard¶
๐งญ The principles transfer; the keystrokes don't. This appendix is a reusable decision instrument for Chapter 1's "Tool Selection is Part of Engineering Structure" (ยง5.4) and the quarterly evaluation loop (ยง12). Use it to score any candidate tool against your permanent framework โ before you commit a single workflow to it.
1. The Five Criteria¶
Every candidate tool is scored on five criteria. Each is a question your permanent framework can answer, independent of the tool's marketing:
| # | Criterion | The question it answers |
|---|---|---|
| 1 | Task fit | Does this tool fit the task's complexity, team size, and workflow? |
| 2 | Repo-respect | Does it treat the repository as the source of truth (reads AGENTS.md / AI_CONTEXT.md), or does it hide structure in its own config? |
| 3 | Portability | Can you migrate away without losing structure? Is everything it needs in plain files? |
| 4 | Security & cost posture | Does it follow least privilege? Can you route models per task, control token spend, and keep sensitive data local? |
| 5 | Migration cost | What is the real cost of switching in: install, config, team retraining, integration with the existing Makefile surface? |
2. Scoring Rubric¶
Score each criterion 1โ5:
| Score | Meaning |
|---|---|
| 5 | Excellent โ no friction, fully aligned with the framework |
| 4 | Good โ minor friction, easy to work around |
| 3 | Acceptable โ some friction, needs a workaround |
| 2 | Weak โ significant friction, structure at risk |
| 1 | Poor โ misaligned with the framework |
3. The Scorecard Table¶
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโฌโโโโโโโโโฌโโโโโโโโ
โ Criterion โ Weightโ Score โ Total โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโผโโโโโโโโโผโโโโโโโโค
โ 1. Task fit โ W1 โ S1 โ W1รS1 โ
โ 2. Repo-respect โ W2 โ S2 โ W2รS2 โ
โ 3. Portability โ W3 โ S3 โ W3รS3 โ
โ 4. Security & cost posture โ W4 โ S4 โ W4รS4 โ
โ 5. Migration cost โ W5 โ S5 โ W5รS5 โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโผโโโโโโโโโผโโโโโโโโค
โ TOTAL โ 1.0 โ โ ฮฃ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโดโโโโโโโโโดโโโโโโโโ
How to set weights. Weights sum to 1.0 and reflect your context โ the weighting is part of the engineering decision. A solo author building a book might weight Repo-respect high; a team under audit might weight Security & cost posture highest; a team that switches tools often weights Portability and Migration cost highest.
How to compare. Score every candidate in one sitting, with the same weights, against the same task. The highest total wins โ but only adopt if the winner also passes the non-scored gate below.
4. The Non-Scored Gate¶
A scorecard can't capture everything. Before adopting any tool, the winning candidate must also pass the Orchestrator's Checklist (Chapter 1, ยง10.7):
โก Am I choosing this tool for the right reason? (not hype or novelty)
โก Does it respect my repository as source of truth?
โก Can I migrate to another tool without losing my structure?
โก Am I building software, or am I building a tool portfolio?
If any answer is "no", the score is not enough โ fix the structure gap or reject the tool.
5. Worked Example: Executing "Improve a Module"¶
Scoring three candidates for the execute phase of a task ("improve module X so it passes make validate"), weights tuned for a small, cost-conscious team:
| Criterion | Weight | Aider | Trae (IDE) | Plain bash |
|---|---|---|---|---|
| Task fit | 0.30 | 5 | 3 | 1 |
| Repo-respect | 0.25 | 5 | 3 | 5 |
| Portability | 0.15 | 4 | 3 | 5 |
| Security & cost | 0.20 | 4 | 3 | 5 |
| Migration cost | 0.10 | 4 | 3 | 4 |
| Weighted total | 1.00 | 4.55 | 3.00 | 3.40 |
Reading the result: Aider wins for AI-judgment tasks (needs no UI, reads the repo, routes models per task); bash wins for deterministic steps; the IDE agent remains the right choice for the build phase (Chapter 2, ยง1.4). The correct answer is usually not one tool โ it's the right tool per phase.
6. When to Run It¶
- Quarterly โ alongside the Tool Selection Update Pattern (Chapter 1, ยง12): new tools and major updates get scored against the same weights, keeping the comparison stable over time.
- Before adoption โ any tool that enters your stack must be scored and documented, with the reasoning recorded next to the tool configuration.
- On pain points โ when a workflow fights you, re-score the incumbent vs. alternatives before migrating.
The scorecard documents the why, so the decision outlasts the tool โ exactly the point of the tool-agnostic foundation.