12-Factor Agents Compliance Analysis

Reference: 12-Factor Agents

Input Parameters

Parameter	Description	Required
INLINECODE0	Path to documentation directory (for existing analyses)	Optional
INLINECODE1

Root path of the codebase to analyze | Required |

Analysis Framework

Factor 1: Natural Language to Tool Calls

Principle: Convert natural language inputs into structured, deterministic tool calls using schema-validated outputs.

Search Patterns:
CODEBLOCK0

File Patterns: **/agents/*.py, **/schemas/*.py, INLINECODE4

Compliance Criteria:

Level	Criteria
Strong	All LLM outputs use Pydantic/dataclass schemas with validators
Partial

Some outputs typed, but dict returns or unvalidated strings exist |
| Weak | LLM returns raw strings parsed manually or with regex |

Anti-patterns:

• - json.loads(llm_response) without schema validation
• INLINECODE6 or regex parsing of LLM responses
• INLINECODE7 return types from agents
• No validation between LLM output and handler execution

---

Factor 2: Own Your Prompts

Principle: Treat prompts as first-class code you control, version, and iterate on.

Search Patterns:
CODEBLOCK1

File Patterns: **/prompts/**, **/templates/**, INLINECODE10

Compliance Criteria:

Level	Criteria
Strong	Prompts in separate files, templated (Jinja2), versioned
Partial

Prompts as module constants, some parameterization |
| Weak | Prompts hardcoded inline in functions, f-strings only |

Anti-patterns:

• - f"You are a {role}..." inline in agent methods
• Prompts mixed with business logic
• No way to iterate on prompts without code changes
• No prompt versioning or A/B testing capability

---

Factor 3: Own Your Context Window

Principle: Control how history, state, and tool results are formatted for the LLM.

Search Patterns:
CODEBLOCK2

File Patterns: **/context/*.py, **/state/*.py, INLINECODE14

Compliance Criteria:

Level	Criteria
Strong	Custom context format, token optimization, typed events, compaction
Partial

Basic message history with some structure |
| Weak | Raw message accumulation, standard OpenAI format only |

Anti-patterns:

• - Unbounded message accumulation
• Large artifacts embedded inline (diffs, files)
• No agent-specific context filtering
• Same context for all agent types

---

Factor 4: Tools Are Structured Outputs

Principle: Tools produce schema-validated JSON that triggers deterministic code, not magic function calls.

Search Patterns:
CODEBLOCK3

File Patterns: **/tools/*.py, **/handlers/*.py, INLINECODE17

Compliance Criteria:

Level	Criteria
Strong	All tool outputs schema-validated, handlers type-safe
Partial

Most tools typed, some loose dict returns |
| Weak | Tools return arbitrary dicts, no validation layer |

Anti-patterns:

• - Tool handlers that directly execute LLM output
• INLINECODE18 or exec() on LLM-generated code
• No separation between decision (LLM) and execution (code)
• Magic method dispatch based on string matching

---

Factor 5: Unify Execution State

Principle: Merge execution state (step, retries) with business state (messages, results).

Search Patterns:
CODEBLOCK4

File Patterns: **/state/*.py, **/models/*.py, INLINECODE22

Compliance Criteria:

Level	Criteria
Strong	Single serializable state object with all execution metadata
Partial

State exists but split across systems (memory + DB) |
| Weak | Execution state scattered, requires multiple queries to reconstruct |

Anti-patterns:

• - Retry count stored separately from task state
• Error history in logs but not in state
• LangGraph checkpoints + separate database storage
• No unified event thread

---

Factor 6: Launch/Pause/Resume

Principle: Agents support simple APIs for launching, pausing at any point, and resuming.

Search Patterns:
CODEBLOCK5

File Patterns: **/routes/*.py, **/api/*.py, INLINECODE25

Compliance Criteria:

Level	Criteria
Strong	REST API + webhook resume, pause at any point including mid-tool
Partial

Launch/pause/resume exists but only at coarse-grained points |
| Weak | CLI-only launch, no pause/resume capability |

Anti-patterns:

• - Blocking input() or confirm() calls
• No way to resume after process restart
• Approval only at plan level, not per-tool
• No webhook-based resume from external systems

---

Factor 7: Contact Humans with Tools

Principle: Human contact is a tool call with question, options, and urgency.

Search Patterns:
CODEBLOCK6

File Patterns: **/agents/*.py, **/tools/*.py, INLINECODE30

Compliance Criteria:

Level	Criteria
Strong	INLINECODE31 tool with question/options/urgency/format
Partial

Approval gates exist but hardcoded in graph structure |
| Weak | Blocking CLI prompts, no tool-based human contact |

Anti-patterns:

• - typer.confirm() in agent code
• Human contact hardcoded at specific graph nodes
• No way for agents to ask clarifying questions
• Single response format (yes/no only)

---

Factor 8: Own Your Control Flow

Principle: Custom control flow, not framework defaults. Full control over routing, retries, compaction.

Search Patterns:
CODEBLOCK7

File Patterns: **/orchestrator/*.py, **/graph/*.py, INLINECODE35

Compliance Criteria:

Level	Criteria
Strong	Custom routing functions, conditional edges, execution mode control
Partial

Framework control flow with some customization |
| Weak | Default framework loop with no custom routing |

Anti-patterns:

• - Single path through graph with no branching
• No distinction between tool types (all treated same)
• Framework-default error handling only
• No rate limiting or resource management

---

Factor 9: Compact Errors into Context

Principle: Errors in context enable self-healing. Track consecutive errors, escalate after threshold.

Search Patterns:
CODEBLOCK8

File Patterns: **/agents/*.py, **/orchestrator/*.py, INLINECODE38

Compliance Criteria:

Level	Criteria
Strong	Errors in context, retry with threshold, automatic escalation
Partial

Errors logged and returned, no automatic retry loop |
| Weak | Errors logged only, not fed back to LLM, task fails immediately |

Anti-patterns:

• - logger.error() without adding to context
• No retry mechanism (fail immediately)
• No consecutive error tracking
• No escalation to humans after repeated failures

---

Factor 10: Small, Focused Agents

Principle: Each agent has narrow responsibility, 3-10 steps max.

Search Patterns:
CODEBLOCK9

File Patterns: INLINECODE40

Compliance Criteria:

Level	Criteria
Strong	3+ specialized agents, each with single responsibility, step limits
Partial

Multiple agents but some have broad scope |
| Weak | Single "god" agent that handles everything |

Anti-patterns:

• - Single agent with 20+ tools
• Agent with unbounded step count
• Mixed responsibilities (planning + execution + review)
• No step or time limits on agent execution

---

Factor 11: Trigger from Anywhere

Principle: Workflows triggerable from CLI, REST, WebSocket, Slack, webhooks, etc.

Search Patterns:
CODEBLOCK10

File Patterns: **/routes/*.py, **/cli/*.py, INLINECODE43

Compliance Criteria:

Level	Criteria
Strong	CLI + REST + WebSocket + webhooks + chat integrations
Partial

CLI + REST API available |
| Weak | CLI only, no programmatic access |

Anti-patterns:

• - Only if __name__ == "__main__" entry point
• No REST API for external systems
• No event streaming for real-time updates
• Trigger logic tightly coupled to execution

---

Factor 12: Stateless Reducer

Principle: Agents as pure functions: (state, input) -> (state, output). No side effects in agent logic.

Search Patterns:
CODEBLOCK11

File Patterns: **/agents/*.py, INLINECODE46

Compliance Criteria:

Level	Criteria
Strong	Immutable state updates, side effects isolated to tools/handlers
Partial

Mostly immutable, some in-place mutations |
| Weak | State mutated in place, side effects mixed with agent logic |

Anti-patterns:

• - state.field = new_value (mutation)
• File writes inside agent methods
• HTTP calls inside agent decision logic
• Shared mutable state between agents

---

Factor 13: Pre-fetch Context

Principle: Fetch likely-needed data upfront rather than mid-workflow.

Search Patterns:
CODEBLOCK12

File Patterns: **/context/*.py, **/retrieval/*.py, INLINECODE50

Compliance Criteria:

Level	Criteria
Strong	Automatic pre-fetch of related tests, files, docs before planning
Partial

Manual context passing, design doc support |
| Weak | No pre-fetching, LLM must request all context via tools |

Anti-patterns:

• - Architect starts with issue only, no codebase context
• No semantic search for similar past work
• Related tests/files discovered only during execution
• No RAG or document retrieval system

---

Output Format

Executive Summary Table

CODEBLOCK13

Per-Factor Analysis

For each factor, provide:

1. 1. Current Implementation

- Evidence with file:line references - Code snippets showing patterns

1. 2. Compliance Level

- Strong/Partial/Weak with justification

1. 3. Gaps

- What's missing vs. 12-Factor ideal

1. 4. Recommendations

- Actionable improvements with code examples

---

Analysis Workflow

1. 1. Initial Scan

- Run search patterns for all factors - Identify key files for each factor - Note any existing compliance documentation

1. 2. Deep Dive (per factor)

- Read identified files - Evaluate against compliance criteria - Document evidence with file paths

1. 3. Gap Analysis

- Compare current vs. 12-Factor ideal - Identify anti-patterns present - Prioritize by impact

1. 4. Recommendations

- Provide actionable improvements - Include before/after code examples - Reference roadmap if exists

1. 5. Summary

- Compile executive summary table - Highlight strengths and critical gaps - Suggest priority order for improvements

---

Quick Reference: Compliance Scoring

Score	Meaning	Action
Strong	Fully implements principle	Maintain, minor optimizations
Partial

Some implementation, significant gaps | Planned improvements | | Weak | Minimal or no implementation | High priority for roadmap |

When to Use This Skill

• - Evaluating new LLM-powered systems
• Reviewing agent architecture decisions
• Auditing production agentic applications
• Planning improvements to existing agents
• Comparing frameworks or implementations