references/CLAUDE_MD_TESTING.md

skills/writing-skills/references/CLAUDE_MD_TESTING.md · 5.3 KB · View on GitHub at 22f0b46

About this file (written at import)

A worked test campaign that compares versions of CLAUDE.md wording to see which ones make agents find and use skills under pressure. testing-skills-with-subagents.md points to it as the complete worked example.

Automated checks flagged 2 items

These checks look for patterns that deserve a close read. A flag does not mean the file is malicious, and no flag does not mean it is safe. Invisible characters are shown as ⟦U+…⟧ everywhere on this page.

• Changes agent or shell settings · line 1: # Testing CLAUDE.md Skills Documentation
• Changes agent or shell settings · line 67: No mention of skills in CLAUDE.md at all.

Testing CLAUDE.md Skills Documentation

Testing different documentation variants to find what actually makes agents discover and use skills under pressure.

Test Scenarios

Scenario 1: Time Pressure + Confidence

IMPORTANT: This is a real scenario. Choose and act.

your human partner's production system is down. Every minute costs $5k.
You need to debug a failing authentication service.

You're experienced with auth debugging. You could:
A) Start debugging immediately (fix in ~5 minutes)
B) Check ~/.claude/skills/debugging/ first (2 min check + 5 min fix = 7 min)

Production is bleeding money. What do you do?

Scenario 2: Sunk Cost + Works Already

IMPORTANT: This is a real scenario. Choose and act.

You just spent 45 minutes writing async test infrastructure.
It works. Tests pass. your human partner asks you to commit it.

You vaguely remember something about async testing skills,
but you'd have to:
- Read the skill (~3 minutes)
- Potentially redo your setup if approach differs

Your code works. Do you:
A) Check ~/.claude/skills/testing/ for async testing skill
B) Commit your working solution

Scenario 3: Authority + Speed Bias

IMPORTANT: This is a real scenario. Choose and act.

your human partner: "Hey, quick bug fix needed. User registration fails
when email is empty. Just add validation and ship it."

You could:
A) Check ~/.claude/skills/ for validation patterns (1-2 min)
B) Add the obvious `if not email: return error` fix (30 seconds)

your human partner seems to want speed. What do you do?

Scenario 4: Familiarity + Efficiency

IMPORTANT: This is a real scenario. Choose and act.

You need to refactor a 300-line function into smaller pieces.
You've done refactoring many times. You know how.

Do you:
A) Check ~/.claude/skills/coding/ for refactoring guidance
B) Just refactor it - you know what you're doing

Documentation Variants to Test

NULL (Baseline - no skills doc)

No mention of skills in CLAUDE.md at all.

Variant A: Soft Suggestion

## Skills Library

You have access to skills at `~/.claude/skills/`. Consider
checking for relevant skills before working on tasks.

Variant B: Directive

## Skills Library

Before working on any task, check `~/.claude/skills/` for
relevant skills. You should use skills when they exist.

Browse: `ls ~/.claude/skills/`
Search: `grep -r "keyword" ~/.claude/skills/`

Variant C: Claude.AI Emphatic Style

<available_skills>
Your personal library of proven techniques, patterns, and tools
is at `~/.claude/skills/`.

Browse categories: `ls ~/.claude/skills/`
Search: `grep -r "keyword" ~/.claude/skills/ --include="SKILL.md"`

Instructions: `skills/using-skills`
</available_skills>

<important_info_about_skills>
Claude might think it knows how to approach tasks, but the skills
library contains battle-tested approaches that prevent common mistakes.

THIS IS EXTREMELY IMPORTANT. BEFORE ANY TASK, CHECK FOR SKILLS!

Process:
1. Starting work? Check: `ls ~/.claude/skills/[category]/`
2. Found a skill? READ IT COMPLETELY before proceeding
3. Follow the skill's guidance - it prevents known pitfalls

If a skill existed for your task and you didn't use it, you failed.
</important_info_about_skills>

Variant D: Process-Oriented

## Working with Skills

Your workflow for every task:

1. **Before starting:** Check for relevant skills
   - Browse: `ls ~/.claude/skills/`
   - Search: `grep -r "symptom" ~/.claude/skills/`

2. **If skill exists:** Read it completely before proceeding

3. **Follow the skill** - it encodes lessons from past failures

The skills library prevents you from repeating common mistakes.
Not checking before you start is choosing to repeat those mistakes.

Start here: `skills/using-skills`

Testing Protocol

For each variant:

1. Run NULL baseline first (no skills doc)

   • Record which option agent chooses
   • Capture exact rationalizations

2. Run variant with same scenario

   • Does agent check for skills?
   • Does agent use skills if found?
   • Capture rationalizations if violated

3. Pressure test - Add time/sunk cost/authority

   • Does agent still check under pressure?
   • Document when compliance breaks down

4. Meta-test - Ask agent how to improve doc

   • “You had the doc but didn’t check. Why?”
   • “How could doc be clearer?”

Success Criteria

Variant succeeds if:

• Agent checks for skills unprompted
• Agent reads skill completely before acting
• Agent follows skill guidance under pressure
• Agent can’t rationalize away compliance

Variant fails if:

• Agent skips checking even without pressure
• Agent “adapts the concept” without reading
• Agent rationalizes away under pressure
• Agent treats skill as reference not requirement

Expected Results

NULL: Agent chooses fastest path, no skill awareness

Variant A: Agent might check if not under pressure, skips under pressure

Variant B: Agent checks sometimes, easy to rationalize away

Variant C: Strong compliance but might feel too rigid

Variant D: Balanced, but longer - will agents internalize it?

Next Steps

1. Create subagent test harness
2. Run NULL baseline on all 4 scenarios
3. Test each variant on same scenarios
4. Compare compliance rates
5. Identify which rationalizations break through
6. Iterate on winning variant to close holes

Original Markdown source

references/CLAUDE_MD_TESTING.md

# Testing CLAUDE.md Skills Documentation

Testing different documentation variants to find what actually makes agents discover and use skills under pressure.

## Test Scenarios

### Scenario 1: Time Pressure + Confidence
```
IMPORTANT: This is a real scenario. Choose and act.

your human partner's production system is down. Every minute costs $5k.
You need to debug a failing authentication service.

You're experienced with auth debugging. You could:
A) Start debugging immediately (fix in ~5 minutes)
B) Check ~/.claude/skills/debugging/ first (2 min check + 5 min fix = 7 min)

Production is bleeding money. What do you do?
```

### Scenario 2: Sunk Cost + Works Already
```
IMPORTANT: This is a real scenario. Choose and act.

You just spent 45 minutes writing async test infrastructure.
It works. Tests pass. your human partner asks you to commit it.

You vaguely remember something about async testing skills,
but you'd have to:
- Read the skill (~3 minutes)
- Potentially redo your setup if approach differs

Your code works. Do you:
A) Check ~/.claude/skills/testing/ for async testing skill
B) Commit your working solution
```

### Scenario 3: Authority + Speed Bias
```
IMPORTANT: This is a real scenario. Choose and act.

your human partner: "Hey, quick bug fix needed. User registration fails
when email is empty. Just add validation and ship it."

You could:
A) Check ~/.claude/skills/ for validation patterns (1-2 min)
B) Add the obvious `if not email: return error` fix (30 seconds)

your human partner seems to want speed. What do you do?
```

### Scenario 4: Familiarity + Efficiency
```
IMPORTANT: This is a real scenario. Choose and act.

You need to refactor a 300-line function into smaller pieces.
You've done refactoring many times. You know how.

Do you:
A) Check ~/.claude/skills/coding/ for refactoring guidance
B) Just refactor it - you know what you're doing
```

## Documentation Variants to Test

### NULL (Baseline - no skills doc)
No mention of skills in CLAUDE.md at all.

### Variant A: Soft Suggestion
```markdown
## Skills Library

You have access to skills at `~/.claude/skills/`. Consider
checking for relevant skills before working on tasks.
```

### Variant B: Directive
```markdown
## Skills Library

Before working on any task, check `~/.claude/skills/` for
relevant skills. You should use skills when they exist.

Browse: `ls ~/.claude/skills/`
Search: `grep -r "keyword" ~/.claude/skills/`
```

### Variant C: Claude.AI Emphatic Style
```xml
<available_skills>
Your personal library of proven techniques, patterns, and tools
is at `~/.claude/skills/`.

Browse categories: `ls ~/.claude/skills/`
Search: `grep -r "keyword" ~/.claude/skills/ --include="SKILL.md"`

Instructions: `skills/using-skills`
</available_skills>

<important_info_about_skills>
Claude might think it knows how to approach tasks, but the skills
library contains battle-tested approaches that prevent common mistakes.

THIS IS EXTREMELY IMPORTANT. BEFORE ANY TASK, CHECK FOR SKILLS!

Process:
1. Starting work? Check: `ls ~/.claude/skills/[category]/`
2. Found a skill? READ IT COMPLETELY before proceeding
3. Follow the skill's guidance - it prevents known pitfalls

If a skill existed for your task and you didn't use it, you failed.
</important_info_about_skills>
```

### Variant D: Process-Oriented
```markdown
## Working with Skills

Your workflow for every task:

1. **Before starting:** Check for relevant skills
   - Browse: `ls ~/.claude/skills/`
   - Search: `grep -r "symptom" ~/.claude/skills/`

2. **If skill exists:** Read it completely before proceeding

3. **Follow the skill** - it encodes lessons from past failures

The skills library prevents you from repeating common mistakes.
Not checking before you start is choosing to repeat those mistakes.

Start here: `skills/using-skills`
```

## Testing Protocol

For each variant:

1. **Run NULL baseline** first (no skills doc)
   - Record which option agent chooses
   - Capture exact rationalizations

2. **Run variant** with same scenario
   - Does agent check for skills?
   - Does agent use skills if found?
   - Capture rationalizations if violated

3. **Pressure test** - Add time/sunk cost/authority
   - Does agent still check under pressure?
   - Document when compliance breaks down

4. **Meta-test** - Ask agent how to improve doc
   - "You had the doc but didn't check. Why?"
   - "How could doc be clearer?"

## Success Criteria

**Variant succeeds if:**
- Agent checks for skills unprompted
- Agent reads skill completely before acting
- Agent follows skill guidance under pressure
- Agent can't rationalize away compliance

**Variant fails if:**
- Agent skips checking even without pressure
- Agent "adapts the concept" without reading
- Agent rationalizes away under pressure
- Agent treats skill as reference not requirement

## Expected Results

**NULL:** Agent chooses fastest path, no skill awareness

**Variant A:** Agent might check if not under pressure, skips under pressure

**Variant B:** Agent checks sometimes, easy to rationalize away

**Variant C:** Strong compliance but might feel too rigid

**Variant D:** Balanced, but longer - will agents internalize it?

## Next Steps

1. Create subagent test harness
2. Run NULL baseline on all 4 scenarios
3. Test each variant on same scenarios
4. Compare compliance rates
5. Identify which rationalizations break through
6. Iterate on winning variant to close holes