> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ensemble.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Rollback & Time Travel

> Instant rollbacks (under 50ms) and perfect reproducibility. Recreate any state from any point in history.

Production is broken. You need to rollback. Now.

## Instant Rollback

### The Problem

Production breaks at 2:47 PM on Tuesday. A customer reports a bug. Your AI system is giving wrong answers.

**Traditional approach:**

```bash theme={null}
git revert abc123  # Revert the commit
git push           # Push to trigger CI/CD
# Wait 5-10 minutes for deploy pipeline
# Hope you reverted the right commit
```

**Edgit approach:**

```bash theme={null}
# Check current deployment
edgit tag list prompts/extraction

# Rollback by setting env tag to previous version
edgit tag set prompts/extraction production v1.0.0
edgit push --tags --force

# Under 50ms later, you're back to working version
```

### How It Works

Edgit uses a 4-level tag format: `components/prompts/extraction/production`

Environment tags (like `production`, `staging`) are **mutable** - they move to point at different versions. This is why you need `--force` when pushing: Git requires it to update remote tags that already exist.

```bash theme={null}
# Current state (broken)
components/prompts/extraction/production → v1.1.0

# Rollback (instant)
edgit tag set prompts/extraction production v1.0.0
# Moves tag: components/prompts/extraction/production → v1.0.0

# Push tag (--force required for mutable tags)
edgit push --tags --force

# GitHub Actions detects tag push and deploys
# Conductor reads new version from runtime
# Under 50ms globally, you're back to v1.0.0
```

**Philosophy:** Edgit creates/moves tags. GitHub Actions detects the tag push and deploys. Separation of concerns.

### Why --force is Required

Environment tags are mutable pointers. When you rollback:

1. `production` tag already exists pointing to v1.1.0
2. You move it to point to v1.0.0
3. Git sees this as "rewriting" the tag
4. `--force` tells Git: "Yes, I want to move this tag"

This is safe - you're not rewriting history, just moving an environment pointer.

## Emergency Rollback Playbook

### Step 1: Check Current Deployment

```bash theme={null}
edgit tag list prompts/extraction
# Output:
# components/prompts/extraction/production → v1.1.0
# components/prompts/extraction/staging → v1.1.0
# components/prompts/extraction/v1.1.0 → abc123
# components/prompts/extraction/v1.0.0 → def456
```

### Step 2: Find Last Known Good Version

```bash theme={null}
# View available versions
edgit tag list prompts/extraction
# Shows: production → v1.1.0 (current/broken)
# Shows: v1.1.0, v1.0.0, v0.9.0 available

# Choose last known good version (e.g., v1.0.0)
```

### Step 3: Rollback

```bash theme={null}
# Rollback to last known good
edgit tag set prompts/extraction production v1.0.0

# Push tags (--force required for mutable env tags)
edgit push --tags --force

# Verify
edgit tag list prompts/extraction
# Output:
# components/prompts/extraction/production → v1.0.0
```

### Step 4: Monitor

```bash theme={null}
# Watch error rates
curl https://metrics.example.com/error-rate

# Test manually
curl https://prod.example.com/test
```

### Step 5: Investigate

```bash theme={null}
# What changed between v1.0.0 and v1.1.0?
git diff components/prompts/extraction/v1.0.0 components/prompts/extraction/v1.1.0

# View commit message
git show components/prompts/extraction/v1.1.0 --format=%B --no-patch
```

**Total time:** Under 2 minutes from "production is broken" to "production is fixed."

## Partial Rollback

You don't have to rollback everything. Roll back only what's broken.

### Scenario

You deployed 3 components at once:

* `prompts/extraction` v1.1.0  Works fine
* `prompts/analysis` v2.0.0  BROKEN
* `configs/validation` v1.5.0  Works fine

```bash theme={null}
# Only rollback the broken component
edgit tag set prompts/analysis production v1.9.0
edgit push --tags --force

# Keep the others
# prompts/extraction still at v1.1.0
# configs/validation still at v1.5.0
```

**This is the power of independent versioning.**

## Time Travel: Recreate Any State

### The Problem

Customer reports: "On Tuesday at 2:47 PM, the AI gave me a wrong answer."

You need to reproduce the exact environment that was running at that moment.

### Solution: Time Travel

```bash theme={null}
# What versions were deployed on Tuesday at 2:47 PM?
# Use Git to find tags created around that time
git for-each-ref --sort=creatordate \
  --format '%(creatordate:iso) %(refname:short)' \
  refs/tags/components/ | grep "2024-11-12"

# Identify versions that were active:
# prompts/extraction: v1.0.5
# prompts/analysis: v1.9.2
# configs/validation: v1.4.0
# scripts/transform: v2.1.3
```

Now recreate that exact state:

```bash theme={null}
# Deploy to debug environment
edgit tag set prompts/extraction debug v1.0.5
edgit tag set prompts/analysis debug v1.9.2
edgit tag set configs/validation debug v1.4.0
edgit tag set scripts/transform debug v2.1.3

# Push tags
edgit push --tags --force
```

Test in debug environment:

```bash theme={null}
curl https://debug.example.com/test \
  -H "Content-Type: application/json" \
  -d '{"input": "customer data from Tuesday"}'

# You're now running the EXACT versions from Tuesday at 2:47 PM
# Should reproduce the bug
```

### Find the Bug

```bash theme={null}
# Test each version individually
# Was it prompts/extraction v1.0.5?
edgit tag set prompts/extraction debug v1.0.6  # Try next version
# Test...

# Was it prompts/analysis v1.9.2?
edgit tag set prompts/analysis debug v1.9.3  # Try next version
# Test...

# Found it! prompts/analysis v1.9.2 has the bug
```

Now fix and deploy:

```bash theme={null}
# Fix the bug in prompts/analysis
vim components/prompts/analysis.md

# Create new version
edgit tag create prompts/analysis v1.9.4

# Test in debug
edgit tag set prompts/analysis debug v1.9.4
edgit push --tags --force
# Verify fix...

# Deploy to production
edgit tag set prompts/analysis production v1.9.4
edgit push --tags --force
```

## Rollback Strategies

### Strategy 1: Immediate Rollback

**When:** Production is on fire, customers are affected

```bash theme={null}
# Don't investigate, just rollback
edgit tag set prompts/extraction production v1.0.0
edgit push --tags --force

# Investigate later in debug environment
edgit tag set prompts/extraction debug v1.1.0
edgit push --tags --force
```

### Strategy 2: Gradual Rollback

**When:** Not sure if current version is the problem

```bash theme={null}
# Reduce canary percentage from 50% to 10%
wrangler kv:key put --namespace-id=$KV_ID "rollout-percentage" "10"

# Monitor error rates...

# If still high, full rollback
edgit tag set prompts/extraction production v1.0.0
edgit push --tags --force
```

### Strategy 3: Targeted Rollback

**When:** Only affecting specific users/regions

```bash theme={null}
# Rollback for specific region
edgit tag set prompts/extraction production-us-west v1.0.0
edgit push --tags --force

# Keep other regions on new version
edgit tag set prompts/extraction production-eu v1.1.0
edgit tag set prompts/extraction production-asia v1.1.0
edgit push --tags --force
```

## Rollback Best Practices

### 1. Always Have a Rollback Plan

Before deploying:

```bash theme={null}
# Document rollback command
echo "If issues: edgit tag set prompts/extraction production v1.0.0 && edgit push --tags --force" > ROLLBACK.md

# Test rollback in staging
edgit tag set prompts/extraction staging v1.1.0
edgit tag set prompts/extraction staging v1.0.0  # Verify rollback works
edgit push --tags --force
```

### 2. Keep Multiple Versions Running

```bash theme={null}
# During rollout, keep old version in canary
production: v1.0.0        # 90% traffic
production-canary: v1.1.0 # 10% traffic

# If canary has issues, just remove it (don't affect production)
```

### 3. Automate Rollback

In CI/CD:

```yaml theme={null}
- name: Deploy with Auto-Rollback
  run: |
    # Deploy new version
    edgit tag set prompts/extraction production v1.1.0
    edgit push --tags --force

    # Wait for deploy
    sleep 60

    # Check error rate
    ERROR_RATE=$(curl https://metrics.example.com/error-rate)

    if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
      echo "Error rate too high! Auto-rollback..."
      edgit tag set prompts/extraction production v1.0.0
      edgit push --tags --force
      exit 1
    fi
```

### 4. Test Rollback Regularly

```bash theme={null}
# Chaos engineering: practice rollbacks
# Every month, do a surprise rollback drill

# 1. Deploy test version
edgit tag set prompts/extraction production-drill v999.0.0
edgit push --tags --force

# 2. "Discover" the issue
# 3. Practice rolling back
edgit tag set prompts/extraction production-drill v1.0.0
edgit push --tags --force

# 4. Time how long it takes
```

## Time Travel Use Cases

### Use Case 1: Reproduce Customer Bug

Customer: "I got wrong results yesterday at 3 PM"

```bash theme={null}
# Find versions from yesterday 3 PM using Git
git for-each-ref --sort=creatordate \
  --format '%(creatordate:iso) %(refname:short)' \
  refs/tags/components/ | grep "2024-11-11"

# Identify and deploy versions that were active
edgit tag set prompts/extraction debug v1.0.4
edgit tag set prompts/analysis debug v1.9.1
edgit push --tags --force

# Test with customer's input
curl https://debug.example.com/test -d @customer-input.json
```

### Use Case 2: Performance Regression Analysis

"Performance was better last week. What changed?"

```bash theme={null}
# Find what was deployed last week
git for-each-ref --sort=creatordate \
  --format '%(creatordate:iso) %(refname:short)' \
  refs/tags/components/ | grep "2024-11-05"

# Compare to current versions
edgit tag list prompts/analysis
# Shows: prompts/analysis changed from v1.8.0 to v1.9.0

# Test both versions
edgit tag set prompts/analysis benchmark-old v1.8.0
edgit tag set prompts/analysis benchmark-new v1.9.0
edgit push --tags --force

# Benchmark performance difference
```

### Use Case 3: A/B Test Historical Versions

"Was v1.5.0 actually better than v1.8.0?"

```bash theme={null}
# Deploy both for comparison
edgit tag set prompts/extraction ab-test-a v1.5.0
edgit tag set prompts/extraction ab-test-b v1.8.0
edgit push --tags --force
```

Ensemble configuration:

```yaml theme={null}
ensemble: ab-test-historical

agents:
  - name: variant-a
    operation: think
    component: prompts/extraction@v1.5.0
    condition: ${input.user_id % 2 === 0}

  - name: variant-b
    operation: think
    component: prompts/extraction@v1.8.0
    condition: ${input.user_id % 2 === 1}
```

### Use Case 4: Audit Trail

"Who deployed what and when?"

```bash theme={null}
# Full history of a component's tags
git log --all --oneline -- 'refs/tags/components/prompts/extraction/*'

# Who created v1.1.0?
git show components/prompts/extraction/v1.1.0

# All tags for a component
edgit tag list prompts/extraction
```

## Advanced: Time Travel with Git

Git tags retain the commit history. To see what was deployed at a specific time, use Git directly:

```bash theme={null}
# Find tag state at a specific date
git for-each-ref --sort=creatordate --format '%(creatordate:iso) %(refname:short)' refs/tags/components/

# See all version tags for a component
git tag -l 'components/prompts/extraction/v*'
```

## Rollback Safety Checks

Before rolling back:

```bash theme={null}
#!/bin/bash
# safe-rollback.sh

COMPONENT=$1
VERSION=$2
ENV=$3

echo "Checking if rollback is safe..."

# 1. Check version exists
if ! edgit tag list $COMPONENT | grep -q "$VERSION"; then
  echo "Version $VERSION doesn't exist"
  exit 1
fi

# 2. Check version exists in available versions
edgit tag show $COMPONENT@$VERSION > /dev/null 2>&1
if [ $? -ne 0 ]; then
  echo "Warning: Cannot verify $VERSION details"
  read -p "Continue anyway? (y/n) " -n 1 -r
  echo
  if [[ ! $REPLY =~ ^[Yy]$ ]]; then
    exit 1
  fi
fi

# 3. Check current version
CURRENT=$(edgit tag list $COMPONENT --env $ENV)
echo "Current version: $CURRENT"
echo "Rolling back to: $VERSION"

read -p "Confirm rollback? (y/n) " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
  exit 1
fi

# 4. Do rollback
edgit tag set $COMPONENT $ENV $VERSION
edgit push --tags --force

echo "Rolled back $COMPONENT to $VERSION in $ENV"
```

## Troubleshooting

### Rollback Not Taking Effect

```bash theme={null}
# Check tag was updated
git show components/prompts/extraction/production
# Should point to rollback version

# Check CI/CD deployed to runtime
curl https://api.example.com/version
# Should show rollback version

# If using Cloudflare KV, check it was updated
wrangler kv:key get --namespace-id=$KV_ID "component-versions" | jq '.["prompts/extraction"].production'
# Should show rollback version
```

### Can't Find Version to Rollback To

```bash theme={null}
# List all versions
edgit tag list prompts/extraction

# If version was deleted, check Git history
git log --all --oneline -- components/prompts/extraction.md

# Restore deleted tag
git tag components/prompts/extraction/v1.0.0 <commit-hash>
```

### Rolled Back But Issue Persists

```bash theme={null}
# Might not be the component you think
# Check all component versions
edgit tag list prompts/extraction
edgit tag list prompts/analysis

# Rollback multiple components if needed
edgit tag set prompts/extraction production v1.0.0
edgit tag set prompts/analysis production v2.0.0
edgit push --tags --force
```

## Next Steps

<CardGroup cols={2}>
  <Card title="Deployment Strategies" icon="rocket" href="/edgit/guides/deployment-strategies">
    Canaries, blue-green, progressive rollouts
  </Card>

  <Card title="A/B Testing" icon="flask" href="/edgit/guides/ab-testing-multivariate">
    Test multiple versions simultaneously
  </Card>

  <Card title="CI/CD Integration" icon="workflow" href="/edgit/getting-started/cicd-integration">
    Automate deployments and rollbacks
  </Card>

  <Card title="Versioning Guide" icon="tags" href="/edgit/guides/versioning-components-agents">
    Master version management
  </Card>
</CardGroup>