Rollback & Time Travel - Ensemble Edge

Production is broken. You need to rollback. Now.

Instant Rollback

The Problem

Production breaks at 2:47 PM on Tuesday. A customer reports a bug. Your AI system is giving wrong answers. Traditional approach:

git revert abc123  # Revert the commit
git push           # Push to trigger CI/CD
# Wait 5-10 minutes for deploy pipeline
# Hope you reverted the right commit

Edgit approach:

# Check current deployment
edgit tag list prompts/extraction

# Rollback by setting env tag to previous version
edgit tag set prompts/extraction production v1.0.0
edgit push --tags --force

# Under 50ms later, you're back to working version

How It Works

Edgit uses a 4-level tag format: components/prompts/extraction/production Environment tags (like production, staging) are mutable - they move to point at different versions. This is why you need --force when pushing: Git requires it to update remote tags that already exist.

# Current state (broken)
components/prompts/extraction/production → v1.1.0

# Rollback (instant)
edgit tag set prompts/extraction production v1.0.0
# Moves tag: components/prompts/extraction/production → v1.0.0

# Push tag (--force required for mutable tags)
edgit push --tags --force

# GitHub Actions detects tag push and deploys
# Conductor reads new version from runtime
# Under 50ms globally, you're back to v1.0.0

Philosophy: Edgit creates/moves tags. GitHub Actions detects the tag push and deploys. Separation of concerns.

Why —force is Required

Environment tags are mutable pointers. When you rollback:

production tag already exists pointing to v1.1.0
You move it to point to v1.0.0
Git sees this as “rewriting” the tag
--force tells Git: “Yes, I want to move this tag”

This is safe - you’re not rewriting history, just moving an environment pointer.

Emergency Rollback Playbook

Step 1: Check Current Deployment

edgit tag list prompts/extraction
# Output:
# components/prompts/extraction/production → v1.1.0
# components/prompts/extraction/staging → v1.1.0
# components/prompts/extraction/v1.1.0 → abc123
# components/prompts/extraction/v1.0.0 → def456

Step 2: Find Last Known Good Version

# View available versions
edgit tag list prompts/extraction
# Shows: production → v1.1.0 (current/broken)
# Shows: v1.1.0, v1.0.0, v0.9.0 available

# Choose last known good version (e.g., v1.0.0)

Step 3: Rollback

# Rollback to last known good
edgit tag set prompts/extraction production v1.0.0

# Push tags (--force required for mutable env tags)
edgit push --tags --force

# Verify
edgit tag list prompts/extraction
# Output:
# components/prompts/extraction/production → v1.0.0

Step 4: Monitor

# Watch error rates
curl https://metrics.example.com/error-rate

# Test manually
curl https://prod.example.com/test

Step 5: Investigate

# What changed between v1.0.0 and v1.1.0?
git diff components/prompts/extraction/v1.0.0 components/prompts/extraction/v1.1.0

# View commit message
git show components/prompts/extraction/v1.1.0 --format=%B --no-patch

Total time: Under 2 minutes from “production is broken” to “production is fixed.”

Partial Rollback

You don’t have to rollback everything. Roll back only what’s broken.

Scenario

You deployed 3 components at once:

prompts/extraction v1.1.0 Works fine
prompts/analysis v2.0.0 BROKEN
configs/validation v1.5.0 Works fine

# Only rollback the broken component
edgit tag set prompts/analysis production v1.9.0
edgit push --tags --force

# Keep the others
# prompts/extraction still at v1.1.0
# configs/validation still at v1.5.0

This is the power of independent versioning.

Time Travel: Recreate Any State

The Problem

Customer reports: “On Tuesday at 2:47 PM, the AI gave me a wrong answer.” You need to reproduce the exact environment that was running at that moment.

Solution: Time Travel

# What versions were deployed on Tuesday at 2:47 PM?
# Use Git to find tags created around that time
git for-each-ref --sort=creatordate \
  --format '%(creatordate:iso) %(refname:short)' \
  refs/tags/components/ | grep "2024-11-12"

# Identify versions that were active:
# prompts/extraction: v1.0.5
# prompts/analysis: v1.9.2
# configs/validation: v1.4.0
# scripts/transform: v2.1.3

Now recreate that exact state:

# Deploy to debug environment
edgit tag set prompts/extraction debug v1.0.5
edgit tag set prompts/analysis debug v1.9.2
edgit tag set configs/validation debug v1.4.0
edgit tag set scripts/transform debug v2.1.3

# Push tags
edgit push --tags --force

Test in debug environment:

curl https://debug.example.com/test \
  -H "Content-Type: application/json" \
  -d '{"input": "customer data from Tuesday"}'

# You're now running the EXACT versions from Tuesday at 2:47 PM
# Should reproduce the bug

Find the Bug

# Test each version individually
# Was it prompts/extraction v1.0.5?
edgit tag set prompts/extraction debug v1.0.6  # Try next version
# Test...

# Was it prompts/analysis v1.9.2?
edgit tag set prompts/analysis debug v1.9.3  # Try next version
# Test...

# Found it! prompts/analysis v1.9.2 has the bug

Now fix and deploy:

# Fix the bug in prompts/analysis
vim components/prompts/analysis.md

# Create new version
edgit tag create prompts/analysis v1.9.4

# Test in debug
edgit tag set prompts/analysis debug v1.9.4
edgit push --tags --force
# Verify fix...

# Deploy to production
edgit tag set prompts/analysis production v1.9.4
edgit push --tags --force

Rollback Strategies

Strategy 1: Immediate Rollback

When: Production is on fire, customers are affected

# Don't investigate, just rollback
edgit tag set prompts/extraction production v1.0.0
edgit push --tags --force

# Investigate later in debug environment
edgit tag set prompts/extraction debug v1.1.0
edgit push --tags --force

Strategy 2: Gradual Rollback

When: Not sure if current version is the problem

# Reduce canary percentage from 50% to 10%
wrangler kv:key put --namespace-id=$KV_ID "rollout-percentage" "10"

# Monitor error rates...

# If still high, full rollback
edgit tag set prompts/extraction production v1.0.0
edgit push --tags --force

Strategy 3: Targeted Rollback

When: Only affecting specific users/regions

# Rollback for specific region
edgit tag set prompts/extraction production-us-west v1.0.0
edgit push --tags --force

# Keep other regions on new version
edgit tag set prompts/extraction production-eu v1.1.0
edgit tag set prompts/extraction production-asia v1.1.0
edgit push --tags --force

Rollback Best Practices

1. Always Have a Rollback Plan

Before deploying:

# Document rollback command
echo "If issues: edgit tag set prompts/extraction production v1.0.0 && edgit push --tags --force" > ROLLBACK.md

# Test rollback in staging
edgit tag set prompts/extraction staging v1.1.0
edgit tag set prompts/extraction staging v1.0.0  # Verify rollback works
edgit push --tags --force

2. Keep Multiple Versions Running

# During rollout, keep old version in canary
production: v1.0.0        # 90% traffic
production-canary: v1.1.0 # 10% traffic

# If canary has issues, just remove it (don't affect production)

3. Automate Rollback

In CI/CD:

- name: Deploy with Auto-Rollback
  run: |
    # Deploy new version
    edgit tag set prompts/extraction production v1.1.0
    edgit push --tags --force

    # Wait for deploy
    sleep 60

    # Check error rate
    ERROR_RATE=$(curl https://metrics.example.com/error-rate)

    if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
      echo "Error rate too high! Auto-rollback..."
      edgit tag set prompts/extraction production v1.0.0
      edgit push --tags --force
      exit 1
    fi

4. Test Rollback Regularly

# Chaos engineering: practice rollbacks
# Every month, do a surprise rollback drill

# 1. Deploy test version
edgit tag set prompts/extraction production-drill v999.0.0
edgit push --tags --force

# 2. "Discover" the issue
# 3. Practice rolling back
edgit tag set prompts/extraction production-drill v1.0.0
edgit push --tags --force

# 4. Time how long it takes

Time Travel Use Cases

Use Case 1: Reproduce Customer Bug

Customer: “I got wrong results yesterday at 3 PM”

# Find versions from yesterday 3 PM using Git
git for-each-ref --sort=creatordate \
  --format '%(creatordate:iso) %(refname:short)' \
  refs/tags/components/ | grep "2024-11-11"

# Identify and deploy versions that were active
edgit tag set prompts/extraction debug v1.0.4
edgit tag set prompts/analysis debug v1.9.1
edgit push --tags --force

# Test with customer's input
curl https://debug.example.com/test -d @customer-input.json

Use Case 2: Performance Regression Analysis

“Performance was better last week. What changed?”

# Find what was deployed last week
git for-each-ref --sort=creatordate \
  --format '%(creatordate:iso) %(refname:short)' \
  refs/tags/components/ | grep "2024-11-05"

# Compare to current versions
edgit tag list prompts/analysis
# Shows: prompts/analysis changed from v1.8.0 to v1.9.0

# Test both versions
edgit tag set prompts/analysis benchmark-old v1.8.0
edgit tag set prompts/analysis benchmark-new v1.9.0
edgit push --tags --force

# Benchmark performance difference

Use Case 3: A/B Test Historical Versions

“Was v1.5.0 actually better than v1.8.0?”

# Deploy both for comparison
edgit tag set prompts/extraction ab-test-a v1.5.0
edgit tag set prompts/extraction ab-test-b v1.8.0
edgit push --tags --force

Ensemble configuration:

ensemble: ab-test-historical

agents:
  - name: variant-a
    operation: think
    component: prompts/extraction@v1.5.0
    condition: ${input.user_id % 2 === 0}

  - name: variant-b
    operation: think
    component: prompts/extraction@v1.8.0
    condition: ${input.user_id % 2 === 1}

Use Case 4: Audit Trail

“Who deployed what and when?”

# Full history of a component's tags
git log --all --oneline -- 'refs/tags/components/prompts/extraction/*'

# Who created v1.1.0?
git show components/prompts/extraction/v1.1.0

# All tags for a component
edgit tag list prompts/extraction

Advanced: Time Travel with Git

Git tags retain the commit history. To see what was deployed at a specific time, use Git directly:

# Find tag state at a specific date
git for-each-ref --sort=creatordate --format '%(creatordate:iso) %(refname:short)' refs/tags/components/

# See all version tags for a component
git tag -l 'components/prompts/extraction/v*'

Rollback Safety Checks

Before rolling back:

#!/bin/bash
# safe-rollback.sh

COMPONENT=$1
VERSION=$2
ENV=$3

echo "Checking if rollback is safe..."

# 1. Check version exists
if ! edgit tag list $COMPONENT | grep -q "$VERSION"; then
  echo "Version $VERSION doesn't exist"
  exit 1
fi

# 2. Check version exists in available versions
edgit tag show $COMPONENT@$VERSION > /dev/null 2>&1
if [ $? -ne 0 ]; then
  echo "Warning: Cannot verify $VERSION details"
  read -p "Continue anyway? (y/n) " -n 1 -r
  echo
  if [[ ! $REPLY =~ ^[Yy]$ ]]; then
    exit 1
  fi
fi

# 3. Check current version
CURRENT=$(edgit tag list $COMPONENT --env $ENV)
echo "Current version: $CURRENT"
echo "Rolling back to: $VERSION"

read -p "Confirm rollback? (y/n) " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
  exit 1
fi

# 4. Do rollback
edgit tag set $COMPONENT $ENV $VERSION
edgit push --tags --force

echo "Rolled back $COMPONENT to $VERSION in $ENV"

Troubleshooting

Rollback Not Taking Effect

# Check tag was updated
git show components/prompts/extraction/production
# Should point to rollback version

# Check CI/CD deployed to runtime
curl https://api.example.com/version
# Should show rollback version

# If using Cloudflare KV, check it was updated
wrangler kv:key get --namespace-id=$KV_ID "component-versions" | jq '.["prompts/extraction"].production'
# Should show rollback version

Can’t Find Version to Rollback To

# List all versions
edgit tag list prompts/extraction

# If version was deleted, check Git history
git log --all --oneline -- components/prompts/extraction.md

# Restore deleted tag
git tag components/prompts/extraction/v1.0.0 <commit-hash>

Rolled Back But Issue Persists

# Might not be the component you think
# Check all component versions
edgit tag list prompts/extraction
edgit tag list prompts/analysis

# Rollback multiple components if needed
edgit tag set prompts/extraction production v1.0.0
edgit tag set prompts/analysis production v2.0.0
edgit push --tags --force

Next Steps

Deployment Strategies

Canaries, blue-green, progressive rollouts

A/B Testing

Test multiple versions simultaneously

CI/CD Integration

Automate deployments and rollbacks

Versioning Guide

Master version management

​Instant Rollback

​The Problem

​How It Works

​Why —force is Required

​Emergency Rollback Playbook

​Step 1: Check Current Deployment

​Step 2: Find Last Known Good Version

​Step 3: Rollback

​Step 4: Monitor

​Step 5: Investigate

​Partial Rollback

​Scenario

​Time Travel: Recreate Any State

​The Problem

​Solution: Time Travel

​Find the Bug

​Rollback Strategies

​Strategy 1: Immediate Rollback

​Strategy 2: Gradual Rollback

​Strategy 3: Targeted Rollback

​Rollback Best Practices

​1. Always Have a Rollback Plan

​2. Keep Multiple Versions Running

​3. Automate Rollback

​4. Test Rollback Regularly

​Time Travel Use Cases

​Use Case 1: Reproduce Customer Bug

​Use Case 2: Performance Regression Analysis

​Use Case 3: A/B Test Historical Versions

​Use Case 4: Audit Trail

​Advanced: Time Travel with Git

​Rollback Safety Checks

​Troubleshooting

​Rollback Not Taking Effect

​Can’t Find Version to Rollback To

​Rolled Back But Issue Persists

​Next Steps

Deployment Strategies

A/B Testing

CI/CD Integration

Versioning Guide

Instant Rollback

The Problem

How It Works

Why —force is Required

Emergency Rollback Playbook

Step 1: Check Current Deployment

Step 2: Find Last Known Good Version

Step 3: Rollback

Step 4: Monitor

Step 5: Investigate

Partial Rollback

Scenario

Time Travel: Recreate Any State

The Problem

Solution: Time Travel

Find the Bug

Rollback Strategies

Strategy 1: Immediate Rollback

Strategy 2: Gradual Rollback

Strategy 3: Targeted Rollback

Rollback Best Practices

1. Always Have a Rollback Plan

2. Keep Multiple Versions Running

3. Automate Rollback

4. Test Rollback Regularly

Time Travel Use Cases

Use Case 1: Reproduce Customer Bug

Use Case 2: Performance Regression Analysis

Use Case 3: A/B Test Historical Versions

Use Case 4: Audit Trail

Advanced: Time Travel with Git

Rollback Safety Checks

Troubleshooting

Rollback Not Taking Effect

Can’t Find Version to Rollback To

Rolled Back But Issue Persists

Next Steps