Rollback & Time Travel

Production is broken. You need to rollback. Now. With Edgit, rollback is instant (<50ms if using Cloudflare KV). And you can recreate any state from any point in history. Perfect reproducibility.

Instant Rollback

The Problem

Production breaks at 2:47 PM on Tuesday. A customer reports a bug. Your AI system is giving wrong answers. Traditional approach:

git revert abc123  # Revert the commit
git push           # Push to trigger CI/CD
# Wait 5-10 minutes for deploy pipeline
# Hope you reverted the right commit

Edgit approach:

edgit deploy set my-prompt v1.0.0 --to prod
# &lt;50ms later, you're back to working version

How It Works

Edgit deployments are just Git tags:

# Current state (broken)
components/my-prompt/prod  v1.1.0

# Rollback (instant)
edgit deploy set my-prompt v1.0.0 --to prod
# Updates tag: components/my-prompt/prod  v1.0.0

# Push tag
git push --tags

# CI/CD updates Cloudflare KV (or your runtime)
# Conductor reads new version from KV
# &lt;50ms globally, you're back to v1.0.0

Emergency Rollback Playbook

Step 1: Identify the Problem

# Check current deployed versions
edgit components list --format tree
# Output:
# my-prompt (prompt)
#  v1.1.0 [prod]  This is currently deployed
#  v1.0.0
#  v0.9.0

Step 2: Find Last Known Good Version

# View deployment history
edgit history my-prompt
# Output:
# prod: v1.1.0 (deployed 2024-11-12 14:45)  Current (broken)
# prod: v1.0.0 (deployed 2024-11-08 10:30)  Last good
# prod: v0.9.0 (deployed 2024-11-01 09:15)

Step 3: Rollback

# Rollback to last known good
edgit deploy set my-prompt v1.0.0 --to prod

# Push tags
git push --tags

# Verify
edgit components list --format tree
# Output:
# my-prompt (prompt)
#  v1.1.0
#  v1.0.0 [prod]  Now deployed
#  v0.9.0

Step 4: Monitor

# Watch error rates
curl https://metrics.example.com/error-rate

# Test manually
curl https://prod.example.com/test

Step 5: Investigate

# What changed between v1.0.0 and v1.1.0?
git diff components/my-prompt/v1.0.0 components/my-prompt/v1.1.0

# View commit message
git show components/my-prompt/v1.1.0 --format=%B --no-patch

Total time: <2 minutes from “production is broken” to “production is fixed.”

Partial Rollback

You don’t have to rollback everything. Roll back only what’s broken.

Scenario

You deployed 3 components at once:

extraction-prompt v1.1.0 Works fine
analysis-prompt v2.0.0 BROKEN
validation-config v1.5.0 Works fine

# Only rollback the broken component
edgit deploy set analysis-prompt v1.9.0 --to prod

# Keep the others
# extraction-prompt still at v1.1.0
# validation-config still at v1.5.0

git push --tags

This is the power of independent versioning.

Time Travel: Recreate Any State

The Problem

Customer reports: “On Tuesday at 2:47 PM, the AI gave me a wrong answer.” You need to reproduce the exact environment that was running at that moment.

Solution: Time Travel

# What versions were deployed on Tuesday at 2:47 PM?
edgit history --date="2024-11-12T14:47:00Z"

# Output:
# extraction-prompt: v1.0.5
# analysis-prompt: v1.9.2
# validation-config: v1.4.0
# transform-script: v2.1.3

Now recreate that exact state:

# Deploy to debug environment
edgit deploy set extraction-prompt v1.0.5 --to debug
edgit deploy set analysis-prompt v1.9.2 --to debug
edgit deploy set validation-config v1.4.0 --to debug
edgit deploy set transform-script v2.1.3 --to debug

# Push tags
git push --tags

Test in debug environment:

curl https://debug.example.com/test \
  -H "Content-Type: application/json" \
  -d '{"input": "customer data from Tuesday"}'

# You're now running the EXACT versions from Tuesday at 2:47 PM
# Should reproduce the bug

Find the Bug

# Test each version individually
# Was it extraction-prompt v1.0.5?
edgit deploy set extraction-prompt v1.0.6 --to debug  # Try next version
# Test...

# Was it analysis-prompt v1.9.2?
edgit deploy set analysis-prompt v1.9.3 --to debug  # Try next version
# Test...

# Found it! analysis-prompt v1.9.2 has the bug

Now fix and deploy:

# Fix the bug in analysis-prompt
vim components/prompts/analysis.md

# Create new version
edgit tag create analysis-prompt v1.9.4

# Test in debug
edgit deploy set analysis-prompt v1.9.4 --to debug
# Verify fix...

# Deploy to production
edgit deploy set analysis-prompt v1.9.4 --to prod
git push --tags

Rollback Strategies

Strategy 1: Immediate Rollback

When: Production is on fire, customers are affected

# Don't investigate, just rollback
edgit deploy set my-prompt v1.0.0 --to prod
git push --tags

# Investigate later in debug environment
edgit deploy set my-prompt v1.1.0 --to debug

Strategy 2: Gradual Rollback

When: Not sure if current version is the problem

# Reduce canary percentage from 50% to 10%
wrangler kv:key put --namespace-id=$KV_ID "rollout-percentage" "10"

# Monitor error rates...

# If still high, full rollback
edgit deploy set my-prompt v1.0.0 --to prod

Strategy 3: Targeted Rollback

When: Only affecting specific users/regions

# Rollback for specific region
edgit deploy set my-prompt v1.0.0 --to prod-us-west

# Keep other regions on new version
edgit deploy set my-prompt v1.1.0 --to prod-eu
edgit deploy set my-prompt v1.1.0 --to prod-asia

Rollback Best Practices

1. Always Have a Rollback Plan

Before deploying:

# Document rollback command
echo "If issues: edgit deploy set my-prompt v1.0.0 --to prod" > ROLLBACK.md

# Test rollback in staging
edgit deploy set my-prompt v1.1.0 --to staging
edgit deploy set my-prompt v1.0.0 --to staging  # Verify rollback works

2. Keep Multiple Versions Running

# During rollout, keep old version in canary
prod: v1.0.0        # 90% traffic
prod-canary: v1.1.0 # 10% traffic

# If canary has issues, just remove it (don't affect prod)

3. Automate Rollback

In CI/CD:

- name: Deploy with Auto-Rollback
  run: |
    # Deploy new version
    edgit deploy set my-prompt v1.1.0 --to prod
    git push --tags

    # Wait for deploy
    sleep 60

    # Check error rate
    ERROR_RATE=$(curl https://metrics.example.com/error-rate)

    if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
      echo "Error rate too high! Auto-rollback..."
      edgit deploy set my-prompt v1.0.0 --to prod
      git push --tags
      exit 1
    fi

4. Test Rollback Regularly

# Chaos engineering: practice rollbacks
# Every month, do a surprise rollback drill

# 1. Deploy test version
edgit deploy set my-prompt v999.0.0 --to prod-drill

# 2. "Discover" the issue
# 3. Practice rolling back
edgit deploy set my-prompt v1.0.0 --to prod-drill

# 4. Time how long it takes

Time Travel Use Cases

Use Case 1: Reproduce Customer Bug

Customer: “I got wrong results yesterday at 3 PM”

# Recreate environment from yesterday 3 PM
edgit history --date="2024-11-11T15:00:00Z"

# Deploy to debug
edgit deploy set extraction-prompt v1.0.4 --to debug
edgit deploy set analysis-prompt v1.9.1 --to debug
git push --tags

# Test with customer's input
curl https://debug.example.com/test -d @customer-input.json

Use Case 2: Performance Regression Analysis

“Performance was better last week. What changed?”

# Get versions from last week
edgit history --date="2024-11-05T12:00:00Z"

# Compare to current
edgit components list --format json > current-versions.json
edgit history --date="2024-11-05T12:00:00Z" --format json > last-week-versions.json

# Diff
diff current-versions.json last-week-versions.json
# Shows: analysis-prompt changed from v1.8.0 to v1.9.0

# Test both versions
edgit deploy set analysis-prompt v1.8.0 --to benchmark-old
edgit deploy set analysis-prompt v1.9.0 --to benchmark-new

# Benchmark performance difference

Use Case 3: A/B Test Historical Versions

“Was v1.5.0 actually better than v1.8.0?”

# Deploy both for comparison
edgit deploy set my-prompt v1.5.0 --to ab-test-a
edgit deploy set my-prompt v1.8.0 --to ab-test-b
git push --tags

Ensemble configuration:

ensemble: ab-test-historical

agents:
  - name: variant-a
    operation: think
    component: my-prompt@v1.5.0
    condition: ${input.user_id % 2 === 0}

  - name: variant-b
    operation: think
    component: my-prompt@v1.8.0
    condition: ${input.user_id % 2 === 1}

Use Case 4: Audit Trail

“Who deployed what and when?”

# Full history of a component
git log --all --decorate --oneline --grep="components/my-prompt"

# Who deployed v1.1.0 to prod?
git log --all components/my-prompt/prod

# What was running during incident?
edgit history --date="2024-11-10T16:30:00Z"

Advanced: Multi-Component Time Travel

Recreate entire system state from a specific time.

#!/bin/bash
# recreate-environment.sh

TARGET_DATE=$1
TARGET_ENV=$2

echo "Recreating environment from $TARGET_DATE to $TARGET_ENV..."

# Get all components
COMPONENTS=$(edgit components list --format json | jq -r '.[] | .name')

# For each component, get version at target date
for component in $COMPONENTS; do
  VERSION=$(edgit history $component --date="$TARGET_DATE" --from prod)

  if [ ! -z "$VERSION" ]; then
    echo "Deploying $component $VERSION to $TARGET_ENV"
    edgit deploy set $component $VERSION --to $TARGET_ENV
  fi
done

git push --tags

echo " Environment $TARGET_ENV now matches $TARGET_DATE"

Usage:

./recreate-environment.sh "2024-11-10T14:30:00Z" debug

# Output:
# Deploying extraction-prompt v1.0.5 to debug
# Deploying analysis-prompt v1.9.2 to debug
# Deploying validation-config v1.4.0 to debug
#  Environment debug now matches 2024-11-10T14:30:00Z

Rollback Safety Checks

Before rolling back:

#!/bin/bash
# safe-rollback.sh

COMPONENT=$1
VERSION=$2
ENV=$3

echo "=
 Checking if rollback is safe..."

# 1. Check version exists
if ! edgit tag list $COMPONENT | grep -q "$VERSION"; then
  echo "L Version $VERSION doesn't exist"
  exit 1
fi

# 2. Check version was previously deployed
if ! edgit history $COMPONENT | grep -q "$VERSION"; then
  echo "  Warning: $VERSION was never deployed to $ENV before"
  read -p "Continue anyway? (y/n) " -n 1 -r
  echo
  if [[ ! $REPLY =~ ^[Yy]$ ]]; then
    exit 1
  fi
fi

# 3. Check current version
CURRENT=$(edgit deploy get $COMPONENT --from $ENV)
echo "Current version: $CURRENT"
echo "Rolling back to: $VERSION"

read -p "Confirm rollback? (y/n) " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
  exit 1
fi

# 4. Do rollback
edgit deploy set $COMPONENT $VERSION --to $ENV
git push --tags

echo " Rolled back $COMPONENT to $VERSION in $ENV"

Troubleshooting

Rollback Not Taking Effect

# Check tag was updated
git show components/my-prompt/prod
# Should point to rollback version

# Check CI/CD deployed to runtime
curl https://api.example.com/version
# Should show rollback version

# If using Cloudflare KV, check it was updated
wrangler kv:key get --namespace-id=$KV_ID "component-versions" | jq '.["my-prompt"].prod'
# Should show rollback version

Can’t Find Version to Rollback To

# List all versions
edgit tag list my-prompt

# If version was deleted, check Git history
git log --all --oneline -- components/prompts/my-prompt.md

# Restore deleted tag
git tag components/my-prompt/v1.0.0 <commit-hash>

Rolled Back But Issue Persists

# Might not be the component you think
# Check all component versions at time of incident
edgit history --date="<incident-time>"

# Rollback all changed components
edgit deploy set component-a v1.0.0 --to prod
edgit deploy set component-b v2.0.0 --to prod

Next Steps

Deployment Strategies

Canaries, blue-green, progressive rollouts

A/B Testing

Test multiple versions simultaneously

CI/CD Integration

Automate deployments and rollbacks

Versioning Guide

Master version management

Edgit

Getting Started

Guides

Reference

​Rollback & Time Travel

​Instant Rollback

​The Problem

​How It Works

​Emergency Rollback Playbook

​Step 1: Identify the Problem

​Step 2: Find Last Known Good Version

​Step 3: Rollback

​Step 4: Monitor

​Step 5: Investigate

​Partial Rollback

​Scenario

​Time Travel: Recreate Any State

​The Problem

​Solution: Time Travel

​Find the Bug

​Rollback Strategies

​Strategy 1: Immediate Rollback

​Strategy 2: Gradual Rollback

​Strategy 3: Targeted Rollback

​Rollback Best Practices

​1. Always Have a Rollback Plan

​2. Keep Multiple Versions Running

​3. Automate Rollback

​4. Test Rollback Regularly

​Time Travel Use Cases

​Use Case 1: Reproduce Customer Bug

​Use Case 2: Performance Regression Analysis

​Use Case 3: A/B Test Historical Versions

​Use Case 4: Audit Trail

​Advanced: Multi-Component Time Travel

​Rollback Safety Checks

​Troubleshooting

​Rollback Not Taking Effect

​Can’t Find Version to Rollback To

​Rolled Back But Issue Persists

​Next Steps

Deployment Strategies

A/B Testing

CI/CD Integration

Versioning Guide

Rollback & Time Travel

Instant Rollback

The Problem

How It Works

Emergency Rollback Playbook

Step 1: Identify the Problem

Step 2: Find Last Known Good Version

Step 3: Rollback

Step 4: Monitor

Step 5: Investigate

Partial Rollback

Scenario

Time Travel: Recreate Any State

The Problem

Solution: Time Travel

Find the Bug

Rollback Strategies

Strategy 1: Immediate Rollback

Strategy 2: Gradual Rollback

Strategy 3: Targeted Rollback

Rollback Best Practices

1. Always Have a Rollback Plan

2. Keep Multiple Versions Running

3. Automate Rollback

4. Test Rollback Regularly

Time Travel Use Cases

Use Case 1: Reproduce Customer Bug

Use Case 2: Performance Regression Analysis

Use Case 3: A/B Test Historical Versions

Use Case 4: Audit Trail

Advanced: Multi-Component Time Travel

Rollback Safety Checks

Troubleshooting

Rollback Not Taking Effect

Can’t Find Version to Rollback To

Rolled Back But Issue Persists

Next Steps