Katastrofåterställning: AWS-native Resiliens med Chaos Engineering

AWS-Native Recovery: When (Not If) Everything Burns

Nothing is true. Everything is permitted. Including—especially—complete infrastructure failure, entire cloud regions burning, and vendors who promised "five nines" explaining why "nines don't count during maintenance windows." Murphy was an optimist who never worked in ops. The question isn't "if everything burns"—it's "when everything burns" and "are you paranoid enough to have actually tested your escape plan instead of just hoping your backups work?" Hope is not a recovery strategy. Tested procedures are. Choose accordingly.

Think for yourself, schmuck! Question authority. Question disaster recovery plans gathering dust in SharePoint that nobody's read since the compliance audit. Question "backup strategies" that have never attempted an actual restore. Question RTO/RPO targets pulled from someone's ass during a compliance meeting because "4 hours sounds reasonable." FNORD. Your DR plan is probably a comfortable lie you tell auditors to make them go away. Disaster recovery theater: expensive documentation pretending to be preparedness while actual disasters expose that nobody tested shit.

At Hack23, we're paranoid enough to assume everything fails. Disaster recovery isn't hypothetical documentation filed under "Things We Hope We Never Need"—it's continuously validated through automated chaos engineering because we're psychotic enough to deliberately break our own infrastructure monthly. AWS Fault Injection Service (FIS) terminates our databases. Crashes our APIs. Severs our network connections. We weaponize chaos to prove recovery automation works before disasters prove it doesn't.

ILLUMINATION: You've entered Chapel Perilous, the place where paranoia meets preparation. Untested DR plans are just bedtime stories CIOs tell themselves. We inject deliberate failures monthly—terminating databases, breaking networks, deleting volumes—because trusting unvalidated recovery is how you discover during actual disasters that your plan was fiction all along. Are you paranoid enough yet?

Our approach combines AWS-native resilience tooling (Resilience Hub, FIS, Backup) with systematic chaos engineering and paranoid-level recovery validation. Because in the reality tunnel we inhabit, everything fails. Clouds crash. Regions burn. Ransomware encrypts. The only question is whether you've actually tested your ability to survive it. Full technical details—because transparency beats security theater—in our public Disaster Recovery Plan. Yes, it's public. No, that doesn't help attackers. FNORD.

Need expert guidance implementing your ISMS? Discover why organizations choose Hack23 for transparent, practitioner-led cybersecurity consulting.

The Five Disaster Scenarios: What Can (Will) Go Wrong

Nothing is true. Everything fails. Our Disaster Recovery Plan covers five distinct failure modes, each requiring different technical recovery approaches:

The Five-Tier Recovery Architecture: Classification-Driven RTO/RPO

Resilience Hub Policy Matrix: Classification-Driven Recovery

Deployment Gating: AWS Resilience Hub assesses application resilience before production deployment. Applications failing RTO/RPO compliance thresholds are blocked from deployment until resilience requirements are met. This ensures disaster recovery capabilities are architectural requirements, not operational afterthoughts.

GATE ILLUMINATION: Deployment gates enforce resilience at build time. Fix architecture before production, not after outages.

SSM Automation: Recovery Procedures That Actually Execute

Manual runbooks are fiction. Under disaster conditions (panic, missing personnel, degraded systems), humans executing manual procedures fail spectacularly. AWS Systems Manager automation documents encode recovery procedures as executable code that works when humans can't think straight.

🔧 Example: IAM Policy Injection for Chaos Testing

Our FIS experiments use SSM automation to inject failures realistically. Here's how we temporarily deny Lambda access during API Gateway chaos testing:

SsmAutomationIamAttachDetachDocument:
  Type: AWS::SSM::Document
  Properties:
    Name: FISAPI-IamAttachDetach
    DocumentType: Automation
    Content:
      schemaVersion: '0.3'
      assumeRole: '{{ AutomationAssumeRole }}'
      parameters:
        TargetResourceDenyPolicyArn:
          type: String
          description: ARN of Deny IAM Policy for AWS Resource
        Duration:
          type: String
          description: The Duration in ISO-8601 format
        TargetApplicationRoleName:
          type: String
          description: The name of the Target Role
      mainSteps:
        - name: AttachDenyPolicy
          action: 'aws:executeAwsApi'
          inputs:
            Service: iam
            Api: AttachRolePolicy
            RoleName: '{{TargetApplicationRoleName}}'
            PolicyArn: '{{TargetResourceDenyPolicyArn}}'
          description: Inject failure by attaching Deny policy
        - name: ExperimentDurationSleep
          action: 'aws:sleep'
          inputs:
            Duration: '{{Duration}}'
          description: Maintain fault injection
          onFailure: 'step:RollbackDetachPolicy'
          onCancel: 'step:RollbackDetachPolicy'
        - name: RollbackDetachPolicy
          action: 'aws:executeAwsApi'
          inputs:
            Service: iam
            Api: DetachRolePolicy
            RoleName: '{{TargetApplicationRoleName}}'
            PolicyArn: '{{TargetResourceDenyPolicyArn}}'
          description: Automatically restore access

Why This Matters: SSM automation provides automatic rollback even if experiment fails. The onFailure and onCancel handlers ensure we don't leave production in broken state. When disaster strikes, this same automation pattern enables recovery procedures that execute reliably under chaos.

🔄 Example: DynamoDB Point-in-Time Recovery

Restoring corrupted DynamoDB table to exact second before corruption using AWS-native SSM automation:

FisRecoverDynamodbTablePITRTemplate:
  Type: AWS::FIS::ExperimentTemplate
  Properties: 
    Actions:
      RecoverDynamodbTablePITR:  
        ActionId: aws:ssm:start-automation-execution
        Parameters:
          documentArn: !Sub 'arn:aws:ssm:${AWS::Region}::document/AWSResilienceHub-RestoreDynamoDBTableToPointInTimeSOP_2020-04-01'
          documentParameters: !Sub |
            {
              "DynamoDBTableSourceName":"global-table",
              "DynamoDBTableTargetName":"global-table-pitr",
              "RecoveryPointDateTime":"${RecoveryPointDateTime}",
              "CopyAllProperties": true,
              "AutomationAssumeRole":"arn:aws:iam::${AWS::AccountId}:role/AWSResilienceHub-RestoreDDBTblFromPointInTimeSOPAssumeRole"
            }
          maxDuration: "PT30M"
    Description: Restore DynamoDB to specific point in time
    RoleArn: !Sub 'arn:aws:iam::${AWS::AccountId}:role/FISAPI-FIS-Role'

Recovery Window: DynamoDB PITR maintains 35 days of continuous backup. You can restore to any second within that window. Data corruption detected at 10:37:23 UTC? Restore to 10:37:22 UTC before corruption began.

FNORD: Manual restore procedures require documentation, calm humans, and correct AWS console clicking under pressure. SSM automation requires one command: "execute this document." Choose accordingly.

Monthly Chaos Engineering: FIS Experiment Portfolio

We don't trust—we verify. Monthly FIS experiments deliberately inject failures to validate recovery automation:

🔴 Critical System Experiments (Monthly):

• Database Disaster: RDS primary instance termination → validates automatic failover to read replica < 5 min
• API Unavailability: 100% Lambda error rate injection → validates circuit breaker activation and graceful degradation
• Network Partition: VPC subnet isolation → validates cross-AZ redundancy and connection retry logic
• Regional Impairment: DNS resolution failure → validates Route 53 health check failover to backup region

🟠 High Priority Experiments (Quarterly):

• Storage Outage: EBS volume unavailability → validates backup volume mount and data recovery
• CDN Degradation: CloudFront cache invalidation → validates origin server direct access
• Compute Failure: EC2 instance termination → validates Auto Scaling group replacement

Evidence Collection: Every FIS experiment generates timestamped logs (CloudWatch, VPC Flow Logs, RDS events, Route 53 health checks). Experiment artifacts prove actual recovery time vs. RTO target. Failures trigger incident response and architectural remediation.

🧪 Example: Complete FIS Experiment CloudFormation

Here's actual CloudFormation defining API Gateway failure injection experiment:

FisDenyApigatewayLambdaTemplate:
  Type: AWS::FIS::ExperimentTemplate
  Properties: 
    Actions:
      InjectAccessDenied:  
        ActionId: aws:ssm:start-automation-execution
        Description: Deny API Gateway Lambda access via IAM policy
        Parameters:
          documentArn: !Sub 'arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:document/FISAPI-IamAttachDetach'
          documentParameters: !Sub |
            {
              "TargetResourceDenyPolicyArn":"${AwsFisApiPolicyDenyApiRoleLambda}", 
              "Duration": "${FaultInjectionExperimentDuration}", 
              "TargetApplicationRoleName":"${ApiRole}", 
              "AutomationAssumeRole":"arn:aws:iam::${AWS::AccountId}:role/FISAPI-SSM-Automation-Role"
            }
          maxDuration: "PT8M"
    Description: Test API resilience via Lambda access denial
    RoleArn: !Sub 'arn:aws:iam::${AWS::AccountId}:role/FISAPI-FIS-Injection-ExperimentRole'
    StopConditions:
      - Source: none
    Tags: 
      Name: DENY-API-LAMBDA

What This Tests: Does your API gracefully degrade when Lambda backend fails? Do circuit breakers activate? Do health checks detect failure and reroute traffic? Does monitoring alert within SLA? Manual testing can't answer these questions reliably. Monthly automated chaos experiments can.

CHAOS ILLUMINATION: Chaos engineering in production proves resilience. Chaos engineering only in staging proves nothing about production.

Multi-Region Recovery: Route 53 Health Checks Save Your Ass

Single-region deployment is single point of failure. AWS regions fail. Us-east-1 has failed multiple times, taking down half the internet each time. If your architecture assumes regions never fail, you're building on wishful thinking instead of engineering.

Route 53 Health Check-Driven Failover:

HealthCheckApi: 
  Type: 'AWS::Route53::HealthCheck'
  Properties: 
    HealthCheckConfig: 
      Port: 443
      Type: HTTPS
      EnableSNI: True
      ResourcePath: "v1/healthcheck"
      FullyQualifiedDomainName: "api.hack23.com"
      RequestInterval: 10
      FailureThreshold: 2

DeliveryApiRoute53RecordSetGroup:
  Type: AWS::Route53::RecordSetGroup
  Properties:
    HostedZoneName: "hack23.com."
    RecordSets:
      - Name: "api.hack23.com."
        Type: A
        SetIdentifier: apizone1a
        HealthCheckId: !Ref HealthCheckApi
        Weight: '50'
        AliasTarget:
          HostedZoneId: !Ref RestApiDomainNameRegionalHostedZoneId
          DNSName: !Ref RestApiDomainNameRegionalDomainName

How This Works: Route 53 health checks hit your API endpoint every 10 seconds. Two consecutive failures (20 seconds) marks endpoint unhealthy. DNS automatically routes traffic to healthy region. Total failover time: <30 seconds including DNS propagation.

Critical Detail: Health checks must validate actual functionality, not just "service responds 200 OK." Our /v1/healthcheck endpoint validates database connectivity, Lambda execution, DynamoDB access—proving the entire stack works, not just that nginx is running.

Think for yourself: Your health check returning 200 OK while database is on fire is useless. Test the whole stack or test nothing.

Backup Validation: Schrödinger's Backup Problem

A backup you haven't restored is both working and broken simultaneously. It exists in quantum superposition until disaster strikes and you attempt restore, collapsing the waveform into "oh thank god" or "we're completely fucked." Testing restores is how you collapse that waveform on your schedule instead of disaster's schedule.

Automated Restore Testing:

Monthly automated restore validation proves backups actually restore. Here's Lambda function that automatically validates DynamoDB backups:

import boto3
from datetime import datetime, timedelta

def validate_backup(event, context):
    dynamodb = boto3.client('dynamodb')
    backup = boto3.client('backup')
    
    # List recent backups
    backups = dynamodb.list_backups(
        TableName='global-table',
        TimeRangeLowerBound=datetime.utcnow() - timedelta(days=7)
    )['BackupSummaries']
    
    if not backups:
        raise Exception("No recent backups found!")
    
    latest_backup = sorted(backups, key=lambda x: x['BackupCreationDateTime'])[-1]
    
    # Restore to test table
    restore_response = dynamodb.restore_table_from_backup(
        TargetTableName=f"restore-test-{datetime.now().strftime('%Y%m%d-%H%M%S')}",
        BackupArn=latest_backup['BackupArn']
    )
    
    test_table = restore_response['TableDescription']['TableName']
    
    # Validate restore
    waiter = dynamodb.get_waiter('table_exists')
    waiter.wait(TableName=test_table)
    
    # Verify data integrity
    response = dynamodb.describe_table(TableName=test_table)
    item_count = response['Table']['ItemCount']
    
    # Cleanup test table
    dynamodb.delete_table(TableName=test_table)
    
    return {
        'statusCode': 200,
        'backup_arn': latest_backup['BackupArn'],
        'restore_validated': True,
        'item_count': item_count,
        'test_timestamp': datetime.utcnow().isoformat()
    }

What This Validates: Backup exists. Backup can be restored. Restored table contains data. Entire restore process completes within RTO. Automated monthly execution means you discover backup problems during testing, not during disasters.

FNORD exists in every untested backup procedure. You can't see it until disaster reveals it. Test restores make FNORD visible.

Evidence Collection: Audit Trail That Survives Disasters

Recovery without evidence is recovery you can't prove. Auditors, insurers, regulators, customers—they all want proof you can actually recover. Collecting evidence during recovery validates capabilities and provides compliance documentation.

Automated Evidence Pipeline:

import json
import boto3
from datetime import datetime

def collect_disaster_recovery_evidence(event, context):
    fis = boto3.client('fis')
    ssm = boto3.client('ssm')
    backup = boto3.client('backup')
    s3 = boto3.client('s3')
    
    evidence = {
        'collection_date': datetime.utcnow().isoformat(),
        'fis_experiments': collect_fis_experiments(fis),
        'ssm_executions': collect_ssm_executions(ssm),
        'backup_jobs': collect_backup_jobs(backup),
        'compliance_status': validate_compliance()
    }
    
    # Store in immutable Glacier storage
    s3.put_object(
        Bucket='hack23-audit-evidence',
        Key=f'dr-evidence/{datetime.now().strftime("%Y-%m")}.json',
        Body=json.dumps(evidence, indent=2),
        StorageClass='GLACIER_IR'  # Immutable storage
    )
    
    return evidence

def collect_fis_experiments(fis_client):
    """Collect last 30 days FIS experiment results"""
    experiments = fis_client.list_experiments()['experiments']
    return [{
        'id': exp['id'],
        'state': exp['state'],
        'creation_time': exp['creationTime'].isoformat(),
        'tags': exp.get('tags', {})
    } for exp in experiments]

def validate_compliance():
    """Validate RTO/RPO compliance"""
    return {
        'rto_compliance': check_rto_targets(),
        'rpo_compliance': check_rpo_targets(),
        'backup_coverage': check_backup_coverage(),
        'chaos_testing': check_chaos_testing_frequency(),
        'timestamp': datetime.utcnow().isoformat()
    }

Evidence Retention: All disaster recovery evidence stored in immutable Glacier storage for 3 years (regulatory requirement). Evidence includes: FIS experiment logs, SSM execution outputs, backup validation results, restore test outcomes, RTO/RPO achievement metrics.

Why Immutable Storage: Regular S3 can be deleted by compromised credentials. Glacier with vault lock cannot be deleted even by root account. Ransomware encrypting evidence? Glacier doesn't care—evidence survives.

Chapel Perilous moment: Discovering during audit that you can't prove your DR capabilities because evidence was deleted. Immutable storage prevents this nightmare.

Our Approach: Automated Recovery Through AWS-Native Tooling

At Hack23, disaster recovery is systematic implementation leveraging AWS managed services:

🔰 AWS Resilience Hub Policy Enforcement:

• Resilience Policies: Define RTO/RPO requirements per application tier mapped to Classification Framework.
• Application Assessment: Continuous resilience analysis identifies gaps, missing redundancy, single points of failure.
• Deployment Gating: Production releases require "GREEN" Resilience Hub assessment status.
• Evidence Documentation: Audit trail of resilience assessments, remediation actions, compliance validation.

🧪 AWS Fault Injection Service Integration:

• Experiment Templates: Pre-configured chaos scenarios (instance termination, API throttling, network blackhole).
• SSM Automation: FIS experiments trigger AWS Systems Manager documents for complex failure scenarios.
• Safeguards: CloudWatch alarm integration stops experiments if critical thresholds breached.
• Validation: Automated verification of recovery time vs. RTO target with pass/fail criteria.

💾 AWS Backup Orchestration:

• Central Backup Plans: Automated scheduling (hourly/daily/weekly) per data classification tier.
• Immutable Vaults: Vault lock prevents backup deletion for ransomware protection. Cross-region replication to separate AWS account.
• Point-in-Time Recovery: Continuous backups enable restoration to any point within retention window.
• Backup Audit Manager: Compliance reporting validates backup coverage, retention policies, restore testing.

☁️ Multi-Region Resilience Architecture:

• Route 53 Health Checks: Automated DNS failover when primary region health checks fail.
• Multi-AZ Deployment: Lambda, RDS, DynamoDB deployed across availability zones for automatic failover.
• S3 Cross-Region Replication: Critical data replicated asynchronously for regional disaster recovery.
• CloudFormation StackSets: Infrastructure-as-code deployed identically across regions for consistent recovery.

Full technical implementation including FIS experiment templates, SSM automation documents, and Resilience Hub policies in our public Disaster Recovery Plan.

Testing Under Realistic Disaster Conditions: Murphy's Law Applied

Your DR plan works perfectly in the lab. Reality isn't a lab. Real disasters happen at 3 AM when the expert is on vacation, network is degraded, half your team is unreachable, and stress makes people stupid. Testing DR under comfortable conditions proves nothing about disaster performance.

Realistic Disaster Testing Scenarios:

Testing Philosophy: Comfortable tests prove comfortable capabilities. Stressful tests prove actual disaster resilience. We test under Murphy's Law assumptions: everything that can go wrong will go wrong, simultaneously, at worst possible time. If recovery succeeds under these conditions, it'll probably succeed during actual disasters.

FNORD is invisible in comfortable testing. Stress reveals FNORD. Test under stress or discover FNORD during disaster.

Welcome to Chapel Perilous: Chaos As Resilience Strategy

Nothing is true. Everything is permitted. Including—especially—your entire infrastructure burning to ash while you discover your "tested" DR plan was fiction. The only question is: are you paranoid enough to have actually proven you can recover, or are you trusting unvalidated hope?

Most organizations write disaster recovery plans, file them in SharePoint next to the business continuity plan nobody's read since the consultant delivered it, and pray to the infrastructure gods they never need them. They talk about RTO/RPO targets pulled from "industry best practices" (translation: someone's ass). They mention "high availability" (translation: we pay for multi-AZ but haven't tested failover). They claim "redundant architecture" (translation: we have backups somewhere, probably). None of it is tested. None of it is proven. It's hopeful fiction masquerading as operational capability. FNORD.

We weaponize chaos because paranoia without action is just anxiety. Monthly FIS experiments deliberately terminate our databases, inject API errors, break our network connections—because if we don't break it first, reality will break it later when you're on vacation. AWS Resilience Hub gates block production deployments that don't meet RTO/RPO requirements—because shipping features that can't survive failures isn't velocity, it's technical debt with catastrophic interest rates. Immutable cross-region backups protect against ransomware—because trusting that attackers won't encrypt your backups is optimism we can't afford. SSM automation documents encode recovery procedures as executable code—because manual runbooks fail spectacularly when executed by panicked humans at 3 AM. Realistic disaster testing with degraded networks, missing personnel, and corrupted backups—because testing under comfortable conditions proves nothing about disaster performance. This isn't theory. It's continuously validated operational resilience. Or as we call it: applied paranoia.

Think for yourself. Question DR plans that have never failed over. Question RTO targets without automation sophisticated enough to meet them. Question "disaster recovery" that's really "disaster hope with extra steps." Question backup strategies that have never attempted restore under realistic conditions (degraded network, stressed personnel, corrupted data). Question health checks that return 200 OK while database is on fire. Question multi-region architecture that's never tested cross-region failover. (Spoiler: Hope isn't a strategy. It's what you do when you don't have a strategy.)

Our competitive advantage: We demonstrate cybersecurity consulting expertise through provable recovery capabilities that survive public scrutiny. <5 min RTO for critical systems with monthly chaos validation and timestamped evidence. Resilience Hub deployment gating that blocks hope-based deployments. Public DR documentation with FIS experiment evidence, SSM automation templates, CloudFormation infrastructure-as-code, Lambda validation functions, and health check configurations because obscurity isn't security. Realistic disaster testing that proves recovery works under Murphy's Law conditions. This isn't DR theater performed for auditors. It's operational proof we're paranoid enough to survive reality.

ULTIMATE ILLUMINATION: You are now deep in Chapel Perilous, the place where all comfortable lies dissolve. You can continue hoping your untested DR plan works while filing it under "Things We'll Never Need." Or you can embrace paranoia, deliberately break your own infrastructure monthly with FIS experiments, encode recovery as SSM automation that executes reliably under chaos, test restoration under realistic disaster conditions (degraded network, missing personnel, corrupted backups), validate multi-region failover with Route 53 health checks, prove backup integrity with automated restore validation, and collect immutable evidence that survives disasters. Your systems. Your choice. Choose evidence over hope. Choose automation over manual procedures. Choose chaos engineering over wishful thinking. Choose realistic disaster testing over comfortable lab tests. Choose survival over comfortable delusion. Are you paranoid enough yet?

All hail Eris! All hail Discordia!

"Think for yourself, schmuck! Untested disaster recovery is disaster theater performed for compliance auditors. We inject deliberate chaos monthly with AWS Fault Injection Service to prove recovery works, encode procedures as SSM automation that executes under pressure, test multi-region failover with Route 53 health checks, validate backup restoration automatically, and test everything under realistic disaster conditions (3 AM, degraded network, missing personnel, corrupted data)—because in the reality tunnel we inhabit, everything fails eventually, Murphy's Law compounds failures simultaneously, and hope is what you feel right before learning your DR plan was comfortable fiction. Restore or regret. Test or discover. Your infrastructure. Your disaster. Your 3 AM phone call. Choose paranoia."

— Hagbard Celine, Captain of the Leif Erikson 🍎 23 FNORD 5

Related ISMS Policies

• Business Continuity Plan — When everything fails at once
• Incident Response Plan — Classification-driven response SLAs
• Backup Recovery Policy — Immutable backups that actually restore
• Back to Main Manifesto — Everything you know about security is a lie