Tutorial 9: Store Remote State

Learning Objectives

• Understand the importance of remote state storage
• Learn to configure S3 backend for state storage
• Implement state locking with DynamoDB
• Practice team collaboration with shared state
• Understand security considerations for remote state

Why Remote State?

Problems with Local State

By default, Terraform stores state in a local file called terraform.tfstate. This works for individual development but has significant limitations:

• No Collaboration: Multiple team members can't work on the same infrastructure
• No Locking: Risk of concurrent modifications corrupting state
• No Security: State files contain sensitive information in plain text
• No Backup: Risk of losing state file means losing infrastructure tracking
• No Versioning: No history of state changes

Benefits of Remote State

• Team Collaboration: Shared access to infrastructure state
• State Locking: Prevents concurrent modifications
• Security: Encryption at rest and in transit
• Backup and Versioning: Automatic backups and change history
• Auditing: Track who made changes and when

Remote State Backends

Available Backends

Terraform supports multiple remote state backends:

• S3: AWS Simple Storage Service (most common)
• Azure Storage: Azure Blob Storage
• Google Cloud Storage: GCS buckets
• Terraform Cloud: HashiCorp's managed service
• Consul: HashiCorp Consul
• etcd: Distributed key-value store

Choosing a Backend

For this tutorial, we'll use AWS S3 with DynamoDB for state locking, as it's the most widely used configuration.

Setting Up S3 Backend

Prerequisites

# Ensure AWS CLI is configured
aws configure list

# Verify permissions
aws sts get-caller-identity

Step 1: Create S3 Bucket for State Storage

Using AWS CLI

# Create unique bucket name
BUCKET_NAME="terraform-state-$(date +%s)-$(whoami)"
REGION="us-west-2"

# Create S3 bucket
aws s3 mb s3://$BUCKET_NAME --region $REGION

# Enable versioning
aws s3api put-bucket-versioning \
  --bucket $BUCKET_NAME \
  --versioning-configuration Status=Enabled

# Enable server-side encryption
aws s3api put-bucket-encryption \
  --bucket $BUCKET_NAME \
  --server-side-encryption-configuration '{
    "Rules": [
      {
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "AES256"
        }
      }
    ]
  }'

# Block public access
aws s3api put-public-access-block \
  --bucket $BUCKET_NAME \
  --public-access-block-configuration \
    BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

Using Terraform (Bootstrap Configuration)

# bootstrap/main.tf
terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-west-2"
}

# Random ID for unique bucket name
resource "random_id" "bucket_suffix" {
  byte_length = 8
}

# S3 bucket for state storage
resource "aws_s3_bucket" "terraform_state" {
  bucket = "terraform-state-${random_id.bucket_suffix.hex}"
  
  tags = {
    Name        = "Terraform State Storage"
    Environment = "shared"
    Purpose     = "terraform-backend"
  }
}

# Enable versioning
resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  
  versioning_configuration {
    status = "Enabled"
  }
}

# Enable server-side encryption
resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
    }
  }
}

# Block public access
resource "aws_s3_bucket_public_access_block" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

# DynamoDB table for state locking
resource "aws_dynamodb_table" "terraform_locks" {
  name         = "terraform-state-locks"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
  
  attribute {
    name = "LockID"
    type = "S"
  }
  
  tags = {
    Name        = "Terraform State Locks"
    Environment = "shared"
    Purpose     = "terraform-backend"
  }
}

# Outputs for backend configuration
output "s3_bucket_name" {
  description = "Name of the S3 bucket for Terraform state"
  value       = aws_s3_bucket.terraform_state.bucket
}

output "dynamodb_table_name" {
  description = "Name of the DynamoDB table for state locking"
  value       = aws_dynamodb_table.terraform_locks.name
}

output "region" {
  description = "AWS region"
  value       = "us-west-2"
}

Step 2: Create DynamoDB Table for State Locking

# Create DynamoDB table for state locking
aws dynamodb create-table \
  --table-name terraform-state-locks \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST \
  --region $REGION

Step 3: Configure Backend in Terraform

Backend Configuration Block

# main.tf
terraform {
  required_version = ">= 1.0"
  
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
  
  # Backend configuration
  backend "s3" {
    bucket         = "your-terraform-state-bucket-name"
    key            = "infrastructure/terraform.tfstate"
    region         = "us-west-2"
    encrypt        = true
    dynamodb_table = "terraform-state-locks"
    
    # Optional: Add additional security
    # kms_key_id = "arn:aws:kms:us-west-2:123456789012:key/12345678-1234-1234-1234-123456789012"
  }
}

provider "aws" {
  region = "us-west-2"
}

Migrating from Local to Remote State

Step 1: Backup Current State

# Create backup of local state
cp terraform.tfstate terraform.tfstate.local.backup

Step 2: Add Backend Configuration

Add the backend configuration to your existing main.tf:

terraform {
  # ... existing configuration
  
  backend "s3" {
    bucket         = "your-terraform-state-bucket-name"
    key            = "infrastructure/terraform.tfstate"
    region         = "us-west-2"
    encrypt        = true
    dynamodb_table = "terraform-state-locks"
  }
}

Step 3: Initialize with Backend

# Reinitialize Terraform with backend
terraform init

# Terraform will ask if you want to copy existing state
# Choose 'yes' to migrate local state to S3

Expected output:

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.

  Enter a value: yes

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Step 4: Verify Remote State

# Verify state is now remote
terraform state list

# Check S3 bucket
aws s3 ls s3://your-terraform-state-bucket-name/infrastructure/

Working with Remote State

State Locking in Action

# Terminal 1: Start a long-running operation
terraform plan

# Terminal 2: Try to run another operation (will be blocked)
terraform plan
# This will show: "Error locking state: ConditionalCheckFailedException"

Viewing Remote State

# Show current state
terraform show

# List resources
terraform state list

# Show specific resource
terraform state show aws_instance.web

Force Unlock (Emergency Use Only)

# If a lock gets stuck, you can force unlock
# WARNING: Only do this if you're sure no other process is running
terraform force-unlock LOCK_ID

Advanced Backend Configuration

Using Variables for Backend Config

Create a backend.hcl file:

# backend.hcl
bucket         = "my-terraform-state-bucket"
key            = "infrastructure/terraform.tfstate"
region         = "us-west-2"
encrypt        = true
dynamodb_table = "terraform-state-locks"

Initialize with backend config file:

terraform init -backend-config=backend.hcl

Environment-Specific Backend Configuration

Development Backend

# backend-dev.hcl
bucket         = "terraform-state-dev"
key            = "dev/terraform.tfstate"
region         = "us-west-2"
encrypt        = true
dynamodb_table = "terraform-state-locks-dev"

Production Backend

# backend-prod.hcl
bucket         = "terraform-state-prod"
key            = "prod/terraform.tfstate"
region         = "us-west-2"
encrypt        = true
dynamodb_table = "terraform-state-locks-prod"

Usage:

# Initialize for development
terraform init -backend-config=backend-dev.hcl

# Initialize for production
terraform init -backend-config=backend-prod.hcl

Workspace-Based Organization

terraform {
  backend "s3" {
    bucket         = "terraform-state-main"
    key            = "infrastructure/terraform.tfstate"
    region         = "us-west-2"
    encrypt        = true
    dynamodb_table = "terraform-state-locks"
    
    # Workspace-specific paths
    workspace_key_prefix = "workspaces"
    # Results in: workspaces/{workspace}/infrastructure/terraform.tfstate
  }
}

Security Best Practices

S3 Bucket Security

# Comprehensive S3 bucket security
resource "aws_s3_bucket" "terraform_state" {
  bucket = "terraform-state-${random_id.bucket_suffix.hex}"
}

# Enable versioning for rollback capability
resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  versioning_configuration {
    status = "Enabled"
  }
}

# Server-side encryption
resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm = "AES256"
      # Or use KMS:
      # sse_algorithm     = "aws:kms"
      # kms_master_key_id = aws_kms_key.terraform_state.arn
    }
    bucket_key_enabled = true
  }
}

# Block all public access
resource "aws_s3_bucket_public_access_block" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

# Lifecycle configuration
resource "aws_s3_bucket_lifecycle_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  
  rule {
    id     = "cleanup_old_versions"
    status = "Enabled"
    
    noncurrent_version_expiration {
      noncurrent_days = 90
    }
    
    abort_incomplete_multipart_upload {
      days_after_initiation = 7
    }
  }
}

# Logging (optional)
resource "aws_s3_bucket_logging" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  
  target_bucket = aws_s3_bucket.access_logs.id
  target_prefix = "terraform-state-access/"
}

IAM Permissions for Terraform State

# IAM policy for Terraform state access
data "aws_iam_policy_document" "terraform_state_access" {
  statement {
    effect = "Allow"
    
    actions = [
      "s3:ListBucket",
    ]
    
    resources = [
      aws_s3_bucket.terraform_state.arn,
    ]
  }
  
  statement {
    effect = "Allow"
    
    actions = [
      "s3:GetObject",
      "s3:PutObject",
      "s3:DeleteObject",
    ]
    
    resources = [
      "${aws_s3_bucket.terraform_state.arn}/*",
    ]
  }
  
  statement {
    effect = "Allow"
    
    actions = [
      "dynamodb:GetItem",
      "dynamodb:PutItem",
      "dynamodb:DeleteItem",
    ]
    
    resources = [
      aws_dynamodb_table.terraform_locks.arn,
    ]
  }
}

resource "aws_iam_policy" "terraform_state_access" {
  name   = "terraform-state-access"
  path   = "/"
  policy = data.aws_iam_policy_document.terraform_state_access.json
}

Using KMS Encryption

# KMS key for state encryption
resource "aws_kms_key" "terraform_state" {
  description             = "KMS key for Terraform state encryption"
  deletion_window_in_days = 7
  
  tags = {
    Name = "terraform-state-key"
  }
}

resource "aws_kms_alias" "terraform_state" {
  name          = "alias/terraform-state"
  target_key_id = aws_kms_key.terraform_state.key_id
}

# Update backend configuration to use KMS
terraform {
  backend "s3" {
    bucket         = "terraform-state-bucket"
    key            = "infrastructure/terraform.tfstate"
    region         = "us-west-2"
    encrypt        = true
    kms_key_id     = "alias/terraform-state"
    dynamodb_table = "terraform-state-locks"
  }
}

Multi-Team State Organization

Project-Based State Organization

terraform-state-bucket/
├── team-a/
│   ├── dev/terraform.tfstate
│   ├── staging/terraform.tfstate
│   └── prod/terraform.tfstate
├── team-b/
│   ├── dev/terraform.tfstate
│   ├── staging/terraform.tfstate
│   └── prod/terraform.tfstate
└── shared/
    ├── networking/terraform.tfstate
    ├── security/terraform.tfstate
    └── monitoring/terraform.tfstate

Backend Configuration for Teams

# Team A - Development
terraform {
  backend "s3" {
    bucket         = "company-terraform-state"
    key            = "team-a/dev/terraform.tfstate"
    region         = "us-west-2"
    encrypt        = true
    dynamodb_table = "terraform-state-locks"
  }
}

# Shared Infrastructure
terraform {
  backend "s3" {
    bucket         = "company-terraform-state"
    key            = "shared/networking/terraform.tfstate"
    region         = "us-west-2"
    encrypt        = true
    dynamodb_table = "terraform-state-locks"
  }
}

State Management Commands

Backend Initialization

# Initialize with new backend
terraform init

# Reconfigure backend
terraform init -reconfigure

# Migrate from one backend to another
terraform init -migrate-state

# Initialize without backend
terraform init -backend=false

State Operations

# Pull remote state to local file
terraform state pull > state.json

# Push local state to remote
terraform state push state.json

# Refresh state from real infrastructure
terraform refresh

# Show state statistics
terraform show -json | jq '.values.root_module.resources | length'

Troubleshooting Remote State

Common Issues and Solutions

State Lock Issues

# Check lock status
aws dynamodb get-item \
  --table-name terraform-state-locks \
  --key '{"LockID":{"S":"terraform-state-bucket/infrastructure/terraform.tfstate-md5"}}'

# Force unlock if needed (dangerous!)
terraform force-unlock LOCK_ID

State File Corruption

# Download previous version from S3
aws s3api list-object-versions \
  --bucket terraform-state-bucket \
  --prefix infrastructure/terraform.tfstate

# Restore previous version
aws s3api get-object \
  --bucket terraform-state-bucket \
  --key infrastructure/terraform.tfstate \
  --version-id VERSION_ID \
  terraform.tfstate.restored

Permission Issues

# Test S3 access
aws s3 ls s3://terraform-state-bucket/

# Test DynamoDB access
aws dynamodb describe-table --table-name terraform-state-locks

# Check IAM permissions
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123456789012:user/terraform \
  --action-names s3:GetObject \
  --resource-arns arn:aws:s3:::terraform-state-bucket/*

Best Practices Summary

1. State File Organization

• Use descriptive state file paths
• Separate environments and teams
• Use consistent naming conventions

2. Security

• Enable encryption at rest and in transit
• Use IAM policies for least privilege access
• Enable versioning for rollback capability
• Block public access to state buckets

3. Backup and Recovery

• Enable S3 versioning
• Implement lifecycle policies for old versions
• Regular state file backups
• Document recovery procedures

4. Team Collaboration

• Use state locking to prevent conflicts
• Implement consistent backend configuration
• Use separate state files for different environments
• Document backend configuration changes

5. Monitoring

• Monitor state file access
• Set up alerts for lock timeouts
• Track state file size growth
• Monitor backend costs

Key Takeaways

• Remote state enables team collaboration and provides security
• S3 with DynamoDB is the most common backend configuration
• State locking prevents concurrent modifications and corruption
• Proper IAM permissions are crucial for security
• Organization and naming conventions matter for team productivity
• Always backup state files and have recovery procedures
• Use encryption and versioning for production environments

Next Steps

1. Complete Tutorial 10: Lock and Upgrade Provider Versions
2. Learn about state management best practices
3. Explore Terraform Cloud as an alternative backend
4. Practice with multi-environment configurations

Additional Resources

• Terraform Backend Configuration
• S3 Backend Documentation
• State Management Best Practices