When terragrunt plan on a single module takes over two minutes before showing a single AWS API call, something is wrong. This post covers two distinct performance problems I found and fixed in a large multi-account Terragrunt setup (~15 AWS accounts, 400+ modules).

The Problem

Running terragrunt plan on our EKS cluster module was taking 2 minutes 27 seconds just to start. With debug logging enabled, the culprit was clear:

$ time terragrunt plan --terragrunt-log-level debug 2>&1 | head -80

09:20:53 INFO  Detected 14 modules in the dependency graph
09:20:53 INFO  sops decrypt: acquiring lock for .../secrets/db-secret.enc.yaml
09:20:54 INFO  sops decrypt: decrypting .../secrets/db-secret.enc.yaml
09:20:55 INFO  sops decrypt: acquiring lock for .../secrets/payment-api-secret.enc.yaml
09:20:56 INFO  sops decrypt: decrypting .../secrets/payment-api-secret.enc.yaml
...
09:21:35 INFO  Running module .../global/iam/policies
09:21:35 INFO  terraform init
09:21:48 INFO  terraform output
09:21:52 INFO  Running module .../eventbridge/infra-changes-bus
09:21:52 INFO  terraform init
09:22:05 INFO  terraform output
...
09:23:20 INFO  Initializing backend for target module...

real    2m27s

Two separate phases caused the slowness. Both are fixable.

Phase 1: SOPS Decryption During Graph Traversal (~40s)

Root Cause

Terragrunt builds a dependency graph by parsing all transitive dependencies before executing anything. During this traversal it evaluates every module’s locals block. If any local calls sops_decrypt_file(), SOPS decrypts the file right then, even if that module is only a transitive dependency and won’t execute at all.

Our ssm/secrets-manager was a transitive dependency of the EKS cluster. It had 11 SOPS calls in locals:

# BEFORE: sops_decrypt_file in locals runs during graph traversal
locals {
  secret_db      = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets/db-secret.enc.yaml"))
  secret_payment = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets/payment-api-secret.enc.yaml"))
  secret_auth    = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets/auth-secret.enc.yaml"))
  # ... 8 more sops calls
}

inputs = {
  input_secrets = {
    "db-secret" = {
      secret_string = jsonencode({
        username = local.secret_db.username
        password = local.secret_db.password
      })
    }
    # ...
  }
}

Every time any module that transitively depended on ssm/secrets-manager was planned, all 11 files were decrypted, whether or not the module itself was being executed.

Fix: Move SOPS Calls into inputs

The inputs block is only evaluated when the module itself executes, not during graph traversal:

# AFTER: sops_decrypt_file in inputs only runs when this module executes
inputs = {
  input_secrets = {
    "db-secret" = {
      secret_string = jsonencode({
        username = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets/db-secret.enc.yaml")).username
        password = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets/db-secret.enc.yaml")).password
      })
    }
    # ...
  }
}

“But aren’t you decrypting the same file twice?”

No. Terragrunt caches sops_decrypt_file() results within a single run. From Terragrunt source config_helpers.go :

var sopsCache = cache.NewCache[string](sopsCacheName)

func decryptFile(ctx context.Context, path string) (string, error) {
    if val, ok := sopsCache.Get(ctx, path); ok {
        return val, nil
    }
    locks.EnvLock.Lock()
    defer locks.EnvLock.Unlock()
    if val, ok := sopsCache.Get(ctx, path); ok {
        return val, nil
    }
    decrypted := decrypt(path)
    sopsCache.Put(ctx, path, decrypted)
    return decrypted, nil
}

The double-check locking pattern ensures that even concurrent goroutines evaluating the same file only decrypt once. The repeated yamldecode(sops_decrypt_file(...)) pattern in inputs is cosmetic verbosity. The first call per path decrypts, every subsequent call in the same run returns the cached result instantly.

Tip

Rule: Never put sops_decrypt_file() in a locals block. Always inline it at the point of use in inputs or generate blocks.

We migrated 81 files across 15 AWS accounts following this rule.

Pattern Variations

Standard case:

# Before:
locals {
  secrets = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets.enc.yaml"))
}
inputs = {
  password = local.secrets.password
}

# After:
inputs = {
  password = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets.enc.yaml")).password
}

Chained locals (a derived local referencing the SOPS local):

# Before:
locals {
  secrets            = yamldecode(sops_decrypt_file("${path}/secrets.enc.yaml"))
  api_token          = local.secrets.logging_api_tokens.prod.devops
  notification_slack = local.secrets.notification_endpoints.slack_channels.prod
}
inputs = {
  api_token              = local.api_token
  notification_endpoints = local.notification_slack
}

# After: inline the full chain
inputs = {
  api_token              = yamldecode(sops_decrypt_file("${path}/secrets.enc.yaml")).logging_api_tokens.prod.devops
  notification_endpoints = yamldecode(sops_decrypt_file("${path}/secrets.enc.yaml")).notification_endpoints.slack_channels.prod
}

Generate blocks (SOPS feeding template rendering):

# Before:
locals {
  secrets = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets.enc.yaml"))
}
generate "kustomization" {
  contents = templatefile("kustomization.yml.tpl", {
    username = local.secrets.dashboard_username
    password = local.secrets.dashboard_password
  })
}

# After:
generate "kustomization" {
  contents = templatefile("kustomization.yml.tpl", {
    username = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets.enc.yaml")).dashboard_username
    password = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets.enc.yaml")).dashboard_password
  })
}

Phase 2: Dependency Output Fetching (~70s)

Root Cause

For each dependency block, Terragrunt by default:

  1. Runs terraform init in the dependency directory (downloads providers, initializes backend)
  2. Runs terraform output -json to extract outputs

With 8+ transitive dependencies, this meant ~70 seconds of repeated terraform init and S3/network calls before the target module could even start:

09:21:35 INFO  Running module .../global/iam/policies
09:21:35 INFO  terraform init      # provider downloads, backend init (~8s)
09:21:48 INFO  terraform output    # reads state via backend (~5s)
09:21:52 INFO  Running module .../eventbridge/infra-changes-bus
09:21:52 INFO  terraform init      # again, every dependency
09:22:05 INFO  terraform output

Fix: Read Outputs Directly from S3

Terragrunt can read dependency outputs straight from S3 state files, skipping terraform init and terraform output entirely:

export TG_DEPENDENCY_FETCH_OUTPUT_FROM_STATE=true

Add to ~/.zshrc or ~/.bashrc to make it permanent.

After setting the flag:

09:21:35 INFO  Fetching outputs for .../global/iam/policies from S3 state
09:21:35 INFO  Reading s3://acme-terraform-state/global/iam/policies/terraform.tfstate
09:21:36 INFO  Fetching outputs for .../eventbridge/infra-changes-bus from S3 state
09:21:36 INFO  Reading s3://acme-terraform-state/security/us-east-1/eventbridge/.../terraform.tfstate

One S3 GetObject per dependency instead of a full terraform init + terraform output subprocess.

Warning

It’s TG_DEPENDENCY_FETCH_OUTPUT_FROM_STATE, not TG_FETCH_DEPENDENCY_OUTPUT_FROM_STATE. Easy to mix up. Double-check if it doesn’t seem to work.

Bonus: S3 Native State Locking

While in root.hcl, we also dropped DynamoDB for state locking. Terragrunt supports S3-native lock files: a .tflock file written alongside the state. Make sure your version of Terraform or OpenTofu supports it as well.

# Before: requires a DynamoDB table provisioned in every account
remote_state {
  backend = "s3"
  config = {
    encrypt        = true
    bucket         = "acme-terraform-state-${local.account_id}"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = local.aws_region
    dynamodb_table = "terraform-locks"
    profile        = local.aws_profile
  }
}

# After: S3-native locking, no DynamoDB needed
remote_state {
  backend = "s3"
  config = {
    encrypt      = true
    bucket       = "my-terraform-state-${local.account_id}"
    key          = "${path_relative_to_include()}/terraform.tfstate"
    region       = local.aws_region
    use_lockfile = true
    profile      = local.aws_profile
  }
}

Benefits: no DynamoDB table per account, simpler bootstrap for new accounts.

Results

# Before (any module with 8+ transitive deps):
$ time terragrunt plan
real    2m27s

# After (both fixes applied):
$ time terragrunt plan
real    0m23s
FixTime saved
SOPS locals to inputs~40s
TG_DEPENDENCY_FETCH_OUTPUT_FROM_STATE=true~70s
Total~110s

Summary

Three changes, one dramatic improvement:

  1. Move sops_decrypt_file() from locals to inputs: prevents SOPS from decrypting during dependency graph traversal. ~40s saved per plan on any module with SOPS-using transitive dependencies.

  2. export TG_DEPENDENCY_FETCH_OUTPUT_FROM_STATE=true: reads dependency outputs directly from S3 state, skips terraform init and terraform output subprocesses per dependency. ~70s saved on modules with many dependencies.

  3. use_lockfile = true in remote_state: removes DynamoDB for state locking, simplifies multi-account setup.

Total: 2m27s to 23s on a module with 8 transitive dependencies and 11 SOPS-encrypted secret files in the dependency graph.