🐶
Terraform

Terraform CloudWatch Log Groups for ECS Tasks and Containers

By Ondřej Dolanský on 12/30/2024

Learn how to effortlessly centralize and monitor your ECS task and container logs by creating a dedicated AWS CloudWatch log group with Terraform.

Terraform CloudWatch Log Groups for ECS Tasks and Containers

Table of Contents

Introduction

This guide provides a step-by-step approach to configuring your Amazon Elastic Container Service (ECS) tasks to send logs to Amazon CloudWatch. By following these steps, you'll be able to centralize your ECS task logs, making it easier to monitor, troubleshoot, and analyze your application's behavior in a production environment.

Step-by-Step Guide

  1. Create a CloudWatch Log Group:

    resource "aws_cloudwatch_log_group" "ecs_log_group" {
  name = "/ecs/my-app"
}

    Replace /ecs/my-app with your desired log group name.

  2. Define an IAM Role for ECS Tasks: This role allows ECS tasks to write logs to CloudWatch.

    resource "aws_iam_role" "ecs_task_role" {
  name = "ecs-task-role"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "ecs-tasks.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

  3. Attach the AmazonECS_FullAccess policy to the role:

    resource "aws_iam_role_policy_attachment" "ecs_task_role_attachment" {
  role       = aws_iam_role.ecs_task_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonECS_FullAccess"
}

  4. Configure the ECS Task Definition: Specify the log group and IAM role in your ECS task definition.

    resource "aws_ecs_task_definition" "my_task_definition" {
  family = "my-app"
  # ... other task definition settings ...

  container_definitions = <<EOF
[
  {
    "name": "my-container",
    "image": "nginx:latest",
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": aws_cloudwatch_log_group.ecs_log_group.name,
        "awslogs-region": "us-west-2",
        "awslogs-stream-prefix": "my-app"
      }
    }
  }
]
EOF

  execution_role_arn = aws_iam_role.ecs_task_role.arn
}
    • Replace "nginx:latest" with your container image.
    • Ensure "awslogs-group" points to the CloudWatch Log Group created earlier.
    • Adjust "awslogs-region" and "awslogs-stream-prefix" as needed.

  5. Deploy your ECS service: Use the defined task definition when creating or updating your ECS service.

Now, logs from your ECS tasks will be sent to the specified CloudWatch Log Group.

Code Example

This Terraform code defines infrastructure for running a Docker container on AWS ECS Fargate. It sets up a CloudWatch Log Group to store container logs, an IAM role with necessary permissions for ECS tasks, and an ECS task definition specifying the container image and log configuration. Finally, it creates an ECS cluster and service to run the defined task, ensuring logs are sent to the designated CloudWatch Log Group.

# Create a CloudWatch Log Group
resource "aws_cloudwatch_log_group" "ecs_log_group" {
  name = "/ecs/my-app"
}

# Define an IAM Role for ECS Tasks
resource "aws_iam_role" "ecs_task_role" {
  name = "ecs-task-role"
  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": "sts:AssumeRole",
      "Principal": {
        "Service": "ecs-tasks.amazonaws.com"
      },
      "Effect": "Allow",
      "Sid": ""
    }
  ]
}
EOF
}

# Attach the `AmazonECS_FullAccess` policy to the role
resource "aws_iam_role_policy_attachment" "ecs_task_role_attachment" {
  role       = aws_iam_role.ecs_task_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonECS_FullAccess"
}

# Configure the ECS Task Definition
resource "aws_ecs_task_definition" "my_task_definition" {
  family = "my-app"

  container_definitions = <<EOF
[
  {
    "name": "my-container",
    "image": "nginx:latest",
    "logConfiguration": {
      "logDriver": "awslogs",
      "options": {
        "awslogs-group": aws_cloudwatch_log_group.ecs_log_group.name,
        "awslogs-region": "us-west-2",
        "awslogs-stream-prefix": "my-app"
      }
    }
  }
]
EOF

  execution_role_arn = aws_iam_role.ecs_task_role.arn
}

# Example ECS Service using the defined task definition
resource "aws_ecs_cluster" "example" {
  name = "my-cluster"
}

resource "aws_ecs_service" "example" {
  name            = "my-service"
  cluster         = aws_ecs_cluster.example.id
  task_definition = aws_ecs_task_definition.my_task_definition.arn
  desired_count   = 1
  launch_type     = "FARGATE"

  network_configuration {
    subnets          = ["subnet-1234567890abcdef0"]
    security_groups = ["sg-1234567890abcdef0"]
  }
}

Explanation:

  1. CloudWatch Log Group: We define a log group named /ecs/my-app to store logs from our ECS tasks.
  2. IAM Role: We create an IAM role ecs-task-role that allows ECS tasks to assume it.
  3. Policy Attachment: We attach the AmazonECS_FullAccess policy to the role, granting it permissions to write logs to CloudWatch.
  4. Task Definition:
    • We define an ECS task definition my_task_definition for our application.
    • Inside the container_definitions, we specify the logConfiguration block:
      • logDriver: Set to awslogs to use CloudWatch Logs.
      • options:
        • awslogs-group: Points to the CloudWatch Log Group created earlier.
        • awslogs-region: Specifies the AWS region.
        • awslogs-stream-prefix: Adds a prefix to the log stream names.
    • We set the execution_role_arn to the created IAM role.
  5. ECS Service:
    • This code block defines an example ECS service that uses the created task definition. You can adjust the service configuration based on your needs.

Important:

  • Replace placeholders like "nginx:latest", "us-west-2", subnet IDs, and security group IDs with your actual values.
  • This code assumes you have an existing VPC and subnets configured.
  • After deploying, logs from your ECS tasks will be sent to the /ecs/my-app log group in CloudWatch.

Additional Notes

General:

  • Log Group Naming: While /ecs/my-app is a common convention, you can choose any valid CloudWatch Log Group name. Consider organizing log groups by application, environment, or team.
  • IAM Best Practices: It's generally recommended to follow the principle of least privilege. Instead of attaching AmazonECS_FullAccess, consider creating a custom IAM policy with only the necessary permissions for writing logs.
  • Log Retention: By default, CloudWatch Log Groups retain logs indefinitely. You can configure log retention policies to automatically delete older logs and manage storage costs.
  • Log Formatting: Ensure your application logs are formatted in a way that's easy to parse and analyze. Consider using structured logging formats like JSON.

Terraform Specific:

  • Terraform Modules: For larger deployments, consider using Terraform modules to encapsulate and reuse your ECS and CloudWatch logging configurations.
  • Variables and Outputs: Use Terraform variables to make your code more reusable and configurable. Output the CloudWatch Log Group name so it can be easily referenced by other parts of your infrastructure or monitoring tools.
  • Resource Dependencies: Ensure that your ECS service depends on the task definition and IAM role to guarantee they are created in the correct order.
  • State Management: Use a remote backend for Terraform state to enable collaboration and prevent accidental state corruption.

Troubleshooting:

  • Log Streams Not Appearing: If logs aren't appearing in CloudWatch, verify that the ECS task has the correct IAM role and that the awslogs driver configuration in the task definition is accurate.
  • Permission Errors: Check the ECS task execution role if you encounter permission errors when writing logs.
  • CloudWatch Logs Insights: Use CloudWatch Logs Insights to query and analyze your ECS logs for troubleshooting and performance monitoring.

Security:

  • Log Encryption: Enable encryption at rest for your CloudWatch Log Groups to protect sensitive information in your logs.
  • Access Control: Restrict access to your CloudWatch Log Groups using IAM policies to control who can view and manage your logs.

Summary

This guide outlines the process of configuring AWS ECS tasks to send logs to a CloudWatch Log Group using Terraform.

Steps:

  1. Create a CloudWatch Log Group: Define a resource for your desired log group using aws_cloudwatch_log_group.
  2. Define an IAM Role for ECS Tasks: Create an IAM role with aws_iam_role and attach the AmazonECS_FullAccess policy using aws_iam_role_policy_attachment. This allows ECS tasks to write logs to CloudWatch.
  3. Configure the ECS Task Definition:
    • In your aws_ecs_task_definition resource, specify the created log group name in the logConfiguration section of your container definition.
    • Set the logDriver to awslogs.
    • Configure awslogs-region and awslogs-stream-prefix as needed.
    • Assign the created IAM role to the execution_role_arn attribute.
  4. Deploy your ECS service: Utilize the configured task definition when creating or updating your ECS service.

By following these steps, logs generated by your ECS tasks will be automatically sent and stored in the designated CloudWatch Log Group, enabling you to monitor and analyze your application logs effectively.

Conclusion

This Terraform code provides a robust and repeatable solution for configuring your ECS tasks to send logs to CloudWatch. By centralizing your logs, you gain improved observability into your application's behavior, simplifying troubleshooting, analysis, and ultimately contributing to a more reliable and maintainable production environment.

References

Were You Able to Follow the Instructions?

😍Love it!
😊Yes
😐Meh-gical
😞No
🤮Clickbait

Related posts

Terraform for_each Error: Cannot Determine Resource Attributes

Learn how to resolve Terraform errors when your "for_each" value depends on resource attributes that can't be determined until runtime.

Terraform: Undeclared Managed Resource Error - How to Fix

Learn how to troubleshoot and resolve the "A managed resource has not been declared in the root module" error in Terraform, ensuring your infrastructure deployments run smoothly.

Terraform: ${path.module} Resolves to "." Not Module Path

Learn why Terraform's ${path.module} variable always references the current working directory instead of the module's location, and how this understanding affects your infrastructure code.