Terraform Cloud Fails with Relative Module Path

• Introduction
• Step-by-Step Guide
• Code Example
• Additional Notes
• Summary
• Conclusion
• References

When working with Terraform Cloud, it's crucial to understand how module sources are managed. Unlike local Terraform runs, Terraform Cloud doesn't automatically upload your entire project directory. This means that if you're using relative paths for your modules, you need to ensure they are explicitly included in the package uploaded to Terraform Cloud. Otherwise, you might encounter errors during the apply stage. Let's explore different ways to manage module sources effectively and troubleshoot potential issues.

Terraform Cloud doesn't automatically upload your entire project directory. When using relative paths for modules (source = "../module"), ensure your modules are included in the uploaded package.

Here's how to manage module sources effectively:

1. Local Paths:

• Best for modules within the same repository, promoting code reuse and organization.
• Example: source = "./modules/my-module"

2. Remote Sources:

• Ideal for sharing modules across projects and teams.
• Supports various repositories (Git, Mercurial, S3, etc.).
• Example: source = "git::https://github.com/your-org/terraform-modules.git//vpc?ref=v1.0.0"

Troubleshooting:

• Terraform Cloud Uploads: Double-check your CI/CD pipeline or upload process to ensure the module directory is included.
• Relative Path Issues: Verify the relative path accuracy from the root of your Terraform configuration.
• Terragrunt: If using Terragrunt, configure it to correctly fetch and package modules.

Best Practices:

• Modularize for Reusability: Break down infrastructure into reusable modules.
• Version Control: Store modules in version-controlled repositories for better management.
• Clear Documentation: Document module usage and inputs for clarity.

This code demonstrates a Terraform project structure using local and remote modules. It includes a network module defined locally and a VPC module fetched from a GitHub repository. The local module is used within the production environment to create a network and subnet. The code highlights the use of relative paths for local modules and provides guidance on configuring Terraform Cloud for module usage. It emphasizes best practices for managing module sources and troubleshooting common issues related to module paths and Terraform Cloud workspaces.

├── environments
│   └── production
│       └── main.tf
├── modules
│   └── network
│       ├── variables.tf
│       └── main.tf
└── main.tf

1. Local Path Module Example (modules/network/main.tf):

# modules/network/main.tf

resource "google_compute_network" "main" {
  name                    = var.network_name
  auto_create_subnetworks = false
  project                 = var.project_id
}

resource "google_compute_subnetwork" "main" {
  name          = "${var.network_name}-subnet"
  ip_cidr_range = var.subnet_ip_range
  network       = google_compute_network.main.name
  region        = var.region
  project       = var.project_id
}

2. Remote Source Module Example (main.tf):

# main.tf

terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 4.0"
    }
  }
}

module "vpc" {
  source  = "git::https://github.com/terraform-google-modules/terraform-google-vpc.git//modules/simple-vpc?ref=v3.5.0"
  name    = "my-vpc"
  project = "your-project-id"
}

3. Using the Local Module (environments/production/main.tf):

# environments/production/main.tf

module "network" {
  source = "../../modules/network"

  network_name    = "prod-network"
  subnet_ip_range = "10.0.0.0/24"
  project_id      = "your-project-id"
  region          = "us-central1"
}

Explanation:

• Project Structure: This example demonstrates a common project structure with separate directories for environments and modules.
• Local Module: The network module defines a network and subnet. It's referenced in environments/production/main.tf using a relative path (source = "../../modules/network").
• Remote Module: The main.tf file utilizes a remote module from GitHub for creating a VPC.
• Terraform Cloud Upload: When using Terraform Cloud, ensure your CI/CD pipeline or upload process includes both the environments and modules directories to avoid "module not found" errors.

Troubleshooting Tips:

• Verify Relative Paths: Double-check the relative paths used for local modules. They should be relative to the file where the module is being called.
• Terraform Cloud Workspaces: If using Terraform Cloud workspaces, ensure the correct working directory is set to the root of your project.
• Terragrunt: If using Terragrunt, configure its source parameter correctly to fetch and package modules.

By following these examples and best practices, you can effectively manage module sources in your Terraform projects and leverage the power of modularity for reusable and maintainable infrastructure code.

Understanding the "Context"

• Local Terraform: When running Terraform locally, it has access to your entire project directory. It can easily resolve relative paths because it starts from the location of your Terraform configuration file.
• Terraform Cloud: Think of Terraform Cloud as a remote machine. It only receives the files you explicitly upload. This is usually handled by your CI/CD pipeline or a manual upload.

Why This Matters

If you use a relative path like source = "../module" and Terraform Cloud doesn't have the parent directory that contains your module, the apply will fail. It's like giving someone directions starting from a place they aren't at.

Impact on CI/CD

• Configuration: Your CI/CD pipeline needs to be configured to package the necessary module directories along with your main Terraform code.
• Potential Issues: If you add new modules with relative paths, remember to update your CI/CD configuration to include them in future uploads.

Additional Best Practices

• Module Registry: For widely used modules, consider publishing them to a private or public Terraform Registry. This simplifies sharing and versioning.
• Module Testing: Test your modules independently to ensure they function as expected before integrating them into larger projects.
• Security: Be mindful of the sensitivity of any data or credentials hardcoded within your modules, especially if you plan to share them.

Beyond the Basics

• Terraform Workspaces: Terraform Cloud workspaces can further isolate environments and module usage. Ensure your workspace settings align with your module structure.
• Terragrunt: Terragrunt can help manage complex Terraform deployments with multiple modules and environments. It has its own mechanisms for handling module sources.

This article provides a concise guide for managing Terraform modules in Terraform Cloud, emphasizing the importance of proper module inclusion during uploads.

Effectively managing module sources is essential for successful Terraform Cloud deployments. Remember that Terraform Cloud doesn't automatically upload your entire project directory, so you must ensure all modules, especially those referenced using relative paths, are included in your uploads. Leverage local paths for modules within the same repository and remote sources for sharing across projects. When troubleshooting, verify your CI/CD pipeline or upload process, double-check relative path accuracy, and ensure Terragrunt is configured correctly if used. Adhering to best practices like modularization, version control, and clear documentation will further streamline your infrastructure management with Terraform Cloud.

• module source in parent folder fails · Issue #28755 · hashicorp ... | Terraform Version First seen in 0.14.x, and still present in 0.15.3 Terraform Configuration Files Generate config files by running this script from a new folder: mkdir a mkdir b mkdir c cat > a/mai...
• terraform - Relative path in local-exec - Stack Overflow | Apr 14, 2019 ... I'm trying to reference a local script inside a local-exec provisioner. The script is located several levels above the module directory. Using ...
• terragrunt and terraform cloud : r/Terraform | Posted by u/mclanem - 3 votes and 9 comments
• Problem with dependant module resolution if the path is relative ... | Terraform Version 0.12.13 Terraform Configuration Files Here you can see two examples: https://github.com/xocasdashdash/terraform-test-case One works perfectly with 0.11, same one fails on 0.12.13 ...
• Module Sources | Terraform | HashiCorp Developer | We recommend using local file paths for closely-related modules used primarily for the purpose of factoring out repeated code elements, and using a native ...
• Terraform Folder Structure : r/Terraform | Posted by u/bitfire184 - 10 votes and 21 comments
• How to work with JSON - Terraform - HashiCorp Discuss | Hello everyone, hope somebody can help me with my issue. I need some help on understanding how to work with a JSON full of variables that I’d like to use in my terraform file in order to create my infrastructure. Is this even possible? from what I was able to see, it is possible but I was not able to find a good example that shows me how to do it. The JSON that I want to use has simple varibles and arrays of arrays (meaning that is not just a simple JSON). Do I need to declare, in a variables.t...
• How do I just get Terraform to look into a subdirectory with a Module ... | Posted by u/MohnJaddenPowers - 7 votes and 10 comments
• Configuration Blocks and Attributes | Learn about all the blocks and attributes supported in the terragrunt configuration file.