My pipeline builds, plans and applies just fine for my dev branch. When I push to my master branch, I get "Error refreshing state: HTTP remote state endpoint requires auth". See pipeline logs: (and since someone will ask, the token "$onbaord_cloud_account_into_monitoring" has complete read/write access to the scoped project API.)...
Running with gitlab-runner 13.4.1 (REDACTED) on runner-gitlab-runner-REDACTED-REDACTED REDACTED Resolving secrets 00:00 Preparing the "kubernetes" executor 00:00 Using Kubernetes namespace: gitlab-managed-apps Using Kubernetes executor with image my-gitlab.io:5005/team-cloud-platform-team/terraform-modules/root-module-deployment/python-terraform14 ... Preparing environment 00:03 Waiting for pod gitlab-managed-apps/runner-REDACTED-project-REDACTED-concurrent-REDACTED to be running, status is Pending Running on runner-REDACTED-project-REDACTED-concurrent-REDACTED via runner-gitlab-runner-REDACTED-REDACTED... Getting source from Git repository 00:02 Fetching changes with git depth set to 50... Initialized empty Git repository in /builds/team-cloud-platform-team/teamcloudv2/workers/onboard-cloud-account-into-monitoring/.git/ Created fresh repository. Checking out REDACTED as master... Skipping Git submodules setup Restoring cache 00:00 Checking cache for REDACTEDREDACTEDREDACTEDREDACTED... No URL provided, cache will not be downloaded from shared cache server. Instead a local version of cache will be extracted. Successfully extracted cache Executing "step_script" stage of the job script 00:05 $ git config --global credential.helper cache $ git fetch From * [new branch] dev -> origin/dev $ if [ "$CI_COMMIT_REF_NAME" == "master" ]; then TF_ACCOUNT="REDACTED";fi $ if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then TF_ACCOUNT="REDACTED";fi $ if [ "$CI_COMMIT_REF_NAME" == "master" ]; then ORG_ACCOUNT="REDACTED";fi $ if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then ORG_ACCOUNT="REDACTED";fi $ echo ${TF_ACCOUNT} REDACTED $ echo ${TF_ADDRESS} $ echo TF_HTTP_ADDRESS=${TF_ADDRESS} TF_HTTP_ADDRESS= $ echo TF_HTTP_LOCK_ADDRESS=${TF_LOCK} TF_HTTP_LOCK_ADDRESS= $ echo TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK} TF_HTTP_UNLOCK_ADDRESS= $ echo TF_ADDRESS=${TF_ADDRESS} TF_ADDRESS= $ export TF_HTTP_ADDRESS=${TF_ADDRESS} $ export TF_HTTP_LOCK_ADDRESS=${TF_LOCK} $ export TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK} $ export TF_HTTP_LOCK_METHOD="POST" $ export TF_HTTP_UNLOCK_METHOD="DELETE" $ export TF_HTTP_RETRY_WAIT_MIN='5' $ export TF_HTTP_USERNAME='JOHN.DOE' $ export TF_HTTP_PASSWORD=${onbaord_cloud_account_into_monitoring} $ export TF_VAR_ACCOUNT=${TF_ACCOUNT} $ export TF_VAR_ORG_ACCOUNT=${ORG_ACCOUNT} $ export AWS_DEFAULT_REGION='us-east-1' $ terraform init Initializing modules... Downloading git:: for lambda... - lambda in .terraform/modules/lambda Downloading git:: for lambda_iam... - lambda_iam in .terraform/modules/lambda_iam Initializing the backend... Successfully configured the backend "http"! Terraform will automatically use this backend unless the backend configuration changes. Error refreshing state: HTTP remote state endpoint requires auth Cleaning up file based variables 00:00 ERROR: Job failed: command terminated with exit code 1 A peek at my gitlab-ci.yml:
image: name: my-gitlab.io:5005/team-cloud-platform-team/terraform-modules/root-module-deployment/python-terraform14 entrypoint: - '/usr/bin/env' - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' stages: - build - plan - apply variables: PLAN: plan.tfplan JSON_PLAN_FILE: tfplan.json WORKSPACE: "dev" TF_IN_AUTOMATION: "true" ZIP_FILE: "lambda_package.zip" STATE_FILE: ${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH} TF_ACCOUNT: "" ORG_ACCOUNT: "" TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH} TF_LOCK: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}/lock TF_UNLOCK: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME}-${CI_COMMIT_BRANCH}/lock cache: key: "$CI_COMMIT_SHA" paths: - .terraform before_script: - git config --global credential.helper cache - git fetch - if [ "$CI_COMMIT_REF_NAME" == "master" ]; then TF_ACCOUNT="REDACTED";fi - if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then TF_ACCOUNT="REDACTED";fi - if [ "$CI_COMMIT_REF_NAME" == "master" ]; then ORG_ACCOUNT="REDACTED";fi - if [ "$CI_COMMIT_REF_NAME" == "dev" ]; then ORG_ACCOUNT="REDACTED";fi - echo ${TF_ACCOUNT} - echo ${TF_ADDRESS} - echo TF_HTTP_ADDRESS=${TF_ADDRESS} - echo TF_HTTP_LOCK_ADDRESS=${TF_LOCK} - echo TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK} - echo TF_ADDRESS=${TF_ADDRESS} - export TF_HTTP_ADDRESS=${TF_ADDRESS} - export TF_HTTP_LOCK_ADDRESS=${TF_LOCK} - export TF_HTTP_UNLOCK_ADDRESS=${TF_UNLOCK} - export TF_HTTP_LOCK_METHOD="POST" - export TF_HTTP_UNLOCK_METHOD="DELETE" - export TF_HTTP_RETRY_WAIT_MIN='5' - export TF_HTTP_USERNAME='MATTHEW.FETHEROLF' - export TF_HTTP_PASSWORD=${onbaord_cloud_account_into_monitoring} - export TF_VAR_ACCOUNT=${TF_ACCOUNT} - export TF_VAR_ORG_ACCOUNT=${ORG_ACCOUNT} - export AWS_DEFAULT_REGION='us-east-1' - terraform init # -----BUILD----- # 1 build job per lambda function buildLambda: stage: build tags: - cluster script: - echo "Beginning Build" - cd lambda_code/ - echo "Switched into Lambda dir" - pip3 install -r requirements.txt --target . - echo "Requirements installed" - echo $ZIP_FILE - zip -r $ZIP_FILE * - echo "Zip file packaged up" artifacts: paths: - lambda_code/ only: - dev - master - merge_requests # -----PLAN----- planMerge: stage: plan tags: - cluster script: - terraform plan dependencies: - buildLambda only: - merge_requests planCommit: stage: plan tags: - cluster script: - terraform plan dependencies: - buildLambda # -----APPLY----- # dev branch will auto deploy applyDev: stage: apply tags: - cluster script: - echo "Running Terraform Apply" - terraform apply -auto-approve dependencies: - buildLambda only: - dev #when: manual # prod branch will require manual deployment approval applyProd: stage: apply tags: - cluster script: - echo "Running Terraform Apply" - terraform apply -auto-approve when: manual dependencies: - buildLambda only: - master backend.tf:
terraform { backend "http" { } } main.tf:
locals { common_tags = { SVC_ACCOUNT_ID = "REDACTED", CLOUD_PLATFORM_PROD_ID = "REDACTED" } } module "lambda" { source = "git::" lambda_name = var.name lambda_role = "arn:aws:iam::${var.ACCOUNT}:role/${var.lambda_role}" lambda_handler = var.handler lambda_runtime = var.runtime default_lambda_timeout = var.timeout env = local.common_tags ACCOUNT = var.ACCOUNT } module "lambda_iam" { source = "git::" lambda_policy = var.lambda_policy ACCOUNT = var.ACCOUNT lambda_role = var.lambda_role } inputs.tf:
variable "handler" { type = string default = "handler.lambda_handler" } variable "runtime" { type = string default = "python3.8" } variable "name" { type = string default = "onboard-cloud-account-into-monitoring" } variable "timeout"{ type = string default = "120" } variable "lambda_role" { type = string default = "cloud-platform-onboard-to-monitoring" } variable "ACCOUNT" { type = string } variable "ORG_ACCOUNT" { type = string } variable "lambda_policy" { default = "{\"Version\": \"2012-10-17\",\"Statement\": [{\"Sid\": \"VisualEditor0\",\"Effect\": \"Allow\",\"Action\": [\"logs:CreateLogStream\",\"logs:CreateLogGroup\"],\"Resource\": \"*\"},{\"Sid\": \"VisualEditor1\",\"Effect\": \"Allow\",\"Action\": \"logs:PutLogEvents\",\"Resource\": \"*\"},{\"Sid\": \"VisualEditor2\",\"Effect\": \"Allow\",\"Action\": \"sts:AssumeRole\",\"Resource\": \"*\"},{\"Sid\": \"VisualEditor3\",\"Effect\": \"Allow\",\"Action\": \"secretsmanager:GetSecretValue\",\"Resource\": \"*\"}]}" } provider.tf:
provider "aws" { region = "us-east-1" assume_role { role_arn ="arn:aws:iam::${var.ACCOUNT}:role/team-platform-onboard-to-monitoring" } default_tags { tags = { owner = "[email protected]" altowner = "[email protected]" blc = "REDACTED" costcenter = "REDACTED" itemid = "PlatformAutomation" } } } Perhaps I also need to share the terraform modules being invoked?
32 Answers
We found the problem: The pipeline referenced a protected environment variable. The master branch was not a protected branch. The solution was to unprotect the environment variable or protect the branch. Instantly solved the problem.
in my case there was no any protected environment variables but it was required to rename TF_STATE_NAME
