Terraform Enterprise Integration in Cloud Management

[Ashok Madhavan1]

Terraform Enterprise Integration in Cloud Management

Introduction

HashiCorp's Terraform Enterprise is a HashiCorp supported multi-cloud cloud provisioning and management system. There are two flavors in this 

1. Terraform Cloud - It is SaaS and hosted by Hashicorp for you.
2. Terraform Enterprise (TFE)  - Hosted on premise by the customer

The API is consistent across Terraform Cloud and Enterprise. In this article when i mean Terraform Enterprise, Terraform Cloud is included as well.

This article covers how Cloud Management in ServiceNow integrates with TFE.

Cloud Management support for Cloud-Native and Multi-Cloud Templating Systems

In Cloud Management, we want to make sure that our customers have a consistent way of dealing with various templating systems. This includes the specifying the templates, creating/generating catalog items from the templates and consuming the catalog items.

The below diagram explains it:

We want and have a consistent way to ingest any template and use appropriate and consistent execution mechanism for provisioning/managing any template. This implies there will be consistent treatment for CFT, ARM, Terraform opensource, Terraform Enterprise, GDM and any other supported template. The consistent treatment implies that discovery, catalog item creation, template ingestion, catalog item consumption and the provisioned stack lifecycle are all consistent across all these clouds.

The overall integration flow with TFE would be like this:

The Admin persona would setup the Terraform enterprise CI record and associate the credentials, URL. She would then discover the TFE related resources like VCS, Repo, workspaces etc. 

The Designer persona would then create a catalog item and point to the right Repo within an appropriate VCS. ServiceNow will auto-populate the catalog item based on the variables et al. The auto-population would take into consideration any metadata info provided. The designer then does any addition decoration like icons etc. She would then mark the catalog item as available for the customer.

The End-user persona would then get to the portal and choose the catalog item. He will fill in the values based on the choices the designer has made. He will then submit it. He can use API to do the same. The system would then execute the backing terraform template and invoke the TFE API to create workspace and apply it as well. The system would create a stack representing the resources that got provisioned in this call. The CMDB would also get populated properly.

Setup and Discovery - Admin Persona

This is the first part of the TFE integration. You would need to

1. Create a credential 
2. Then use that credential to create a TFE provider organization record in Cloud Management 
3. Then discover the TFE organization

Create Credential

We need to create the credential record for the TFE API keys. You would need to get to the Cloud Admin Portal and get to Credentials section.

You would need to create the credential record. For this you need to choose then need to specify the 'API Key Credentials'.

Then create the credentials by providing the API keys from TFE for the Terraform Organization. Provide a meaningful name and the API Key in the creation screen. The API key would be available from the Terraform Enterprise organization setting screen. The credential creation screen will look like this:

TFE Credential key (API Key) must start with Bearer<SPACE><GENERATED-KEY>. (For example something like this ==> Bearer mabcdefgh…)

ThenClick submit.

After you create the credential, make sure that you create an alias for the credential as well. The alias should be of type 'Credential'.

Click on the 'lock' icon next to the credential alias.

Click Submit.

Choose the credential alias you just created.

Click update.

 This alias association is critical as the underlying IntegrationHub calls depend on the credential alias.

Create TFE Organization Record  

You would need to get to the Cloud Admin Portal and get to Config Management section.

Click on the 'New' button to create a new TFE provider record. This will open the create popup screen like this:

Provide a name unique to this TFE. 

Choose the Provider as 'Terraform Enterprise'.

Provide the org name that corresponds to the TFE organization.

For Terraform Cloud, provide  'https://app.terraform.io/api/v2' as the URL. For Terraform Enterprise get the appropriate URL from your terraform administrator.

Choose the Server Type as either 'Cloud' or 'Enterprise'.

Choose the credential that corresponds to this TFE instance.

Then save it.

Discover TFE Organization

Post creation of the TFE organization, it would show up in the landing page. Click on the TFE organization record you just created and it would show up like this.

 

You would see the resource types which we discover in a TFE organization. It would be empty in the beginning. Now click on the 'Discover Now' button to start the discovery of this organization. The discovery would take a few moments to complete. Then you would see something like this:

Click on the 'Tfe VCS'  to get the list of VCS providers associated to this organization.

Any repo under any of these VCS providers can be used by Cloud Management. For this we will need credential information about the VCS system so that we can read the Terraform files for the ingestion process which will come later.  This is something which is very critical for catalog item creation. You would need to create the Github credential record for the VCS system(s).  As usual you create the API key credential record and associate it to the VCS records. 

Create a API Key Credential by going to the credentials menu. The click New. Then choose an API Key Credential. 

Key for VCS system should be like this token<SPACE>< GENERATED-KEY >. (token fabcdefgh123.....…)

Click submit.

Click on the Terraform Enterprise VCS record. Make sure that the proper credential record is associated. 

Click on the 'Find Branches and Repositories'. This will discover all the repos under the the VCS system at a high level. This is so that the catalog item designer can choose to expose any of the repo as a catalog item later. You would do this to each of the VCS system that you want to.

Catalog Item Creation - Designer Persona

One of the most important value points of TFE integration with Cloud Management is the ease with which one can expose a terraform configuration as a full fledged catalog item for the end user to consume. And with the catalog item comes all the goodness of ServiceNow with respect to Governance and CMDB support etc.

 Get to the Cloud Admin Portal and from there to the Cloud Catalog Item in the Design section.

Click on 'New' button to create a new catalog item. 

Choose the source as 'Configuration Management Template' and Provider Type as 'Terraform Enterprise'.

Then choose your specific TFE provider as the provider.

Click save to save the record. Then you would see a screen like this.

So far we have just created the catalog item and it is almost an empty record. Now we will have to specify the particular github repo so that this catalog item can front end it. Click on the 'New' button in the 'Cloud Template' tab in the lower part of the screen.

 The 'Configuration Installable' will show the list of repos which are pertinent to the VCS systems that are associated with this Terraform organization. The 'Configuration Installable' is nothing but the provisionable terraform templates in this context. 

Click on the search icon next to the Configuration Installable. A screen to pick the github repo will show up and will look like this:

Choose the appropriate repo and the version. Then click submit. On submission, the system takes in the github repo info, uses the credential info from the associated VCS system and reads the terraform files. It gets the info about the provider, variables etc and then populates the template version parameters.

Click the 'Activate' button. This will take in the template version parameters  and other associated information and populate the catalog item appropriately. It will create the mgmt variables as well as the terraform related variables and create any catalog client scripts etc as well. It will make the catalog item as a completely working one. 

Then check the 'Active' flag on the catalog item and save. Now your catalog item is ready for end user consumption.

Consumption - End User Persona 

Provisioning

 The end user can get to the Cloud User portal to order this catalog item (offering). 

Fill in information and click 'Next'.

Fill-in/choose the appropriate info and click submit. The system will then communicate with Terraform Enterprise, create the workspace, apply it. When the apply is complete, the system will do a pointed discovery of the provisioned resources and populate the CMDB. In addition it will create the stack and associate these CIs to the stack as well.

 

Click on the 'View stack details' to get the details of the stack that was provisioned.

The stack info would show up like this:

Deprovisioning

A provisioned stack can be deprovisioned later by the user. She can choose the stack and then click on the 'Deprovision' operation and click submit. The system will then communicate with theTerraform Enterprise system and decommission the workspace.

In addition to deprovisioning the stack/workspace, CMP provides you the ability to do Day-2 operations directly on the resource contained within the stack itself. For these operations, it would use the provider's API to do the needful.

Additional Info

In this article the TFE integration with CMP was described. We went over the various steps that needs to be done to make it happen. In coming days we will be adding articles that explain how to use the 'Metadata' snippets to enrich the terraform templates and how it reduces the TCO. We will also go over the additional advantages CMP provides on top of the TFE integration.

[GMerino]

Great post Ashok!

Do you have any shareable template using CMP variables in it?

Thanks 

[Piyush Dhoke2]

Hi Ashok,

I have done below steps but failing while creating catalog items (Not able to activate the catalog item). Please have a look below step by step implementation:

1. I am able to discover the .tf files from the terraform server.

2. Then created the catalog item & added required values

3. Then I have added cloud template from the related list of Catalog item above 7 activated it.

4. Still I am not seeing any Template version parameters, so I am not able to activate the catalog item.

Please suggest if I am missing anything in the configuration.

Thanks in Advance.

Piyush Dhoke

[Rebeca G_llego]

Hello, Ashok

Nice thread! Could you please share the slidedeck you shown in the screenshots?

Thank you!

[arielgritti]

Hi Ashok

Thanks for this wonderful article.

I have an issue with the credentials. The Discovery ends but don't see any results.

I'd created an API Token in the Terraform Cloud.

When I'd created the credentials use: "Bearer mabcdefgh<space><API Token generated from Terraform>"

It's right?

Thanks

Ariel

[arielgritti]

Hi

I solved it, with your help. The correct is Bearer<space><API Token generated from Terrradorm>

Thanks,

Ariel

[arielgritti]

Now I'm in the step:'Find Branches and Repositories' and obtain this error: Failed to retrieve repository and file details, please check the orchestration logs and flow logs for more details.

Any idea? What step I missed?

Thanks,

Ariel

[Chaytali]

Hi,

I have followed the steps till 'Discover TFE Organization' but Discover Now is long running and has been running without throwing any error.

Is there any additional steps required? Do we need to setup a MID Server?

 

[madhavabharadwa]

Hi,

Yes, you need MID Server.

Regards,

Madhava

[madhavabharadwa]

Hi,

I am getting error at "Catalog Item" steps. Trying to create a Catalog Item based on TFE Module for AWS S3. Is AWS support is available through TFE Connector? Could you please look at the screen shot and help?

Regards,

Madhava

[Ram Devanathan1]

did you get past this issue @arielgritti 

[Ram Devanathan1]

no - you have to follow steps listed in this blog article to work with AWS.

or alternately you can wait for the jan store release when we support AWS out of the box.

[arielgritti]

Hi @Ram Devanathan 

Happy New Year!

Not yet, I had to stop doing the tests for other activities and I have not resumed it yet 😞

Thanks,

Ariel

[madhavabharadwa]

Thank you Ram for the updates.

When is Jan store release is planned? How to apply store release updates to PDI?

[Chaytali]

Hi,

I am not able to activate the cloud catalog item even when the cloud template is active. Please find the below screenshots.

Since I am unable to activate cloud catalog item I can't see that in the active window.

[Chaytali]

Hi,

Where to find Discovery Logs in servicenow and also does the login credentials(user) requires any role to view discovery logs? We have added discovery_admin role to the login user but still unable to see discovery logs. 

[Sami Koskivaara]

Note that you need to create Terraform Cloud User token, not Terraform cloud organization token since it needs to access the runs! For GitHub you need to create personal access token.

Terraform cloud workspaces are automatically created based on the provisioning (so each provisioning is a new unique workspace) and the credentials (at least for Azure) are injected from the template as such:

main.tf

provider "azurerm" {
  features {}
  subscription_id = var.subscription_id
  client_id       = var.client_id
  client_secret   = var.client_secret
  tenant_id       = var.tenant_id
}

variables.tf

variable "subscription_id" {}
variable "client_id" {}
variable "client_secret" {}
variable "tenant_id" {}
variable "region" {}

ServiceNow then will "magically" inject the correct credentials and such to the template.

There is also a bug! You can't run two or more provisioning jobs at the same time from ServiceNow. Terraform Cloud and the target cloud are ok but the provisioning job in ServiceNow is stuck forever. Please Ashok can you take a look?

[Jagadeesh4]

I guess this magic (inject the correct credentials from ServiceNow into Terraform Enterprise Variables / Environment Variables) is not working. Refer the below error message

[1]Terraform v0.14.0

Configuring remote state backend...

Initializing Terraform configuration...

[31m

[1m[31mError: [0m[0m[1mInvoking Azure CLI failed with the following error: [0m

[0m[0m[0m

[31m

[1m[31mError: [0m[0m[1mError building AzureRM Client: obtain subscription() from Azure CLI: Error parsing json result from the Azure CLI: Error launching Azure CLI: exec: "az": executable file not found in $PATH[0m

[0m  on .terraform/modules/landingzone/providers.tf line 1, in provider "azurerm":

   1: provider "azurerm" [4m{[0m

[0m

[0m[0m

[Sami Koskivaara]

Not sure Jagadeesh what is your Terraform environment (not cloud?) but it seems the problem is that you don't have az cli installed:

Error launching Azure CLI: exec: "az": executable file not found in $PATH

Try to install it and put it to env vars to your path: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli

[Jagadeesh4]

I am using Terraform Enterprise and using CPG - Terraform 0.14 Support (https://developer.servicenow.com/connect.do#!/share/contents/5714057_cpg_terraform_014_supprot?v=1&t...)

As per my understanding, ServiceNow is wrapping up this Azure credentials through azurerm provider, however this details has to be passed through mid-server script named “TerraformEnterpriseVariable” from ServiceNow cloud service account. This is not happening in my point of view, given that we are passing it through Terraform template.

Any further insights would be appreciated.

Regards,

Jagadeesh

[BhupeshG]

Hi

One of my TF expert asked this question

, there are multiple factors to check for compatibility.

1. Terraform enterprise version  
2. Terraform version
3. Terraform Provider version

All 3 are logically different.

you need to check with vendor for all 3 above compatibility with the plugin version you are using.

Since the snow tool and plugin is owned by you, I expect that you will check these details and confirm whether the plugin will support my version or not.

Terraform Enterprise - v202106-1 ReleaseSequence  544”. And terraform version( for coding & modules) which we would be leveraging is “Terraform 1.0.1”

As per below doc link tf version mentioned - is it tfe version or tfe provider version ??

https://docs.servicenow.com/bundle/quebec-it-operations-management/page/product/cloud-management-v2/...

https://registry.terraform.io/providers/hashicorp/tfe/latest/docs

as per above link----Since v0.24.0 tfe provider version, this provider requires Terraform >= 0.12.

Also for the VCS, can we Azure devops or only git hub is supported??

[Ram Devanathan1]

believe i answered most question in the other post.

Terraform version as in our sample version.tf -

terraform {
  required_version = ">= 0.12"
}

tfe provider version as in our sample providers.tf -

provider "azurerm" {
  subscription_id = var.subscriptionId
  client_id       = var.clientId
  client_secret   = var.clientSecret
  tenant_id       = var.tenantId
  #features {}
  version = "=1.44.0"
}

terraform enterprise version does not matter as long as terraform version 0.12.xx is supported

HTH.

Ram

[BhupeshG]

Thanks Ram

what is the difference between Terraform Version(version.tf) and Terraform Enterprise version ?

I have also have HI case but not yet got the appropriate answer.

[Madhava Bharad1]

@Ram Devanathan Hi Ram, In last engagement where we implemented CPG with Terraform connector. We reported a issue where ServiceNow was not able to process Terraform template with Map variable, only String variables were supported. Is that fixed now? what variable types are supported now? Is below Terraform variable types supported? I mean, ServiceNow can parse the TF templates with these variable types? Please let me know.

• list(<TYPE>)
• set(<TYPE>)
• map(<TYPE>)
• object({<ATTR NAME> = <TYPE>, ... })
• tuple([<TYPE>, ...])

[Ram Devanathan1]

yes support for complex vars, along with Terraform 1.x support  is coming with the next release due this quarter.

we are opening for design-partner (beta) program now, reach out offline to me at ramkumar<dot>devanathan<at>servicenow<dot>com if you are interested.