Provisioning an AKS Cluster in Azure using ARM

Ashok Madhavan1 · ‎08-04-2019

This article talks about provisioning an AKS Cluster using ARM template. It uses the Cloud Catalog Item with template ingestion.

Background: (this is the most important. As long as we get this, all other things are straight forward)

For provisioning an AKS cluster, Azure needs a service principal client and a client secret. Without these we cannot provision an AKS cluster. There are two popular ways to do this:

Use Azure CLI/Powershell to create principal, secret and then the AKS cluster.
Create a service principal client & secret up front and store the secret info in Azure Vault. Use the vault in the ARM template and provision the cluster. This is a nested ARM execution.

We will be using the second option in our article. It is a nested ARM execution. There would be a source ARM template which will do the provisioning of the cluster. A wrapper ARM template will get the value of the vault and shoot it to the source ARM template. This way of an ARM template calling another ARM template is called a nested ARM execution and is popular in the field and used by practitioners.

The source template is here : https://github.com/aashok-madhavan/armtemplates/blob/master/source_aks_cluster.json (this is pretty much the one from the azure-quickstart-templates provided by Microsoft)

The wrapper template is here : https://github.com/aashok-madhavan/armtemplates/blob/master/wrapper_aks_cluster.json

Source Template

If you look at the parameters of the source ARM template, you would see the follwoing:

"parameters": {
        "clusterName": {
            "type": "string"
        },
        "location": {
            "type": "string"
        },
        "dnsPrefix": {
            "type": "string"
        },
        "osDiskSizeGB": {
            "type": "int",
            "defaultValue": 032
        },
        "agentCount": {
            "type": "int"
        },
        "agentVMSize": {
            "type": "string",
            "defaultValue": "Standard_DS2_v2"
        },
        "linuxAdminUsername": {
            "type": "string"
        },
        "sshRSAPublicKey": {
            "type": "string"
        },
        "servicePrincipalClientId": {
            "metadata": {
                "description": "Client ID (used by cloudprovider)"
            },
            "type": "securestring"
        },
        "servicePrincipalClientSecret": {
            "type": "securestring"
        },
        "osType": {
            "type": "string",
            "defaultValue": "Linux"
        },
        "kubernetesVersion": {
            "type": "string",
            "defaultValue": "1.13.5"
        }
    }

As you see, there are parameters for servicePrincipalClient and servicePrincipalClientSecret. Both of them are taken as secure strings.

------------------------------------------------------------------------------------

"servicePrincipalClientId": {
"metadata": {
      "description": "Client ID (used by cloudprovider)"
  },
  "type": "securestring"
},
"servicePrincipalClientSecret": {
  "type": "securestring"
},

------------------------------------------------------------------------------------

In addition to these two, we will need to send values for all the parameters.

Wrapper Template

Now let us Look at the wrapper template. You will see that it takes in a set of parameters. These parameters match to those of the source template. In addition it asks for the vault information as well.

"parameters": {
        "clusterName": {
            "type": "string"
        },
        ....
        ....
        "vaultSubscription": {
            "metadata": {
                "description": "The subscription to which this vault belongs to"
            },
            "type": "string"
        },
        "vaultResourceGroupName": {
            "metadata": {
                "description": "The resource group to which this vault belongs to"
            },
            "type": "string"
        },
        "vaultName": {
            "metadata": {
                "description": "The resource group to which this vault belongs to"
            },
            "type": "string"
        },

        "servicePrincipalClientId": {
            "type": "string"
        },
        "servicePrincipalClientSecret": {
            "type": "string"
        },
        ....
        ....
    }

The vaultSubscription, vaultResourceGroupName, vaultName are the parameters that are extra. They are used to arrive at the proper vault secret so that the AKS cluster can be provisioned.

Now let us look at the resource section of the wrapper template where the real delegation/nesting happens.

"resources": [
        {
            "apiVersion": "2018-05-01",
            "name": "dynamicSecret",
            "type": "Microsoft.Resources/deployments",
            "properties": {
                "mode": "Incremental",
                "templateLink": {
                    "contentVersion": "1.0.0.1",
                    "uri": "https://raw.githubusercontent.com/aashok-madhavan/armtemplates/master/source_aks_cluster.json"
                },
                "parameters": {
                    "clusterName": {
                        "value": "[parameters('clusterName')]"
                    },
                    "location": {
                        "value": "[parameters('location')]"
                    },
                    "dnsPrefix": {
                        "value": "[parameters('dnsPrefix')]"
                    },
                    "osDiskSizeGB": {
                        "value": "[parameters('osDiskSizeGB')]"
                    },
                    "agentCount": {
                        "value": "[parameters('agentCount')]"
                    },
                    "agentVMSize": {
                        "value": "[parameters('agentVMSize')]"
                    },
                    "linuxAdminUsername": {
                        "value": "[parameters('linuxAdminUsername')]"
                    },
                    "sshRSAPublicKey": {
                        "value": "[parameters('sshRSAPublicKey')]"
                    },
                    "servicePrincipalClientId": {
                        "value": "[parameters('servicePrincipalClientId')]"
                    },
                    "servicePrincipalClientSecret": {
                        "reference": {
                            "keyVault": {
                                "id": "[resourceId(parameters('vaultSubscription'), parameters('vaultResourceGroupName'), 'Microsoft.KeyVault/vaults', parameters('vaultName'))]"
                            },
                            "secretName": "[parameters('servicePrincipalClientSecret')]"
                        }
                    },
                    "osType": {
                        "value": "[parameters('osType')]"
                    },
                    "kubernetesVersion": {
                        "value": "[parameters('kubernetesVersion')]"
                    }
                }
            }
        }
    ]

You will see that in the resource section, it uses a type to imply that it is a deployment ==> "type": "Microsoft.Resources/deployments".

And the template is a link to another template ==>

"templateLink": {
"contentVersion": "1.0.0.1",
"uri": "https://raw.githubusercontent.com/aashok-madhavan/armtemplates/master/source_aks_cluster.json"
}

This basically tells us that this wrapper ARM template calls/delegates it to the proper source AKS cluster ARM template.

Next, let us look at how the parameters are passed from the wrapper to the source template.

The wrapper template acts as expected as a wrapper and does one more important thing. It does the job of reading the serviePrincipalClientSecret from the given Azure Vault secret.

"servicePrincipalClientSecret": {
"reference": {
      "keyVault": {
          "id": "[resourceId(parameters('vaultSubscription'), parameters('vaultResourceGroupName'), 'Microsoft.KeyVault/vaults', parameters('vaultName'))]"
},
          "secretName": "[parameters('servicePrincipalClientSecret')]"
      }
}

By doing this, there is no need for ServiceNow CMP to know what the real value of the secret is and makes sure there is no security issues.

Cloud Catalog Item

To provide Azure AKS as a self service item, we need to do the following in the cloud management portal.

Get to the Cloud Admin Portal ==> Design ==> Cloud Catalog Items.

Create a new catalog item and choose 'Cloud Template' and 'ARM Template' as the template type.

Get to the 'Cloud Templates' sub-tab of the same catalog item. Click New so that we can upload/reference the wrapper ARM template.

Then provide the wrapper ARM template. It can be an url or a file or content.

The system automatically figures out all the parameters of the template.

Populate the catalog item with the parameters and the needed catalog client scripts by clicking the 'Activate' button.

Make the catalog item to show up in the portal by checking the 'Active' box and saving.

Thatz it. you are now ready to order a brand new Azure AKS cluster.

Azure Setup

Vault Settings

App Registration

Cloud Catalog Item Configuration

To make the ordering experience better and to make the administration better, we might want to make some changes to the catalog item. In the beginning we hinted about asking the questions about vault Subscription ID, vault Resource Group, vault Name and the secret name. These are typically done by the admin and is pre-created. Then this could be

a) set as default in the catalog item variables and hidden from end user or

b) Use a policy to set these variables dynamically based on some conditions.

In this sample, we will set the default value for the vault Subscription, Resource Group, vault name. In a different article, we can show a policy way of setting this.

If we do the above two, it then becomes way more simpler and easier to order a kubernetes cluster in Azure.

anilkumarveeram · ‎02-04-2020

very nice Ashok, I have done this an year back but I have not used keyvault for secrets , I did with some lookup from the table.

ServiceNow Community servicenow community

Provisioning an AKS Cluster in Azure using ARM