Provisioning S3 with Replication

[Ashok Madhavan1]

Provisioning S3 with Replication

S3 can be deployed with replication across regions. This article talks about how to achieve this using ServiceNow CMP.

This article will use the Pre-Provision capabilities of CMP in ServiceNow.

Things to know about this: 

• A single CFT can provision S3 buckets in the same region and not across. This implies that we will need to execute two different S3 buckets in two different regions. 
• CMP supports ingesting templates like CFT, ARM, GDM, Terraform and create user offerings in a matter of minutes. This article will explain how we ingest the concerned CFTs and achieve our goal.
•  There are two CFTs invloved. One for creating the backup S3 bucket. The other now to create the primary one. In the primary CFT will also have IAM Roles defined so that the primary S3 changes can be replicated to the backup bucket.

Pre-Requisities:

The pre-requisite update-set for making sure the Cloud API input parameters are visible is here :https://developer.servicenow.com/app.do#!/share/contents/4708895_cloud_api_stack_orchestration_creat... Please apply this update set before you start. 

We are going to do the following:

• Step 1 - Create Cloud Script Template for the Backup S3 bucket template
• Step 2 - Create a Template Cloud Catalog Item for the Primary S3 Bucket template
• Step 3 - Setup a pre-provisioning operation on the Template Cloud Catalog Item to provision the Backup S3 bucket template
• Step 4 - Configure the Template Cloud Catalog Item to collect and populate all the necessary parameters

The backup CFT is here : https://raw.githubusercontent.com/aashok-madhavan/CFTs/master/BackupS3.yaml . This file is also attached to this article.

The primary CFT is here: https://raw.githubusercontent.com/aashok-madhavan/CFTs/master/PrimaryS3WithReplication.yaml . This file is also attached to this article.

The interaction between these two CFTs is as follows:

• Backup S3 CFT will provision a S3 bucket in region 1 with a name like : '$common_name - region 1'. For example, myBucket - us-west-2.
• The primary S3 CFT will create the source bucket with name '$common_name - region 2'. For example, myBackupBucket - us-west-1. This primary template expects the name of the backup bucket to be myBucket - us-west-2. 
• The catalog item will take in a common name and create two different buckets and create replication from primary to backup S3 buckets.

Any template execution in CMP follows the following order:

 

PreProvision: This is the operation where you could do up-front activities. Like making a call to an IPAM system to get the IP info so that the same can be sent to Provision operation as inputs. In many cases this could be a no-op. In our case, we will make the call to execute the backup S3 in this operation.

Provision: This is the operation where the main template (CFT in this case) will be executed. The system will automatically populate the CMDB post the provision operation.

PostProvision: This is the operation where you would do any tasks post the successful provision. These could be things like hardening the VMs that were provisioned by the CFT or register the provisioned resources on some system etc. In our use case this will be a no-op. 

The steps for this use case would be :

With the background set, let us jump into doing it.

Step 1:Create Cloud Script Template for the Backup S3 bucket template

Let us create the Backup S3 CFT in the 'Cloud Script Templates' table. You can get to that from the cloud admin portal --> Design --> Cloud Scripts --> Cloud Script Templates.

Create the template. The name of the 'Cloud Scripts Template' table is 'sn_cfg_cloud_script_template'. The table name and the sys_id will be later needed in the PreProvision operation for accessing the content of the template.

Step 2: Create a Template Cloud Catalog Item for the Primary S3 Bucket template

Have the CFT ready (file or url). Get to the 'Cloud Catalog Items' screen and create a new catalog item.

Give it a proper name. Choose the 'source' as 'Cloud Template' and 'Template Type' as 'CloudFormation Template' and save it.

Then associate the CFT to it by clicking the 'New' button in the 'Cloud Templates' tab in the bottom part of the screen. 

Upload/Copy Paste/Provide URL for the primary template and save the record. System will ingest the CFT and figure out the parameters that are needed for this template.

Then click on the 'Activate' button on the template version record.

 

This will make sure the catalog item gets populated properly with the variables and do all the needed plumbing needed like creating an empty Pre/Post Provision operations etc.

Step 3:Setup a pre-provisioning op to provision the Backup S3 Bucket Template

Now we are at a state where this primary CFT can be provisioned. But for this CFT to work properly, a backup S3 bucket needs to be created before. For this, we will use the PreProvision operation to do the needful.

Get to the Resource Blocks from cloud admin portal --> Design --> Resource Blocks. Choose the Resource Block that has the same name as that of your catalog item. If your catalog item name was 'S3 with Replication', look for a resource block 'S3 with Replication'. Then click on that card.

To make edits to the Resource Block, un-publish the block by clicking the 'Published' toggle button on the right.

We need to get to the 'PreProvision' operation. For that you would need to click on the 'Operations' tab first. Then choose the 'PreProvision' operation on the Operation pulldown.

After you choose the 'PreProvision' operation, you will see that there are 3 default input parameters created for you. All Day-2 operations need them and you can keep it as is. You would also see that the 'Steps' tab is empty. We will shortly add the step to provision the backup S3 CFT.

We need to get the Location for the backup S3 template. Let us create a new parameter to gather this and let us call it 'BackupLocation'.

Create the parameter and save it.

Similarly, add a parameter for the backup S3 as well. We will call it 'Name'.

These parameters (Name, BackupLocation) are now available to be used in the catalog item.

Now get back to your catalog item. Click on the 'PreProvision Operation' tab. Then click on 'New' button.

A form will show up and enter the information to execute a Cloud API to execute a CFT.

• For Step Type, use 'CAPI'. This is the default in the pulldown.
• For CAPI Provider, use 'AWS CloudFormation'. This is the existing provider for CFT execution.
• For CAPI Interface, use 'Stack Orchestration Interface'. This is existing and has methods for executing CFT based operations.
• For CAPI Method, use 'CreateStack'. This is the method that executes a CFT and creates a cloudformation stack in AWS.
• For CAPI Version, use '1.0'.

Then save.

Now we need to map the inputs to this call. Click on the 'Key Values' tab (this is the first tab and will be by default visible).

• Set the 'orchestrationTemplate' to '$(ci.sn_cfg_cloud_script_template[sys_id_here].content)'. This expression will get the content of the backup template and push it into the Cloud API call. You would need to get the sys_id from the table. 
• Set the providerAPIParameters with proper StackName mapping '{"StackName":"${parameter.StackName}"}'.
• Set the templateParameters to '{"Name":"${parameter.Name}-${parameter.BackupLocation}"}'. This mapping is important as the name of the primary S3 bucket CFT expects the name to be the name+region. So we map it to the name +backup location. 

 

The mapping part is all done now. 

Step 4: Decorate Catalog Item

Decoration for BackupLocation

We need to decorate the catalog item so that end user can choose values instead of entering. The BackupLocation variable we would want it to be a context driven pulldown than a user entry text box. We will associate the BackupLocation to a resource pool so that it shows only those valid backup regions.

Get to the catalog item you have created. Click on the 'Manage Attributes' as we need to expose the 'BackupLocation' as a user input.

 

We need to expose BackupLocation as a user entry field and decorate it. For that, choose the 'Blueprint Container Resource.Provision' operation. Set the 'Generate Variable' of the BackupLocation to be true. This will create a new catalog item variable for the 'BackupLocation'.

Click New and create the attribute for BackupLocation

Then make the 'Generate Variable' as true for the BackupLocation.

Now click on the 'Variable Sets' tab --> 'General Info'. Then click the 'BackupLocation' variable.

Choose 'Lookup Select Box' as the type. Check the 'Use Pool Filter' and use the 'Location Pool' and 'GetLocationsForCloudAccount' filter. Then save it. This pool and filter is not present out of the box. Import and commit this update-set and then you would get this pool filter. If you are going to use the consolidated update-set that is given in the end of this article, you do not need this. The link is here : https://developer.servicenow.com/app.do#!/share/contents/4708895_custom_location_resource_pool?t=PRO.... This pool filter will make sure that proper locations shows up for the end user to choose than type. We would want the BackupLocation to show up right after the Location variable. To do that set the 'order' to '40'.

We also want to set the Destination Region of the primary CFT to be based on the BackupLocation. We will setup a form rule on the BackupLocation. This rule will fire when the BackupLocation changes and will set the value of the destination region as per the BackupLocation.

Under the Rules Tab, click New.

Give it a meaningful name. I am using 'Set Destination Region Rule'. Add an action under that region that would call a script include that return the region of the destination from the BackupLocation's value. The value should be set to '$(Script:sn_cmp.CMPScriptHelper.getRegionFromAssociatedDatacenter[arg=${parameter.BackupLocation}])'

To make it even better, we can add an action to make the destination region as read only.

Decoration for IAM Capability to the CFT execution

The primary S3 CFT does IAM role activities and for this we need to send the appropriate capabilities. If a CFT is doing any IAM activities, we need to send IAM capability information. For this, we need to map the following

Set 'AdditionalAPIParameters' to '{"Capabilities.member.1":"CAPABILITY_IAM"}'

This step is ultra-critical as without this, the primary CFT execution will fail. 

Get to the 'Manage Attributes' and then to the 'Blueprint Container Resource.Provision'. You would see that there is an attribute called 'AdditionalAPIParameters'. You would see the 'Generate Variable' is set to false. Double click on that and set it to 'true'.

Then click the 'Apply Changes' button.

Now click on the 'Variable Sets' tab. Choose the 'General Info' variable set. Then click the 'AdditionalAPIParameters' variable.

Click on the 'Default Value' tab and set the default value. Set it to '{"Capabilities.member.1":"CAPABILITY_IAM"}'.

The end user does not need to see this. You can un-set the 'Visible' flag in the Availability tab and then save it.

With this, we are ready for the end user to order a S3 bucket with replication.

Step 5: Consumption

Get to the end user portal (https://YOUR_SERVER_HERE.service-now.com/cloud_portal) and click  'Launch a Stack'. Then choose your catalog item.

The General Info tab will show like this:

The BackupLocation loads based on the choice of Cloud Account. Also when the BackupLocation value is chosen, the destination region in the Provision tab is set appropriately and made read-only also.

The end user needs to give in proper StackName, Location, BackupLocation and the S3 name. Then when they submit, the system will create the backup bucket first and then the primary bucket and then the whole 9 yards. 

You can then click the 'View stack details' link and see the provisioned resources.

You can decommission the S3 bucket by clicking the 'Deprovision' operation of the stack. 

The update-set for a sample S3 with Replication catalog item can be found here: https://developer.servicenow.com/app.do#!/share/contents/4708895_aws_s3_bucket_with_replication?v=1....

[Ram Devanathan1]

Awesome article, ton of information in there. Thanks Ashok.

[Ashutosh Munot1]

Fantastic.

[Ashok Madhavan1]

thank you

[Ashok Madhavan1]

Had missed to include an update-set for the location pool. Edited the article to include it.

[Ashok Madhavan1]

thank you Ram.