New York and Scoped Applications - some new features

[coryseering]

New York and Scoped Applications - some new features

Deletes

When ServiceNow introduced Scoped Applications (waaaaaay back in Fuji) we wanted the package to always represent the complete and current state of the app. To this end, the Publish mechanism (whether you publish to the private repo for your company, or to the Store, or even to an Update Set) will do a fresh query against all of the metadata tables to serialize your app's records for unloading. Your published app is always a reflection of what the app looks like now, and fresh installs of your application should look the same on any target instance that installs it.

Part of the nature of the platform is that any record that exists in your instance continues to exist, through platform upgrades and plugin installs and app updates- unless there is a specific command to delete that record. This can be done through the use of Fix Scripts or by loading a new copy of the record, but with a specific "action" attribute to tell the system you want to delete it.

These two things combined actually make for one of the features of Scoped Applications that people don't like. Scoped app developers want their installed app versions to look like (to have the same files) as when they clicked Publish for that version on their development system. If the develop version 1 of their app, then delete a Business Rule and a Script Include and replace it with an Event + Script Action for version 2 of their app, then they want the Business Rule and the Script Include to be deleted when their instances upgrade from version 1 of the app to version 2. However, since we don't capture that "delete" action as a Fix Script or as a copy of the record with a Delete action (because the records no longer exist), you lose that history. In this case, user's instances which had version 1 installed will continue to have the Business Rule and the Script Include, even though version 2 of the application doesn't ship them anymore. This makes for an interesting comparison with instances that installed version 2 and never had version 1 at all. In those instances, that Business Rule and Script Include would not be present.

The strategy for managing this has been to modify the records instead of deleting them. As an application developer, if you want to stop using a Business Rule, set the "active" flag on the rule to false. If you want to stop using a Script Include, just stop calling it in your other active code. This is a bit unsatisfactory, since it means that you continue to ship things that you don't want to have to care about anymore. Your app can become cluttered over time. You are, in essence, responsible for maintaining the "history" of your application with every release.

For the New York release, we introduced a new concept to Scoped Applications, called the author_elective_update folder. When your application is packaged for the repo or for committing to Source Control, we will unload additional things into this folder that we would previous have ignored. The author_elective_update folder is where we will unload Deletes. When your application is installed on your own instances, we will automatically load these Deletes from the author_elective_update folder. For our example above, this would result in the Business Rule and the Script Include being deleted from the system when the application is upgraded from version 1 to version 2. Additionally, if you make changes to the schema of your application by deleting a table or a column, that delete will be tracked in author_elective_update but will not be applied by default.

If your application comes from a vendor and they have Deletes in some new version of the application, we will not automatically apply them when you upgrade the application. You have the option of applying the Deletes manually post-upgrade just like you can do when the instance itself is upgraded. We will record an Upgrade History for this application, and write Skips into the log for each application upgrade. You can go through those Skips and choose to apply them (such as accepting a Delete for a Script Include or a Business Rule) or leave them be.

One other type of metadata that will also be put into author_elective_update are choice sets. We will now be putting choice sets for your app's tables into the author_elective_update folder. We do this so that you can modify your choice lists- for instance, by removing one option and adding another- and seeing the option actually get removed when your app is updated on your production instance. Previously, in this scenario we would only have added the new option, leaving the option you wanted to delete there too.

These changes mean that updates to your applications should work more in line with how Update Set updates work. Since app updates do not give you an interface to preview changes yet, we have built-in some properties that you can use to control how updates from author_elective_updates are treated. We have four levels of control currently:

- To turn off the feature completely, set the system property com.glide.apps.include_my_deletes to false. This will prevent the system from loading anything out of the author_elective_update folder. This will also prevent records being added to the author_elective_update folder, if you have in-development applications on this instance which you intent to Publish.

- To only save and load Choice sets, set the system property com.glide.apps.include_only_sys_choice to true.

- To allow automatically apply schema drops (automatically delete tables or columns which no longer exist in your application), set the system property com.glide.apps.include_my_schema to true.

- To decide on a case-by-case basis, set com.glide.apps.force_skips to true.

Property Name	Behavior	Version Introduced	Used in Source Control	Default	Additional Information
com.glide.apps.include_only_sys_choice	Loads only deletes and updates to sys_choice fields from author_elective_update	New York Patch 1	False	False
com.glide.apps.include_my_schema	Loads only deletes to schema files from author_elective_update	New York Patch 1	False	False	In Orlando Patch 4+ this property will default to true for new instances Instances upgrading to Paris will have this property default to true if no company applications are on the instance at upgrade time.
com.glide.apps.force_skips	Skips all deletes in author_elective_update allows user to choose to load each on a case by case basis in Upgrade History.	New York Patch 4	False	False
com.glide.apps.include_my_deletes	Automatically loads deletes for all record types included in author_elective_update	New York	True	True

Scenarios

Consider the following scenarios:

1. You are building a new application to put on the Store, and storing your work in Source Control

2. You are a vendor and already have versions of your application on the Store, and you are providing a new version

3. You have an existing application that you distribute to your company's instances using the repo, and you are providing a new version

4. You work on the same application across multiple instances, and you sync to the same Source Control branch(es)

New Vendor-Application Scenario

Since you are building a new application, there really isn't much that has changed for you. During the course of building your application, you may end up creating and deleting multiple kinds of metadata. As you commit your changes to Source Control, you will find that the author_elective_update folder starts to accumulate some records. You can turn this off by setting the system property com.glide.apps.include_my_deletes to false. On your next commit, you will find that the author_elective_update folder no longer exists. If you choose not to turn it off then the folder will be present when the application is Published to the store. When your customer installs it for the first time, and if they have not set com.glide.apps.include_my_deletes to false, the Deletes will be loaded into their system but should have no effect since the original files the Deletes represent don't exist on their system.

Existing Vendor-Application Scenario

If you have an existing application on the repo and you decide to develop new features for it, we will record the Deletes for your application into the author_elective_update folder. The actual contents of this folder will vary based upon the history of actions taken on this development instance.

If you are working on this application for the original instance that the application was developed on, and you haven't cloned-over or otherwise deleted the application (or its records in sys_metadata_delete) then the history of all of your deletes will be included. If you have cloned the application, or you've performed Source Control operations that uninstall and reinstall the application (such as switching branches) then the history of Deletes for this application has probably already been removed. We will track new Deletes in the author_elective_update folder, but we cannot retroactively create Deletes for metadata that no longer exists.

When you publish the new version of your app, the Deletes that have been recorded on the instance are included. If you want to know what those are, you can browse the author_elective_update folder in your git repository, or look at the sys_metadata_delete table and filter on your application. From the time you upgrade to New York (and start committing to your repo) your Deletes will be stored with your app as you commit to Source Control or Publish to the Store.

When your customers upgrade to this new version of your application, the Deletes in your author_elective_update folder will be written as Skips in the Upgrade History entry for this application upgrade. Your users can browse the Skip list, and apply the changes on a case-by-case basis. Your customers have the option of setting com.glide.apps.include_my_deletes to false, which will skip writing Skip records as well (emulating the behavior of Madrid-and-earlier instances, which don't have a concept of author_elective_update at all).

Existing Application For Your Company Instances Scenario

As in the above scenario, if you have an existing application on the repo and you decide to develop new features for it, we will record the Deletes for your application into the author_elective_update folder. The actual contents of this folder will vary based upon the history of actions taken on this development instance. The only difference between this scenario and the one above is what happens when the instance gets an upgrade for the app from the repo (not from Source Control).

When your instances upgrade to this new version of your application, the Deletes in your application will be applied based upon the preferences below. You have 4 options:

1. The default setting: Accept any Deletes except schema changes (Skip deleting columns or tables)

2. Accept only Choice updates, and Skip the rest by setting com.glide.apps.include_only_sys_choice to true

3. Accept all Deletes, including schema deletes, by setting com.glide.apps.include_my_schema to true

4. Don't accept any Deletes, but write out Skips so you can apply them on a case-by-case basis by setting com.glide.apps.force_skips to true

5. Don't accept any Deletes and skip writing Skips (thus emulating the behavior of Madrid-and-earlier instances) by setting com.glide.apps.include_my_deletes to false.

Working On An Application Across Multiple Instances With Source Control Scenario

In this scenario, you are working on an application across multiple instances, and keeping the application in-sync across instances by using the same branch during development. In this scenario, your Deletes are captured as you work on (and commit) your changes. All changes in the author_elective_update folder will be pulled into the instance when you Apply Remote Changes in Source Control, or when you switch to a branch which contains author_elective_update. You can accept these changes, or you can set com.glide.apps.include_my_deletes to false and when you perform a Source Control operation that would have loaded them, we will instead write our Skip records in an Upgrade History entry.

TL;DR (Too Long, Didn't read)

In short, you have control over how the author_elective_update folder is handled. You can turn it off totally by setting com.glide.apps.include_my_deletes to false. You can set the specific level of records you want to apply (everything, everything except schema changes, or only Choice Sets). You can also make sure that the apps you develop will ship Deletes in the author_elective_update folder, but you skip all incoming changes from any vendor app's author_elective_update folder by setting com.glide.apps.disable_elective_skip_history to true and also setting com.glide.apps.force_skips to true.

I know the settings are complex, but so are the scenarios this feature should handle. You have different levels of trust between applications you develop within your own company and applications you get from vendors via the Store; you need applications imported via Source Control to load everything possible as the source of truth.

Glossary of Terms

- Metadata are the records that define an application. You may also see it called file, application file, sys_metadata or app record. It includes record types like Business rules, Script includes, Table definitions, Client scripts, and so on. This is distinct from data which are things applications read, write, update and delete (like Incidents, Change Requests, Problems, Projects, Users, Locations, and so on).

- Package is the term used to describe the the collection of records that make up the application, specifically as it applies to installing or updating that application on instances other than where it was originally developed. Typically, this means the package built and stored in the repo, for distribution through the Store or the "My Applications" page, but also loosely used to describe the contents of a branch in a git repository when the application is connected to Source Control, or to an Update Set when the app has specifically been Published to an Update Set.

- Repo has a couple of different uses, depending on context.

- With regards to Publishing applications, the repo is a storage mechanism provided by ServiceNow and made available to all of the instances that belong to a given company. All Applications published to this repo can be installed on all of that company's instances, but those applications are not made available to other company's instances.

- When used with regards to the Source Control integration for scoped apps, a repo is any repository which you have linked the app's source code to. This is typically a Github or Gitlabs repository, within which you can store your app's source code, and which you can push changes to or import changes from, using the controls within Studio.

- Store is where applications can be put up for sale to ServiceNow customers. A scoped app can be published to the Store for other customers to purchase, and you can see the store at https://store.servicenow.com

- Update sets are one of the ways that changes to records in ServiceNow can be tracked. When Publishing to an Update set, a new set is created and every file in your application will be copied here.

- Serialize means taking the record from the database and writing it out as XML. This occurs whenever an update is written to an Update set, or when you right-click a record header and choose Export -> XML, or when you commit changes via the Source Control integration. Serialized data may include more than one record in the output (such as when a record's attachments are included in the output, or when a Form layout, List layout, or Choice set is exported).

- Deletes are a way of tracking when a specific application file has been deleted from a system. They are recorded as records in the table sys_metadata_delete.

- Unloading is the act of serializing a record and storing the result- either in an Update Set, in a Source Control commit, or into an application package being put on the repo or Store.

- Fix script is a way of running code after an upgrade, a plugin update, or an app update.

- Schema is a term that encompasses your tables and columns. Dropping (deleting) a table or an individual column is considered a schema change.

- Upgrade History is a module in ServiceNow that lets you see the history of upgrades to your platform, as well as individual plugin activation histories, and individual application upgrades.

- Module is an entry in the Navigator list (also referred to as the "left-hand nav") of the ServiceNow interface

- Choice sets are records in ServiceNow that contain multiple individual choices for a single table's choice field.

- Skips are records in the sys_upgrade_history_log table which are related to an Upgrade History entry, and detail whether a vendor-supplied (or ServiceNow-supplied) file was not loaded during an upgrade, but can be applied manually.