Using scripted search sources to search external websites and applications.

[jesseadams]

In my previous post, I introduced you to Search Sources in Service Portal. Search Sources are new to Istanbul in the Service Portal application. We talked about the components that make a search source work and walked through the basics of configuring a search source for a portal. In this post we're going to look at scripted search sources and learn how to use them to search an external application or website from inside your portal. This is a very powerful feature because it can allow you to create a single portal. Users can visit the portal to look for answers from any resource used by your organization that you have access to via a REST api.

Script Search Sources to search external websites and apps

To enable Search Sources to crawl external sites, you'll need to have read part 1 so that you're up to speed on the basics. You'll also need some external resource that you have access to via REST, basic knowledge of AngularJS, and an understanding of creating pages and widgets in Service Portal.

Now, we're going to be doing some custom scripting here so it is important to note that this is not a solution that is officially supported by ServiceNow. My goal with this post is to provide an example to help you get started in creating your own search sources and to share some of the things I learned while configuring one of these in my own instance. This is not intended to be a solution to all your searching needs, just a springboard into your quest to find the solution to all of your searching needs.

I repeat, this customization is not officially supported by ServiceNow

There are a couple of questions we need to ask before we begin:

1. Do we want to link to the external content in its native application/site? Or do we want to bring the content in and display it in ServiceNow?
2. Where are we pulling the data from?

For our example, we're going to search the knowledge base from another ServiceNow instance and display it inside our portal as if it were native content.

The first steps here are really the same as what we did in part 1. The big differences between what we did in part 1 and what we are doing in part 2, is the search page template, the data fetch script, and the page and widget that will display our external content.

Obtain the data that the search source will crawl

Let's start with the data fetch script. Once you click the "Is scripted source" checkbox on the search source form this field is displayed. This is where you'll setup the search request for your external site. For this example, we're going to use the ServiceNow table api. This part will vary greatly depending on the application you're querying.

We'll start from this basic template and add in the necessary fields as we go:

(function(query) {
   var results = [];
   /* Calculate your results here. */
   return results;
})(query);

This defines the search function that is executed by the instance. In order for this to work we'll need the URL for our request endpoint, and a sn_ws.RESTMessage object. I'm using a recordless REST message but this could be modified to use a pre-configured outbound REST message record if needed. Setting up the request should look like this:

var url= "https://myInstance.service-now.com/api/now/table/kb_knowledge?sysparm_query=GOTO123TEXTQUERY321%3D" + encodeURI(query) + "&sysparm_fields=sys_id%2Cnumber%2Cshort_description%2Ccategory%2Ctext";
var ws = new sn_ws.RESTMessageV2();
     ws.setBasicAuth("search_user", "search");
   ws.setHttpMethod("get");
   ws.setEndpoint(url);
       var jsonOutput = ws.execute();

Here we're using basic authentication for our call. We're passing in the username and password for a user on that instance in order to gain access to the knowledge records. There are probably better and more secure ways to handle the authentication but I'm not going to get in the weeds on that. I'm really trying to keep things simple here that we can focus on the basics of how to set this up. If you want to try a different approach for authentication, you might start here: Create a basic auth profile..

Once we have our request setup we need to define how we're going to handle the response data. I'm specifying the fields to be returned in my query like this:

"&sysparm_fields=sys_id%2Cnumber%2Cshort_description%2Ccategory%2Ctext";

Next, we need to decode the JSON object we'll get in response. Once we decode the JSON object, then we iterate through each result and set some fields

1. result.url — if you were linking to the external site rather than opening the record in ServiceNow this would be where the link leads to.
2. Result.target — this is the target for the link. Here we're using _blank to open in a new window
3. Result.primary — this sets what our primary display field will be.

The code to do that will look like this:

if (jsonOutput) {
var response = new JSON().decode(jsonOutput.getBody());
results = response.result;
results.forEach(function(result) {
           result.url = result.svn_url;
           result.target = "_blank";
           result.primary = result.short_description;
       });
}

Here is the finished product for the data source:

Displaying the search results:

Next, we'll setup the search page template. We really only need to tweak this a little bit in order to open the search results in a page that is designed to handle our external content. In fact, all we really need to do it modify the link. The default template link looks like this:

<a href="?id=form&sys_id={{item.sys_id}}&table={{item.table}}" class="h4 text-primary m-b-sm block">

We just need to set this up to open a different portal page, and pass in whatever parameters that page will need to get and display the article. We're going to call this page ext_knowledge and pass a parameter "record" which should be the sys_id of our external article. We'll also add a target from the variable we set in our results object earlier.

<a href="?id=ext_knowledge&record={{item.sys_id}}" target="{{item.target}}" class="h4 text-primary m-b-sm block">

Again, you could choose the simpler path here and just make this link to the external application or site this item came from but I find the idea of displaying this content as if it is native to our instance to be interesting and that might provide a better user experience so we'll go that route here. So, the final product for the search source record looks like this:

Displaying the searched content on the Service Portal

So far things have been pretty straightforward. You need an API to search some external site. You need to decide what data to bring over and setup the data fetch script and you need to make a small modification to the search page template. Displaying the content in my portal is where I really ran into a couple of gotchas that proved a little tricky. We'll do this by creating a new widget called External Knowledge and adding it to the ext_knowledge page that we referred to earlier in our search page template.

The issue I ran into displaying this content

• First, you can't simply pass the article text via the URL and display it.. you have to actually get the record again.
• Second, the content won't be rendered as HTML by default. You have to use a little angular magic to allow that.

Get the record using a Server Script

We'll deal with our first issue in the server script for the widget. First, define a function called getRecord(). That is going to be mostly identical to the data fetch script from before but with a few changes.

1. Because we're not doing a search, the URL should change to this:

   var url= "https://myInstance.service-now.com/api/now/table/kb_knowledge/"+rec +"?sysparm_fields=short_description%2Ctext%2Cnumber%2Ccategory";

2. You'll notice we are using the variable rec instead of query in our query string.
3. We don't need the url and target variables from before so when we handle the response data the only variable we're setting is result.primary. Then, we just define record variable and get the value from the URL parameter and pass that into our getRecord function. The server script ends up something like this:

(function() {
   data.record = $sp.getParameter('record');
   getRecord(data.record);
   function getRecord(rec){
       var results = [];
       /* Calculate your results here. */
       var url= "https://myInstance.service-now.com/api/now/table/kb_knowledge/"+rec +"?sysparm_fields=short_description%2Ctext%2Cnumber%2Ccategory";
var ws = new sn_ws.RESTMessageV2();
     ws.setBasicAuth("search_user", "search");
   ws.setHttpMethod("get");
   ws.setEndpoint(url);
       var jsonOutput = ws.execute();
       if (jsonOutput) {
           var response = new JSON().decode(jsonOutput.getBody());
           results = response.result;
           results.forEach(function(result) {
               result.primary = result.short_description;
           });
       }
       data.article = results;
   }
})();

Basically all this does is make a REST call to get the record and store the result data in data.record. This will make it available to our controller via the $scope.

Use AngularJS to create a HTML Template

Our HTML template is very simple here. We want to show the article title and render the HTML content from the text field in a similar manner to how we display knowledge articles that are in our instance. This takes 4 lines:

<div>
<!-- your widget template -->
<h1>{{::c.data.article.primary}}</h1>
<div class="kb_article" ng-bind-html="::data.article.text" style="overflow-x:auto;"></div>
</div>

The most important thing to note here is that we're using ng-bind instead of the usual curly braces to bind our article text here. This is basically the angular magic I mentioned earlier. Once we use ng-bind we can add a single line of code to the client controller to allow the HTML content to be rendered as HTML and then we're done.

Allowing HTML content via the Client Controller

As I said, aside from the boilerplate code declaring the controller function and the c variable we have one line of code here:

$scope.data.article.text = $sce.trustAsHtml($scope.data.article.text);

This tells angular that $scope.data.article.text is HTML content and it is okay to render it as such.

Now when we search in the portal, our results will look like this:

When we open one of the search results, it looks like this:

Again, USE AT YOUR OWN RISK! This is not a ServiceNow official customization.

Again, I have to re-iterate that this is use at your own risk and is not supported by ServiceNow. Hopefully it is a good enough example to help you get started using search sources but keep in mind that my goal here is to share what I learned by setting one of these up myself and my hope is to help you avoid spending time trying to solve the same problems I ran into; not to give you the holy grail of searching. So let's recap what we went over here:

• We configured a scripted search source using the ServiceNow table api to crawl a knowledge base from another ServiceNow instance.
• We decoded the JSON response to set some fields that the search widget expects to be set.
• We created a widget that we could use to display the article content in the portal as if it were native content and the adjusted the search page template to link to a page using that widget.   (again, keep in mind you could just link to the external site if you want to keep things simple)
• We looked at using ng-bind along with angular's trustAsHTML method to display HTML content.

This can be adapted to work for any REST endpoint you have access to and could really enhance the search on your portal by allowing you to combine all of your resources into a single portal. For more information on search sources please refer to our documentation: Configure search in Service Portal   and for more information on the ServiceNow table apis used in this demo see this article: Table API.