Harness the power of Knowledge Search

[Sarup Paul]

Time is of the essence when customers come to your Support portal, and more often their eyes try to locate the search box instead of browsing through text-heavy pages. If you are promoting self-service, search is an important vehicle for achieving that goal.The ServiceNow platform provides a rich set of features to tune Knowledge Search. This post covers how to make search work better for your implementation. ServiceNow has some awesome resources that can help you implement search in a way that works for you by considering pre and post use. There also some links here that refer to customizations that have leveraged the platform's capabilities to get even better results.

There are several aspects to consider that can improve the search in your ServiceNow instance when considered and done right. I would suggest creating a guide that knowledge creators can follow to make sure their content is findable in the Knowledge search.

Things to consider when building a searchable knowledge base:

1. Zing search
2. Field Weights
3. Stop Words
4. Short Description (Title)
5. Meta tags
6. Boolean search capabilities
7. Knowledge "pinning"
8. Synonyms
9. Inverse Document Frequency (IDF)
10. Configure Query mode and Partial Matching
11. Search across Multiple tables

Zing text search

Zing is the text indexing and search engine that performs all searches in ServiceNow. In order to tune Zing text search, it's important to understand how the search engine processes the queries and determines the relevant articles. At the core of any search engine, like ours, is a scoring mechanism that evaluates the query against the Articles in the Knowledge base. The process starts with the indexing of the Articles. Knowledge is setup OOB to be indexed, by configuring text indexes on kb_knowledge table.

You should also understand how Zing scores each of the articles during a search. The factors that contribute to the score are Frequency, Sequence and Weights and determines a document's score. The "Document Scoring Diagram" shows you the flow in process, and what elements we can use to better tune our knowledge search.

Field weights

One of the first methods is using Field weights. You can set the relative weight of a field to ensure that relevant information, or content you want to appear, ranks higher than others. On HI, we have tweaked these weights on the Short Description field to enhance our relevancy. We increased the field weight of our short descriptions from 10 to 15.

Stop words

As a part of configuring Knowledge search for your implementation, you should consider reviewing the base system stop words. In some cases, you may find commonly used terms in your environment are configured as Stop words. For example, a company like ServiceNow probably wouldn't be happy with "broken" or "problem" as set as Stop words as these are often used when searching. You should review the Stop words that are setup for kb_knowledge table and make any necessary configurations based on your organization's needs. It may be helpful to pull the most searched words in relations to your company and industry and make sure they are not on the Stop words list for that specific table. Note: Text search uses the global list plus the table-specific list of stop words when indexing the table.

To review the Global Stop Words:

1. Navigate to System Definition > Text Index Stop Words.
2. Add or remove stop words from the list.

If a message appears at the top of the list, contact Technical Support to regenerate all indexes. You must regenerate indexes whenever words may be missing from an index. For example, if you delete, inactivate, or change an active global stop word, the word may be missing from the index. An after business rule checks these conditions and generates the notification message when index regeneration is necessary.

To configure your Knowledge Stop Words:

1. Navigate to System Definition > Text Indexes.
2. Open the text index entry for the table kb_knowledge.
3. Add or remove stop words from the Index Stop Words related list. For each word, select the desired Stop mode (Not a Stop Word (inactivates the stop word), Neither Index nor Query, or Index but do not Query).

If a message appears at the top of the form, click Regenerate Text Index and click OK. You must regenerate indexes whenever words may be missing from an index. For example, if you delete or change a stop word for which the mode was Neither Index nor Query, the word may be missing from the index. An after business rule checks these conditions and generates the notification message when index regeneration is necessary.

Short Description (Title)

I can't stress enough the importance of having an appropriate and optimized Short Description for your Article. This not only helps in search, but helps a lot on the readability of the Article. You may want to reference about the Product/Service that the Article is about, and the symptoms/Issue that the Article is trying to address. A good title will have keywords and will be concise and to the point.

Meta tags

Another improvement on the Article is adding keywords and phrases in the META section of the Article form. Meta is a hidden field on the Knowledge form that you can enable on your instance. This field is setup OOB to have a weight of 50, and hence keywords and phrases matches to meta, vastly helps in improving the search relevancy.

You can refer to this post on configuring your knowledge meta tags for best practices to improve how your users find your content. Insert phrases and search terms for which you would expect this article to be applicable.

Public Search Engine setup

If your Knowledge Base is public, and you want to configure public search engines you will find the configuration instruction here- Google custom search integration. On HI, we have implemented some search best practices, that would be useful to knowledge managers to handle content being crawled by search engines.

Boolean search

Your power users will like the Boolean search capabilities that are extended out of the platform capabilities; which combines search terms with logic operators. Advanced uses of Boolean search operators allow super searchers to use AND, OR, and NOT to narrow down and hone in on search results.   We are exploring ways of exposing the boolean search as a Advanced Search UI.

Pinning

Sometimes, you will have a scenario for "high-value" support issues, that are often related to a keyword e.g. "password" or "email". In such scenarios, you may want to use the Pinning feature in Knowledge v3. Pinning an article will feature it in search results by putting it at the top of a user's search results. In such cases, you may want to create "Hub Articles" that refers to other Articles that cover a specific topic such as "password reset."

Synonyms

Too often your end users may search for terms that may not exactly match the terms in the article. The meta tags help a long way- but it may become a challenge to scale over a large number of articles. Synonyms help to manage the terms centrally across all objects within the system, including Knowledge. See here

Enable Inverse Document Frequency

Enable term frequency—inverse document frequency (TF-IDF) to increase the search result scores of search terms that appear more frequently in a document, but less frequently in the whole collection of searchable documents. Let's say you are searching for "Collision Insurance"- the word "Collision" will be more frequent in a document that refers to that topic, but the word "insurance" would be more prevalent across all documents in the KB. When enabled IDF algorithms will boost documents with "Collision". This feature is available for Knowledge search starting Jakarta, and enabled by default on new and z-booted instances. More details here - Score search terms by inverse document frequency (IDF)

Configure Query mode and Partial Matching

Multi-word searches can be tricky, and quite often too many results are returned based on all the searched terms. Changing the query mode, and configuring partial match can reduce the search result set, leading to more relevant search results. This feature is available for Knowledge search starting Jakarta, and enabled by default on new and z-booted instances. More details here - Change the query mode of an indexed table

Configure multiple tables for indexing and searching

Performing searches across multiple tables requires normalizing the tables first so the relevance scores mean the same thing for all of the tables. V4 indexing provides that normalization. Zing uses the Okapi BM25 ranking function to calculate the relevancy score of search results.

So that's it for now. Hope this post helps you get the most out of Knowledge Search. For times that you are end up running into issues, see Debugging Search.

Cheers !!