Configuring auto stop words and regenerate text indexes

[nisha_narayanak]

Text searching is one of the powerful tools within the ServiceNow application, which searches multiple tables within ServiceNow via a single transaction. Text search enables us to find things easily without knowing which table or column the data is stored in. Search brings a logical collection of results grouped by table. I am going to show you how to configure the auto stop words and rebuild text indexes for a table.

You will find all the tables that are indexed under System definition > Text indexes.

A few months ago, Britt Champeau introduced us to text index stop word searches. Stop words are common words that are not indexed and won't appear in search results or be queried for search. Stop words are of 2 types, global stop words (implemented out of box) and automatic table specific stop words. Once the frequency of any word on a table reach a particular threshold, the system automatically designates it as a stop word. This auto stop threshold can be enabled by clicking the table names under System definition > Text indexes. Enabling Stop Words on a table helps to reduce the size of the ts_ tables which will help to speed up the search process as well. The ServiceNow product documentation has additional information on text search and stop words.

How to configure auto stop words followed by regeneration of text indexes:

I am using this Task table as an example here

1. Set threshold for auto stop words:

    a. Navigate to System Definition > Text Indexes > For eg: Open Task table record

    b. Check 'AutoStop' checkbox   and then set "Auto threshold" to 25,000.

    c. Save.

 

2. Reset Text Search Caches

        Navigate to System Definition > Text Indexes

        In Related links select "Reset Text Search Caches"

3. Run "TS Index Stats" job

      a. Navigate to System Scheduler > Scheduled Jobs > Open the "TS Index Stats" record

      b. Select "Execute Now"

      c. Monitor job until it completes (either by watching the worker on the node that picks it up or checking the "state" from sys_trigger_list)

4. Set Stop Mode if stop words are generated. If the stop words are not generated lower the "auto threshold" value and repeat the above 3 steps.

      a. Once "TS Index Stats" finishes, go back to the ts_index_name table

      b. Select the task record

      c. Scroll down to the stop words that were generated and go to the "Stop Mode" column, do a multi-select and change all the records to:"Neither Index Nor Query".

See detailed instructions below:

                              4.c.i.     Copy URL only for the stop words of a table and open in another window .

                              4.c.ii.     Right click on column header   "Stop Mode"

                              4.c.iii.   Select "Update All" ( This appears only if you open the stop words in new window )

                              4.c.iv.   A new form opens where select "Neither Index Nor Query" from drop down and click on "Update" ) (Neither index nor query tells the system not to index the word or be queried.)

5. Regenerate Indexes

      a. Go back to the ts_index_name > task record.   Select the related link 'Regenerate Text Index'

      b. Verify that the "text index event process" is processing and finishes, Monitor the job until it completes

Repeat above steps for all tables you are planning to configure auto stop words and regenerate text indexes.

Regeneration of indexes may take a few hours depending on the size of tables and other factors. This can cause performance degradation during the process and should be planned after business hours.