Speed up email delivery by disabling notifications for users with bounced emails

[jonnyseymour]

At a time when Twitter, Facebook, Google and Yahoo give you flexibility to select what you want and don't want to read, the email is still one of the places where "unsubscribe" is not always an option. As users go on holidays, move jobs (e.g. the account get disabled), mailboxes get full, emails are marked as spam, etc, some email relays would start bouncing back those emails, and those users would not be able to unsubscribe to those senders. On your instance, it is a good practice to review the emails bounced back and review the users for which no further notification should be sent. This requires an efficient and regular intervention, as bounced emails would slowly increase if no action is taken. From the administration point of view, if users are empower themselves to unsubscribe, you will have less work to do, and wanted emails would go out faster.

Following up my previous blog Speed up your email delivery by validating recipients, here is my review on the matter.

3 ways to minimize bounced emails

1. Educate your users to unsubscribe from the unwanted notifications instead of just ignoring them
2. Regularly execute a validation of the emails bounced back to the instance
3. Archive junk emails to improve the email table performance

Educate your users to unsubscribe from unwanted notifications

To educate your users to unsubscribe, ensure to inform them this functionality is available on your instance. Users can create filters within their preference to only receive the notification if the condition set on their personalised filter are met. They could also choose to disable per notification, or disable notifications received by the user altogether. This does not affect the user active status - just the notifications targeted to them.

Here is one example from a notifications received by one user: angelo.ferentz.

For this, users can go to their notification preferences, then select the notification they have received and set the dial to "off." That would avoid further notifications from being sent to this user only.

They can also decide to leave them ON, but set a personalised filter that avoids them under certain conditions.

Finally, if users decide to stop any notification from being sent to their devices (e.g. before going on holiday), users can select to disable them altogether.

From the performance point of view, if users only enable the notifications they require, there would be less emails to send. This is a much better option than generating tons of emails that go out and back again, when these settings are ignored.

If you are reading this, unsubscribe of all the unwanted notifications   to help improve performance on the services you receive and provide.

Regularly execute a validation of the emails bounced back to the instance

There is not standard practice to parse the emails bounced back. However, as a cloud, instance administrators are limited on the options available to avoid bounced emails.

Administrators can see the emails received and marked as Junk. If you recognise those are emails as bounced back, the safest option is to set the user 'notification' to Disabled. Then contact the user to inform them of the change. You can do this regularly as it improves your email reliability because it reduces the amount of emails bouncing back. If your instance is having a larger portions of emails bounced back, I have created some tools to help.

retrieveBouncedBackEmail is a script when executed   that would match a query and parse the emails on the body_text, which usually contains the bounced emails. It ignored the emails after the "Received:" text, as it usually contains the original senders.

setUserNotificationbyEmail is a script that would enable or disable the notification on the user that matches the email address.

Here is the script for retrieveBouncedBackEmail:

// Retrieve the list of emails on bounced emails 
// Note: The normal practice here isn't to attempt to parse the bounce message at all.
//
// Returns an array with the list of [0] messages [1] Array of emails bounced
function retrieveBouncedBackEmail(queryToMatch, maxToFindPerEmail, limitquery) {
       return _retrieveBouncedBackEmail(queryToMatch, maxToFindPerEmail, limitquery);
       function _retrieveBouncedBackEmail(queryToMatch, maxToFindPerEmail, limitquery) {
               // Set max emails retrieved from each bounced email parsed - default 10 
               maxToFindPerEmail = maxToFindPerEmail || 10;
               // Limit parsing more than limitquery emails - default 100
               limitquery = limitquery ? limitquery : 100;
               // Set the query that matches the bounced emails - default: "type=received-ignored^sys_created_onONThis week
               queryToMatch = queryToMatch || "type=received-ignored^sys_created_onONThis week@javascript:gs.beginningOfThisWeek()@javascript:gs.endOfThisWeek()";
               // Set the response message and the query to perform
               var vmessages = ["Query limit set " + limitquery, "Searching on sys_email", "Query: " + queryToMatch, "Threshold: " + maxToFindPerEmail],
                       b = new GlideRecord("sys_email");
               b.addEncodedQuery(queryToMatch);
               b.setLimit(limitquery);
               b.query();
               vmessages.push(" sys_email records returned: " + b.getRowCount());
               // Loop on the matched emails and assume the body contains the list of emails bounced back
               for (var lemails = []; b.next();) {
                       var a = b.body_text,
                               // Ignores the text after "Received:" on the body_text as it contains the emails from the original message
                               // that you do not want to parse. You want to retrieve only the ones related to the bounced email
                               g = a.indexOf("Received:");
                       0 < g && (a = a.substr(0, g));
                       a = (a = extractEmails(a)) ? uniq(a) : [];
                       // It would report on email bound back on which we could not extract an email 
                       // so you need to inspected it, to validate why it did not retrieve an email
                       0 == a.length && vmessages.push("Email we could not extract an email addresses: " +
                               b.sys_id + " - count: " + a.length);
                       // If the email contains too many emails, it will add a message
                       // so you need to inspected it, to validate why there were so many emails
                       a.length > maxToFindPerEmail && vmessages.push("Email ignored by threshold: " + b.sys_id + " - count: " + a.length);
                       (0 < a.length && a.length) <= maxToFindPerEmail && (lemails = uniq(lemails.concat(a)))
               }
               // returns an array with the list of arrays: 
               // [0] array of message [1] array of emails
               return [vmessages, lemails]
       };
       // Generate an array of emails found on the text provided
       function extractEmails(text)
       {
               return text.match(/([a-zA-Z0-9._-]+@[a-zA-Z0-9._-]+\.[a-zA-Z0-9._-]+)/gi);
       }
       // Sort and remove non-unique strings on the array
       function uniq(a) {
               return a.sort().filter(function(item, pos, ary) {
                       return !pos || item != ary[pos - 1];
               })
       }
}

Here is the script for setUserNotificationbyEmail:

// Set the User 'notification' to Enable/Disabled for the matching email addresses
// vemailarray is the array of emails to disable
// setEnable is either "enabled" or "disabled". Disabled is the defaul
// limitquery is the max number of users to set. It has a limitquery of 100
// 
function setUserNotificationbyEmail(vemailarray, setEnable, limitquery) {

       // It will only query the users that need to be enabled or disabled, and ignore the ones already set.
       var c = (setEnable = /^(enable|enabled|true|1|yes|on)$/i.test(setEnable)) ? "notification=1^emailIN" + vemailarray.join(",") : "notification=2^emailIN" + vemailarray.join(","),
               b = new GlideRecord("sys_user");
       limitquery = limitquery ? limitquery : 100;
       b.addEncodedQuery(c);
       b.setLimit(limitquery);
       b.query();
       // create the message to provide back
       c = ['Query Limit set ' + limitquery, 'Records found: ' + b.getRowCount(), 'Query executed: ' + c];
       // Here the users are being set
       for (; b.next();) c.push((setEnable ? "Enabled " : "Disabled ") + "notification: " + b.sys_id + " - user: " + b.user_name), b.notification = setEnable ? 2 : 1, b.update();
       return c
};

For an example, here is a bounced email found on an instance:

Executing:

// set the qualification to match your bounced emails.   
var result = retrieveBouncedBackEmail("sys_idSTARTSWITHc3a63a97dbc14bc0d975f1c41d9619b7", 45, 1000);
gs.print("\n Results:\n" + result[0].join("\n") + "\n\nFinal list of emails " + result[1].length + "\n\n\n" + result[1].join("\n"));

RESULT:

*** Script: 
 Results:
Query limit set 1000
Searching on sys_email
Query: sys_idSTARTSWITHc3a63a97dbc14bc0d975f1c41d9619b7
Threshold: 45
 sys_email records returned: 1

Final list of emails 3


andrea.sisco@abc.com.net
test-bb.aa@abc.com.net
test.aaa@abc.com.net

To disable the users, you can execute the following:

// create an array of emails, for the users you want to disable.
var todisablelist = ["andrea.sisco@abc.com.net","test-bb.aa@abc.com.net","test.aaa@abc.com.net"]
// To disable the User Notification uncomment the next line
gs.print( '\n\nSetting user disabled:\n\n' + setUserNotificationbyEmail(todisablelist,'Disabled').join('\n'));

RESULT:

*** Script: 
Setting user disabled:
Query Limit set 100
Records found: 1
Query executed: notification=2^emailINandrea.sisco@abc.com.net,test-bb.aa@abc.com.net,test.aaa@abc.com.net
Disabled notification: 8d256345dbe983002fd876231f96196e - user: andrea.sisco

Or you can use them together as follow:

// set the qualification to match your bounced emails.   
var result = retrieveBouncedBackEmail("sys_idSTARTSWITHc3a63a97dbc14bc0d975f1c41d9619b7", 45, 1000);
gs.print("\n Results:\n" + result[0].join("\n") + "\n\nFinal list of emails " + result[1].length + "\n\n\n" + result[1].join("\n"));
// result[1] hold the array of emails, for the users you want 
// To disable *after you validate* the User Notification result[1] contains an array of the emails
gs.print( '\n\nSetting user disabled:\n\n' + 
     setUserNotificationbyEmail(todisablelist,'Disabled').join('\n'));

RESULT:

*** Script:
Results:
Query limit set 1000
Searching on sys_email
Query: sys_idSTARTSWITHc3a63a97dbc14bc0d975f1c41d9619b7
Threshold: 45
sys_email records returned: 1

Final list of emails 3


andrea.sisco@abc.com.net
test-bb.aa@abc.com.net
test.aaa@abc.com.net

Setting user disabled:


Query Limit set 100
Records found: 1
Query executed: notification=2^emailINandrea.sisco@abc.com.net,test-bb.aa@abc.com.net,test.aaa@abc.com.net
Disabled notification: 8d256345dbe983002fd876231f96196e - user: andrea.sisco

Please note the user notification value does not affect the "active" or "lock" status on the account. It will only affect the notifications.

I hope these actions empower the administrator to set a user account's notifications to Disable. This would dramatically reduce the amount of emails being bounced back.

Archive junk emails

ServiceNow can archive and eventually destroy email messages that you no longer need, in the case Junk emails or if your Email table is excessively large. This is especially important if you depend on emails as very large tables tend to degrade over time and delay upgrades.

As per performance, we are looking for email retention rules: "Emails - Ignored and over 90 days old". This rule archives email message records that were created more than 90 days prior to the current date and are of type received-ignored or sent-ignored.

Once the emails are archived, then they stay on the archive for a year before being destroyed.

Now, it is time to put this all in place and allow your users to unsubscribe. For those which the notifications bounce back, then set the sys_user 'notification' to Disabled. Finally, for those emails ending up on your instance Junk, archive and destroy them. If you keep doing this, your email delivery will be reliable and delays will be a thing of the past.

Here are some useful resources for more information :

• Speed up your email delivery by validating recipients

• Email Resources Page (KB0540674)

• Apply notification conditions (Docs - Jakarta)
• User notification preferences in UI16 (Docs - Jakarta)
• Notifications (Docs - Jakarta)
• Create a notification filter (Docs - Jakarta)

• 3 steps to determine if and why the event processor is unable to start

• Diagnosing and remedying stuck event processes

• Docs: Additional email properties

• Docs: Available system properties

• Docs: Create an email notification

• Docs: Notification examples

• Docs: Notifications

• Docs: Scripting for email notifications

• Troubleshooting email notifications - Send to the Event Creator

• Troubleshooting Outbound Email (KB0521382)

• Troubleshooting stuck event process (KB0547784)

• Verifying the recipient has an email address (KB0528669)

• Verifying the recipient's email address is properly formatted (KB0528671)

• My other blogs