Speed up your email delivery by validating recipients

[jonnyseymour]

An extremely practical and well presented email relay system, is comprised of a fast cloud SMTP server. It furthers the ability to integrate with external SMTP servers that it is suitable for most business needs. You can reduce the number of invalid emails and speed up delivery of your emails by ensuring to validate the recipients. To ensure flexibility, you can add the possibility to unsubscribe from an email notification. I will try to focus on identifying invalid email addresses. However, ensuring valid email addresses is like trying to eradicate a virus, it requires an efficient and regular intervention. Note, this is just the tip of the iceberg on outbound email performance as several other factors count as well.

Here are 6 tips reduce the number of invalid emails and speed up delivery:

1. Educate your users and include information on the notification about how to unsubscribe.
2. Validate that the target emails are valid.
3. Regularly execute email verification and check the list of incorrect or undelivered emails.
4. Prepare for emails that bounce back (e.g. out of the office, undelivered, etc.).
5. Plan delivery of bulk emails during non-busy times.
6. Ensure the target email servers have white-listed the relay SMTP server.

For these examples, I have set the following emails for testings:

Email	Syntax	Valid	Reason
abel.t uter@exa mple.com	Error	No	Space on example
adela.cervantszexample.com	Error	No	No @
aileen.mottern@ example.NOEXISTANT	Error	No	NOEXISTANT is not a domain
'alejandra.prenatt@ example.com'	Error	No	single quotes ' at beginning and end
test@gmail.com	OK	Yes	Email is valid

I added the recipients to an incident Assigned to and Watch list and added additional comments.

1. Educate your users and allow the option to unsubscribe.

It is difficult to know the user that need to read an email notification before many implementations. For flexibility, our system allows your to enable or disable your email notification as part of the user preferences. Provide information for how your users can disable the unwanted notifications.

Valid emails addresses are the recipients that will receive and potentially read the email notification.

Target recipient	Correct syntax	Target exist	User read email
Valid	Yes	Yes	Yes
Invalid	Yes	Yes	No
Invalid	Yes	No	No
Invalid	No	No	No

In addition, to ensure flexibility, your email notification   can have an unsubscribe link on the notification itself or to the user notification preference settings.This will avoid users marking your emails as spam on the target email, or triggering unwanted emails back, allow the sense of agreement with the target user, etc. Less emails means less time to deliver future notifications.

e.g On your notifications, you can add a mail script to include an unsubscribe link:

<mail_script>
template.print('<a href="' + gs.getProperty("glide.servlet.uri") + "unsubscribe.do?sysparm_notification=" + email_action.sys_id + '">Unsubscribe</a><br/>\n');
</mail_script>

Alternatively, an email script that will allow you to unsubscribe.

2. Validate that the target emails are correct

This is one of the most important steps. Check and double check that the email syntax and a basic domain are correct. This is target to the users that have incorrectly written emails.

Ensure your email addresses lists is tuned, and the syntax is correct. Below is a sample findInvalidEmailAddresses script the checks if the email address is valid. To validate the domain, you could use the instance search queries. e.g. emails that ends with "@acme.com", etc.

Invalid emails are all those emails that will never get delivered (email is incorrect) or read on by the user the email address was provided (e.g. email account is abandoned)

In our examples:

The email system will perform a basic syntax check and will pick very few emails as follow:

>Notification 'Incident commented' (0d51edbac0a80164006963b000ff644e) excluded recipients because user's device contains an empty or invalid email address (see "cmn_notif_device.email_address"): 'Adela Cervantsz' (0a826bf037102000e0bfc8bcbe5d7a)

Any other user which email were accepted will show on the email logs as:

> Notification 'Incident commented' (0d51edbac0a80164006963b000ff644e) included recipients via notification's "Users/Groups in fields" (e.g., watchlist, assigned_to, etc): 'Alejandra Prenatt' (22826bf03710200044e0bf8bcbe5dec), 'Jonny Seymour' (b6392da30f1856146cce1050efd), 'Aileen Mottern' (71826bf03710200044ebfc8bcbe5d3b), 'Abel Tuter' (62826bf037100044e0bfc8bcbe5df1)

On the email itself the recipients will look like: aileen.mottern@example.NOEXISTANT, abel.t uter@exa mple.com, test@gmail.com, 'alejandra.prenatt@example.com'

To validate the syntax of emails stored on the sys_user table, here is a background script that can help you find the some of those emails with the incorrect syntax:

findInvalidEmailAddresses(!1); // false !1 to NOT set notification off automatically   
function findInvalidEmailAddresses(setnotifyoff) {   
       var a = [], // holds the message back   
               b = [], // holds the sys_id's for the link   
               f = "emailISNOTEMPTY^active=true^locked_out!=true^notification=2",   
               h = 10000, // max results for the query to avoid unlimited queries   
               e = setnotifyoff == !0; // check if want to set notification off on the user   
       a.push("*** Scripts findInvalidEmailAddresses ***");   
       a.push("-----Searching for invalid email addresses: ----- setting notification off: " + e + "\nsys_user query: " + f);   
       // c is the regex to validate the email is valid. Note this is a guide   
       var c = /^\b[a-z0-9!#$%&'*+\/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+\/=?^_`{|}~-]+)*@(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+(?:[A-Z]{2}|com|org|net|gov|tv|biz|info|mobi|name|aero|jobs|museum)\b$/i,   
               d = new GlideRecord("sys_user");   
       // query for relevant users.   
       d.addEncodedQuery(f);   
       d.setLimit(h); // do not query more than 10K records   
       for (d.query(),g=1; d.next();) {   
               d.email && !c.test(d.email) && (b.push(d.sys_id + ""), a.push("User #"+ g +" Found: " + d.sys_id + ' - email = "' + d.email + '"')) && g++ &&   e && (d.notification = 1) && d.update();   
       }   
       (d.getRowCount() >= h)   && a.push("\n**** WARNING results are higher or equal than the limit "+ h +" set for the query");   
       b.length ? a.push("\n**** Found "+ b.length + " possible invalid email(s) of " + d.getRowCount() + " emails checked. Run 'findInvalidEmailAddresses(!0)' to disable them" + "\nTo list them use the following link:\n"+ gs.getProperty('glide.servlet.uri') + "sys_user_list.do?sysparm_query=sys_idIN" + b.join(",") 😞 a.push("\n**** All emails compliant: " + d.getRowCount() + " emails checked." );   
       // print the message for the background script   
       gs.print(a.join("\n"))   
}; 

For our examples, it will discover the following problematic emails:

---background script result-----
*** Script: *** Scripts findInvalidEmailAddresses ***
-----Searching for invalid email addresses: ----- setting notification off: false
sys_user query: emailISNOTEMPTY^active=true^locked_out!=true^notification=2
User #1 Found: 0a826bf03710200044e0bf10bcbe5d7a - email = "adela.cervantszexample.com"
User #2 Found: 22826bf03710200044e0bf10bcbe5dec - email = "'alejandra.prenatt@example.com'"
User #3 Found: 62826bf03710200044e0bf10bcbe5df1 - email = "abel.t uter@exa mple.com"
User #4 Found: 71826bf03710200044e0bf10bcbe5d3b - email = "aileen.mottern@example.NOEXISTANT"
**** Found 4 possible invalid email(s) of 527 emails checked. Run 'findInvalidEmailAddresses(!0)' to disable them
To list them use the following link:
<instance>/sys_user_list.do?sysparm_query=sys_idIN0a826bf03710200044e0bfc8bc5d7a,22826bf037102044e0bfc8bcbe5dec,628bf03710200044e0b8bcbe5df1,71826bf030200044e0bfc8bcbe5d3b
---background script -----

You will need to search for those users and then set the notification = Disable. After that, you may need to contact the users to correct their email addresses.

3. Regularly execute email verification and check the list of incorrect or undelivered emails

Regular expressions and email checks will not filter all invalid emails. Even if your email syntax is correct and validated by regular expressions, you do not really know whether an address is valid until you send an email to it. Even if the email arrives in the inbox, that doesn't mean it has been read.

Invalid emails could cause delays to the overall email delivery.

• Invalid email could cause SMTP retries event if only ONE email of 100 on the recipient list does not exit.
• Bounced emails can execute inbound actions and potentially create email loops that cause some email outage, than bounce back, then again.
• Extra processing for the irrelevant emails could be relevant.
• Target email systems could black-list the SMTP relay, delaying the delivery even further.

To avoid those problems, there are several email validations that can be performed. The automatic validation is not available on ServiceNow yet: if you have an external email provider that can test the target emails, you can provide them with your email list and they will validate if they exist on the final exchange or not. They will extract the MX records from the email address and connect to mail server (over SMTP and also simulates sending a message) to make sure the mailbox really exist for that user/address.   If you do not have those services, you can still review the bounced emails on your instance to manually validate those emails. For those emails that bounced back, set the notification = disable on the sys_user form to avoid further unwanted emails. Then, contact the user owners for the correct addresses.

Even then, you will need to regularly check for incoming bounced back emails as users leave companies, email changes, etc. There are several other cases some where email will not get delivered. e.g. Emails bounced back because the target email has full mailboxes could indicates that no ones read that mailbox.

This is important as your instance will retry to send on certain cases, even if it is an invalid email depending on the SMTP response.

You will need to search your skipped emails regularly to fix the problematic emails manually. To avoid them, set the notification - disable on the related sys_user record.

This is an example email:

>Subject:Undelivered Mail Returned to Sender
>From:MAILER-DAEMON (Mail Delivery System)
>Auto-Submitted:auto-replied
State: Ignored
Mailbox: Junk
User: MAILER-DEAMON
Body = [ This is the mail system at host bulk.service-now.com. I'm sorry to have
to inform you that your message could not be delivered to one or more recipients.
It's attached below. For further assistance, please send mail to postmaster.
If you do so, please include this problem report. You can delete your own text
from the attached returned message. The mail system <aileen.mottern@example.NOEXISTANT>:
Host or domain name not found. Name service error for name=example.NOEXISTANT type=A: Host not found

You will need to search for email aileen.mottern@example.NOEXISTANT, and set the notification = Disabled for the associated user.

4. Be prepared for emails bounced back (e.g. out of the office, undelivered, etc)

Each email you send is potentially one email back to be received. Be prepared. To ignore emails, you either configure your email filters on the email configuration or install the "Email Filters" plugin. If using email configuration, ensure the properties to ignore unwanted emails are set to cover all your bounced email cases.

For example: glide.pop3.ignore_headers, glide.pop3.ignore_subjects, glide.pop3.ignore_senders

For most sophisticated filters, use the "Email Filters" plugin. Email filters take over the email configuration filter. Once enabled, administrators can configure specific email filtering preferences by using a condition builder or a condition script.

With Email filters enabled, glide.pop3.ignore_headers, glide.pop3.ignore_subjects and glide.pop3.ignore_senders   will get ignored.

See Email Spam Scoring and Filtering (will need to login to HI to view) to know how to use the Email Filters plugin to filter emails which have been scored as spam, when using the ServiceNow email infrastructure.

If your email does   meet the filters/Email filters conditions, you will receive a message "ignored by filter" on the logs, an the email will get skipped.

* Information       - Skipping 'Create Incident', ignored by filter

If your email filters are NOT configured correctly, the bounced emails will execute your inbound actions

5. Plan delivery of bulk emails on non-busy times

Emails sent by bulk may delay critical business emails. Many of those emails will get rejected by the target SMTP server and resend once again. Validate with your administrator the best times to send bulk emails. If it's the first time you will send a bulk, ensure you monitor the bounce-back emails to disable the notification on the related users for those problematic email addresses.

You can also trigger some notification at an specific time using our gs.eventQueueScheduled(event, record, parm1, param2, date-time) function that could trigger your notification at the required date-time. For example: gs.eventQueueScheduled("incident.reminder", current, gs.getUserID(), gs.getUserName(), current.u_reminder);

Keep in mind, there are five (5) email properties that will potentially add a relevant delay on email delivery because of invalid emails causing defer retries:
glide.smtp.default_retry, glide.smtp.defer_retry_ids, glide.smtp.fail_message_ids, glide.email.smtp.max_recipients and glide.email.smtp.max_send

Please set the properties to to meet your business requirements. The emails will be queued and re-queued several times for the problematic email addresses.

#1 glide.smtp.default_retry: Enables (true) or disables (false) resending email when an unknown SMTP error code is encountered.

The instance only recognizes the SMTP error codes defined in the glide.smtp.defer_retry_ids property. Default value: true

#2 glide.smtp.defer_retry_ids: Specifies the comma-separated list of SMTP error codes that force the instance to resend email.

Default value: 421,450,451,452

421 -       <domain> Service not available, closing transmission channel-

450 -       Requested mail action not taken: mailbox unavailable- e.g. SMTP server could not access a mailbox to deliver your message

451 -       Requested action aborted: local error in processing- e.g. This error is usually SMTP relaying service from too many messages.

452 -       Requested action not taken: insufficient system storage-

#3 glide.smtp.fail_message_ids: Specifies the comma-separated list of SMTP error codes that prevent the instance from resending email.

Default value: 500,501,502,503,504,550,551,552,553,554

500 -         Syntax error, command unrecognised - e.g. Your antivirus/firewall interfering with incoming and/or outgoing SMTP communications

501 -         Syntax error in parameters or arguments - e.g. Invalid email addresses or an invalid domain name recipient. Error can indicate bad connection

502 -         Command not implemented

503 -         Bad sequence of commands - e.g. Error, particularly if repeated, indicates bad connection. Verify authentication settings

504 -         Command parameter not implemented

551 -         User not local; please try <forward path>

552 -         Requested mail action aborted: exceeded storage allocation - e.g. The recipient's mailbox has reached its maximum allowed size.

553 -         Requested action not taken: mailbox name not allowed - e.g. Invalid email address.

554 -         Transaction failed - e.g. when its anti-spam firewall does not like the sender's email address, or the sender's IP address, or the sender's ISP server

#4 glide.email.smtp.max_recipients: Specifies the maximum number of recipients the instance can list in the To: line for a single email notification. Notifications that would exceed this limit instead create duplicate email notifications addressed to a subset of the recipient list. Each email notification has the same maximum number of recipients. Default value: 100. This is ONLY valid for recipients created by the email notification system (it does not include emails added by scripts or email client).

On each duplicated email, email logs will show:

Information

Email with 4 recipients is split into 4 separate emails because property glide.email.smtp.max_recipients=1 (Email 4 of 4)

On our example, the final emails look as follow:

#5 glide.email.smtp.max_send: Specifies how many emails to send through each new SMTP connection. The instance establishes a new SMTP connection if there are more emails to send than the specified value. Default value: 100

#4 and #5 can be used to set email throttling. e.g 100 email with 50 recipients max each vs 100 emails with 1 recipient max each. It completely depend on your target emails and email capacity of the target exchanges.

In a nutshell, plan your email delivery, set the resend properties accordingly with a reasonable to email throttling to protect system performance.

6. Ensure the target email servers have white-listed the relay SMTP server.

If your SMTP server is not white-listed, spam system could penalize it by adding transaction time to the relay or delaying delivery. This is VERY common. If the number of emails bounced back or the delivery time increases after bulk emails, check the target email address SMTP servers have white-listed the relay SMTP server on your instance. If you are using our SMTP relay system, ServiceNow highly recommends that you configure your target mail exchange systems to retrieve our Sender Policy Framework (SPF) records dynamically, to white-list our Email servers. See Allowing email delivery from ServiceNow to your mail servers for more information.

A final word. Less is better. Less email recipients means you spend less, you need less storage on the target, you need a smaller processes and

you sent them faster. To make it less, you need to regularly validate your target emails even if it means donkey work. If you do not check regularly, as more invalid emails populate your system, the more time your email will take to get delivered.

Hope it helps. I have performed the tests on Geneva, using Chrome as browser.

Additional information can be found here:

• Speed up email delivery by disabling notifications for users with bounced emails

• Email Resources Page (KB0540674)

• 3 steps to determine if and why the event processor is unable to start

• Diagnosing and remedying stuck event processes

• Docs: Additional email properties

• Docs: Available system properties

• Docs: Create an email notification

• Docs: Notification examples

• Docs: Notifications

• Docs: Scripting for email notifications

• Troubleshooting email notifications - Send to the Event Creator

• Troubleshooting Outbound Email (KB0521382)

• Troubleshooting stuck event process (KB0547784)

• Verifying the recipient has an email address (KB0528669)

• Verifying the recipient's email address is properly formatted (KB0528671)

• My other blogs