An extremely practical and well presented email relay system, is comprised of a fast cloud SMTP server. It furthers the ability to integrate with external SMTP servers that it is suitable for most business needs. You can reduce the number of invalid emails and speed up delivery of your emails by ensuring to validate the recipients. To ensure flexibility, you can add the possibility to unsubscribe from an email notification. I will try to focus on identifying invalid email addresses. However, ensuring valid email addresses is like trying to eradicate a virus, it requires an efficient and regular intervention. Note, this is just the tip of the iceberg on outbound email performance as several other factors count as well.

 

speed-up-emails.png

 

Here are 6 tips reduce the number of invalid emails and speed up delivery:

  1. Educate your users and include information on the notification about how to unsubscribe.
  2. Validate that the target emails are valid.
  3. Regularly execute email verification and check the list of incorrect or undelivered emails.
  4. Prepare for emails that bounce back (e.g. out of the office, undelivered, etc.).
  5. Plan delivery of bulk emails during non-busy times.
  6. Ensure the target email servers have white-listed the relay SMTP server.

 

For these examples, I have set the following emails for testings:

Email

Syntax

Valid

Reason

abel.t uter@exa mple.com

Error

No

Space on example

adela.cervantszexample.com

Error

No

No @

aileen.mottern@ example.NOEXISTANT

Error

No

NOEXISTANT is not a domain

'alejandra.prenatt@ example.com'

Error

No

single quotes ' at beginning and end

test@gmail.com

OK

Yes

Email is valid

 

I added the recipients to an incident Assigned to and Watch list and added additional comments.

assigned watch list.jpg

 

1. Educate your users and allow the option to unsubscribe.

It is difficult to know the user that need to read an email notification before many implementations. For flexibility, our system allows your to enable or disable your email notification as part of the user preferences. Provide information for how your users can disable the unwanted notifications.

 

Valid emails addresses are the recipients that will receive and potentially read the email notification.

 

Target recipient

Correct syntax

Target exist

User read email

Valid

Yes

Yes

Yes

Invalid

Yes

Yes

No

Invalid

Yes

No

No

Invalid

No

No

No

 

In addition, to ensure flexibility, your email notification  can have an unsubscribe link on the notification itself or to the user notification preference settings.This will avoid users marking your emails as spam on the target email, or triggering unwanted emails back, allow the sense of agreement with the target user, etc. Less emails means less time to deliver future notifications.

 

e.g On your notifications, you can add a mail script to include an unsubscribe link:

<mail_script>
template.print('<a href="' + gs.getProperty("glide.servlet.uri") + "unsubscribe.do?sysparm_notification=" + email_action.sys_id + '">Unsubscribe</a><br/>\n');
</mail_script>

 

Alternatively, an email script that will allow you to unsubscribe.

email script unsubscribe.jpg

 

2. Validate that the target emails are correct

This is one of the most important steps. Check and double check that the email syntax and a basic domain are correct. This is target to the users that have incorrectly written emails.

Ensure your email addresses lists is tuned, and the syntax is correct. Below is a sample findInvalidEmailAddresses script the checks if the email address is valid. To validate the domain, you could use the instance search queries. e.g. emails that ends with "@acme.com", etc.

Invalid emails are all those emails that will never get delivered (email is incorrect) or read on by the user the email address was provided (e.g. email account is abandoned)

In our examples:

validate target email.jpg

 

The email system will perform a basic syntax check and will pick very few emails as follow:

>Notification 'Incident commented' (0d51edbac0a80164006963b000ff644e) excluded recipients because user's device contains an empty or invalid email address (see "cmn_notif_device.email_address"): 'Adela Cervantsz' (0a826bf037102000e0bfc8bcbe5d7a)

Any other user which email were accepted will show on the email logs as:

> Notification 'Incident commented' (0d51edbac0a80164006963b000ff644e) included recipients via notification's "Users/Groups in fields" (e.g., watchlist, assigned_to, etc): 'Alejandra Prenatt' (22826bf03710200044e0bf8bcbe5dec), 'Jonny Seymour' (b6392da30f1856146cce1050efd), 'Aileen Mottern' (71826bf03710200044ebfc8bcbe5d3b), 'Abel Tuter' (62826bf037100044e0bfc8bcbe5df1)

On the email itself the recipients will look like: aileen.mottern@example.NOEXISTANT, abel.t uter@exa mple.com, test@gmail.com, 'alejandra.prenatt@example.com'

email recipients.jpg

 

To validate the syntax of emails stored on the sys_user table, here is a background script that can help you find the some of those emails with the incorrect syntax:

 

findInvalidEmailAddresses(!1); // false !1 to NOT set notification off automatically  
function findInvalidEmailAddresses(setnotifyoff) {  
    var a = [], // holds the message back  
        b = [], // holds the sys_id's for the link  
        f = "emailISNOTEMPTY^active=true^locked_out!=true^notification=2",  
        h = 10000, // max results for the query to avoid unlimited queries  
        e = setnotifyoff == !0; // check if want to set notification off on the user  
    a.push("*** Scripts findInvalidEmailAddresses ***");  
    a.push("-----Searching for invalid email addresses: ----- setting notification off: " + e + "\nsys_user query: " + f);  
    // c is the regex to validate the email is valid. Note this is a guide  
    var c = /^\b[a-z0-9!#$%&'*+\/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+\/=?^_`{|}~-]+)*@(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+(?:[A-Z]{2}|com|org|net|gov|tv|biz|info|mobi|name|aero|jobs|museum)\b$/i,  
        d = new GlideRecord("sys_user");  
    // query for relevant users.  
    d.addEncodedQuery(f);  
    d.setLimit(h); // do not query more than 10K records  
    for (d.query(),g=1; d.next();) {  
        d.email && !c.test(d.email) && (b.push(d.sys_id + ""), a.push("User #"+ g +" Found: " + d.sys_id + ' - email = "' + d.email + '"')) && g++ &&  e && (d.notification = 1) && d.update();  
    }  
    (d.getRowCount() >= h)  && a.push("\n**** WARNING results are higher or equal than the limit "+ h +" set for the query");  
    b.length ? a.push("\n**** Found "+ b.length + " possible invalid email(s) of " + d.getRowCount() + " emails checked. Run 'findInvalidEmailAddresses(!0)' to disable them" + "\nTo list them use the following link:\n"+ gs.getProperty('glide.servlet.uri') + "sys_user_list.do?sysparm_query=sys_idIN" + b.join(",") ): a.push("\n**** All emails compliant: " + d.getRowCount() + " emails checked." );  
    // print the message for the background script  
    gs.print(a.join("\n"))  
}; 

For our examples, it will discover the following problematic emails:

---background script result-----

*** Script: *** Scripts findInvalidEmailAddresses ***

-----Searching for invalid email addresses: ----- setting notification off: false

sys_user query: emailISNOTEMPTY^active=true^locked_out!=true^notification=2

User #1 Found: 0a826bf03710200044e0bf10bcbe5d7a - email = "adela.cervantszexample.com"

User #2 Found: 22826bf03710200044e0bf10bcbe5dec - email = "'alejandra.prenatt@example.com'"

User #3 Found: 62826bf03710200044e0bf10bcbe5df1 - email = "abel.t uter@exa mple.com"

User #4 Found: 71826bf03710200044e0bf10bcbe5d3b - email = "aileen.mottern@example.NOEXISTANT"

 

 

**** Found 4 possible invalid email(s) of 527 emails checked. Run 'findInvalidEmailAddresses(!0)' to disable them

To list them use the following link:

<instance>/sys_user_list.do?sysparm_query=sys_idIN0a826bf03710200044e0bfc8bc5d7a,22826bf037102044e0bfc8bcbe5dec,628bf03710200044e0b8bcbe5df1,71826bf030200044e0bfc8bcbe5d3b

 

---background script -----

You will need to search for those users and then set the notification = Disable. After that, you may need to contact the users to correct their email addresses.

 

3. Regularly execute email verification and check the list of incorrect or undelivered emails

Regular expressions and email checks will not filter all invalid emails. Even if your email syntax is correct and validated by regular expressions, you do not really know whether an address is valid until you send an email to it. Even if the email arrives in the inbox, that doesn't mean it has been read.

 

Invalid emails could cause delays to the overall email delivery.

  • Invalid email could cause SMTP retries event if only ONE email of 100 on the recipient list does not exit.
  • Bounced emails can execute inbound actions and potentially create email loops that cause some email outage, than bounce back, then again.
  • Extra processing for the irrelevant emails could be relevant.
  • Target email systems could black-list the SMTP relay, delaying the delivery even further.

 

To avoid those problems, there are several email validations that can be performed. The automatic validation is not available on ServiceNow yet: if you have an external email provider that can test the target emails, you can provide them with your email list and they will validate if they exist on the final exchange or not. They will extract the MX records from the email address and connect to mail server (over SMTP and also simulates sending a message) to make sure the mailbox really exist for that user/address.  If you do not have those services, you can still review the bounced emails on your instance to manually validate those emails. For those emails that bounced back, set the notification = disable on the sys_user form to avoid further unwanted emails. Then, contact the user owners for the correct addresses.

 

Even then, you will need to regularly check for incoming bounced back emails as users leave companies, email changes, etc. There are several other cases some where email will not get delivered. e.g. Emails bounced back because the target email has full mailboxes could indicates that no ones read that mailbox.

 

This is important as your instance will retry to send on certain cases, even if it is an invalid email depending on the SMTP response.

 

You will need to search your skipped emails regularly to fix the problematic emails manually. To avoid them, set the notification - disable on the related sys_user record.

 

This is an example email:

>Subject:Undelivered Mail Returned to Sender

>From:MAILER-DAEMON (Mail Delivery System)

>Auto-Submitted:auto-replied

State: Ignored

Mailbox: Junk

User: MAILER-DEAMON

Body = [ This is the mail system at host bulk.service-now.com. I'm sorry to have

to inform you that your message could not be delivered to one or more recipients.

It's attached below. For further assistance, please send mail to postmaster.

If you do so, please include this problem report. You can delete your own text

from the attached returned message. The mail system <aileen.mottern@example.NOEXISTANT>:

Host or domain name not found. Name service error for name=example.NOEXISTANT type=A: Host not found

You will need to search for email aileen.mottern@example.NOEXISTANT, and set the notification = Disabled for the associated user.

 

4. Be prepared for emails bounced back (e.g. out of the office, undelivered, etc)

Each email you send is potentially one email back to be received. Be prepared. To ignore emails, you either configure your email filters on the email configuration or install the "Email Filters" plugin. If using email configuration, ensure the properties to ignore unwanted emails are set to cover all your bounced email cases.

 

For example: glide.pop3.ignore_headers, glide.pop3.ignore_subjects, glide.pop3.ignore_senders

 

For most sophisticated filters, use the "Email Filters" plugin. Email filters take over the email configuration filter. Once enabled, administrators can configure specific email filtering preferences by using a condition builder or a condition script.

With Email filters enabled, glide.pop3.ignore_headers, glide.pop3.ignore_subjects and glide.pop3.ignore_senders  will get ignored.

See Email Spam Scoring and Filtering (will need to login to HI to view) to know how to use the Email Filters plugin to filter emails which have been scored as spam, when using the ServiceNow email infrastructure.

 

If your email does  meet the filters/Email filters conditions, you will receive a message "ignored by filter" on the logs, an the email will get skipped.

* Information    - Skipping 'Create Incident', ignored by filter

If your email filters are NOT configured correctly, the bounced emails will execute your inbound actions

5. Plan delivery of bulk emails on non-busy times

Emails sent by bulk may delay critical business emails. Many of those emails will get rejected by the target SMTP server and resend once again. Validate with your administrator the best times to send bulk emails. If it's the first time you will send a bulk, ensure you monitor the bounce-back emails to disable the notification on the related users for those problematic email addresses.

 

You can also trigger some notification at an specific time using our gs.eventQueueScheduled(event, record, parm1, param2, date-time) function that could trigger your notification at the required date-time. For example: gs.eventQueueScheduled("incident.reminder", current, gs.getUserID(), gs.getUserName(), current.u_reminder);

 

Keep in mind, there are five (5) email propertieshttps://docs.servicenow.com/administer/reference_pages/reference/r_OutboundMailConfiguration.htmlthat will potentially add a relevant delay on email delivery because of invalid emails causing defer retries:

glide.smtp.default_retry, glide.smtp.defer_retry_ids, glide.smtp.fail_message_ids, glide.email.smtp.max_recipients and glide.email.smtp.max_send

Please set the properties to to meet your business requirements. The emails will be queued and re-queued several times for the problematic email addresses.

 

#1 glide.smtp.default_retry: Enables (true) or disables (false) resending email when an unknown SMTP error code is encountered.

The instance only recognizes the SMTP error codes defined in the glide.smtp.defer_retry_ids property. Default value: true

 

#2 glide.smtp.defer_retry_ids: Specifies the comma-separated list of SMTP error codes that force the instance to resend email.

Default value: 421,450,451,452

421 -    <domain> Service not available, closing transmission channel-

450 -    Requested mail action not taken: mailbox unavailable- e.g. SMTP server could not access a mailbox to deliver your message

451 -    Requested action aborted: local error in processing- e.g. This error is usually SMTP relaying service from too many messages.

452 -    Requested action not taken: insufficient system storage-

 

#3 glide.smtp.fail_message_ids: Specifies the comma-separated list of SMTP error codes that prevent the instance from resending email.

Default value: 500,501,502,503,504,550,551,552,553,554

 

500 -     Syntax error, command unrecognised - e.g. Your antivirus/firewall interfering with incoming and/or outgoing SMTP communications

501 -     Syntax error in parameters or arguments - e.g. Invalid email addresses or an invalid domain name recipient. Error can indicate bad connection

502 -     Command not implemented

503 -     Bad sequence of commands - e.g. Error, particularly if repeated, indicates bad connection. Verify authentication settings

504 -     Command parameter not implemented

551 -     User not local; please try <forward path>

552 -     Requested mail action aborted: exceeded storage allocation - e.g. The recipient’s mailbox has reached its maximum allowed size.

553 -     Requested action not taken: mailbox name not allowed - e.g. Invalid email address.

554 -     Transaction failed - e.g. when its anti-spam firewall does not like the sender’s email address, or the sender’s IP address, or the sender’s ISP server

 

#4 glide.email.smtp.max_recipients: Specifies the maximum number of recipients the instance can list in the To: line for a single email notification. Notifications that would exceed this limit instead create duplicate email notifications addressed to a subset of the recipient list. Each email notification has the same maximum number of recipients. Default value: 100. This is ONLY valid for recipients created by the email notification system (it does not include emails added by scripts or email client).

 

On each duplicated email, email logs will show:

InformationEmail with 4 recipients is split into 4 separate emails because property glide.email.smtp.max_recipients=1 (Email 4 of 4)

 

On our example, the final emails look as follow:

email properties.jpg

 

#5 glide.email.smtp.max_send: Specifies how many emails to send through each new SMTP connection. The instance establishes a new SMTP connection if there are more emails to send than the specified value. Default value: 100

 

#4 and #5 can be used to set email throttling. e.g 100 email with 50 recipients max each vs 100 emails with 1 recipient max each. It completely depend on your target emails and email capacity of the target exchanges.

 

In a nutshell, plan your email delivery, set the resend properties accordingly with a reasonable to email throttling to protect system performance.

 

6. Ensure the target email servers have white-listed the relay SMTP server.

If your SMTP server is not white-listed, spam system could penalize it by adding transaction time to the relay or delaying delivery. This is VERY common. If the number of emails bounced back or the delivery time increases after bulk emails, check the target email address SMTP servers have white-listed the relay SMTP server on your instance. If you are using our SMTP relay system, ServiceNow highly recommends that you configure your target mail exchange systems to retrieve our Sender Policy Framework (SPF) records dynamically, to white-list our Email servers. See Allowing email delivery from ServiceNow to your mail servers for more information.

 

 

 

 

 

A final word. Less is better. Less email recipients means you spend less, you need less storage on the target, you need a smaller processes and

you sent them faster. To make it less, you need to regularly validate your target emails even if it means donkey work. If you do not check regularly, as more invalid emails populate your system, the more time your email will take to get delivered.

 

Hope it helps. I have performed the tests on Geneva, using Chrome as browser.

 

Additional information can be found here: