Diagnosing and remedying stuck event processes

[kim_purcell]

If you are having trouble with processing events, such as inbound or outbound emails not processing or a scheduled events showing the past date/time, this may be because your scheduled events or other events are stuck in the queue as a result of an error processing one of the events.

Using the Diagnostics page to troubleshoot a stuck event:

1. Navigate to System Diagnostics > Diagnostics Page.
2. In the system overview section, check the number of events pending. This number should not exceed 1000. If it does, this likely means a large number of events are stuck in the instance. The system overview below shows only one event pending:
   
   

See also Troubleshooting email notifications - Send to the Event Creator if you are experiencing trouble confirming that notifications from an event are being sent.

Check the list of system events to identify what's not processing:

1. In the application navigator, enter sysevent.list to display all the events.
2. Check the created and processed time for each event (this shouldn't be more than 30-40 seconds).
3. Add the Claimed By column to the list view. This displays the node where the events are processed (and helps you check for a particular node if there are a large number of active nodes on an instance.)

4. Go to stats.do of that particular node and check if the events process is stuck:
   1. In the stats.do, open the link for events process worker page. This displays the stacktrace for eventprocess.java.
5. If your scheduled event process is stuck, you should kill the event process in the Claimed By node:
   1. Search for All Active Transaction in the left navigation filter.
   2. Select the stuck events process row and from the Actions on the Selected Rows drop-down.
   3. Select Kill.

You can analyze the worker stack from the schedule events process to identify how the events got stuck. Below is an example of the stack trace where an event is stuck because of a business rule:

To fix an event stuck on the oldest unprocessed event:

1. In the navigation filter, enter sysevent.list to display all events.
2. Filter by column [Processed] [is] [empty]. This lists all the unprocessed events.
   
   
   
   
3. Sort the events by the oldest date/time at the top.
4. Mark the State field of the oldest event as Error. This helps process the other unprocessed events.
5. After this is fixed, change the State field of the oldest event to ready.
   
   

Show the correct time on a scheduled event process:

If the scheduled events process shows the past date/time, this indicates errors in the events process schedule.

1. In the navigation filter, enter sys_trigger.list to view all schedules.
2. Search for events process.
3. Check if the Next action field is showing as future date/time and the State field is not queued or error.
   
   
   
   
4. If the next action date/time is showing the past date/time and error as seen above, click Execute Now to execute the event. This should execute the event and put the future date/time in the Next action field.
5. If the State field is showing as queued or error, change the field to ready and Save the record.

Working with notifications and your scheduled events can be tricky. My best advice to share with you regarding preventing events from getting stuck is to check back regularly to see if events are getting processed. If you have checked your diagnostics page and your list of system events, check the knowledge base and the community to see if others have experienced a similar situation and have found a way to work around it.