Discovering Eugene's “Special” Application

[Charles C_ Ruff]

Hello Everyone,

Here’s a quick story about using ServiceNow’s Pattern Designer and Process Classifier to show how we can customize Discovery to detect a custom application and add it to the CMDB.

Disclaimer: This is a fictional story with fake characters but a real solution.  If your name is *Eugene, this story is not necessarily about you, but if the shoe fits…

I need to discover a custom application that is nowhere else except in my company… but my buddy (he’s not really my buddy, but we all call him that because he can’t be fired since he’s the CIO’s nephew), Eugene wrote it, AND NOTHING IS STANDARD!  I won’t get into the details of how his application was submitted as an “actual production solution” to a problem we weren’t sure we really had. We’ll just call the app “Eugene’s Special Application,” as we do in all of our management meetings and every side conversation about server performance.  

I must begin with some foundational questions, but at the root, how do I discover this thing and get a better understanding of the impact it has on my organization?  While I’m initially only interested in discovering this application, understanding the reach it has within my enterprise is the next step. I’ve read "How to write a new Service Mapping pattern - Where's the starting point?” but this application is unique, and I only want to know where it is right now (mapping will come later).  I asked Eugene for some help with identifying the key items needed to discover his application and I’m not sure if he was just messing with me or he actually built it this way:

Me: “Eugene, can you describe what your application does?”
Eugene: “It does some stuff to help me with my weekly status reports.  Basically, it pulls certain inventory data from a server and feeds it into a custom database I have running on my local system.”
My “security-based" spidey-sense is getting alerted.
Me: “What type of software is this?”
Eugene: “Well, I like to call it 'Eugene’s status report helper,' it’s part database, part file server, and part music finder, haha!”
Wait… did he build a Napster clone?
Me: “Which platforms does it run on?”
Eugene: “Windows only, I don’t understand Linux”
It’s a good thing too… who knows what kind of nightmare he could have unleashed.
Me: “Can you tell me any identifying attributes of it, like ports it uses?”
Eugene: “That’s the cool part, it runs on port 1337, since I’m soooo leet. But if it crashes, I couldn’t figure out how to use that port again, so I just pick a random one.”
Eugene, seriously…
Me: “What’s the name of the main executable?”
Eugene: “OneEug.exe.”
Of course it is...
Me: “Now that I know the name, I must ask, how big is it, or better yet, what kind of resources does it consume?”
Eugene: “That depends on the type of server it’s running on. I usually don’t see any issues with it running on a box with lots of memory and fast CPU’s.  It has a memory leak which I’ve struggled to pinpoint, so I just cycle the service when it starts to consume too much memory. That’s why it switches the port it’s using sometimes.”
I’m struggling to contain my absolute lack of surprise and my eyes from rolling out of my head.
Me: “Are there any additional configuration files used to run the application?”
Eugene: “Yes, I use a special file called TwoEugConf.conf… I named it that since I have ‘conf in the name and the extension, get it?”
<insert facepalm meme here>
Me: “When the application is installed do you place it in the same file location each time or is that randomized too?”
Eugene: “I can randomize the installation location?!? Oh man, I’ll have to put that into my next version ‘SoEug.exe’”
What have I done, did I just give him an idea?!?
Eugene: “It’s not installed, it’s copied from my system to C:\ProgramFiles\uGene\OneEug.exe then I use some PowerShell to start it.”
Me: “Does it connect to any other systems or services?”
Eugene: “It runs in a distributed manner across all other nodes and automatically installs itself on another server if the load gets too heavy.  I maintain a list of servers on my local system called OneEugPlayground.conf that each one of my apps reads to know which server to install on next.”
He’s created a worm and somehow convinced management [his uncle] that it’s a needed application.
Me: “Do you know which servers it is running on?”
Eugene: “No, most of the time it’s only on one or two that I know of; win2012r2 and win1.  Although, I saw it running on almost a hundred boxes when we were crunching our year end financials… those applications really slow mine down.”

As I pick myself up off the floor, I think I have enough information needed to discover his application, and maybe even begin to take some steps to reduce the impact it has on the overall performance of OUR ENTIRE INFRASTRUCTURE! First, I need to ensure I can find it and report on where it is.  I’ll attempt to remove it later.  

Here’s what I know:
Executable name: OneEug.exe (and potentially SoEug.exe if he’s allowed to keep hacking).
Possible Port Number: 1337 (potentially any)
Runs on: Windows (currently only two systems called win2012r2 and win1)
Installed in: C:\ProgramFiles\uGene\
Has a config file: C:\ProgramFiles\uGene\TwoEugConf.conf

I want to jump in and create a discovery pattern for this application, but I will need to run a few tests and teach the system how to find what I’m looking for first.  

What’s needed:

• Extend the Application (cmdb_ci_appl) table by creating a child with all of the attributes I want to find in my discovery.
• Modify the form to show the data I want to see.
• Create a CI Classification for the process I’m discovering and map it to the pattern
• Create a pattern to find all the data I can conjure out of his application.
• Run a discovery!

I know his application is running on two of our Windows boxes, so I will run a Quick Discovery on one of them to help me build everything.

In the Filter navigator, search for “Discovery Schedules” and select “Discovery Schedules” under the “Discovery” application, then click “Quick Discovery” in the navigation bar. I used the IP address of the Windows server I know this application is on, and I specified a MID server that can discover on that subnet.

Open the Discovery Status record and wait until the discovery has completed (you will see “Completed” in the “State” field). Click the CMDB CI listed under the “Devices” section, below "Related links.”

Scroll down to “Related links > Running Processes” and find the application we’re looking for: “OneEug.exe”

Click the PID (Process ID) and it will show the port this application is currently listening on and any additional parameters which may be important for our targeting purposes.  I wrote down the process ID for helping me debug the pattern a little later.

I need to extend the cmdb_ci_appl (Application) table before I can make the process classification.

In the CI Class Manager (Use Filter navigator), click the “Hierarchy” button and search for Application.  Selecting the three lines to the right of “Application” will present you with the ability to add a child class.

I’ll name the new class “Eugene” and set all the attributes I’m looking for… but I don’t really know what they are yet, so I’ll have to come back to this part a little later. Having the table (u_cmdb_ci_eugene) created will help me put all the pieces together.

Since Eugene mentioned his inability to maintain the same port after the application recycles, I’ll need to create a CI Classification for the process.

In Discovery Definition > CI Classification > Processes, click “New” and fill in the details.

• Name: Eugene Special
• Table: Eugene
• Relation type: Runs on::Runs
• Condition: Parameters - contains - OneEug.exe or SoEug.exe (using the details in the Parameters section from the previous discovery)

Next, in the Filter navigator, search for “Pattern Designer” and the “Discovery Patterns” table displays a list of out of the box patterns. In the London release of ServiceNow (and with all the plugins I have enabled), there are 395 defined patterns… none of which will find Eugene’s worm. Select “New” and a new window or tab is opened which displays a new record for the pattern:

Enter the pattern name. 
Select the pattern type from the list
    Since I’ve chosen “Application" – a pattern for performing the top-down and horizontal discovery of an application, I can use this pattern in the future for a full service map.
Select the CI type for this pattern.
    In this case, I’m sticking with “Eugene (u_cmdb_ci_eugene)"
Select “New” in the “Identification Section” at the bottom of the form and it will open a new modal dialogue window:
Enter the Name of the Identification Section
Specify the Entry Point Types
    In this case, I chose All Endpoints (just in case I run into anything weird)
Select the Find Process Strategy
    I used “None” as there is really no way to know if the application process is running on a system or not (he may have copied the file over but didn’t start it).
And last, the Order
    This is used if you have more than one Identification section
Click “Save” in the top right corner of the window and the pattern steps will be displayed.  This is where we begin to build the [seemingly] impossible steps needed to finally find this application.

I click the “Discovering Eugene’s Worm” link under the Identification Section to display the pattern designer.
In the left column you can create, edit, or view steps that the pattern will take to create or update a CI.
In the middle column you can select and define operations for gathering data and creating variables.
In the right column you can view and interact with temporary variables after each pattern step completes an action.

In the left column, I’m going to rename the first step to "Check process name to match OneEug.exe."
In the middle column, there are currently 28 operations to choose from, ranging between Structure, Query, Parse, Table Operations, and Others.

To start, I’m selecting the “match” operation because the one thing I know to be true is the name of the executable (I swear I heard Eugene gloating about the name of this app… not sure what he’s so proud of). I selected “Any” as the “Meet” property because I want to future proof this pattern for when Eugene writes version 2.  I will use a variable in the right column to fill in the details of the match operation.  Expand “process” under the “Temporary Variables” section and click+drag the “executable” label to the first text box in the middle column below “Meet," “Contains” as the operator, and “OneEug.exe” as the item I’m searching for.  Click the “+” to add another match operation and repeat the previous steps for Eugene’s next version; “SoEug.exe.

Before I move on, I’m going to cheat a little and get some help from the debugger.  Click the “Debug Mode” button at the top and switch the “Debug type” to “Horizontal,” add the IP and the PID (Process ID written down previously) and click “Connect."

Next, I want to add a new step below the first one (“Get Executable Information”) and utilize the “Parse Variable” operation. This is where the debugger can really come in handy. In the right column, under “Temporary Variables” I expand “process” and drag the “executable” entry to the “1. Enter Variable” text box and set “2. Define Parsing” to “Delimited Text.”  By clicking the “Retrieve” button, I’m able to capture only the name of the current executable in use (in case Eugene is allowed to make a version 2, SoEug.exe, we can see which one is in use upon discovery).  The executable name is displayed in the “Output” section where I can click+drag my mouse over the portion of the variable I want to capture (_Name_ is displayed but I can change that to whatever I want the variable name to be); “exec_name” is what I used.  The Delimiters and Positions fields are populated automatically.

I’ll add another step below (“Parse TwoEugConf.conf”) using the “Parse File” operation. This time, I’ll drag the “executableDir” Temporary Variable to the “1. Select File” text box, add a “+” at the end and add the name of the conf file. Ensure “Delimited Text” is selected for the parsing strategy again, then click the “Retrieve File Content” box.  The “Output” field is populated with the contents of the conf file.  I chose “list” instead of “table” in the “Variables” section and named each of the variables in the format I’ll use in the attributes section of the Eugene table (when I get back to it). Utilizing the same click+drag method from the previous step, I’m able to create variables for each section in the conf file.  I also specified the line separator as being “CARRIAGE RETURN,” and changed the name of the temporary table to “eugenes_conf" (delimiters and positions were filled in automatically).

There are a few things that stand out to me upon importing this file; <excessive sarcasm>Eugene is clearly strong with his “Security-foo” considering his username and password are written in clear text</excessive sarcasm>, He thinks way too highly of himself based on his password (EugeneCodesLikeaBoss… in “hacker type” really?!?), I should have built this discovery process ages ago.

After wiping my tears away (a mixture of horror and humor), I want to get some additional information from this pattern to better understand the impact this application has on the server it’s running on.  I’ll add a “WMI Query” operation below the 3rd step called “Get process information.”  Since I’m still in debug mode, I can just click the “Get Data” button next to the “1. Namespace” dropdown and select “\root\CIMV2.”  This populates the entire Windows Management Instrumentation database but for my project, all I need is the Win32_Process table and a few fields: MaximumWorkingSetSize, PageFaults, ThreadCount, and VirtualSize (https://docs.microsoft.com/en-us/windows/desktop/cimwin32prov/win32-process).  After I select the fields I want to pull into the table (5. Target Table - "$process_info"), I need to add a condition clause “Name like ‘OneEug%’ OR Name like ‘SoEug%’” to ensure the full statement is created appropriately:

"SELECT MaximumWorkingSetSize,PageFaults,ThreadCount,VirtualSize FROM Win32_Process WHERE " + "Name like 'OneEug%' OR Name like 'SoEug%’"

Next, I want to ensure this data is viewable on the form I create for the Eugene table (so when someone drills into the results, they can see this information… and heckle Eugene).  I need to go back to the CI Class Manager (use the filter navigator > “Hierarchy” button, search for “Eugene”), click the “Attributes” section, “Added” button.  Here I can add all of the attributes I want captured.  Quite simply, click the words “Insert a new row…” and type each column label, which will automatically populate the rest of the fields (make any adjustments based on the data we captured earlier, such as “Max length” and column name if it is different from the pattern).

Next, I go back to the pattern, right click the “4. Get process information” step and choose “Create Shared Library” in case Eugene makes more applications, I can re-use this library. Name it “Normalize Process Information” and I’ll add the four fields from the previous step. I’ll use a “Set Parameter Value” operation for each, drag the “process_info” variable from the Temporary Variables section in the right column to the “Value” box and add a “[“ to the end to auto-complete the associated values for each entry (MaximumWorkingSetSize, PageFaults, ThreadCount, and VirtualSize).  Next, I search for each related column name (based on the attributes I just added to the Eugene CI Class) listed above the Temporary Variables and drag the attribute to the “Name” textbox (repeated for each entry).

I want to see which executable name is being used (in case he makes a version 2) and change the name of the application that is stored in the record, so I’ll add two more “Set Parameter Value” operations steps.  The first I’ll call “Add Executable” and following the same steps as I did for the shared library, drag the “u_exe_name” attribute from the Eugene table to the “Name” textbox and drag the “exec_name” variable from the Temporary Variables section to the “Value” textbox.

And finally, repeat the actions from the last step for “Set application name for CI.”

I need to go back to the Process Classification to update the pattern being used.  In Filter navigator, search for “Processes” under “Discovery Definition > CI Classification” and select “Eugene Special.”  Under Related Links, in the “Triggers probes” section, click “Edit” and search for “Horizontal Pattern” to add it to the Triggers probes list (click “save”).

Next, add the pattern.  Click “HorizontalDiscoveryProbe-Horizontal Patt” and select “Eugene’s Special Application” for the “Pattern” field and click “Update.”

Once the record is updated, it’s time to run another discovery.  From here I modify the form to show what I want to see.  Back to “Discovery Schedules,” I kick off a quick discovery and watch the “Discovery Log” under “Related Links” to watch the magic.  Then I drill into the Pattern Log (click the “Here” link in the log entry for “Short Message”).

This shows that all my steps have completed successfully!

Going back to the “Devices” tab under “Related Links,” I can select the CMDB CI to see the details (and make changes to the form if I so desire).

Under the “Additional Actions” flyout, select Configure > Form Design, I set it up how I like it.    

 

Which results in a form we can use to heckle Eugene at our next meeting (which is technically an HR violation, so I’ll only really heckle to myself 😛. Thanks Eugene).

And the dependency view!

I hope you've enjoyed this little story about finding a rogue application using ServiceNow’s Pattern Designer, Process Classifier and Discovery.  Who knows, one day it may help you to reduce the impact when the "Eugene" of your organization is allowed to do dumb things.

* Thank you Matt M. for the inspiration from all your stories about Eugene.

[johnnyjava]

Fun stuff. Art imitating life?

[Ryan Lee2]

Great article Charles!    This is a great read for anyone wanting a crash course in Discovery Pattern creation.   

Now, when do we report Eugene to Security?!