Agent Guidance System Performance & Error Monitoring
For proactive issue handling, the Agent Guidance Application and its technology stack should be monitored at the level of the server environment and various log sources. There are no specific requirements for this from the Agent Guidance System itself, and it is at the customer's discretion as to the level and implementation of this monitoring.
Server Resource Monitoring
Its advised that all servers in the Agent Guidance technology stack (IIS, SQL, Redis) have their resource usage monitored, and alerts enabled for peak or sustained high loads. Typically, no unusual metrics are needed for this monitoring, and the standard server resource metrics are listed below for reference:
-
-
-
-
Disk read/write throughput
-
Network in/out throughput
The below sources are common targets for monitoring the Agent Guidance webserver(s). When multiple webservers are being used, each webserver should be individually monitored.
As the Agent Guidance application runs as an IIS website, requests made to the application are stored within the IIS logs. IIS logs are enabled by default, but they may have been disabled or reconfigured. For details on how IIS logging works and any configurations options, please refer to
appropriate Microsoft documentation.
A common reason for checking the IIS logs can be to help investigate performance-related issues reported by users. There are several factors that can affect a user’s experience of application performance, including server load and network delays. By default, IIS logs capture each request's total processing time in milliseconds from when the Microsoft IIS server receives the request to when the response is sent by the server. This time excludes any network delays, and so provides an initial view of system response performance.
Agent Guidance Website Logs
Each Agent Guidance website outputs its core logging stream to a series of log files named
web.log within its installed
\\data\logs\ directory. The default configuration includes events across the entire application at the
Error level and specific inclusion for login and licensing matters at the
Warn level, and will generate up to ten numbered log files before the oldest file is discarded. Details on configuring this logging stream can be found in the
Agent Guidance Logging knowledge base article.
Agent Guidance Service and Plugin Logs
Each Agent Guidance Service and Plugin outputs its logging stream to a series of log files named for the individual Service or Plugin, e.g.
CSEFMessageProcessing.log. The default configuration includes events across all Services and Plugins at the
Error level, and will generate up to ten numbered log files (for each Service or Plugin) before the oldest file (for each Service or Plugin) is discarded. Details on configuring this logging stream can be found in the
Agent Guidance Logging knowledge base article.
If an error occurs that isn't handled by the Agent Guidance application or the relevant Service or Plugin, then it may still be captured in the relevant webserver's Event Viewer. The exact section within Event Viewer will depend upon the context of the failing module: it will either be under the Application and Services Logs > Intelligent Agent log folder, or it will be under the generic Windows Logs > Application or Windows Logs > System log folders.
Redis can be configured to set the logging level/verbosity at the desired level. Details can be found in the self-documented Redis configuration file in your Redis installation, as can be seen in this
example configuration file for Redis 6.2.
During periods of high load, Agent Guidance performance can deteriorate if bottlenecks within Microsoft SQL Server are encountered. To identify such a scenario, Microsoft SQL Server performance metrics should be monitored in addition to Microsoft SQL Server resource usage. Below are some advised key metrics to monitor within Microsoft SQL Server:
-
Deadlocks: deadlocks are when 2 SQL queries are mutually blocking each other, and one has to be terminated to allow the continuation of operation. The Agent Guidance System should never encounter this situation with default core-application activity, and should it occur it will likely lead to user errors and small data loss. Microsoft provide articles on
deadlocks and their monitoring.
-
Outbound Messages (tbl_message)
When an outbound Email or SMS message encounters a sending error, the error is logged in the Agent Guidance database against the message's entry within
tbl_message. Only the most recent error for a message will be maintained in this table; if the message is subsequently sent successfully, then the error message will remain logged but the status of the message will update to reflect its success. When each sending error occurs, the full error trace will be recorded to the
Message Processing Service logs, however, these don't include any context related to the specific message's content.
An example SQL query to select summary information for all currently failed messages from the tbl_message table in the Agent Guidance database.
|
SELECT
id,
(CASE [type] WHEN 1 THEN 'Email' WHEN 2 THEN 'SMS' END) AS messageType,
lastprocesseddate,
lastError
FROM tbl_message
WHERE status = 2
|
If you wish to monitor outbound messaging, or discuss the data available within the Agent Guidance database in your version, please consult your Agent Guidance vendor.
The SQL transactions being made by the Agent Guidance website to the Agent Guidance database can be monitored via
SQL Server Profiler. This will show the execution time of each request, and allow identifying any excessive requests for further investigation. A connection to the Agent Guidance database will be made for each request using the credentials defined in the Agent Guidance website's
web.config file, allowing filtering the query stream by user and database names.