Product and feature support¶
Supported products¶
VCS/Expressway¶
Log file format
Upload the full diagnostic archive collected under Maintenance -> Diagnostics -> Diagnostic logging. Check the “take tcpdump while logging” option on the web interface to also include pcap for extended feature set of the tool. We rely on xconf and xstat (txt and xml) files to get general system information, but also some vital information required for successful main log analysis (loggingsnapshot file). We expect 1 archive per server.
Note
Default logs levels are fine for most of the analysis. Additional XMPP communication can be displayed with Mobile and Remote Access login message flow when “developer.xcp.jabber” is set to DEBUG under Maintenance -> Diagnostics -> Advanced -> Support Log configuration.
Supported features
SIP/H.323 calls
RTP streams (pcap)
SIP registrations
MRA sessions
DNS
XMPP
STUN
TCP/UDP streams (pcap)
CUCM (Cisco Unified Communications Manager)¶
Log file format
Please make sure that the data is correctly prepared as per the following guidelines:
Use RTMT (Real-time monitoring tool) to download the Cisco CallManager service traces from ALL nodes in the cluster.
Traces should be set to detailed which is the default setting from CUCM 9.X unless configured otherwise.
Zip all the files and folders that RTMT created in one archive and upload to the tool.
Leave original filenames(SDL*/calllogs*) and structure of directories created by RTMT.
If there are multiple clusters, upload one archive per cluster.
When troubleshooting CTI-related issues, the CTI Manager traces should be collected together with the CCM traces.
For more information about setting the traces to “detailed” level and collecting them please refer to this document.
Supported features/scenarios
Call flows with different signalling protocols (H323/SIP/MGCP/SCCP/CTI)
Call features: Transfer/Forward, Hold, Recording(Gateway-recording, BIB recording)
Media resource allocation(MTP/Xcoder,etc.)
Mobility(SNR)
Note
Other call scenarios/features should work as well but are subject to the continuous testing.
Analysis
Due to the nature and size of CCM/CTI SDL traces, the data filtering is required and the full analysis is applied on per call basis only. Several steps are executed during the analysis flow:
Extract the archive and map SDL*/calllogs* for the same host
Find calls that are within SDL timelines based on SDLs/calllogs(UCM calls overview).
After selecting the call(manual step), filter only relevant lines
Apply annotations, ladder diagram based on the filtered data
Allow ability to download the filtered data(which includes related raw SDL files).
Note
Please note that the UCM calls overview page may not provide full information about a call, for instance Disconnect reason, CI B, Call-Id can be missing or multiple entries can be present. After selecting the call, all call legs should be linked and complete information is available in Call Details
TC/CE endpoints¶
Log file format
Upload log archive collected over web interface under Diagnostics -> Log Files. For more information about collecting the logs please refer to this document.
Supported features
SIP calls
DNS
SIP registration
HTTP
CMS (Meeting server)¶
Log file format
Upload logbundle.tar.gz file collected over SFTP from your CMS server. Alternatively, upload log archive collected using the CMS log collector tool found here.
Supported features
SIP calls
Jabber client¶
Log file format
Upload the problem report (PRT) generated by the Jabber client. All jabber platforms are supported from version 11.6.
Supported features
SIP calls and registration
DNS requests
HTTP requests
Jabber version CLI script¶
Currently there is no option in IM&P server user interface to list versions of Jabber clients logged into IM&P server at any give time. This tool uses data dump from server CLI to generate the report.
Log file format
There are 2 txt files required, each of which has the following format:
line 1: CLI query
line 2+: the output of CLI query
and contains one of the 2 queries:
run sql select pkid ,userid from enduser (ran on CUCM)
run pe sql ttlogin select * from clientsessions (ran on IM&P)
Example file content
query 1
admin:run sql select pkid ,userid from enduser
pkid userid
==================================== ===============================================
ae003243-0632-4b79-a042-dcbc82b68485 vsidimak@ciscotac.net
query 2
admin:run pe sql ttlogin select * from clientsessions
sqlRv(t) sqlstmt(select * from clientsessions;)
***result set start***
"count(1), success(t)"
<msg><type>TTLOGIN</type><table>clientsessions</table><action>Q</action><time>0</time><old><1>4f40c76b-148d-47cc-a7d4-5d18a83c479c</1><5>1</5><6>11.8.4.52954</6><2>5e029d96-bb8d-8983-5af4-b6d70db3fd33</2><4>null</4><3>4adc1203-0eca-14cd-b5b8-3a78f76cbce6</3><0>00000001:59a31f79:000095bf:00000006</0></old></msg>
Usage tips
Since the output of queries might be large, make sure you increase the ‘Lines of scroll back’ under ‘Window’ section in Putty.
Run the query on each node in the IM&P cluster and copy the output into separate files. You will have to run the tool for each IM&P individually, or alternatively merge the outputs from different servers for the same query into 1 file, following the format above.
Upload the files to the tool, which will automatically detect the product types as IMP_JABBER_VERSION_STAT_1 (query 1) and IMP_JABBER_VERSION_STAT_2 (query 2) respectively.
Select both files and hit “Run analysis”.
Note
The CLI outputs only provide Real-time Jabber statistics and not historical.
Sample tool output
Conductor¶
Log file format Upload the full diagnostic archive collected under Maintenance -> Diagnostics -> Diagnostic logging. Check the “take tcpdump while logging” option on the web interface to also include pcap for extended feature set of the tool. We rely on xconf and xstat (txt and xml) files to get general system information, but also some vital information required for successful main log analysis (loggingsnapshot file).
Supported features
SIP calls
Conferences
RTP streams (pcap)
TCP/UDP streams (pcap)
DNS (pcap)
API/HTTP
CVP (Customer Voice Portal)¶
Log file format
Upload the CVP Call Server logs collected from location C:\Cisco\CVP\logs. The log file names should not be changed. In order to analyze SIP calls from CVP, SipLibrary logging should be enabled on CVP CS server. Please refer to this instruction.
Supported features
SIP calls
GED-125 (communication between CVP and ICM)
ECE (Enterprise Chat and Email)¶
Log file format
The system decodes and parses the ECE logs, which could be collected from the ECE File server. In order to analyze the EAAS/EAMS(Listener) services the trace level has to be INFO. The log file names should not be changed.
Supported features
EAAS (MR events and internal EAAS messages)
EAMS/Listener (ARM events)
Connections/disconnections statistics (ARM/MR)
Dumpcfg Utility¶
Log file format
Upload the dumpcfg utility logs in .txt format. Logs can be collected either from the command line or from Diagnostic Portico -> GetConfigurationCategory of a Logger.
Supported features
List of Transactions (configuration messages)
List of Tables (that had a configuration change)
PerfMon for Voice Operating System¶
Log file format
Upload the archive that contains perfmon logs(Cisco RIS Data Collector PerfMonLog) collected using the RTMT tool from your Voice Operating System (CUCM,CUC,UCCX, etc.) server. The log bundle should contain PerfMon_*.csv files which record performance counters like CPUTime, VmSize(Virtual Memory Size), etc. This tool visualizes specific counters allowing you to easily spot issues.
Supported features
Before analysis, you will need to select one of the available templates:
Performance preview template (CPU, memory, IOWait)
Disk/storage troubleshooting template
CPU template
CUCM call activity template
DB replication
Telepresence Server¶
Log file format
Upload the .zip system logs file which can be downloaded at TPS Status -> System logs Download file
Supported features
SIP calls
MCU¶
Log file format
Upload a .zip file which contains the event logs and the protocol logs.
Supported features
SIP calls
H.323 calls
BROADWORKS (Cisco-BroadWorks Application Server)¶
Log file format
The Cisco-BroadWorks Call Engine application supports the XSLog from the Application Server (AS/XS). The XSLogs are located on the AS under:
/var/broadworks/logs/appserver
bwadmin@as1-r22.bstac.net$ ls -ltr
…
-rw-rw-r--. 1 bworks bwadmin 11559805 Aug 21 09:07 XSLog2019.08.21-06.06.32.txt
XSLog file format is always XSLogYYYY.MM.DD-hh.mm.ss.txt representing the time stamp of the file creation. Please make sure that the XSLog has the correct logging level in order for the Cisco-BroadWorks Call Engine to run correctly :
1. Call Correlation Id (External Tracking Id) Call Correlation Id (Extternal Tracking Id) must be enabled on the AS:
AS_CLI/System/CallP/CallCorrelation> get
callCorrelationIDFormat = clusterUnique OR networkUnique
AS_CLI/Interface/SIP>
…
sendCallCorrelationIDAccess = true
sendCallCorrelationIDNetwork = true
This is required and this is to ensure that different call legs can be linked together representing the same calls.
2. XSLog input channels level The minimum XSLog level for the Cisco-BroadWorks Call engine to work adequately is the following:
AS_CLI/Applications/ExecutionAndProvisioning/XS/Logging/InputChannels> get
Name Enabled Severity
=====================================================================
...
CallP true Info
...
Sip true Info
SipMedia true Info
MGCP true Info
...
OCI-C true Info
...
Diameter true Info
...
MediaCr true Info
...
Note that higher level of logging (e.g. CallP at Debug or FieldDebug) will provide more functionality for Digital Signature (DSig).
Supported features/scenarios
- Call flows with different signaling protocols:
SIP
MGCP
Diameter (Billing)
MediaCr (CFW)
OCI-C
Call features:
Originating calls
Terminating calls
SCAP (Shared Call Appearance)
SimRing (Simultaneous Ringing)
Call Recording
Etc..
Analysis
Unlike other Call Engine product, the Cisco-BroadWorks Call engine uses CallHalfs and Global CallHalfs (CallHalfs without the sub-session i.e. :0, :1, etc) values as the IP addresses for origination and destination.
The CallHalfs are used for messaging to/from a node other than the AppServer. The Global CallHalfs are used on the AppServer linking all the sub-sessions (i.e. CallHallf-XXXXXXX:0, CallHallf-XXXXXXX:1). This will be the case for example when a call is requesting the NS to be routed:
The device (10.192.83.123-085) is originating a call to the AS. The originating callhalf is callhalf-4531085:0 and the AppServer global callhalf is callhalf-4531085. This call implies a routing request to NS-10.8.5.103. The callhalf for this request is callhalf-4531085:1 and is still attached to global callhalf-4531085.
Nodes representations
Whenever possible, the Call engine will display a node with its specific roles (AppServerLegX, NS-IP, MS-IP). In general, it will try to create a new AppServerLegX so the box will represent only one incoming call (call leg) for the user. AppServerLegX still represents the same AS but it is a new call leg in terms of call processing.
Example: Call from UserA/EntA to UserB/EntB. User has call forward to C.
From the AS perspective, calling another user hosted on the same AS but not within the same enterprise is essentially seen as follow in terms of calls legs:
A origination: A to network/pstn
B termination: Network/pstn to B
The call legs are not linked/related to each other. However, the Call Engine is able to link these call legs together as represented above.
Device/Endpoint nodes:
- These nodes represents a device (EndPoint) originating or terminating a call.
Top line: Real IP + XXX where the XXX are the last three digits of the global callhalf. This is to ensure that device/endpoint are unique within the call flow. Otherwise originating and terminating device/endpoint coming/going from/to the same source/destination (e.g. both device/endpoint behind the same SBC) will appear as a single box with two callhalfs.
Middle line: Whenever possible, this will be the user’s DN.
Bottom line: CallHalf associated with this call leg.
AppServerLegX:
- These nodes represent the AS (Application Server) callhalfs call legs associated with a call
Top line: AppServerLegX. As mentioned, the Call engine will try, whenever possible, to create a single box per call originating or terminating at a user.
Middle line: The ORIG or TERM sessions.
Bottom line: Global CallHalf associated with the call legs.
NS node:
- These nodes represent the NS (Network Server) boxes.
Top line: NS + Real IP (NS-IP)
Middle line: Request type ivr, cfw-media, Routing.
Bottom line: CallHalf associated with the request
MS node:
- These nodes represent the MS (Media Server) boxes.
Top line: MS + Real IP (NS-IP)
Middle line: Request type ivr, cfw-media, Routing.
Bottom line: CallHalf associated with the request
Note
CallHalf for CFW protocol requests are prepended with cfw.
ASXintraASY node:
- These nodes are not physical nodes. They represents AS connecting calls to itself. They are created when the CallId of an outgoing INVITE is BW type and the CallId of an INVITE is the same as well as the destination (outgoing) and the origination (incoming) are the same. This is connecting AppServerLegX to AppServerLegY but this is not a physical node. So case where the INVITE is sent to secondary AS and back to primary AS will also be seen as ASXintraASY and in this case will represent a physical node.
Top line: ASXintraASY
Middle line: destination and originating DNs
Bottom line: CallHalf associated with the request
ASXpstnASY node:
- These nodes are physical nodes. They are middle points between two (or more) AppServerLegX. They are created when the CallId of an outgoing INVITE is BW type and the CallId of an INVITE is the same as well as the destination (outgoing) and the origination (incoming) IPs are different. They will also be created when the External Tracking Id is carried into the SIP messages so the terminating AppServerLegY is linked to the originating AppServerLegX (IMS case).
Top line: ASXpstnASY
Middle line: destination and originating DNs
Bottom line: CallHalf associated with the request
Diameter node:
- These nodes represent the Diameter boxes.
Top line: Diameter + Real IP (NS-IP)
Middle line: Billing.
Bottom line: CallHalf associated with the request
Note
CallHalf for Diameter protocol requests are prepended with “diam”.
IMRN node:
- These nodes represent SIP IMRN allocation and released for IMRN (BW Mobility) origination. External Tracking Ids are different for the allocation and released. The Call engine is able to link them.
Top line: IMRN
Middle line: BW Mobile Originator
Bottom line: CallHalf associated with the request
PCAP¶
Log file format
Upload pcap file collected after stopping the trace. Generally the steps given below are followed to collect the trace.
Select the interface to capture the trace
Apply filter (if any)
Start the PCAP trace
Reproduce the issue
Stop the PCAP trace
Export the collected PCAP trace
The exported file will have .pcap file extension.
Supported features
RTP streams
DNS
TCP/UDP streams
Sample tool output
For each source and destination, we can also view the ladder diagram as shown below
Immersive¶
Log file format
Upload the tar file for analysis. The file contains the following
Bootlogs
Logs
Core data
State data
IPCC_RTR¶
Log file format
Upload IPCC RTR call log file.
Supported features
List of RTR calls
RTR call details
RTR call details can be viewed by clicking on a row in the call table. Call details include messages and call leg info.
Sample tool output
Sample RTR call list
Call details include messages and call leg info.
IPCC_AGENT_PIM¶
Log file format
Upload IPCC Agent PIM call log file.
Supported features
List of Agent PIM calls
Agent PIM call details
Agent PIMcall details can be viewed by clicking on a row in the call table. Call details show messages of following types.
Established
RTPStarted
RTPStopped
ConnectionCleared
CallCleared
Sample tool output
Sample Agent PIM call list
Call details show messages as shown in the below figure
IPCC_CTISRV¶
Log file format
Upload IPCC CTI log file.
Supported features
List of IPCC CTI calls
List of IPCC CTI agents
List of IPCC CTI events
CTI call details
CTI agent details
Message and Message body in CTI events
Sample tool output
Sample CTI Event list
Call details show messages as shown in the below figure
IPCC_VRUPIM(GED-125)¶
Log file format
Upload IPCC VRUPIM log file.
Supported features
List of VRU calls
VRU call details
Sample tool output
VRU call details can be viewed by clicking on a row in the call table. Call details include messages and call leg info.
Sample VRU calls list
Call details show messages as shown in the below figure
WEBEXAPP(Webex Teams)¶
Log file format
Upload WEBEXAPP (Webex teams) log file. To download the webex app (webex teams) log file refer this document.
To restore PII in log file, we should upload ced.dat file associated with the collected log.
Supported features
HTTP queries
SIP Registration
DNS queries
Annotations
Sample tool output
SIP Messges can be viewed by clicking on a row in the SIP registration table.
Sample SIP Registration table
SIP Message details will be displayed as shown in the below snapshot
