Product and feature support

Supported products

VCS/Expressway

Log file format

Upload the full diagnostic archive collected under Maintenance -> Diagnostics -> Diagnostic logging. Check the “take tcpdump while logging” option on the web interface to also include pcap for extended feature set of the tool. We rely on xconf and xstat (txt and xml) files to get general system information, but also some vital information required for successful main log analysis (loggingsnapshot file). We expect 1 archive per server.

_images/expressway_diag_archive_content.png

Note

Default logs levels are fine for most of the analysis. Additional XMPP communication can be displayed with Mobile and Remote Access login message flow when “developer.xcp.jabber” is set to DEBUG under Maintenance -> Diagnostics -> Advanced -> Support Log configuration.

Supported features

  • SIP/H.323 calls

  • RTP streams (pcap)

  • SIP registrations

  • MRA sessions

  • DNS

  • XMPP

  • STUN

  • TCP/UDP streams (pcap)

CUCM (Cisco Unified Communications Manager)

Log file format

Please make sure that the data is correctly prepared as per the following guidelines:

  1. Use RTMT (Real-time monitoring tool) to download the Cisco CallManager service traces from ALL nodes in the cluster.

  2. Traces should be set to detailed which is the default setting from CUCM 9.X unless configured otherwise.

  3. Zip all the files and folders that RTMT created in one archive and upload to the tool.

  4. Leave original filenames(SDL*/calllogs*) and structure of directories created by RTMT.

  5. If there are multiple clusters, upload one archive per cluster.

  6. When troubleshooting CTI-related issues, the CTI Manager traces should be collected together with the CCM traces.

For more information about setting the traces to “detailed” level and collecting them please refer to this document.

Supported features/scenarios

  • Call flows with different signalling protocols (H323/SIP/MGCP/SCCP/CTI)

  • Call features: Transfer/Forward, Hold, Recording(Gateway-recording, BIB recording)

  • Media resource allocation(MTP/Xcoder,etc.)

  • Mobility(SNR)

Note

Other call scenarios/features should work as well but are subject to the continuous testing.

Analysis

Due to the nature and size of CCM/CTI SDL traces, the data filtering is required and the full analysis is applied on per call basis only. Several steps are executed during the analysis flow:

  • Extract the archive and map SDL*/calllogs* for the same host

  • Find calls that are within SDL timelines based on SDLs/calllogs(UCM calls overview).

  • After selecting the call(manual step), filter only relevant lines

  • Apply annotations, ladder diagram based on the filtered data

  • Allow ability to download the filtered data(which includes related raw SDL files).

Note

Please note that the UCM calls overview page may not provide full information about a call, for instance Disconnect reason, CI B, Call-Id can be missing or multiple entries can be present. After selecting the call, all call legs should be linked and complete information is available in Call Details

TC/CE endpoints

Log file format

Upload log archive collected over web interface under Diagnostics -> Log Files. For more information about collecting the logs please refer to this document.

Supported features

  • SIP calls

  • DNS

  • SIP registration

  • HTTP

CMS (Meeting server)

Log file format

Upload logbundle.tar.gz file collected over SFTP from your CMS server. Alternatively, upload log archive collected using the CMS log collector tool found here.

Supported features

  • SIP calls

Jabber client

Log file format

Upload the problem report (PRT) generated by the Jabber client. All jabber platforms are supported from version 11.6.

Supported features

  • SIP calls and registration

  • DNS requests

  • HTTP requests

Jabber version CLI script

Currently there is no option in IM&P server user interface to list versions of Jabber clients logged into IM&P server at any give time. This tool uses data dump from server CLI to generate the report.

Log file format

There are 2 txt files required, each of which has the following format:

  • line 1: CLI query

  • line 2+: the output of CLI query

and contains one of the 2 queries:

  • run sql select pkid ,userid from enduser (ran on CUCM)

  • run pe sql ttlogin select * from clientsessions (ran on IM&P)

Example file content

query 1

admin:run sql select pkid ,userid from enduser
pkid                                 userid
==================================== ===============================================
ae003243-0632-4b79-a042-dcbc82b68485 vsidimak@ciscotac.net

query 2

admin:run pe sql ttlogin select * from clientsessions
sqlRv(t) sqlstmt(select * from clientsessions;)
***result set start***
"count(1), success(t)"
<msg><type>TTLOGIN</type><table>clientsessions</table><action>Q</action><time>0</time><old><1>4f40c76b-148d-47cc-a7d4-5d18a83c479c</1><5>1</5><6>11.8.4.52954</6><2>5e029d96-bb8d-8983-5af4-b6d70db3fd33</2><4>null</4><3>4adc1203-0eca-14cd-b5b8-3a78f76cbce6</3><0>00000001:59a31f79:000095bf:00000006</0></old></msg>

Usage tips

  1. Since the output of queries might be large, make sure you increase the ‘Lines of scroll back’ under ‘Window’ section in Putty.

  2. Run the query on each node in the IM&P cluster and copy the output into separate files. You will have to run the tool for each IM&P individually, or alternatively merge the outputs from different servers for the same query into 1 file, following the format above.

  3. Upload the files to the tool, which will automatically detect the product types as IMP_JABBER_VERSION_STAT_1 (query 1) and IMP_JABBER_VERSION_STAT_2 (query 2) respectively.

  4. Select both files and hit “Run analysis”.

_images/csa_callengine_jabber_version_script_files.jpg

Note

The CLI outputs only provide Real-time Jabber statistics and not historical.

Sample tool output

_images/csa_callengine_jabber_version_script_output.jpg

Conductor

Log file format Upload the full diagnostic archive collected under Maintenance -> Diagnostics -> Diagnostic logging. Check the “take tcpdump while logging” option on the web interface to also include pcap for extended feature set of the tool. We rely on xconf and xstat (txt and xml) files to get general system information, but also some vital information required for successful main log analysis (loggingsnapshot file).

Supported features

  • SIP calls

  • Conferences

  • RTP streams (pcap)

  • TCP/UDP streams (pcap)

  • DNS (pcap)

  • API/HTTP

CVP (Customer Voice Portal)

Log file format

Upload the CVP Call Server logs collected from location C:\Cisco\CVP\logs. The log file names should not be changed. In order to analyze SIP calls from CVP, SipLibrary logging should be enabled on CVP CS server. Please refer to this instruction.

Supported features

  • SIP calls

  • GED-125 (communication between CVP and ICM)

ECE (Enterprise Chat and Email)

Log file format

The system decodes and parses the ECE logs, which could be collected from the ECE File server. In order to analyze the EAAS/EAMS(Listener) services the trace level has to be INFO. The log file names should not be changed.

Supported features

  • EAAS (MR events and internal EAAS messages)

  • EAMS/Listener (ARM events)

  • Connections/disconnections statistics (ARM/MR)

Dumpcfg Utility

Log file format

Upload the dumpcfg utility logs in .txt format. Logs can be collected either from the command line or from Diagnostic Portico -> GetConfigurationCategory of a Logger.

Supported features

  • List of Transactions (configuration messages)

  • List of Tables (that had a configuration change)

PerfMon for Voice Operating System

Log file format

Upload the archive that contains perfmon logs(Cisco RIS Data Collector PerfMonLog) collected using the RTMT tool from your Voice Operating System (CUCM,CUC,UCCX, etc.) server. The log bundle should contain PerfMon_*.csv files which record performance counters like CPUTime, VmSize(Virtual Memory Size), etc. This tool visualizes specific counters allowing you to easily spot issues.

Supported features

Before analysis, you will need to select one of the available templates:

  • Performance preview template (CPU, memory, IOWait)

  • Disk/storage troubleshooting template

  • CPU template

  • CUCM call activity template

  • DB replication

Telepresence Server

Log file format

Upload the .zip system logs file which can be downloaded at TPS Status -> System logs Download file

Supported features

  • SIP calls

MCU

Log file format

Upload a .zip file which contains the event logs and the protocol logs.

Supported features

  • SIP calls

  • H.323 calls

BROADWORKS (Cisco-BroadWorks Application Server)

Log file format

The Cisco-BroadWorks Call Engine application supports the XSLog from the Application Server (AS/XS). The XSLogs are located on the AS under:

/var/broadworks/logs/appserver
bwadmin@as1-r22.bstac.net$ ls -ltr
…
-rw-rw-r--. 1 bworks  bwadmin 11559805 Aug 21 09:07 XSLog2019.08.21-06.06.32.txt

XSLog file format is always XSLogYYYY.MM.DD-hh.mm.ss.txt representing the time stamp of the file creation. Please make sure that the XSLog has the correct logging level in order for the Cisco-BroadWorks Call Engine to run correctly :

1. Call Correlation Id (External Tracking Id) Call Correlation Id (Extternal Tracking Id) must be enabled on the AS:

AS_CLI/System/CallP/CallCorrelation> get
   callCorrelationIDFormat = clusterUnique OR networkUnique

AS_CLI/Interface/SIP>
   …
   sendCallCorrelationIDAccess = true
   sendCallCorrelationIDNetwork = true

This is required and this is to ensure that different call legs can be linked together representing the same calls.

2. XSLog input channels level The minimum XSLog level for the Cisco-BroadWorks Call engine to work adequately is the following:

AS_CLI/Applications/ExecutionAndProvisioning/XS/Logging/InputChannels> get
                                            Name  Enabled    Severity
=====================================================================
                                             ...
                                           CallP     true        Info
                                             ...
                                             Sip     true        Info
                                        SipMedia     true        Info
                                            MGCP     true        Info
                                             ...
                                           OCI-C     true        Info
                                             ...
                                        Diameter     true        Info
                                             ...
                                         MediaCr     true        Info
                                             ...

Note that higher level of logging (e.g. CallP at Debug or FieldDebug) will provide more functionality for Digital Signature (DSig).

Supported features/scenarios

Call flows with different signaling protocols:
  • SIP

  • MGCP

  • Diameter (Billing)

  • MediaCr (CFW)

  • OCI-C

Call features:

  • Originating calls

  • Terminating calls

  • SCAP (Shared Call Appearance)

  • SimRing (Simultaneous Ringing)

  • Call Recording

  • Etc..

Analysis

Unlike other Call Engine product, the Cisco-BroadWorks Call engine uses CallHalfs and Global CallHalfs (CallHalfs without the sub-session i.e. :0, :1, etc) values as the IP addresses for origination and destination.

images/csa/callengine/CSA-CE-AppServerLeg-SiptoMgcp.png

The CallHalfs are used for messaging to/from a node other than the AppServer. The Global CallHalfs are used on the AppServer linking all the sub-sessions (i.e. CallHallf-XXXXXXX:0, CallHallf-XXXXXXX:1). This will be the case for example when a call is requesting the NS to be routed:

images/csa/callengine/CSA-CE-AppServerLeg-OriginationAndNSrequest.png

The device (10.192.83.123-085) is originating a call to the AS. The originating callhalf is callhalf-4531085:0 and the AppServer global callhalf is callhalf-4531085. This call implies a routing request to NS-10.8.5.103. The callhalf for this request is callhalf-4531085:1 and is still attached to global callhalf-4531085.

Nodes representations

Whenever possible, the Call engine will display a node with its specific roles (AppServerLegX, NS-IP, MS-IP). In general, it will try to create a new AppServerLegX so the box will represent only one incoming call (call leg) for the user. AppServerLegX still represents the same AS but it is a new call leg in terms of call processing.

Example: Call from UserA/EntA to UserB/EntB. User has call forward to C.

images/csa/callengine/CSA-CE-AppServerLeg-CallFromUserAinEntAtoUserBinEntB.png

From the AS perspective, calling another user hosted on the same AS but not within the same enterprise is essentially seen as follow in terms of calls legs:

A origination: A to network/pstn

B termination: Network/pstn to B

The call legs are not linked/related to each other. However, the Call Engine is able to link these call legs together as represented above.

Device/Endpoint nodes:

images/csa/callengine/CSA-CE-AppServerLeg-DeviceEndpointNode.png
These nodes represents a device (EndPoint) originating or terminating a call.
  • Top line: Real IP + XXX where the XXX are the last three digits of the global callhalf. This is to ensure that device/endpoint are unique within the call flow. Otherwise originating and terminating device/endpoint coming/going from/to the same source/destination (e.g. both device/endpoint behind the same SBC) will appear as a single box with two callhalfs.

  • Middle line: Whenever possible, this will be the user’s DN.

  • Bottom line: CallHalf associated with this call leg.

AppServerLegX:

images/csa/callengine/CSA-CE-AppServerLeg-AppServerOrigTermNode.png
These nodes represent the AS (Application Server) callhalfs call legs associated with a call
  • Top line: AppServerLegX. As mentioned, the Call engine will try, whenever possible, to create a single box per call originating or terminating at a user.

  • Middle line: The ORIG or TERM sessions.

  • Bottom line: Global CallHalf associated with the call legs.

NS node:

images/csa/callengine/CSA-CE-AppServerLeg-NSnode.png
These nodes represent the NS (Network Server) boxes.
  • Top line: NS + Real IP (NS-IP)

  • Middle line: Request type ivr, cfw-media, Routing.

  • Bottom line: CallHalf associated with the request

MS node:

images/csa/callengine/CSA-CE-AppServerLeg-MSnode.png
These nodes represent the MS (Media Server) boxes.
  • Top line: MS + Real IP (NS-IP)

  • Middle line: Request type ivr, cfw-media, Routing.

  • Bottom line: CallHalf associated with the request

Note

CallHalf for CFW protocol requests are prepended with cfw.

ASXintraASY node:

images/csa/callengine/CSA-CE-AppServerLeg-ASXintraASYnode.png
These nodes are not physical nodes. They represents AS connecting calls to itself. They are created when the CallId of an outgoing INVITE is BW type and the CallId of an INVITE is the same as well as the destination (outgoing) and the origination (incoming) are the same. This is connecting AppServerLegX to AppServerLegY but this is not a physical node. So case where the INVITE is sent to secondary AS and back to primary AS will also be seen as ASXintraASY and in this case will represent a physical node.
  • Top line: ASXintraASY

  • Middle line: destination and originating DNs

  • Bottom line: CallHalf associated with the request

ASXpstnASY node:

images/csa/callengine/CSA-CE-AppServerLeg-ASXintraASYnode.png
These nodes are physical nodes. They are middle points between two (or more) AppServerLegX. They are created when the CallId of an outgoing INVITE is BW type and the CallId of an INVITE is the same as well as the destination (outgoing) and the origination (incoming) IPs are different. They will also be created when the External Tracking Id is carried into the SIP messages so the terminating AppServerLegY is linked to the originating AppServerLegX (IMS case).
  • Top line: ASXpstnASY

  • Middle line: destination and originating DNs

  • Bottom line: CallHalf associated with the request

Diameter node:

images/csa/callengine/CSA-CE-AppServerLeg-DiameterNode.png
These nodes represent the Diameter boxes.
  • Top line: Diameter + Real IP (NS-IP)

  • Middle line: Billing.

  • Bottom line: CallHalf associated with the request

Note

CallHalf for Diameter protocol requests are prepended with “diam”.

IMRN node:

images/csa/callengine/CSA-CE-AppServerLeg-IMRNnode.png
These nodes represent SIP IMRN allocation and released for IMRN (BW Mobility) origination. External Tracking Ids are different for the allocation and released. The Call engine is able to link them.
  • Top line: IMRN

  • Middle line: BW Mobile Originator

  • Bottom line: CallHalf associated with the request