Cisco and Lync One-Way Audio Troubleshooting

I had one case recently with Lync and Cisco UCM integration where the Cisco user would periodically be unable to hear the Lync user speaking. We were using media bypass with MTP required on the Cisco trunk, but the problem was fairly sporadic. The SIP Invite messages and SDP looked correct with the proper MTP for media, but the problem would continue to surface randomly. After checking all of the configs we got down to the point of needing a packet capture to determine the audio problem. We captured traces on both the Lync client and the Cisco phone, and compared the audio streams. The goal was to see if the Cisco phone was not actually receiving the Lync user's audio packets. The point of this post is to walk through the troubleshooting process we used to reconstruct the Lync and Cisco audio streams in Wireshark, which ultimately helped us pinpoint the problem location.

To get started, open your packet capture in Wireshark:

clip_image001

In the case of the capture collected on the Cisco phone we found the RTP packets were unable to be identified by Wireshark. They were purely UDP data as shown in the previous screen. In order to play these back we first needed to identify the RTP data. Highlight a UDP packet and then in the Wireshark menu click Analyze, Decode As, select RTP, and press OK.

clip_image002

You'll now see the same UDP data is identified as RTP traffic using the G.711 codec:

clip_image003

From the Wireshark menu now select Telephony, RTP, and Stream Analysis. You'll see the forward (sent) and reverse (received) audio RTP streams here. In this case we saw significant stream data bi-directionally from both capture points. This ruled out any kind of MTP problem and allowed us to validate the audio was being sent and received by both parties.

clip_image004

Press the Player button and click the View as time of day checkbox to listen to the audio stream. I typically select both the forward and reverse checkboxes and then press Play again to listen to both parties:

clip_image005

The interesting discovery was that we actually had two-way audio on both capture points. We could listen to the entire conversation on the trace collected at the Cisco phone port, which pointed us to the actual Cisco handset as the problem.

You can pull up call stats on a Cisco phone by pressing the ? button twice very rapidly. When we did this we found the codecs matched, but the RxDisc (Receive packets discarded) value was extremely high. This confirmed the handset itself was dropping the packets and not playing them back to the user.

clip_image006

After some more testing we isolated the problem to the Cisco 7940 series phones. The latest firmware updates did not resolve the issue and we have a Cisco TAC case still open on the subject. I'll update the post with any additional information we find.

Office 365 Migration with Cisco IronPort

I ran across an interesting issue recently where a client could not get Autodiscover to work properly during their “rich coexistence” period with an on-prem Exchange 2010 during their migration to Office 365. Autodiscover for an on-prem user would work fine, but as soon as the user had their mailbox moved to Office 365 the Autodiscover process wouldn’t work. The DNS records looked fine and when looking at the log we saw the client would connect to the internal SCP, get a redirect to Office 365 for the correct SMTP address, and then fail. We couldn’t set up a brand new profile for the user internally, but we noticed it would work perfectly ok from an Internet client. Must be something internal at that point, right?

After some more testing we learned a Cisco IronPort was being used for outbound web proxy filtering. As soon as we added an exception for the test machine's IP address we found Autodiscover worked just fine for a cloud user. In the end we added an exception for the FQDNs .outlook.com and .online.lync.com. Secure web filtering keeping users safe and admins frustrated. Happy migrating.

Lync and Cisco CUPS RCC

I recently had a project where remote call control with Lync was still a requirement for a customer. While preparing for this integration I found some pretty varying information out there – some even indicating this was no longer possible without TLS, which is untrue, so I thought I would clear this up a bit. For this environment let’s assume the following:

  • CUPS Domain: CUPS.confusedamused.com
  • CUPS Server IP: 10.1.0.7
  • SIP Domain: confusedamused.com
  • Lync FE Pool: LYNCPOOL.ptown.local

The first step here is to build a trusted application pool which is required to trust a specific host. Since we only have a single CUPS server we'll specify a single host pool.

New-CsTrustedApplicationPool 10.1.0.7 -Registrar LYNCPOOL.ptown.local-Site 1 -TreatAsAuthenticated $true -ThrottleAsServer $true -RequiresReplication $false

Next, we’ll configure a trusted application which lives on that “pool” of the CUPS IP address. This is so Lync trust requests from this server and port combination. The application ID parameter can be whatever you like – it just provides a unique way to identify a trusted application in your environment. You can say “yes” to the warning about UCMA applications after running this command.

New-CsTrustedApplication -ApplicationID CUPS -TrustedApplicationPoolFQDN 10.1.0.7 -Port 5060 -EnableTcp

After those two steps we need to publish the topology with these changes:

Enable-CsTopology

So all we’ve done so far is told Lync it can trust an application running on port 5060 on a server with the IP address of the CUPS server. The next step is to tell Lync how to actually send traffic to our CUPS SIP domain. In OCS this was done with the static routes tab which was accessible in the GUI, but with Lync this must be done in the Lync Management Shell. First, create a static route matching the CUPS SIP domain, cups.confusedamused.com in our case, with a next hop of the CUPS server IP address and listening port.

$TCPRoute = New-CsStaticRoute –TCPRoute –Destination 10.1.0.7 -Port 5060 –MatchUri CUPS.confusedamused.com

This step just puts the route in a variable. We need to actually add it to the routing configuration like so:

Set-CsStaticRoutingConfiguration –Route @{Add=$TCPRoute}

Also, don't forget to allow your Lync server to accept TCP requests on port 5060:

Set-CsRegistrar registrar:LYNCPOOL.ptown.local –SipServerTcpPort 5060

At this point everything should be (theoretically) in place, but I found it still didn’t work correctly. Lync was still discarding responses sent from the CUPS server as if it wasn’t trusted. Here’s the key – open Topology Builder, and download a copy of your latest topology. Edit the properties of the new trusted application pool which was created. In order for this to work properly you must limit the service usages to the IP address of the CUPS server.

I’m not sure why this doesn’t work correctly without it, but it won’t. Publish this change, restart your FE services, and you should find RCC working properly.