It's About Time

My latest joy was tracking down a Lync problem where users on just a single pool were seeing the "Limited External Calling" error in their client after signing in. This had been working previously and no changes to the firewalls, certificates, DNS, or servers had taken place. All the usual stuff checked out OK and the logging traces on the client actually looked pretty clean. We saw no errors on the server and all other functionality seemed just fine.

After digging through the server-side traces I finally noticed a few diagnostic messages appearing periodically while looking for MRAS issues (some portions snipped for readability):

LogType: diagnostic
Severity: error
Text: Message expired in the outbound queue before it could be sent
SIP-Start-Line: SERVICE sip:<Edge Internal Pool Name>:5062;grid SIP/2.0
Peer: <Edge Internal Pool Name>:5062

LogType: diagnostic
Severity: warning
Text: Message or one of its headers caused SIP transaction processing error
Result-Code: 0xc3e93f5b SIPPROXY_E_TRANSACTION_TIMEOUT
Data: Transaction Time-out: Type [2] Method [0x40] Call-Id [7d8d76435a474b568f430436e49cfcfd\n] RUri[sip:<SIP URI>;opaque=user:epid:uZWEVFe3nFqIKFtM_NzmFwAA;gruu] From[<SIP URI>;tag=8E780080] To[<SIP URI>;tag=af1b73d16f]

The timeout and expiry messages finally clued me in on the real problem - it turned out the timezone on this server had been flipped incorrectly by an automated process.

After correcting the clock settings and restarting the FE services we found the A/V authentication working properly again. So if everything else looks correct, it may come down to the basics - make sure your clock is set properly.

Lync to Windows Live A/V Federation

One of the coolest new Lync features is that you can now do A/V federation with Windows Live users, but you'll find this does not work out of the box. First of all, your organization must complete the Public IM Connectivity provisioning process. After that, there are two modifications required even if you've enabled Public IM connectivity for the external access policy assigned to users.

First, there is a hidden parameter allowing A/V federation to PIC only available through the Lync Management Shell. This example modifies the global policy to allow both Public IM and Public IM A/V traffic so change the scope appropriately if you're limiting by site or users.

Set-CsExternalAccessPolicy Global -EnablePublicCloudAccess $true -EnablePublicCloudAudioVideoAccess $true

Secondly, the Windows Live network does not support SRTP encryption of the audio/video traffic, but Lync requires this encryption by default. We need to change Lync to support encryption instead of require it. Once that change is made Lync will prefer encrypted sessions and still negotiate those first, but will allow unencrypted media to be exchanged if it can't agree on encryption. The other Lync default is to only allow VGA video quality, but you can do 720p to Windows Live if both endpoints support it. This example changes the media encryption and video quality at the global level.

Set-CsMediaConfiguration Global -EncryptionLevel SupportEncryption -MaxVideoRateAllowed Hd720p15M

That change should be picked up within 5 minutes on the Front-End. After that, sign out of your Lync client and back in. You can verify the change by holding down the CTRL key, right-clicking the Lync task tray icon, and selecting Configuration Information. The PC to PC AV Encryption should say "AV Encryption Supported" now.

If you don't make this change you'll see an error on Front-End servers when trying to initiate an A/V call that the encryption levels don't match:

Start-Line: SIP/2.0 488 Not Acceptable Here
From: "Eddie Vedder"<sip:eddie@confusedamused.com>;tag=ed272dd714;epid=8e3ef28192
To: <sip:user@hotmail.com;mepid=F6333909B2AE4F60A2553FA59913B0A8>; tag=ab1e5513de
USER-AGENT: UCCAPI/4.0.7440.0 WLM/15.4.3502.0922 (Windows Live Messenger)
ms-client-diagnostics: 52017;reason="Encryption levels dont match"

Also, keep in mind the Windows Live user must be using the Windows Live Messenger 2011 version to support Lync A/V federation. When you're connecting a call the Windows Live client will recognize that you're connecting to Lync:

Happy federating!

Your OCS A/V Authentication Certificate Subject Name Doesn't Matter

A few months back I was involved in a discussion about what the subject name of an OCS Edge Server's A/V authentication certificate should be. Some folks were saying to use the Edge server's internal FQDN and others were saying to use the external, public FQDN you define for A/V. I was in the camp using the external name, but the odd thing was both sides said their approach worked. There is definitely some confusion about what name you should use and Microsoft has actually published directly conflicting information which further confuses the issue. Some testing I've recently done clears up why so many documents and people contradict each other - the subject name doesn't matter. Really. You could put whatever you want in that subject name, assign it to A/V authentication and it will work flawlessly. The purpose of this certificate per the Technet documentation:

The private key of the A/V authentication certificate is used to generate authentication credentials.

Specifically, it's not used for encryption or MTLS even if that's not made clear anywhere. Let's take a step back and clarify a few things for some background:

  • There are two services that run on the Edge server with "A/V" in the name. If you’re not familiar with the difference, Jeff Schertz’s More on OCS Edge Server Certificates article has a good explanation for some background on what the difference is between the A/V Authentication and A/V Edge services, but basically - the A/V Authentication service is internal facing and A/V Edge Service is external facing.
  • There is no certificate assigned to the A/V Edge service because encryption for external A/V traffic is provided by SRTP.
  • The certificate for A/V Authentication is only used by internal clients when trying to communicate with an external or federated client. This means you can (and should) use an internal certificate authority to issue this certificate. There is no benefit or need to use a public certificate for A/V authentication.

Let's walk through a little example here as if I was trying to figure out what name to use for my A/V authentication certificate. I have the following environment:

  • Public Domain: confusedamused.com
  • Internal AD Domain: ptown.local
  • SIP Domain: confusedamused.com
  • Edge Server Internal FQDN: edge.ptown.local
  • A/V Edge Service FQDN: av.confusedamused.com

So with that information what should I use as the certificate name for the A/V authentication certificate? If you consult the Technet documentation topic Set up Certificates for A/V Authentication you’ll find this note (emphasis is mine):

The subject name should match the fully qualified domain name (FQDN) of the A/V Edge Service published by the external firewall, or the FQDN of the VIP used by the A/V Edge Service array on the external load balancer (that is, if the Edge Servers are load balanced).

So based on that blurb, my A/V authentication certificate subject name should be av.confusedamused.com. Fair enough.

I ran through the OCS 2007 R2 Edge Planning Tool for a sanity check. You can see the result below, but the tool follows the Technet documentation and uses the external FQDN I defined for the A/V Edge Service when it asked.

tool-av
tool-results

A group of MVPs and Microsoft employees published a document called Deploying Certificates in Office Communications Server 2007 which says the following about the A/V authentication certificate (emphasis is mine again):

Must be the FQDN of Audio/Video authentication server in DNS.

Well that calls out the name of the authentication server, not the A/V Edge Service. I think this comes down to really just poor wording in the document which contributes to confusion, but what is the name of our A/V Authentication server? It would be the same name as the internal Edge interface, right? The A/V Authentication server is the Edge server, not the external FQDN. So now we're being told to use the internal FQDN, edge.ptown.local as the subject name.

Also released by Microsoft was a document called OCS 2007 R2 Walkthrough - Scale to Load Balanced Edge Server which completely contradicts Technet and the Edge Planning Tool (emphasis mine):

  • Access Edge Internal (Corporate Certificate). In our sample topology, the subject name would be set to ocsedge.contoso.com, the FQDN of the Edge Server internal interface.
  • A/V Authentication Internal (Corporate Certificate). In our sample topology, the subject name would be set to ocsedge.contoso.com, the FQDN of the Edge Server internal interface.

This seems to match up with the certificates document and is somewhat backed by the exact same Technet article I referenced earlier which says:

As a security precaution, you should not use the same certificate for A/V authentication that you use for the internal interface of the Edge Server.

This begs the question "Why would I ever even try to use the same certificate?" The only logical reason would be perhaps because they use the same subject name. That jives with the Scale to a Load Balanced Edge Server documentation. If we're thinking about this in terms of MTLS connections, you would have to think that this makes the most sense. In your OCS Forest properties if you added an A/V Edge server with the name edge.ptown.local for port 5062, it's reasonable that you'd expect the A/V Authentication service operating on port 5062 of the internal interface to offer a certificate matching this name. If it presented something wrong, say maybe the external FQDN of the A/V Edge service it should fail, right?

Well, the truth is the name doesn't matter. There isn't MTLS validation happening on port 5062 the same way you'd expect MTLS between servers on 5061. I think the reason the certificate requirement issue hasn't been pointed out yet is because it's never caused a problem - it works either way. I can use a certificate with a subject name gobblygook.confusedamused.com and media relay authentication through the Edge server works just fine. It just needs a certificate to generate authentication credentials for the media relay process. Go ahead and try it out - put whatever name you want on the certificate and it will still work.

So while the subject name doesn't really matter, if you're still interested in adhering to best practices I would recommend using the external facing, public A/V Edge name. In the example earlier this would be av.confusedamused.com. Hopefully Microsoft will update the certificate and scaling documents with a clarification and make them more consistent with the rest of Technet.