Lync 2013 CU3 and Hosting Providers

One change I've noticed with the Lync 2013 CU3 (October 2013) Edge server update is how it validates trusted FQDNs found in the topology. Specifically, a FQDN can now either be configured a hosting provider or a federated partner's Access Edge address, but not both.

Under normal circumstances this shouldn't pose an issue, but the Lync Control Panel will not prevent an administrator from performing this configuration. And until CU3, the services didn't seem to care either way.

As an example, the proper way to allow Lync Mobile push notifications for Windows Phone clients (and Lync 2010 iOS clients), has always been to define sipfed.online.lync.com as a hosting provider like in this screenshot.

11-6-2013 9-20-09 PM

The partner push.lync.com should then be defined as a SIP Federated Domain with no Access Edge service specified. But, if an administrator did accidentally specify sipfed.online.lync.com as the Access Edge service FQDN, like in the screenshot here, it had no negative impact on the configuration. 11-6-2013 9-30-54 PM

Until CU3 this didn't present a problem. But after applying CU3 on an Edge server you'll find the Access Edge service will no longer start. Event 14517 will be logged after it fails:

The server configuration validation mechanism detected some serious problems.

ERRORS:
The server at FQDN [sipfed.online.lync.com] is configured as both type 'allowed partner server' and type 'IM service provider' .

The solution here is to make sure any hosting providers, such as Lync Online, are not also defined as the Access Edge address for a SIP federated domain. This issue isn't unique to push.lync.com - it applies to any Office 365 tenants specified as a SIP federated domain.

Make sure you validate your hosting provider and federated domain configurations prior to deploying CU3 on an Edge server.

Your OCS A/V Authentication Certificate Subject Name Doesn't Matter

A few months back I was involved in a discussion about what the subject name of an OCS Edge Server's A/V authentication certificate should be. Some folks were saying to use the Edge server's internal FQDN and others were saying to use the external, public FQDN you define for A/V. I was in the camp using the external name, but the odd thing was both sides said their approach worked. There is definitely some confusion about what name you should use and Microsoft has actually published directly conflicting information which further confuses the issue. Some testing I've recently done clears up why so many documents and people contradict each other - the subject name doesn't matter. Really. You could put whatever you want in that subject name, assign it to A/V authentication and it will work flawlessly. The purpose of this certificate per the Technet documentation:

The private key of the A/V authentication certificate is used to generate authentication credentials.

Specifically, it's not used for encryption or MTLS even if that's not made clear anywhere. Let's take a step back and clarify a few things for some background:

  • There are two services that run on the Edge server with "A/V" in the name. If you’re not familiar with the difference, Jeff Schertz’s More on OCS Edge Server Certificates article has a good explanation for some background on what the difference is between the A/V Authentication and A/V Edge services, but basically - the A/V Authentication service is internal facing and A/V Edge Service is external facing.
  • There is no certificate assigned to the A/V Edge service because encryption for external A/V traffic is provided by SRTP.
  • The certificate for A/V Authentication is only used by internal clients when trying to communicate with an external or federated client. This means you can (and should) use an internal certificate authority to issue this certificate. There is no benefit or need to use a public certificate for A/V authentication.

Let's walk through a little example here as if I was trying to figure out what name to use for my A/V authentication certificate. I have the following environment:

  • Public Domain: confusedamused.com
  • Internal AD Domain: ptown.local
  • SIP Domain: confusedamused.com
  • Edge Server Internal FQDN: edge.ptown.local
  • A/V Edge Service FQDN: av.confusedamused.com

So with that information what should I use as the certificate name for the A/V authentication certificate? If you consult the Technet documentation topic Set up Certificates for A/V Authentication you’ll find this note (emphasis is mine):

The subject name should match the fully qualified domain name (FQDN) of the A/V Edge Service published by the external firewall, or the FQDN of the VIP used by the A/V Edge Service array on the external load balancer (that is, if the Edge Servers are load balanced).

So based on that blurb, my A/V authentication certificate subject name should be av.confusedamused.com. Fair enough.

I ran through the OCS 2007 R2 Edge Planning Tool for a sanity check. You can see the result below, but the tool follows the Technet documentation and uses the external FQDN I defined for the A/V Edge Service when it asked.

tool-av
tool-results

A group of MVPs and Microsoft employees published a document called Deploying Certificates in Office Communications Server 2007 which says the following about the A/V authentication certificate (emphasis is mine again):

Must be the FQDN of Audio/Video authentication server in DNS.

Well that calls out the name of the authentication server, not the A/V Edge Service. I think this comes down to really just poor wording in the document which contributes to confusion, but what is the name of our A/V Authentication server? It would be the same name as the internal Edge interface, right? The A/V Authentication server is the Edge server, not the external FQDN. So now we're being told to use the internal FQDN, edge.ptown.local as the subject name.

Also released by Microsoft was a document called OCS 2007 R2 Walkthrough - Scale to Load Balanced Edge Server which completely contradicts Technet and the Edge Planning Tool (emphasis mine):

  • Access Edge Internal (Corporate Certificate). In our sample topology, the subject name would be set to ocsedge.contoso.com, the FQDN of the Edge Server internal interface.
  • A/V Authentication Internal (Corporate Certificate). In our sample topology, the subject name would be set to ocsedge.contoso.com, the FQDN of the Edge Server internal interface.

This seems to match up with the certificates document and is somewhat backed by the exact same Technet article I referenced earlier which says:

As a security precaution, you should not use the same certificate for A/V authentication that you use for the internal interface of the Edge Server.

This begs the question "Why would I ever even try to use the same certificate?" The only logical reason would be perhaps because they use the same subject name. That jives with the Scale to a Load Balanced Edge Server documentation. If we're thinking about this in terms of MTLS connections, you would have to think that this makes the most sense. In your OCS Forest properties if you added an A/V Edge server with the name edge.ptown.local for port 5062, it's reasonable that you'd expect the A/V Authentication service operating on port 5062 of the internal interface to offer a certificate matching this name. If it presented something wrong, say maybe the external FQDN of the A/V Edge service it should fail, right?

Well, the truth is the name doesn't matter. There isn't MTLS validation happening on port 5062 the same way you'd expect MTLS between servers on 5061. I think the reason the certificate requirement issue hasn't been pointed out yet is because it's never caused a problem - it works either way. I can use a certificate with a subject name gobblygook.confusedamused.com and media relay authentication through the Edge server works just fine. It just needs a certificate to generate authentication credentials for the media relay process. Go ahead and try it out - put whatever name you want on the certificate and it will still work.

So while the subject name doesn't really matter, if you're still interested in adhering to best practices I would recommend using the external facing, public A/V Edge name. In the example earlier this would be av.confusedamused.com. Hopefully Microsoft will update the certificate and scaling documents with a clarification and make them more consistent with the rest of Technet.

Edge Transport Rules & SCL Values

Ever tried to prepend the subject of every email message with [SPAM] if it meets a certain SCL value? I figured it would be easy enough with a transport rule. Walked through the wizard and set up a basic rule set that I thought seemed reasonable.

Apply rule to messages
With a spam confidence level (SCL) rating that is greater than or equal to 4
And from users outside the organization
Prepend the subject with [SPAM]

Now that should work, right? Wrong. Messages kept flowing through without the subject modified at all. I took a look at the mail headers and I could see the SCL value being set to 4 or 5, so why wasn’t the subject being modified?

Turns out the Content Filtering Agent fires after the Edge Filtering Agent. So messages would flow through my custom rule with an SCL of 0, continue on because they didn’t meet my criteria of 4 or higher, then get passed to the Content Filtering Agent which would set the SCL 4, and then end up in my mailbox. In order to fix this you need to flip the agents around so the Content Filtering Agent is processed before Edge Transport Agent. By default The Edge Rule Agent is 3rd and the Content Filtering Agent is 4th. This Powershell line should flip them around:

Set-TransportAgent ‘Edge Rule Agent’ –Priority 4

Or, if you prefer to actually read documentation before banging your head against the wall wondering why it doesn’t word you could be smarter than me and just read this article in advance: How To Make the SCL Value Available to Edge Transport Rules