Sorry for the off topic post, but my company is growing and looking to add another Microsoft UC Consultant to the team. Please reach out to jobs…at…extrateam.com if interested. The full job posting is below.

Microsoft Unified Communications Consultant @ ExtraTeam
We tend to assume success, and for good reason. We’ve built a bleeding edge technology organization from the ground up. Each and every day we receive validation on our immense value to the world in strategizing and deploying the best of the best technology solutions. Our Microsoft practice has more than doubled over the past year and we continue to expand at a breath-taking pace. You will be joining the top Microsoft consulting team in the Bay Area; our team consists of Microsoft Certified Masters, MVP’s, and published authors.

This is high-performanceville and we just can’t wait to have you here.

Standard description to an exceptional opportunity:

This is a fast moving job where you will be working on all the latest technology from Microsoft.

Typical projects you will be working on include designing, deploying and maintaining;

• Exchange 2010 including Unified Messaging
• Lync 2010 with full voice and video integration

Job Responsibilities:

• Designing: Work closely with our customers to assess their needs and design appropriate solutions as well as being an evangelist for ExtraTeam.
• Implementing: You will be part of a high level team responsible for meeting our customers’ implementation, configuration, installation and management needs.
• Troubleshooting: Work closely with customers to resolve networking problems across a wide range of technologies.
• Documenting: Ensure high quality technical documents are produced quickly and accurately.

Our customer base is a very diverse mix including many household names, defense contractors, retail giants, leading pharmaceuticals as well as local government and education.

Although technical expertise is key, your attitude and aptitude will be far more important. We’re looking for someone with a strong desire to learn from the best, as part of our tightly-knit team.

We are a long standing Microsoft Gold Partner as well as a Cisco Gold Partner.

What’s in it for you:

• Strong base salary, quarterly bonus, benefits, 401K, and much more.
• Stable, fun, and team-oriented work environment.
• Opportunity to innovate with the latest tools at your disposal.
• Opportunity to work remotely on select projects
• Opportunity for growth. This is a full-time, permanent position. We’re thinking long term.

Requirements for you to meet your potential:

• Microsoft MCITP certification in Exchange 2010 and/or Lync 2010
• You will need to be able to handle multiple projects concurrently and drive them to completion (yes, we’re very busy)
• Cisco certification would be desirable

There have been a few instances lately where I’ve needed to add a speech language pack to an Exchange Unified Messaging server after a dial plan and auto-attendants have been created. Installing the language pack is no problem, but what you’ll find is that the new language is only available to new dial plans and any objects tied to them. You cannot simply install the pack and then select the language for an existing user or auto-attendant.

Here is an example case where I’ve installed the Portuguese language pack on a server:

But you can see the pack is not available for a dial plan created prior to the installation:

I’m sure the Microsoft answers are either to A – make sure you install the language packs up front, or B – create a new dial plan and auto-attendants, but in my case A was not possible and I had no interest in the effort involved for B.

So, ADSI Edit to the rescue. You can grab the language codes for installed packs from the UM server object properties at CN=<Server Name>, CN=Servers, CN=Exchange Administrative Group (FYDIBOHF23SPDLT), CN=Administrative Groups, CN=<Exchange Org Name>, CN=Microsoft Exchange, CN=Services, CN=Configuration, DC=<Forest>, DC=<TLD>. The msExchUMAvailableLanguages attribute will list the languages installed on the server (1033 is US English):

Now, armed with the language code for Portuguese (1046) you can modify the existing dial plan or auto-attendants objects in the UM AutoAttendant or UM DialPlan containers to support this language. The containers for these objects are found within CN=Exchange Administrative Group (FYDIBOHF23SPDLT), CN=Administrative Groups, CN=<Exchange Org Name>, CN=Microsoft Exchange, CN=Services, CN=Configuration, DC=<Forest>, DC=<TLD>. You can add the language as an option by modifying the msExchUMAvailableLanguages attribute to include the new language code. Here I have added it to an existing plan called Brazil:

You can now see this language appear as an option for the dial plan within the Exchange Management Console:

You can use this same method for an auto-attendant, but I would add the language first to the dial plan the auto-attendant is associated with. Obviously using ADSI Edit incorrectly has potential for causing some serious issues. Proceed at your own risk.

This afternoon we ran across an issue with a fairly new Exchange 2010 Database Availability Group comprised of 3 nodes all running SP1 with Update Rollup 3. The primary datacenter had 2 nodes with a local file share witness while the 3rd node and alternate file share witness were in a DR site. We also had recently performed a successfull datacenter failover and failback test that went swimmingly so everything was back up and running in the primary datacenter.

What we noticed today was that the cluster quorum and file share witness settings persisted as a node and file share majority after the failback instead of reverting to a node majority model like a 3-node DAG should be using. The only time Exchange should be using this model is when we have an even number of servers in the DAG. So without reproducing this again I can only see this as a timing issue – when one of the primary datacenter nodes gets added back to the DAG the quorum settings are flipped, but once the 3rd and final node joins again the quorum settings are not adjusted. This leaves us with a node and file share majority, and the FSW being our alternate FSW.

You can see here if you open the Cluster MMC our DAG is operating as a node and file share majority model even though all 3 nodes are online:

The fix for the issue is really easy – just run the Set-DatabaseAvailabilityGroup with no parameters. This process does not take the databases or cluster offline, but you’ll see the DAG detect it is using the wrong model for an odd number of nodes and adjust itself accordingly:

After the change you can verify in the cluster MMC that the quorum settings have been corrected to be a node majority:

I’m sure there’s a rational reason behind this behavior, but I haven’t quite nailed down why this happens quite yet. In the meantime it’s just one more step to add to your DR documentation!

Something I’m finding myself doing more and more lately is using multiple IP addresses on a single NIC for a Windows server. The reasons vary, but it’s generally in order to support a single server running 2 different services on the same port. This can happen for Lync with your Edge servers (or for skirting the reverse proxy requirement on Front-Ends), or with Exchange when creating multiple receive connectors on a server.

A behavior that changed with the introduction of Server 2008 is that the source IP address on a NIC will always be the lowest numerical IP. So that whole idea of your primary IP being the first one you put on the NIC – throw that idea out the window.

For example, let’s say we build a new Exchange server and configure the NIC with IP 10.0.0.100. This IP is registered in DNS and the server uses this IP as the source when communicating with other servers. Our fantastic network administrator has also created a NAT rule on the firewall to map this IP to a particular public IP for outbound SMTP so that our PTR lookups match up.

But now we want to add another IP for a custom receive connector and the network admin hands you a free IP which happens to be 10.0.0.50. You add this as an additional IP on the NIC and voila – you have a couple issues:

• You just registered two names for the same server in DNS if dynamic registration is enabled.
• Your server is now sending all outbound traffic from 10.0.0.50! (because 50 is lower than 100)

One of these is easily solved – just turn off dynamic registration and manually create the DNS records for the server. The other one is a little trickier because Server 2008 and 2008 R2 will still be sending traffic as the 10.0.0.50 IP. In the case of Exchange, this could create some ugliness for outgoing SMTP because now your firewall is not NATing to the correct public IP and you start bouncing mail due to PTR lookup failures.

Fortunately, we have a way to tell Windows not to use the lower numbered IP as a source address by adding the IP via the netsh.exe command. For Server 2008 SP2 and 2008 R2 RTM we need to apply a hotfix first. 2008 R2 SP1 included this fix by default so it is no longer required. Without the hotfix or SP1 you’ll find netsh.exe does not display or recognize the special flag.

Hotfix Downloads:

• 2008 SP2: http://support.microsoft.com/kb/975808
• 2008 R2 RTM: http://support.microsoft.com/kb/2386184/

The key to this is the IP address must be added via netsh.exe with a particular flag. So if you’ve already added the IP address via the GUI you’ll need to remove it first. After that, use this command to add the secondary IP:

netsh int ipv4 add address "Local Area Connection" 1.2.3.4/24 SkipAsSource=true

The SkipAsSource flag does two things – first, it instructs Windows not to use this IP as a source IP for outgoing traffic. And secondly, it prevents the registration of this IP in DNS if dynamic registration is enabled. Two birds with one stone!

You can always view the status of the IPs and their SkipAsSource status with the following command:

netsh int ipv4 show ipaddresses level=verbose

This morning I tried doing a desktop share with one of my coworkers and received an error I hadn’t seen before:

Sharing is not supported with this contact.

It was odd because the user was Available and not using a mobile client as far as I could tell from the presence. Turns out, the user was signed in to OWA and using the Lync integration there to chat with me. It would be nice if in a future update (hint, hint) that an OWA user’s presence be published as “IM Only” to indicate it does not support the other modalities.

In the last few Lync deployments I’ve done I’ve run into two different instances where the Lync client was failing to login to Exchange Web Services to retrieve the conversation history and user voicemail. In both cases there wasn’t actually the red exclamation mark on those two tabs in the UI like you’d expect if there were an error; the client just hummed along like nothing was wrong. In each scenario if I viewed the configuration information you would see the client report “EWS Not Deployed”, which was odd because Exchange 2010 was most definitely deployed at both customer sites.

Sidenote: The EWS polling takes roughly 30 seconds to reach this state. If you view the configuration info immediately you’ll see “EWS OK”, which is only because Lync has tried yet. So be careful when testing this and thinking everything is just fine.

Solution 1: Verify the InternalURL and ExternalURL for the Web Services virtual directory are entered
The first fix was incredibly easy and after some more digging we determined this was only occurring when a client was external and logging in through an Edge server. When we looked at the Exchange Client Access Server we found this customer had not actually entered an ExternalURL parameter for the Web Services virtual directory. This works just fine for Outlook clients, but Lync is expecting this value to be filled out. If it’s not entered it assumes EWS is not deployed externally and doesn’t attempt a connection, which is a pretty reasonable action. You might argue the Outlook action is incorrect and it should treat it the same way. But anyway, the fix is to just fill out the ExternalURL and Lync will begin using that value to login to EWS successfully.

Sidenote 2: The information discovered by Lync via Autodiscover is cached in the registry at HKCU\Software\Microsoft\Communicator\<SIP URI>\Autodiscovery (Can you tell a Lync dev wrote the regkey name? Autodiscovery instead of Autodiscover?) You’ll see entries for the internal and external URLs for the Availability Service, Exchange Control Panel, Exchange Web Services, and Out of Office Assistant. I’ve been able to delete this entire registry key for quick testing and found it recreated with no issues.

Solution 2: Place https://<Your SMTP domain>/ in the Local Intranet Zone
The second instance of this issue was a little more complicated, and still doesn’t make much sense to me, but I figured I would share. In this case the customer did not have Outlook Anywhere published so we expected it to fail externally, but this error was actually occurring internally. After verifying the InternalURL was filled out correctly we started doing some traces and noticed the Lync client would make a GET request to the /Autodiscover/Autodiscover.xml file on Exchange, Exchange would return a 401 Unauthorized challenging for credentials like we expected, and then the trace died. There were be no more responses from the Lync client IP address sent to Exchange in the logs. We verified this on multiple machines and operating systems and concluded that the Lync client would never respond to the credential request! For what it’s worth, Autodiscover was working fine for Outlook clients and no special configuration had been done to Exchange.

So we put a call into PSS and they told me Lync will not read the SCP for Autodiscover in AD, even if the Lync client is internal and that it will do its own Autodiscover lookup (Can anyone confirm/deny this?). Therefore, it will fall back to https://domain.com/Autodiscover/Autodiscover.xml, and if that fails it should move on to https://autodiscover.domain.com/Autodiscover/Autodiscover.xml like an Outlook client. This is where it got weird – PSS told me from the ETL trace Lync was not falling back to the 2nd option, yet I could clearly see it make a request to IIS and not respond. From what they saw the Lync client was getting stuck on the 1st option which didn’t really exist. In any event, they had me add http://<domain.com>/ to the Local Intranet Zone on the client. Even though we knew this was not the location of Autodiscover and I really didn’t think it would make a difference it did solve the problem. After adding entry this we saw clients then try to resolve autodiscover.domain.com and grab the Autodiscover.xml file correctly from https://autodiscover.domain.com/Autodiscover/Autodiscover.xml. At this point the EWS status in the configuration information returned to EWS OK.

Sidenote 3: There is a thread on the Technet forums about this issue which suggests editing your applicationhost.config file on the Exchange server. I have to recommend against this and as you can see in the comments it hasn’t really fixed the problem for anyone. The solution is more likely one of the ones presented here.

Something I’ve been noticing in a few Lync (and Exchange) deployments where Forefront TMG is involved is a significant delay in loading websites through the reverse proxy. I generally use Chrome as my primary browser and noticed sites would be extremely slow to load or appear entirely unresponsive when published through TMG. This was happening to both Lync Web App and Outlook Web App scenarios so I figured the issue just had to be with TMG. After some digging it turns out the problem is Chrome because when browsing to these sites in either Internet Explorer or Firefox the pages load just fine.

The issue here is a new feature in Chrome called SSL False Start which is supposed to speed up your SSL connections. Unfortunately, the end result against sites published by TMG is they don’t ever load unless the user manually refreshes the page a 2nd time. Keep in mind this applies to any SSL website published by TMG and accessed by a user with Chrome, not just Lync Web App or Outlook Web App.

There is also an issue open on Google Code about this problem, http://code.google.com/p/chromium/issues/detail?id=67617, but there is no server-side fix. At this time the only solution is to modify the Google Chrome shortcut to disable the SSL False Start feature. Just modify your shortcut to be “chrome.exe -disable-ssl-false-start” and all is well.

I think that one of the coolest features of Exchange 2010 is the seamless free/busy and calendar federation between organizations. In order to get federation provisioned there are a number of steps you need to take which you can find detailed on Technet.

The first step of this setup involves creating a Federation Trust to the Microsoft Federation Gateway (MFG), but in order to create this trust you need to use a public certificate issued by one of the following Certificate Authorities (the haphazard thumbprint formatting is Technet’s, not mine):

I recently was involved an Exchange deployment that involved purchasing a SAN certificate from Comodo. One of the certificate authorities Comodo uses to issue SAN certs is the USERTrust Legacy Secure Server CA, which has its own certificate issued by the Entrust.net Secure Server Certification Authority. Bottom line is the certificate you get verifies up to the Entrust certificate you can see below which the Federation Gateway supports.

After trying to create the Federation Trust we were seeing the following error:

An error occurred while attempting to provision Exchange to the Partner STS. Detailed information “An error occurred accessing Windows Live. Detailed information “The request failed with HTTP status 403: Forbidden.”.”

Basically this is the MFG’s way of saying “I don’t trust this certificate.” It turns out the MFG is geared to only accept certificates issued directly from one of the certificate authorities listed above which is not something I saw in the documentation. So if the Entrust Secure Server Certification Authority had issued our webmail certificate it would have been accepted. But like in our case, if your certificate is issued from a 3rd party intermediate certificate authority it won’t be accepted even if it technically verifies up to a support rooted authority.

The good news is a call to PSS resulted in Microsoft making a change on the MFG to accept certificates issued by this particular intermediate CA going forward for everyone. So if ran into this error previously you should be able to try again with the same certificate and see the trust succeed. As of this writing I’ve requested them to also add support for the AAA Certificate Services intermediate CA Comodo also issues certificates from.

One of the deployments I’ve been working on recently involved using F5 BigIP hardware load balancers to do SSL offloading for a two-node Exchange 2010 design. To give some background here usually you would just pass through port 443 (I’m skipping over the RPC Client Access piece since it’s not relevant here) from your load balancer straight to the Exchange servers, letting the servers handle the SSL encryption like in this diagram:

The benefit of that approach is it’s simple and a very common deployment method. On the flip side, you can benefit from offloading SSL encryption to the BigIPs and gain some more advanced forms of load balancing. In this case the improved load balancing was the goal along with some internal policies forcing this approach. What happens with SSL offloading is the HTTPS traffic ends at the BigIPs which turn around and pass port 80 clear-text traffic back to the Exchange servers so they have a bit less CPU work to do. That strategy looks more like this:

The problem with this configuration is Exchange is really designed to operate with SSL in mind and you have to go out of your way to allow it to operate in clear-text. What you’ll need to configure on each CAS server is:

• Set the Outlook Web Access SSLOffloaded registry key: http://technet.microsoft.com/en-us/library/bb885060(EXCHG.80).aspx
• Remove the SSL requirement from all Exchange virtual directories via IIS Manager.
• Edit the Exchange Web Services web.config file to force HTTP: http://technet.microsoft.com/en-us/library/ee633481.aspx
• Configure Outlook Anywhere SSL Offload: http://technet.microsoft.com/en-us/library/aa998346.aspx

The issue I ran into is after following all of these steps Autodiscover was still not functional through the load balancing. I could enter https://<CAS Array FQDN>/Autodiscover/Autodiscover.xml into a browser and reach the XML file with no problem, but running the Autodiscover test within Outlook would return a 404 error. Every other service was working just fine:

This threw me for awhile and after a bit of searching I ran across KB 980048 where it’s noted that Autodiscover cannot be used on port 80 with an HTTP POST request, which is what Outlook uses. My attempts at accessing the XML directly succeeded because I was only trying to download the file. Supposedly this is going to be fixed in Service Pack 1.

While the KB provides no immediate solution what I found that works is to use the same methodology Technet recommends for the Exchange Web Services web.config file. Go into your /Autodiscover folder and edit the web.config to replace all instances of httpsTransport with httpTransport (a simple search and replace should work). Be sure to save a copy before you make modifications, restart your server after making the change and you should be able to offload SSL for Autodiscover successfully. Since as far as I know this is undocumented today you can try this at your own risk, but it appears to be working.

Edit – post removed!

This post was originally written based on the Exchange 2010 beta bits before the Technet documentation was updated to reflect the actual required permissions for a DAG’s FSW. Consequently, it had a major error. You’ll want to visit Devin’s page for a full explanation and the correct way to set up the DAG. Always helps when you can read the documentation, right?