It's About Time

My latest joy was tracking down a Lync problem where users on just a single pool were seeing the "Limited External Calling" error in their client after signing in. This had been working previously and no changes to the firewalls, certificates, DNS, or servers had taken place. All the usual stuff checked out OK and the logging traces on the client actually looked pretty clean. We saw no errors on the server and all other functionality seemed just fine.

After digging through the server-side traces I finally noticed a few diagnostic messages appearing periodically while looking for MRAS issues (some portions snipped for readability):

LogType: diagnostic
Severity: error
Text: Message expired in the outbound queue before it could be sent
SIP-Start-Line: SERVICE sip:<Edge Internal Pool Name>:5062;grid SIP/2.0
Peer: <Edge Internal Pool Name>:5062

LogType: diagnostic
Severity: warning
Text: Message or one of its headers caused SIP transaction processing error
Result-Code: 0xc3e93f5b SIPPROXY_E_TRANSACTION_TIMEOUT
Data: Transaction Time-out: Type [2] Method [0x40] Call-Id [7d8d76435a474b568f430436e49cfcfd\n] RUri[sip:<SIP URI>;opaque=user:epid:uZWEVFe3nFqIKFtM_NzmFwAA;gruu] From[<SIP URI>;tag=8E780080] To[<SIP URI>;tag=af1b73d16f]

The timeout and expiry messages finally clued me in on the real problem - it turned out the timezone on this server had been flipped incorrectly by an automated process.

After correcting the clock settings and restarting the FE services we found the A/V authentication working properly again. So if everything else looks correct, it may come down to the basics - make sure your clock is set properly.

Moving the Lync RtcReplicaRoot Folder Drive

A fairly frustrating item in any Lync deployment is the fact that you cannot control what drive the Replica service creates its folder and file share on. There is no way to control this during the Lync installation and it seems fairly random as far as I can tell.

In many instances a Lync server will only have a single disk so this isn't a big issue, but if you are collocating a Monitoring or Archiving role on a SQL cluster node you can run into some serious problems. Since the drive selection is random, you may find the RtcReplicaRoot folder created on a shared cluster disk. This works fine when that node is active, but as soon as the cluster fails over that drive letter becomes inaccessible and the replica's topology is out-of-date.

In a scenario where you find the folder on the wrong drive you can move it, but it does require a few manual steps. Much of this is a combination of a Technet forum thread and my own experience, but here is the process which worked for me:

  1. Stop the Replica service on the node you are changing.
  2. Open a command prompt (As Administrator) and use this command to move the content:
    xcopy <Old Drive>\RtcReplicaRoot <New Drive>\RtcReplicaRoot /O /X /E /T /H /K
  3. Manually unshare the old folder. It does not show in the Computer Management shares snap-in, so you should check the folder properties directly and unshare from there.
  4. Take ownership of the old xds-replica folder inside of the old RtcReplicaRoot folder.
  5. Delete the old RtcReplicaRoot folder.
  6. Share the new xds-replica folder inside of the new RtcReplicaRoot folder.
  7. Grant Network Service full control of this share.
  8. Grant (Local)\RTC Local Config Replicator full control of this share.
  9. Open Regedit and navigate to HKLM\System\CurrentControlSet\Services\REPLICA
  10. Modify the drive letter referenced in the ImagePath value to point to the new drive.
  11. Start the replica service.
  12. Open the Lync ManagementShell and run:
    Invoke-CsManagementStoreReplication -ReplicaFQDN <Server FQDN>
  13. Restart the Replica service once more and verify your topology status is up-to-date.

Hope this helps. Pretty sure this falls in the unsupported section since this is not documented anywhere.

ExtraTeam is hiring a Microsoft UC Engineer

Sorry for the off topic post, but my company is growing and looking to add another Microsoft UC Consultant to the team. Please reach out to jobs...at...extrateam.com if interested. The full job posting is below.

Microsoft Unified Communications Consultant @ ExtraTeam
We tend to assume success, and for good reason. We've built a bleeding edge technology organization from the ground up. Each and every day we receive validation on our immense value to the world in strategizing and deploying the best of the best technology solutions. Our Microsoft practice has more than doubled over the past year and we continue to expand at a breath-taking pace. You will be joining the top Microsoft consulting team in the Bay Area; our team consists of Microsoft Certified Masters, MVP's, and published authors.

This is high-performanceville and we just can't wait to have you here.

Standard description to an exceptional opportunity:

This is a fast moving job where you will be working on all the latest technology from Microsoft.

Typical projects you will be working on include designing, deploying and maintaining;

  • Exchange 2010 including Unified Messaging
  • Lync 2010 with full voice and video integration

Job Responsibilities:

  • Designing: Work closely with our customers to assess their needs and design appropriate solutions as well as being an evangelist for ExtraTeam.
  • Implementing: You will be part of a high level team responsible for meeting our customers' implementation, configuration, installation and management needs.
  • Troubleshooting: Work closely with customers to resolve networking problems across a wide range of technologies.
  • Documenting: Ensure high quality technical documents are produced quickly and accurately.

Our customer base is a very diverse mix including many household names, defense contractors, retail giants, leading pharmaceuticals as well as local government and education.

Although technical expertise is key, your attitude and aptitude will be far more important. We're looking for someone with a strong desire to learn from the best, as part of our tightly-knit team.

We are a long standing Microsoft Gold Partner as well as a Cisco Gold Partner.

What's in it for you:

  • Strong base salary, quarterly bonus, benefits, 401K, and much more.
  • Stable, fun, and team-oriented work environment.
  • Opportunity to innovate with the latest tools at your disposal.
  • Opportunity to work remotely on select projects
  • Opportunity for growth. This is a full-time, permanent position. We're thinking long term.

Requirements for you to meet your potential:

  • Microsoft MCITP certification in Exchange 2010 and/or Lync 2010
  • You will need to be able to handle multiple projects concurrently and drive them to completion (yes, we're very busy)
  • Cisco certification would be desirable