Not So Snomtastic

Disclaimer: This post is purely my own opinion and is based on my personal experience using Snom products against Lync Server 2010 circa Fall 2011.

Awhile back I was involved with a migration involving moving a customer from an OCS 2007 R2 Enterprise Voice deployment where they were using a large installation of Snom phones. There were a number of problems which could have probably been alleviated by keeping the Snom device firmware reasonably up to date before we started the project, but the scenario we started with was one where all the phones were running firmware approximately 2 years old. No big deal, right?

Updates
The first major issue with these phones is they are not updated using the same device update feature which is built right in to both Lync and OCS. This means you cannot download a Microsoft-produced UCUpdates.cab and load it on the phones the same way you update all of your other UC devices. All updates for these devices come from Snom and must be distributed to each phone through some other manner.

You have two options - you can manually update each device, or use the freely available Snomtastic server software (which is not actually produced by Snom so good luck on finding quality support or documentation around this product.) Manually updating the phones has the approximate success rate of hitting the jackpot in Vegas, and is probably worth an entire post in itself. Let's just say it involves knowing the secret code to press on the phones as they boot (Up, Up, Down, Down, Left, Right, Left, Right, A, B, A, B, Select, Start... ok, just kidding.) In reality, it's a crapshoot if you pressed the right sequence in time to even kick the phone into an updateable mode. And the odds of the update bricking the phone when coming from these old versions was about 50/50 from our testing.

Updating hundreds of phones manually isn't practical, so you're probably looking at using Snomtastic to perform these updates. In my case Snomtastic was already deployed, but it was an old version which was apparently single-threaded. Sidenote: it also appears to be in contention for the World's Worst User Interface, but it's free so I suppose you take what you get if you don't want to manually update each phone. The single-threaded limitation meant we could only safely update 10-15 phones at a time. If you attempted to do more than that the software would come to a screeching halt with the CPU pegged, and the phones would be stuck trying to load the firmware. The end result was you had to go physically touch and manually kick them out of this mode before they were operational again. Updating to a new version of Snomtastic was out of the question as it was going to involve physically touching each phone once more to change VLANs around.

Any Day Now...
In the end we turned a blind eye to the update process and began validating the product against Lync. This customer had a ton of Snom 3xx devices which did not actually have a firmware release for Lync quite yet. Snom was touting their "UC Edition" firmware which is the version which runs against Lync, but that firmware only existed for their newer 8xx series phones when we started. This meant we were stuck with the older 8.5.8.1 "OCS Edition" version for the 3xx phones. We were told the UC Edition for the 3xx's was coming "any day now" when the project started. 3 months later the story hadn't changed, so the confidence in quick updates and issue resolution was not high. We did notice the 8.5.8.1 phones had MRAS issues aginast our Lync Edge server pool which was using DNS load balancing, but we were assured by support that the UC Edition firmware would solve this issue, whenever it came out. Surely the OCS Edition couldn't be expected to support DNS load balancing when that feature did not exist for OCS.

DNS Load Balancing
While waiting for the 3xx update Snom support encouraged us to test the 8xx devices because the code would be identical. This seemed like a good idea so we started doing some tests against the environment Again we noticed that even the 8xx devices couldn't communicate with remote users. Internal calls worked fine, but any time a call required Edge traversal the call would fail. This was only occurring when the Snoms were involved, and a full Lync or Lync Phone Edition client received MRAS credentials just fine. We eventually tracked this down with Snom support to a problem with DNS load balancing in their code. The gist was if the Snom phone got more than one name back for the Edge pool it freaked out and wouuldn't get any MRAS token. Snom support got us a "fix" in a nightly beta snapshot which I was told ignores any DNS entry other than the first one. Yes, you read that correctly - the fix was to essentially turn off the HA features of the Snom client so there was no resiliency to DNS load balancing of the Edge pool. It would be good to point out this was the initial public release of the UC Edition code, which specifically advertised support for DNS Load Balancing and had been stamped with the official "Qualified with Lync" label.

Snom support eventually gave us a beta version of the UC Edition for 3xx devices which had the exact same problem. The UC Edition introduced support many Lync-specific features, but it was really hard to have any confidence in any of these claims when it was obvious that a pretty major new feature wasn't working at all. I have no clue if this bug has been fixed to this day, or if the code still ignores anything past the first DNS record returned for the pool.

The Big Switch
In light of these issues the customer made the tough call to switch out all the phone hardware and went with one of the devices which carries the "Optimized for Lync" label from one of the major Lync Phone Edition vendors. The end-user and administrator feedback with this switch was overwhelmingly positive. Call quality was perceived as better by the users, and we were able to back up those claims with the quality reports from the Monitoring server. The administrative experience for updating and monitoring phones was also much improved by using the native reporting and management tools.

There are plenty of other Lync Phone Edition options available which may seem likea larger initial investment in total price, but you really need to consider a number of factors in that decision other than per-device pricing. With a lower-end device you're likely to spend an incredible amount of time troubleshooting issues which wouldn't normally be a problem. You also have additional overhead in terms of management for these devices since you now have to maintain a Snomtastic server (with no real support) alongside your Lync deployment, when you could just be using the native Lync tools for updates. The end-users are going to have a much nicer experience via the USB tethering abilities of Lync Phone Edition clients, and it's important to note that the Snoms also do not support RTAudio. They advertise wideband audio support, but it is G.722 only which uses nearly twice the bandwidth of wideband RTAudio. This can be a concern for bandwidth-constrained branch sites.

I'm not advocating a boycott of Snom products here. I'm sure others have had different, and hopefully more positive experiences, but mine was incredibly poor with Lync for a number of reasons I've mentioned above. And I'm sure Snom is making a strong effort to improve these products, but my perspective at this point is that they just doesn't seem Enterprise-class. They are not an option I would present to a customer today over any of the vendors producing Lync Phone Edition devices.

Migrating OCS conference directories to Lync the hard way

A few evenings ago I ran into a scenario where moving a conference directory from OCS to Lync failed, and the conference directory ended up in this limbo state where it wasn't on Lync, I couldn't move it back to OCS, and the conferencing attendant wouldn't recognize any PSTNs IDs which were part of the directory. Not a great scenario.

After running Move-CsConferenceDirectory I could verify the move was in progress, but it never completed. The status would show it was trying to move, and OCS eventually started throwing errors that it no longer had a conference directory, but it never fully made it to Lync. The TargetServerIfMoving parameter stayed populated:

Get-CsConferenceDirectory -Identity 5
Identity: 5
ServiceId: UserServer:OCSPOOL.ptown.local
TargetServerIfMoving: UserServer:LYNCPOOL.ptown.local

Trying to run Move-CsConferenceDirectory again would consistently fail with the following errors:

WARNING: Move operation failed for conference directory with ID "5". Cannot perform a rollback because data migration might have already started. Retry the operation.
WARNING: Before using the -Force parameter, ensure that you have exported the conference directory data using DBImpExp.exe and imported the data on the target pool. Refer to the DBImpExp-Readme.htm file for more information.
Exception from HRESULT: 0xC3EE7950, Microsoft.Rtc.Management.ConferenceDirectoryCmdlets.MoveConferenceDirectoryCmdlet

In the end I needed to export the data from the OCS directory via DbImpExp, force the directory to move, and then import the data. Not the cleanest route, but it works. The order is important, so be patient.

On the OCS pool and database export the conference directory data:

DbImpExp.exe /hrxmlfile:C:\Temp\OCSDirectory5.xml /SQLServer:OCS-SQL.ptown.local /restype:confdir

Only once you're positive you have a good export (Read: opened the file and checked!), and made a copy of it you can force the Move-CsConferenceDirectory operation:

Move-CsConferenceDirectory 5 -TargetPool LYNCPOOL.ptown.local -Force

Congrats. You've moved the directory to Lync, but it's empty. Copy the .xml export file to a FE in the Lync pool. On the Lync pool and database import the directory data while specifying the conference directory ID to recover the old data:

DbImpExp.exe /import /hrxmlfile:C:\Temp\OCSDirectory5.xml /SQLServer:LYNC-SQL.ptown.local /restype:confdir /dirid:5

At this point I could see the directory was no longer moving because TargetServerIfMoving was empty, and the conference attendant was now recognizing PSTN IDs which had been created against this directory.

Get-CsConferenceDirectory -Identity 5
Identity: 5
ServiceId: UserServer:LYNCPOOL.ptown.local
TargetServerIfMoving: 

Again, this is a good reason to always do a DbImpExp.exe dump before moving directories or databases around. Those .XML files can save your skin!

Monitoring OCS and Lync Peak Call Capacity

Recently I had a customer interested in checking how many concurrent calls a particular OCS Mediation Server was handling. The challenge with this is that separate perfmon counters exist for inbound calls and for outbound calls, but there is not a built-in counter which measures both. So while we could monitor the peak capacity of each we had no guarantee that these peak values were occurring at the same time.

In order to track this usage I've come up with a Powershell script which grabs these two counters, parses their values, adds them together, and dumps the output into a CSV file. At the end of the monitoring period you can take this CSV into Excel and easily find the peak total call count.

Here are some notes on the behavior:

  • The CSV output is date and time, inbound calls, outbound calls, and total calls.
  • Data is output to the console and to CSV for real-time monitoring.
  • The default values track usage for a week, polling the counters every 15 seconds. You can change the total number of loops in the script to your liking if you need a longer track record.
  • If you run the script again it will detect if previous data exists and rename the old file so you don't lose anything.
  • I've run this as a logged in user account, but I imagine you could set it up as a scheduled task to run in the background.
  • In order to run the script you should first run Set-ExecutionPolicy Unrestricted

The caveat with the Lync version is now that a Mediation server can use multiple gateways we can't see which gateway is being used for each inbound or outbound call. But this still gives an idea of concurrent call capacity flowing through each Mediation role.

I hope to improve this in the future, but wanted to make it available for everyone sooner than later.