Skype for Business Contact List Changes are Not Saved

As more and more companies adopt a hybrid approach to Office 365 there is an increasing mix of on-premises and online users, but not all policies applied on-premises will carry over to Office 365 as gracefully as you might hope.

As an example, consider an organization that has determined tha users with both a Mac and PC couldn't possibly have the need to update their contact list from a Mac client so they've enabled the Unified Contact Store (UCS). For the nerds, this means that:

Set-CsUserServicesPolicy -Identity Global -UcsAllowed $true

Now consider that user has signed in on a PC and successfully migrated their contact list to UCS at some point. And that same user later migrates to Skype for Business Online where the global UcsAllowed paramater is set to $false by default.

The end result is the user will find they are unable to have changes persist to their contact list. The Skype for Business client honors new additions or changes to group membership within the UI during that session, but those modifications will be removed on the next sign-in or registration refresh. If the user adds a contact, moves them to another group, or removes them from the contact list, the original contact list will be reset on the next registration attempt.

On the other hand, this is a really nice trick to play on someone if you want to make them think they've lost their mind.

The issue here is that the user was previously using UCS, but Office 365 does not allow for this feature by default. When the user signs in UCS kicks in to provide the contact list, but changes are only being saved against the Skype for Business Server because the policy prevents UCS.

In order to resolve the situation you'll need to roll the user back to using the Skype for Business Server contact list provider option via:

Invoke-CsUcsRollBack -Identity [username]

This will force the user to immediately sign out, but will start using the Skype for Business Server as the contact list provider on the next sign-in.

Organizing your OneDrive for Business

We all agree that folders are bad, right? They only make sense at a certain point in time and only mean something to the person who created them. Everyone hates the folders in SharePoint which someone else created, but I find the majority of people still organize their personal OneDrive for Business files within folders.

The problem with this is that OneDrive for Business screams and begs for you to share those files with others, so users create folders as a mechanism to define who has access to sets of data. Folder names like the built-in "Shared with Everyone" or "Shared with Team X" become the ACLs which makes sense for a specific project, but that concept quickly detoriates when project members change or someone else needs access to a subset of content within that folder.

I was guilty of the same crime for a long time, but have recently adopted a new technique I think works really well. It goes like this:

  • Store all your files in a completely flat structure within OneDrive for Business, and set permissions only at the file level. No folders allowed. Sort the view by Modified Date for an extra boost because this way everything you're actually working on is front and center, similar to a "Recent Documents" view.

  • If a group of files lend themselves to being shared together, put them on SharePoint or in an Office 365 Group. Your personal OneDrive for Business isn't the best fit for a set of documents which are related. You should set up a document library, team site, or Office 365 Group to move that effort forward. Sync that location to keep a local copy handy.

  • Create a single folder called Archive. Similar to mail archive concepts the only thing that goes in here are files you haven't accessed in a really long time and need to keep around for reference.

Windows 10 Enterprise & Credential Manager

Updated 4/26/2016: Turns out this is not specific to Windows Enterprise or Professional, but actually related to Azure AD Join and Windows Hello. If your machine is joined to Azure AD during the OOBE installation step and you sign in with a Hello method (face, fingerprint, PIN) then Credential Manager is busted. But if you bypass Hello (cover your face!) and use a password then Credential Manager is "unlocked" and works exactly as expected. From what I can tell this has to do with how Microsoft Passport containers work and because Windows Hello will only unlock a single container. If your machine is joined to Azure AD the Enterprise Container is what Hello authentication unlocks, but Credential Manager only functions when the default (non-Enterprise) container is unlocked.

Credential Manager has been a part of Windows for a long time now and I've noticed it seems to have stopped behaving as expected in builds of Windows 10 Enterprise Edition. To be more specific, defining a saved Windows credential in Credential Manager and then adding the same address to the Local Intranet Zone of Internet Explorer used to result in a SSO experience for the user. The machine would pass the saved credentials automatically without the user seeing any kind of authentication prompt.

This was a very handy trick for machines which weren't joined to a domain and needed to authenticate to various sites or applications without requiring the user to type their password over and over throughout the day. You could save a Windows credential for *.confusedamused.com and add that same wildcard definition to the Local Intranet Zone to cut down on practicing your password-typing skills.

What I've noticed is that while the Professional Edition continues to work like prior versions of Windows the Enterprise Edition will always present the authentication prompt, even with the exact same configuration.

If you try to track down potential differences you'll quickly run across Credential Guard, a feature unique to Enterprise Edition which Microsoft describes as this:

Introduced in Windows 10 Enterprise, Credential Guard uses virtualization-based security to isolate secrets so that only privileged system software can access them. Unauthorized access to these secrets can lead to credential theft attacks, such as Pass-the-Hash or Pass-The-Ticket. Credential Guard prevents these attacks by protecting NTLM password hashes and Kerberos Ticket Granting Tickets.

Sounds great. This feature doesn't run by default, but after a couple of steps via GPO I was able to enable Credential Guard and validate it was running on my Surface Pro 4. But after removing and adding the saved credentials to Credential Manager I was in the same spot with being constantly prompted.

Reading a little further down the Technet page about Credential Guard you'll also find this innocuous-sounding statement:

Starting with Windows 10, version 1511, domain credentials that are stored with Credential Manager are protected with Credential Guard. Credential Manager allows you to store credentials, such as user names and passwords that you use to log on to websites or other computers on a network. The following considerations apply to the Credential Guard protections for Credential Manager:

  • Applications that extract derived domain credentials from Credential Manager will no longer be able to use those credentials.

The emphasis there is mine, but I think that last statement is key. I've been unable to find any documentation which clearly states this means Windows credentials saved in Credential Manager won't work, but I think they fall into the derived domain credentials category.

What doesn't add up is that this feature is supposed to be unique to Credential Guard, but it seems to prevent Credential Manager from working with Windows credentials even when not enabled.

I suppose you could make the argument that only corporate-owned devices would run Enterprise Edition and that any corporate-owned device shouldn't be storing saved credentials, but I'd still prefer to see this as a configurable setting even if the default behavior is to block the usage.

Clearing up the clear and simple guidance about Configuration Manager and Intune

Brad Anderson of Microsoft recently highlighted a new post from the Intune team full of ridiculous flow charts which supposedy deliver "clear and simple guidance" about when to use System Center Configuration Manager vs. Intune for device management.

Hopefully the flow charts didn't require much labor because they are completely unnecessary. Here's the extracted clear and simple guidance about which tool is most appropriate for each scenario:

  • Configuration Manager: Corporate-owned device which is not mobile, already on the domain, or requires fine-grained settings control via GPO.
  • Azure Active Directory Join + Intune MDM: Corporate-owned device which is mobile, not on the domain today, and does not require fine-grained settings control via GPO.
  • Azure Active Directory "Add work account" + Intune MDM: Personal device not owned by the company.

Downloading Office 365 Video

Updated 3/26/2016: Microsoft is now rolling out the ability to download videos in a much more elegant manner. 

One of the more popular features requested in Microsoft's Office 365 video portal is the ability to download the videos for offline use. As it stands today the user experience is that the video streams via your browser or mobile app and the user is not presented with any kind of download option in the UI.

But, we know that the Office 365 video portal is simply built on top of SharePoint so there must be a way to get at the actual video object somehow, right? Good news: you can, as long as you can construct the correct SharePoint URLs.

The root of your O365 Video portal is at this address:

https://[Tenant Name].sharepoint.com/portals/hub

Hitting that URL will simply take you to the O365 Video home page, which isn't super helpful for saving a video. In order to download a video you'll want to first identify the channel the video is in and then access this URL to see a SharePoint view of all content within that channel:

https://[Tenant Name].sharepoint.com/portals/[Channel Name]/pVid/Forms/Thumbnails.aspx 

Note that the /hub part of the previous URL is not necessary here and if your channel names have a space remember to replace the space with a hyphen, not your typical %20 encoding in the URL. SharePoint replaces the spaces with hyphens when it builds the site URL.

As an example, if my tenant name is confusedamused.onmicrosoft.com and my videos are in a channel called PDX then the URL to see the contents of that channel would be this:

https://confusedamused.sharepoint.com/portals/PDX/pVid/Forms/Thumbnails.aspx

Once you land on that URL you should see thumbnails of all your videos. Hover over any of them and hit the ellipses button to present a download option. Be sure to watch your video at least 5 times to justify the extra effort required to save it for offline access.

Alternatively, if you know the channel and video name you can access it directly with this format:

https://[Tenant Name].sharepoint.com/portals/[Channel Name]/pVid/[Video Title].mp4

In the case where your video title has a space in the name you'll want to use a %20 encoding as usual. And one last note - if a video channel name was ever changed the SharePoint URL will retain the original channel name's URL path despite the new display name. You can sort out what the original name was via some Fiddler traces if you're striking out on the URL you've created, but that is outside the scope of this post.

Quirks of Office 365 Groups

(Yes, it has been awhile!)

When Office 365 Groups first launched last year it was a rather underwhelming first stab at combining a lot of functionality which was arguably already in the Microsoft productivity stack, and in pretty much every case, way more refined than Groups.

We've seen incremental changes to the Groups feature over the past few months and the messaging seems to point that all roads from Exchange, SharePoint, and Lync/Skype for Business eventually lead to Groups. There is still a long list of control knobs and discovery refinements needed before this is something an enterprise can fully adopt, but there are some use cases which make Groups appealing today.

One of the main motivators for using a social tool like Yammer instead of a traditional distribution list is that it captured the historical content. So the person who starts the day after someone posts a critical message to the entire team actually has access to the previous conversations and content. Groups solves that problem quite well, even if it's difficult for an administrator to manage that content. But let's face it - Yammer's admin features are sorely lacking as well, so if you're willing to give up tagging and notifications then Groups is a viable alternative.

I wanted to share a few quirks I've noticed if you're headed down this path:

Logo and Image Sizing

The photo you set for the Group logo should be 240x240 pixels. I can't find this documented anywhere, but that seems to be the size of the example or stock image you'll see when the group is created. But, OWA and OneDrive for Business show different quality levels for reasons I can't explain. The first image here is what I see in OWA and the second is the logo in OneDrive for Business when I access the files section, which looks obviously blurry.

Screen Shot 2015-03-31 at 9.42.30 PMScreen Shot 2015-03-31 at 9.42.03 PM

Group Permissions

If you try to manage the Group within the Azure AD Access Panel you may find that you're then unable to manage some options back in the OWA interface. For example, I created this Group within OWA as a public group anyone could join. I modified the permissions in Azure AD to require approval. Now - even as the Group owner - I'm unable to revert that setting in OWA and see the Privacy section is grayed out.

Screen Shot 2015-03-31 at 9.56.34 PM

Default SMTP Address

Groups default to using your tenant ID as the primary SMTP address. Really smart folks on my team like Keif Machado and Joe Stocker have already beaten this one to death, but let's just say it's inconsistent with what anyone would expect when creating a new object. Remember to flip it to an appropriate domain after creation.

Likes

It's getting social, and we're starting to see even more overlap with Yammer functionality. I can "Like" a message anyone posts and they will receive a notification that I did so. But, that pop-up notification is only visible in OWA, so the Outlook stalwarts will surely suffer from a lack of ego-boosting alerts.

Screen Shot 2015-03-31 at 9.45.30 PM

Lync/Skype Presence

There seems to be some semblance of the presence concept, and perhaps we're seeing the starting to see these groups becoming a persistent chat room in Skype for Business. When you view the contact card for a group you'll see the presence indicator on the left side just like any other contact. I've seen it change from Available to Away with not much background on why, but it's obvious that some aspect of the group has presence abilities.

Screen Shot 2015-03-31 at 9.29.51 PM

Hope that helps!

Lync Split-Domain & Static Route Conflicts

Something that is coming up more and more on Lync projects is the concept of integrating with newer video and collaboration services like Acano or Pexip. It's very important to understand that the deployment guides from these services request you create a static route from your Lync Front End pools to their MCUs. This in itself is not a red flag, especially since this is how some of the Polycom DMA or Cisco VCS integration has worked in the past.

However, this does pose a problem for organizations looking to leverage split-domain with Lync Online, or more commonly, Exchange Online being leveraged for voicemail while Lync Enterprise Voice remains on-premises. The issue here is that using your primary SIP domain as the Match URI in the static route to these video services prevents the signaling from getting to your federation Edge servers (at least, as of the latest Lync 2013 CUs.) The static routing configuration seems to kick in before the call is ever routed to the Edge, so the concept of the CsHostingProviders and the shared address space is never respected.

Let's walk through an example:

  • My primary SIP domain in Lync is @confusedamused.com.
  • I migrate my mailboxes to O365 and enable the Exchange Online shared address space. My voicemail is hosted in O365.
  • I want to host large meetings in O365 so I also enable the Lync Online shared address space for hybrid. I have some users homed to O365.

Everything works fine at this point.

  • I now want to integrate with Acano or Pexip. I create a static route for @confusedamused.com which goes from my Lync FEs to their bridges.

This breaks both Exchange Online voicemail routing and communication to Lync Online users.

The real solution here is to use a different SIP domain for your video routing, which as undesirable as this may be to end users, has arguably been a best practice for a long time. Create the static route matching @video.confusedamused.com, @acano.confusedamused.com, or @pexip.confusedamused.com. Take your pick, or create your own variation. It just needs to be different from the SIP domain you actually assign to Lync users.

This is important to grasp if you're using Lync Online or Exchange Online today and looking to add video, but probably even more critical from a roadmap standpoint. Exchange Online Unified Messaging is a fairly common use-case today, and Lync Online hybrid is becoming more and more popular. Planning ahead for these scenarios and any potential video integration will help you avoid these issues.

Lync 2013 SQL Express Instance Memory

Depending on the role, Lync 2013 servers will have one or two SQL Express instances installed as part of the Lync Deployment Wizard steps. All servers deploy an instance called RTCLOCAL, and Lync Front Ends will also deploy an instance called LYNCLOCAL. Generally, there is no configuration or action required on these instances.

If you were to manually install a SQL Express instance the minimum and maximum memory values are essentially unlimited. Technically, this isn't a realistic view because SQL Express is actually limited to 1 GB of memory, but it's important to know the out-of-the-box defaults.

I've noticed that the Lync installation wizard actually configures static minimum and maximum values on each instance of both RTCLOCAL and LYNCLOCAL, and these are based on the amount of RAM presented to a server at installation time. You can see an example here, which is from a FE that uses Hyper-V dynamic memory and only had around 2.19 GB RAM available.
3-12-2014 3-57-38 PM

The instance will always float between 270 MB and 337 MB of memory, which is quite a bit below SQL Express's ability to address up to 1 GB. We want both of these instances to always be able to address 1 GB, so this is not ideal!

Each instance uses slightly different defaults, but for the sake of documentation I've validated the RTCLOCAL instance will be set to a minimum of 12% and maximum of 15% memory. LYNCLOCAL will be set to a minimum of 6% and maximum of 8% memory.

If the server was installed with a minimal amount of RAM, this could negatively impact the Lync services because the SQL instances won't enough memory available to function properly. You'll likely be able to start services, but find they stop after a period of time and log events such as this one:

There is insufficient system memory in resource pool 'internal' to run this query.
Cause: The connection to the database might be broken.

It's fairly obvious that the process needs additional memory, but simply adding memory to the server won't resolve the issue. The "gotcha" here is if you install Lync with minimal memory for some reason and then bump it up later, Lync's SQL Express instances won't actually leverage the memory because of the static max/min values configured at installation time.

You can manually adjust these values within the SQL Express instance, but the Lync process will actually reset your modifications back to these percentages after a period of time.

The permanent fix is to re-run Step 1 and Step 2 in the Lync Deployment Wizard on each machine that had RAM added after the initial installation. This will set the RTCLOCAL and LYNCLOCAL instance max/min memory configuration to the percentages above based on the new amount of installed memory. Now of course, SQL Express still won't be able to access any more than 1 GB of memory, but this fixes any potential ceiling below that number.

Big thanks to my ExtraTeam colleague, Chris Lehr, for helping track this down.

Lync MX and Skype Crash on Windows 8.1

I got around to upgrading my main laptop to Windows 8.1 this week (finally!) and found I was unable to open the Metro-version apps of both Lync and Skype. They would start to open, show me the splash screen, and then exit. After a bit of digging it appears the crash is caused by the display adapter driver. Seems like Intel still has some issues with their driver for the HD 4000 Graphics on 8.1 and there doesn't seem to a fix.

In case it saves someone else the time, I've already tried all of the following revisions, the latest of which was released less than two weeks ago:

  • 10.18.10.3379
  • 10.18.10.3345
  • 10.18.10.3316
  • 9.18.10.3190 (Windows 8 version)

So you may want to hold off on that upgrade if you have an Intel HD 4000 chip and rely on Lync MX or Skype heavily.

Lync 2013 CU3 and Hosting Providers

One change I've noticed with the Lync 2013 CU3 (October 2013) Edge server update is how it validates trusted FQDNs found in the topology. Specifically, a FQDN can now either be configured a hosting provider or a federated partner's Access Edge address, but not both.

Under normal circumstances this shouldn't pose an issue, but the Lync Control Panel will not prevent an administrator from performing this configuration. And until CU3, the services didn't seem to care either way.

As an example, the proper way to allow Lync Mobile push notifications for Windows Phone clients (and Lync 2010 iOS clients), has always been to define sipfed.online.lync.com as a hosting provider like in this screenshot.

11-6-2013 9-20-09 PM

The partner push.lync.com should then be defined as a SIP Federated Domain with no Access Edge service specified. But, if an administrator did accidentally specify sipfed.online.lync.com as the Access Edge service FQDN, like in the screenshot here, it had no negative impact on the configuration. 11-6-2013 9-30-54 PM

Until CU3 this didn't present a problem. But after applying CU3 on an Edge server you'll find the Access Edge service will no longer start. Event 14517 will be logged after it fails:

The server configuration validation mechanism detected some serious problems.

ERRORS:
The server at FQDN [sipfed.online.lync.com] is configured as both type 'allowed partner server' and type 'IM service provider' .

The solution here is to make sure any hosting providers, such as Lync Online, are not also defined as the Access Edge address for a SIP federated domain. This issue isn't unique to push.lync.com - it applies to any Office 365 tenants specified as a SIP federated domain.

Make sure you validate your hosting provider and federated domain configurations prior to deploying CU3 on an Edge server.

LYSS.exe High CPU Usage

Not sure on the root cause, but I ran in to an instance where the LYSS.exe process was consuming 80-90% CPU on a Lync 2013 Front End server, and consequently causing issues with conference joins and other Lync functions. This process name is new to Lync 2013 and represents the Lync Storage Service. LYSS runs within an additional SQL Express instance on each Front End server, and is responsible for providing the magic pixie dust that allows Lync to now leverage either SQL or Exchange 2013 Web Services for contacts and archiving data. LYSS provides a layer of abstraction for the internal Lync components to deliver content to, and then sorts out how to deliver it to the appropriate end-state data store. It's essentially a glorified queuing service which replaced the need for MSMQ, so it temporarily stores the data, and then delivers to the appropriate destination (either SQL or Exchange.)

Back to the intent of the post, LYSS doesn't run as its own service so you cannot simply restart it via the Services MMC. If you do run in to this problem, you can kill the lyss.exe process. The service will restart itself automatically, and you'll hopefully see the CPU usage drop.

Lync Meeting Content and Attachments That Won't Download

My most recent adventure involved a scenario where files uploaded or attached to a Lync meeting couldn't be saved by many of the meeting participants. You could press the Save As or Open buttons, but the progress indicator would just sit at 0% and never move. Some users in each meeting could actually download the content, but the behavior did not follow specific users or meetings. Permissions in the meeting were set to allow anyone to download, so it didn't appear to be an issue specific to the meeting settings. The Lync QoE report submitted by each client gave me a rather unhelpful message:

A resource was unable to be downloaded via HTTP

Time to do some tracing. Firing up Fiddler allowed me to see the client make a download attempt to the external web services site, but I was getting a 404 Not Found response from the server. Sure enough, I was able to find the same hit from my client in the IIS logs on the Front End server.

GET /DataCollabWeb/Fd/05f08f60-e18a-5f5c-b73d-2331aa486896/{ED48B5A7-1CD5-4D5F-9D04-6EC08A6F8FEF}/S4S3N9QD/bc2f8fb2-0948-f75b-9a96-fddf0018beef.bin - 4443 - 192.168.200.224 Mozilla/4.0+(compatible+;+MSIE+9.10.9200.16688+;+Microsoft+Windows+Professional++;+Placeware+RPC+1.0) - 404 0 2 22

Odd. IIS was unable to locate the file the Lync client was asking for. IIS serves up these files from the Lync back-end file share, which pointed me in the direction of the DFS share. This was configured to replicate between two file servers, but nothing looked obviously wrong after opening the share folder.

After some more thought I opened the individual shares on each server. Bingo. They each had different, unique content, only one of which matched what I saw in the actual shared Lync namespace! The Front Ends connections were being distributed across the different DFS members and the content was not being replicated on the back-end, which created scenarios where a file would appear to be missing.

Digging in to the DFS logs allowed me to see the servers had stopped replicating almost 90 days earlier. DFS in 2008 R2 and later uses a feature that prevents replication from restarting if an automatic recovery occurred after a network or power outage. So a brief issue a few months back actually caused DFS to stop replication, and it never resolved itself.

You can switch the automatic recovery back to pre-Server 2008 R2 behavior with this command (from an elevated command prompt) on each DFS node:

wmic /namespace:\\root\microsoftdfs path dfsrmachineconfig set StopReplicationOnAutoRecovery=FALSE

Even still, that change does not magically fix DFS. If a node has been unable to replicate for more than 60 days DFS will also prevent the replication from occurring due to a MaxOfflineTimeInDays parameter. You can view that default value with this command:

wmic.exe /namespace:\\root\microsoftdfs path DfsrMachineConfig get MaxOfflineTimeInDays

And if you need to adjust that value you can use this command on each node:

wmic.exe /namespace:\\root\microsoftdfs path DfsrMachineConfig set MaxOfflineTimeInDays=120

This actually still did not resolve my problem. In the interest of time I was able to use the following steps to get replication flowing again:

  • Disable the secondary node in the Replication Group through the DFS Management Console.
  • On the secondary node, remove the existing file share to close all connections to it.
  • Copy all of the content from the secondary node to a temporary location.
  • Remove all content from the (now unshared) folder on the secondary node.
  • Share the empty folder on the secondary node.
  • Re-enable the secondary node in the DFS Management Console.
  • Copy the temporary saved data to the primary node's file share, skipping any duplicate files.

Issue the following command on both nodes to poll Active Directory for an update:

dfsrdiag.exe PollAD

You should see Event Log entries logged that indicate replication is starting and that an initial sync has completed. After validating replication was working I was able to successfully download newly uploaded meeting content from any connected client.

Lync 2013 and the RTCXDS 16 GB Transaction Log Limit

Be aware that when you deploy Lync 2013 there is a maximum size limit that is configured for the transaction log of the RTCXDS database. The other databases created by Install-CsDatabase all have a maximum size set to about 2 TB (so, essentially unlimited), but the RTCXDS log file is strangely capped at 16 GB.

This isn't normally a problem, but if you use SQL Mirroring and/or the Lync Backup Service, which will both leverage this file more, you may start experiencing problems when this limit is reached. KB 2756725 is posted which indicates you may be unable to start the Front End services once this occurs:

Or you may find users are just unable to connect to Lync, and these errors are logged in your Front Ends.

The RtcDb Sync Agent has encountered an Exception: [System.Data.SqlClient.SqlException (0x80131904): The transaction log for database 'rtcxds' is full due to 'LOG_BACKUP'.

If you open the SQL Management Studio you can check the free space available in each log file with the following query:

DBCC SQLPERF (LOGSPACE)

You'll see this information for each database. In this case you can see the RTCXDS database has used 96.95% of the log space available.

full

You can flush this free space out by running a SQL backup job of the transaction logs. I hope it's obvious, but if you've hit the 16 GB limit you're going to create a 16 GB backup file, so make sure you have at least that much free space on your backup target. This is not your usual database-level backup; you'll need to make sure you run a backup of the transaction logs. The specific steps to run this job are outlined here

After the job completes run the query again and notice the change. RTCXDS now has only 0.14% log space used.

empty

So, the question is how to avoid this problem before you get burned. I see two options:

Back Up the Transaction Log File Regularly

Schedule regular backups of the RTCXDS transaction log file. This will flush the file and keep the Log Space Used % at a low value. It's fairly common for Lync deployments to use various scripts for dumping all the backup data to flat files instead of involving SQL admins and backup software, so this may not be an appealing option.

Increase the Transaction Log Maximum Size

Increase the maximum size of the RTCXDS transaction log to match the other databases at 2 TB. You'll most likely run out of disk space on the server before you ever hit this number, and you can catch this issue from cropping up with a basic disk free space alert.

You can follow these steps to adjust the log file maximum size:

  • Connect to the Database Engine of the SQL instance.
  • Expand Databases.
  • Right-click on RTCXDS and choose Properties.
  • Under Select a page, select Files.
  • Select the rtcxds_log file, then move the scroll bar to the right.
  • In the Autogrowth/Maxsize column click the button with the three dots "…"
  • Under the Enable Autogrowth menu, change the Maximum File Size to Unlimited.
  • Click OK twice to commit changes.

Exchange 2013 Schema Prep Objects to Object References without an Object

When running the Exchange Server 2013 Schema Prep step you may find the Exchange pre-requisites test fails, and an incredibly helpful error message is provided for your reading enjoyment:

The On-Premises test failed with the message: Object reference not set to an instance of an object.

For more information, visit: http://technet.microsoft.com/library(EXCHG.150)/ms.exch.setupreadiness.
DidOnPremisesSettingCreatedAnException.aspx

Well, actually, that was rather unhelpful. And the handy URL provided basically just says the content hasn't been added yet.

Digging in to the ExchangeSetup.log files found in C:\ExchangeSetupLogs is the only way to diagnose what's really going on. This file will list all of the pre-requisite checks the installer is running through, and if you search for the phrase 'HasException:True' you can skip to anything flagged as an issue. This would theoretically identify the exact failure and let you be on your way, right? Nope.

In my case, I ran across this exception:

Evaluated [Setting:IsHybridObjectFoundOnPremises] [HasException:True] [Value:
Microsoft.Exchange.Management.Deployment.HybridConfigurationDetection.HybridConfigurationDetectionException: The On-Premises test failed with the message: Object reference not set to an instance of an object.. ---> System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.Exchange.Management.Deployment.HybridConfigurationDetection.HybridConfigurationDetection.TestOnPremisesOrgRelationshipDomainsCrossWithAcceptedDomain(IOnPremisesHybridDetectionCmdlets onPremCmdlets)
   at Microsoft.Exchange.Management.Deployment.HybridConfigurationDetection.HybridConfigurationDetection.RunOnPremisesHybridTest()
   --- End of inner exception stack trace ---

Ok, so we're getting a little bit closer. You still need to do some Exchange team to real-world language translation, but the gist seems to be that the installer has an issue with the Organizational Relationships and/or Accepted Domains.

Nothing seemed obviously wrong with either of these at first glance, but after combing through each listed object in both sections about ten times I discovered a (non-functional) organizational relationship that had been defined without an Application URI value. Seems like someone had staged this object, but never completed configuration. Deleting the problematic organizational relationship allowed the installer to complete the schema prep step without any further complaints.

It sure would be nice if the Exchange installer was a bit more detailed about exactly which object it was checking or had a problem with, but who doesn't love a good Easter egg hunt from time to time?

Lync 2013 Mobile Client Voicemail

I discovered recently that the Lync 2013 Mobile client needs voicemails to be stored in the MP3 format in order to play them back directly within the application. One of my customers had set the default codec on the Exchange Unified Messaging dial plan to be GSM to provide greater compatibility with mobile devices a few years back, and never updated the setting.

When users attempted to play back their voicemails in the Lync 2013 Mobile client they would see this error:

We can't play this message. Tap Play to receive a call to hear it.

iPhoneVM

You can work around this issue by modifying the audio codec used by the UM dial plan, which affects all users assigned that particular dial plan:

Set-UmDialPlan -Identity MyDialPlan -AudioCodec MP3

Or, you can set the audio codec on a per user-basis with the Set-UmMailbox cmdlet, which will override the dial plan default:

Set-UmMailbox -Identity tom -CallAnsweringAudioCodec MP3 

In either case, you should be find that any voicemails received after the change can now be played directly within the Lync 2013 Mobile client.

Avoiding Lync 2013 Certificate Prompts

Lync 2013 and the introduction of the lyncdiscover client bootstrapping process has introduced a new world of hurt for many Lync deployments, especially those that contain multiple SIP domains. I won't go in to the guts of why or how this happens - Richard Brynteson and Jeff Schertz do a great job explaining this aspect. The gist of the issue is your users will see a certificate warning in either of these two scenarios:

  • The lyncdiscover.<Your SIP Domain Here> query returns a web services FQDN that doesn't end with <Your SIP Domain Here>.
  • The lyncdiscover query returns a web services FQDN that ends with <Your SIP Domain Here> and succeeds, but then returns a SIP pool FQDN that doesn't end with <Your SIP Domain Here>.

So, how to work around this? You have a few options.

Keep all FQDNs within the SIP Domain Namespace

A typical deployment will put the pool FQDN within the namespace of the Active Directory domain, but you can actually name the pool anything you'd like when adding it to Topology Builder. You avoid the prompt by creating a pool FQDN and web services FQDN that are both within the primary SIP domain namespace. So, to provide an example configuration:

  • Active Directory Domain: ptown.local
  • Primary SIP Domain: confusedamused.com
  • Front End Pool FQDN: pool.confusedamused.com
  • Internal Web Services FQDN: webservicesinternal.confusedamused.com

This option only works if you're deploying an Enterprise Edition pool, and only resolves the warnings for your primary SIP domain. Other SIP domains serviced by this pool will still get the certificate warning. While it's technically possible to modify the internal web services FQDN on a Standard Edition pool you can't change the pool FQDN, so there is no point.

Configure the Trust Model Data

For those running Standard Edition pools or Enterprise Edition pools with multiple SIP domains, you'll need to configure the TrustModelData registry key outlined in KB 2531068.

In Lync 2013 the key no longer exists by default, and the location of the registry settings have shifted slightly from the KB article. They can now be found in HKLM\Software\Microsoft\Office\15.0\Lync.

One other change to watch out for is that the client doesn't actually read a TrustModelData key in the HKLM hive (true at least on Windows 8.) It will only read the HKCU version. In order to work around this issue you can deploy the key to the HKCU hive for all users via GPO preferences. The value specified should be a comma separated list of domains for the pool or web services FQDNs, which will typically be your Active Directory namespace.

Capture1

Lync 2013 Mobile Clients and Apache Reverse Proxy

Just a heads up: You may be experiencing issues using the new Lync 2013 mobile apps if your reverse proxy is Apache. I don’t have a solution at this time, but have confirmed replacing Apache with IIS ARR works just fine in the same environment. All other external web services and the old 2010 mobile clients work properly, but we observed the following behavior on the 2013 apps:

  • First client sign-in attempt fails, but second sign-in succeeds.
  • Presence for contacts is very slow to load.
  • Client sign out generates an error that the connection to the server was lost.
  • Voice and video calls will not connect.

There are no clues in the server-side logs, but if you review the mobile client logs you’ll find the following error after Apache sends the mobile client a response:

ERROR TRANSPORT TransportUtilityFunctions.cpp/1749:Accept-types (application/vnd.microsoft.com.ucwa+xml) not found in Content-Type response from server (text/plain).  Not decoding.

Deciphering this, the client is expecting an HTTP header to be returned with the a Content-Type of application/vnd.microsoft.com.ucwa+xml to be returned with the server response. The mobile client logs also show the HTTP headers in each response, which include the following:

HttpHeader:Content-Type text/plain

Since the UCWA MIME type is not a default registration on Apache the server instead reverts to its default behavior of sending text/plain for any content type it does not understand. The response being returned as text/plain causes the mobile client to error out and disconnect the request.

We attempted to register the MIME type with the following command, but it proved unsuccessful.

AddType application/vnd.microsoft.com.ucwa+xml .xml

Maybe some other Apache guru can share a fix here, but I wanted to save others the troubleshooting time if you see this issue.

Cisco and Lync One-Way Audio Troubleshooting

I had one case recently with Lync and Cisco UCM integration where the Cisco user would periodically be unable to hear the Lync user speaking. We were using media bypass with MTP required on the Cisco trunk, but the problem was fairly sporadic. The SIP Invite messages and SDP looked correct with the proper MTP for media, but the problem would continue to surface randomly. After checking all of the configs we got down to the point of needing a packet capture to determine the audio problem. We captured traces on both the Lync client and the Cisco phone, and compared the audio streams. The goal was to see if the Cisco phone was not actually receiving the Lync user's audio packets. The point of this post is to walk through the troubleshooting process we used to reconstruct the Lync and Cisco audio streams in Wireshark, which ultimately helped us pinpoint the problem location.

To get started, open your packet capture in Wireshark:

clip_image001

In the case of the capture collected on the Cisco phone we found the RTP packets were unable to be identified by Wireshark. They were purely UDP data as shown in the previous screen. In order to play these back we first needed to identify the RTP data. Highlight a UDP packet and then in the Wireshark menu click Analyze, Decode As, select RTP, and press OK.

clip_image002

You'll now see the same UDP data is identified as RTP traffic using the G.711 codec:

clip_image003

From the Wireshark menu now select Telephony, RTP, and Stream Analysis. You'll see the forward (sent) and reverse (received) audio RTP streams here. In this case we saw significant stream data bi-directionally from both capture points. This ruled out any kind of MTP problem and allowed us to validate the audio was being sent and received by both parties.

clip_image004

Press the Player button and click the View as time of day checkbox to listen to the audio stream. I typically select both the forward and reverse checkboxes and then press Play again to listen to both parties:

clip_image005

The interesting discovery was that we actually had two-way audio on both capture points. We could listen to the entire conversation on the trace collected at the Cisco phone port, which pointed us to the actual Cisco handset as the problem.

You can pull up call stats on a Cisco phone by pressing the ? button twice very rapidly. When we did this we found the codecs matched, but the RxDisc (Receive packets discarded) value was extremely high. This confirmed the handset itself was dropping the packets and not playing them back to the user.

clip_image006

After some more testing we isolated the problem to the Cisco 7940 series phones. The latest firmware updates did not resolve the issue and we have a Cisco TAC case still open on the subject. I'll update the post with any additional information we find.

Lync Music on Hold with Media Gateways

With the CU7 release Microsoft has added Music on Hold abilities to the Lync Phone Edition clients. Many organizations are starting to turn this feature on now, but it's possible that external PSTN callers still might not not hear any music on hold being played. If you're seeing (or not hearing) this behavior, there may be a gateway setting that's not passing the music on hold to PSTN callers.

On an AudioCodes MSBG 1000 this setting can be found under VOIP\GW and IP to IP\DTMF and Supplementary Services\Supplementary Services. Change the Hold Format parameter to "Send Only" if it is currently set to 0.0.0.0. This parameter controls if the gateway should expect a SDP with fields set as a=sendonly and c= containing the client's IP address, or if it should expect a=inactive and c=0.0.0.0. Setting this value to Send Only allows the Lync PCs and phones to successfully pass music on hold to a PSTN caller.

VMM 2012 SP1 Service Stops Every Night

If you have the System Center Configuration Manager 2012 RTM agent installed on a Windows Server 2012 server also running the System Center Virtual Machine Manager 2012 SP1 agent, you might find the VMM service unexpectedly stops each night. There is nothing obvious in the event logs about why the service stops, but each night the hosts will move to "Not responding" in VMM because the service is not running.

This is caused by an inventory process in SCCM that causes the VMM agent to stop. You can uninstall the SCCM agent using this command:

C:\Windows\ccmsetup\ccmsetup.exe /uninstall