Troubleshooting SharePoint 2013 document rendering through Office Web Apps

Symptoms

The issue was experienced when clicking a document in a SharePoint library, expecting it to be rendered by Office Web Apps in the browser, but instead an error was thrown up in the User Interface saying "Sorry, there was a problem and we can't open this document. If this happens again, try opening the document in Microsoft Word." Similar messages were thrown for PDFs and other Microsoft Office file formats.

There are a whole host of generic, meaningless Office Web Apps errors that can be thrown in the UI by Office Web Apps, some of which are listed on the Microsoft Office Web Apps setup page with common troubleshooting steps. However in my case, as mentioned earlier, I knew my issue was related to SharePoint itself rather than the Office Web Apps farm.

Troubleshooting

My first port of call was to verify the ULS logs on the Office Web Apps servers. Some of the interesting entries identified were the following:

WOPI: Could not deserialize file metadata [url:https:///_vti_bin/wopi.ashx/files/9794e6b1dc944c2aa21806a9221d9d9d, e:There was an error deserializing the object of type Microsoft.Office.OpenWebApplication.WopiCheckFileResponse. Encountered unexpected character '<'.]

This was something I'd never seen before, and didn't seem to be something that was discussed much on the web either. At this point I reached out to my extremely well esteemed MCM class peer Wictor Wilen, who helped me troubleshoot the issue further.

Below the above error, was HTML for an error page SharePoint was returning, when Office Web Apps was expecting an XML response. Decomposing the HTML, we could see the error:

ErrorText=Method+not+found%3a+%27System.String+Microsoft.SharePoint.Utilities.SPUtilityInternal.GetSharingUrl(Microsoft.SharePoint.SPWeb%2c+System.Guid%2c+Int32)%27

As already proven, the issue was not with Office Web Apps, SharePoint was throwing an error when invoking a request to the endpoint that Office Web Apps used when it communicates with SharePoint. We therefore needed to look in the SharePoint ULS logs for when it is invoking the request to /vti_bin/wopi.ashx.

After searching logs of all SharePoint servers, we found the following:

Application error when access /_vti_bin/wopi.ashx, Error=Method not found: 'System.String Microsoft.SharePoint.Utilities.SPUtilityInternal.GetSharingUrl(Microsoft.SharePoint.SPWeb, System.Guid, Int32)'.

at Microsoft.SharePoint.SoapServer.Wopi.RunCheckFile(HttpContext context, CommonRequestData requestData, CommonResponseData responseData)

at Microsoft.SharePoint.SoapServer.Wopi.ProcessRequestCore(HttpContext context, CommonRequestData requestData, CommonResponseData responseData)

at Microsoft.SharePoint.SoapServer.Wopi.ProcessRequest(HttpContext context)

at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()

at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

This was something that we hadn't seen before.

At this stage, firing up Reflector and looking at the method signature for GetSharingUrl, revealed that signature was in fact different to what was being logged! The signature we were looking at through reflector for SharePoint 2013 SP1 was as follows:

internal static string GetSharingUrl(SPWeb web, SPList list, SPListItem listItem)

This started to make us think that one of our key DLLs (such as the STSSOAP.dll or Microsoft.SharePoint.dll) was out of date, or at least an old version was being referenced by our SharePoint environment.

Firing up Resource Monitor on the SharePoint servers in the farm allowed us to verify which DLL was being used by SharePoint which was invoking the Office Web Apps request. Within the tool itself, selecting the CPU tab, and then searching "Associated Handles" for STSSOAP.dll revealed that my worker process was actually loading STSSOAP from the GAC, as per the screenshot below.

As far as we were aware, this DLL should not be in the GAC, and checking a few other SharePoint installations, we could confirm this. Note that any DLL in the GAC will always take precedence of DLLs located elsewhere in your IIS _app_bin directory or your SharePoint hive.

Checking the version of this DLL in the GAC in fact revealed that this was an outdated version - SharePoint 2013 RTM! This was extremely peculiar given this farm was a SharePoint 2013 SP1 farm built with slipstream media.

Resolution

I then went on to check the date/timestamp that this DLL was deployed to the GAC, and after much digging was able to correlate this with a custom solution being deployed to the environment. Cranking open the solution, and looking through the files included, and most importantly what was contained within the manifest file, I could see that the developer had included this old version of the DLL!

The manifest contained the following line:

<Assembly Location="STSSOAP.DLL" DeploymentTarget="GlobalAssemblyCache" />

Clearly, rather than adding a reference to this DLL, the developer had actually added the file instead!

I immediately spoke to the developer about this issue, explained the errors in his ways, and asked him to create a new solution packaged correcting this issue. Retracting the original solution, and deploying the new solution which effectively removed the DLL from the GAC.

Straight away, I then checked to see if the issue had been resolved, and hey presto, documents were now successfully being rendered from the library. Opening resource monitor now also showed that the STSSOAP.dll was no longer being referenced from the GAC.

Lessons learned? Always ensure that before you deploy a custom solution to a farm, you're aware of what it contains and what it actually does. A check as simple as reviewing the manifest file within the solution, can reveal such issues early on, and save hours of time otherwise spent troubleshooting!

As you can see BrightStarr's experts have the all answers when it comes to Microsoft's enterprise technology stack and if you have a solution in place and would like us to check it over for performance, then why not check out our health check and optimization services. We believe that a health check should examine the technical platform from the business requirements down to a server's performance. Without understanding what the business wants the platform to support, you cannot truly examine the technical platform's expected future performance. With this approach we will help keep your solution running 100% of the time and achieving as much as possible.