September 28, 2011

Unhandled Exception System.BadImageFormatException

Today I tried to run a small .NET application, that I am using for more than 5 years already. It serves a small but very important purpose in my personal data management. I guess the author of this app, which I have great respect of, will be really surprised to find out anyone is still using his app out there.

When I started the application I’ve got the following error message:

image

In case you are missing Debug the program option you need to install some JIT debugger like Visual Studio, WinDbg or other. I will get into details of how this works in Windows in another post.

If you open the details of this crash you will get almost no meaningful information:

Description:
  Stopped working

Problem signature:
  Problem Event Name: APPCRASH
  Application Name: pwm.exe
  Application Version: 1.5.0.10
  Application Timestamp: 48014697
  Fault Module Name: KERNELBASE.dll
  Fault Module Version: 6.1.7601.17651
  Fault Module Timestamp: 4e21213c
  Exception Code: e0434f4d
  Exception Offset: 000000000000cacd
  OS Version: 6.1.7601.2.1.0.256.1
  Locale ID: 1033

We can see from the above that the program stopped working (we’ve kind of knew that already) and that the exception goes from within kernelbase.dll. Given that I know upfront this is a managed application, having a crash at kernelbase.dll does not feel good.

I have Visual Studio installed already so I just selected Debug the program option and then Yes on Visual Studio Just-in-Time debugger dialog.

image

From the dialog box above you can now see the exception type in top level text saying: An unhandled exception (‘System.BadImageFormatException’) occurred in pwm.exe [3380].

Digging through MSDN does not help much either: Troubleshooting Exceptions: System.BadImageFormatException - A BadImageFormatException exception is thrown when the file image of a DLL or executable program is not valid. Make sure the file image is a valid managed assembly or module. This exception is thrown when unmanaged code is passed to Load for loading.

At this place I stopped for a moment. I didn’t want to go into the debugger if the problem is not requiring it. As a general rule, use the right tool for the right job. Or you might spent countless hours trying to solve the problem with the wrong tool.

Tip: In case you read my previous blog post you may want to try your SOS skills and debug the application using Visual Studio after all :) Chances are that you will get the following error after you run .load sos command: Unable to evaluate the expression. This error comes to say that at point of the execution of the program there is no managed code loaded yet. In short – loading sos does not make sense here and Visual Studio can not load the debugging extension. I am with you that a better error message should be provided.

So I have BadImageFormatException and MSDN saying that the image may not be valid. I tried restoring the files in the folder from my backup drive. That did not change anything though.

I also tried looking at the assembly binding log using Assembly Binding Log Viewer aka FUSLOGVW. One of my theories was that the machine was missing some of the .NET binaries. Fusion log showed that the problem was situated in an assembly called NativeHelpers:

image

Double clicking on the particular trace, give you more details on the problem:

The operation failed.
Bind result: hr = 0x8007000b. An attempt was made to load a program with an incorrect format.

In case you want to know more about Fusion log viewer please read the all time blog post from Suzanne Cook - Debugging Assembly Loading Failures. And in order to keep this post short and focused I will write more about Fusion log viewer in another post.

Then I asked myself a question – what changed that it was working before and now it is not working. Well, that was obvious to tell, at least to me. Last night I finally managed to reinstall my laptop with fresh new installation of Windows 7. However this time I decided to go with x64 edition as opposed to the x86 version. The last one I was running for more than an year. After small memory upgrade it was about the time for me to make the switch.

Suddenly it occurred to me that BadImageFormatException might actually mean that the assembly is not built to run in x64 environment. So I decided to try a small trick. There a tool in .NET Framework SDK that allows you to change the bitness of the assembly. It is called CorFlags.exe and is usually located at C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin. Your mileage may vary.

So I went ahead and run corflags.exe /32BIT+ pwm.exe. That is the main executable of the application. The parameter /32BIT+ sets the 32 bit of the assembly. Short example on how to use corflags with 32BIT parameter you may find in the Corflags.exe in 64-bit Environment blog post from Matevz Gacnik.

Volla! My application got running and life’s good again. I happen to have the sources of the app so I loaded the project in Visual Studio just to verify the root cause of the problem. I found out that NativeHelpers is a managed C++ project. As such this project is built using the Win32 configuration. Managed C++ projects doesn’t have Any CPU configuration type as C# managed projects do. Needless to say all the rest of the projects were configured as Any CPU type. Hence CLR started pwm.exe as x64 process (like the OS type it was running on) and then CLR tried to load NativeHelpers as a x86 assembly. That causes file format incompatibility and BadImageFormatException is thrown by CLR.

What does it mean to you as a developer – learn what project configurations are for in Visual Studio. Learn how to deal with deployment on x86 and x64 platforms. Also learn what Any CPU means. A nice resource wrapping up all these topics is located at Moving from 32-bit to 64-bit application development on .NET Framework blog post from Gaurav Seth. Pay attention to the resources at the end of the blog post too.

September 25, 2011

TfsComProviderSvr has stopped working

Some time ago I got this error on my machine: “TFS Power Tool Shell Extension (TfsComProviderSvr) has stopped working”. My first reaction to handle this problem would be to reinstall TFS Power Tools 2010 where the problem was obviously located. However I decided to leave the problem so I can write this blog post.

It took few months and hundreds instances of the dialog box below though:

image

This dialog box is part of Windows Error Reporting (WER) which I am about to cover in additional post. One thing you should really do when you see this message is to open the details using the “View problem details” panel at the bottom of the dialog box. In my case this provided me with the following information:

Problem signature:

  Problem Event Name:        CLR20r3

  Problem Signature 01:        tfscomprovidersvr.exe

  Problem Signature 02:        10.0.30319.0

  Problem Signature 03:        4d6d4f92

  Problem Signature 04:        JOPSJ0BBDPTVPWWFWUIIWQ2SYBSSFGDF

  Problem Signature 05:        10.0.0.0

  Problem Signature 06:        4d6d4f78

  Problem Signature 07:        4b

  Problem Signature 08:        24

  Problem Signature 09:        System.NullReferenceException

  OS Version:        6.1.7601.2.1.0.256.4

  Locale ID:        1033

  Additional Information 1:        0a9e

  Additional Information 2:        0a9e372d3b4ad19135b953a78882e789

  Additional Information 3:        0a9e

  Additional Information 4:        0a9e372d3b4ad19135b953a78882e789

There is some vital information we can gather here. That is process name (tfscomprovidersvr.exe), process version (10.0.30319.0) and exception type (System.NullReferenceException). It is good enough to pass this information to the support team and request support. This is not our goal here though.

 

Let’s go ahead and debug the crashed program instead. You can see there is a Debug button as part of the dialog box. When click it, you will be presented with options to select a debugger by your choice. I have Visual Studio installed so I selected it and I got the Disassembly window opened in Visual Studio.

 

image

This window shows us one really important piece of information – the location where the exception has been thrown. The faulting method is Microsoft.TeamFoundation.PowerTools.TfsComProvider.Core.TfsCache.CachedServer.Dispose(). Unfortunately we do not have the pdb and source files (more on this in later post) so we can only see the assembly code. Unless we do some tricks.

 

Some debugging pros my ask – why in the world you did not use WinDbg to debug this managed code instead. I personally also like WinDbg much more when it comes to debugging IL and assembly code. There is one thing that is really easy and straightforward to get from within Visual Studio. Just see the Locals window below:

image

We can now browse through the exception and this objects and see their member variable values. Also hovering over a symbol name, if resolved, will show its value in a tooltip. Pretty easy, right?

Open the Threads window to see where our code is executed. If Threads window is missing for you, go to Debug\Windows and select Threads.

image

Similarly you can open Parallel Stacks window:

image

And also the Call Stack window:

image

All these windows come to show you that the code is currently executed in the CLR Finalizer Thread.

So we’ve seen the assembly code, local variables and stack info. What should we do next? We can give Visual Studio debugger some help in order to digest the assembly code and make it more meaningful to us.There it comes SOS.dll. SOS is a debugger extension that have the capability to read and parse CLR structures and IL code. It is intended to be used with WinDbg however since Visual Studio 2010 it can be also loaded inside Visual Studio also.

Open the Immediate Windows and type .load sos. Type !help to see the list of all SOS commands. Tip: Right click on the Immediate window and select Dock as Tabbed Document to open the window using the full IDE size. Now that we have SOS loaded, lets try to find more information about the code being executed.

Now let’s get the value of IP (Instruction Pointer) register using the Register window:

image

The IP register contains the address of the instruction currently being executed. Now that we have it, we can call SOS method !ip2md that will convert our currently executing address into a CLR method description object:

!ip2md 001E3EF9
MethodDesc: 00139b18
Method Name: Microsoft.TeamFoundation.PowerTools.TfsComProvider.Core.TfsCache+CachedServer.Dispose()
Class: 001b4ab4
MethodTable: 00139b74
mdToken: b18800b20600004b
Module: 00135614
IsJitted: yes
CodeAddr: 001e3ec0
Transparency: Safe critical

Now we have the MethodDesc address that we can use to decompile this method to IL using the !DumpIL command.

!DumpIL 00139b18
ilAddr = 003c38cc
IL_0000: ldarg.0
IL_0001: call CachedServer::get_IsDisposed
IL_0006: brtrue IL_0092
IL_000b: ldarg.0
IL_000c: ldc.i4.0
IL_000d: call CachedServer::set_IsConnected
IL_0012: ldarg.0
IL_0013: ldfld CachedServer::m_vcServer
IL_0018: ldarg.0
IL_0019: ldftn CachedServer::VCServer_PendingChangesChanged
IL_001f: newobj Microsoft.TeamFoundation.VersionControl.Client.Wo::.ctor
IL_0024: callvirt Microsoft.TeamFoundation.VersionControl.Client.Ve::remove_PendingChangesChanged
IL_0029: ldarg.0
IL_002a: ldfld CachedServer::m_vcServer

In case your IL skills are fading, another option is just to use Telerik’s amazing FREE tool JustDecompile (pun intended). It will show us the originating C# code instead and it will help you better understand the IL code above.

public void Dispose()
{
    if (!this.IsDisposed)
    {
        this.IsConnected = false;
        this.m_vcServer.PendingChangesChanged -= new WorkspaceEventHandler(this.VCServer_PendingChangesChanged);
        this.m_vcServer.GetCompleted -= new WorkspaceEventHandler(this.VCServer_GetCompleted);
        this.m_vcServer.CreatedWorkspace -= new WorkspaceEventHandler(this.VCServer_CreatedWorkspace);
        this.m_vcServer.DeletedWorkspace -= new WorkspaceEventHandler(this.VCServer_DeletedWorkspace);
        this.m_vcServer.UpdatedWorkspace -= new WorkspaceEventHandler(this.VCServer_UpdatedWorkspace);
        GC.SuppressFinalize(this);
        this.m_isDisposed = true;
    }
}

Ok, so far so good. But how do we find out where exactly this application crashed? Here it gets a little hairy as there is no 1:1 way to tell that. The best thing we can do is to compare the source code (IL or C#) with the actually assembly code. Here is part of the assembly output taken from the Disassembly window:

00000000  push        ebp
00000001  mov         ebp,esp
00000003  push        edi
00000004  push        esi
00000005  mov         esi,ecx
00000007  cmp         byte ptr [esi+1Dh],0
0000000b  jne         000000FC
00000011  mov         ecx,esi
00000013  xor         edx,edx
00000015  call        FFFFF130
0000001a  mov         ecx,71F8F8B0h
0000001f  call        FFF3E1E4
00000024  mov         edi,eax
00000026  mov         ecx,dword ptr [esi+14h]
00000029  lea         edx,[edi+4]
0000002c  call        701EF170
00000031  mov         eax,13CAF8h
00000036  mov         dword ptr [edi+0Ch],eax
00000039  mov         ecx,dword ptr [ecx+34h]
0000003c  mov         edx,edi
0000003e  cmp         dword ptr [ecx],ecx
00000040  call        71D18A10
00000045  mov         ecx,71F8F8B0h
0000004a  call        FFF3E1E4

The highlighted line is where the debugger says the exception has occurred. That is instruction located at address 00000039. Based on the instruction located on this address, we can safely assume that the address referenced [ecx+34h] is not valid and that is the cause of the exception.

Now that we have the IL and C# source code, let’s try to map them to the faulting instruction using some simple and well known code constructions. This way we can see where ECX is being initialized and with what.

00000000 push ebp
00000001 mov ebp,esp
00000003 push edi
00000004 push esi
  public void Dispose()
{
00000005 mov esi,ecx
00000007 cmp byte ptr [esi+1Dh],0
0000000b jne 000000FC
IL_0000: ldarg.0
IL_0001: call CachedServer::get_IsDisposed
IL_0006: brtrue IL_0092
if (!this.IsDisposed)
{

00000011 mov ecx,esi
00000013 xor edx,edx
00000015 call FFFFF130
IL_000b: ldarg.0
IL_000c: ldc.i4.0
IL_000d: call CachedServer::set_IsConnected
this.IsConnected = false;
0000001a mov ecx,71F8F8B0h
0000001f call FFF3E1E4
IL_0012: ldarg.0
IL_0013: ldfld CachedServer::m_vcServer
 this.m_vcServer
00000024 mov edi,eax
00000026 mov ecx,dword ptr [esi+14h]
00000029 lea edx,[edi+4]
0000002c call 701EF170
00000031 mov eax,13CAF8h
00000036 mov dword ptr [edi+0Ch],eax
00000039 mov ecx,dword ptr [ecx+34h]
0000003c mov edx,edi
0000003e cmp dword ptr [ecx],ecx
00000040 call 71D18A10
IL_0018: ldarg.0
IL_0019: ldftn CachedServer::VCServer_PendingChangesChanged
IL_001f: newobj Microsoft.TeamFoundation.VersionControl.Client.Wo::.ctor
IL_0024: callvirt Microsoft.TeamFoundation.VersionControl.Client.Ve::remove_PendingChangesChanged

… .PendingChangesChanged -= new WorkspaceEventHandler(this.VCServer_PendingChangesChanged);
00000045 mov ecx,71F8F8B0h
0000004a call FFF3E1E4
IL_0029: ldarg.0
IL_002a: ldfld CachedServer::m_vcServer
 this.m_vcServer…

As you can see here, ECX is being initialized as part of the following statement:

this.m_vcServer.PendingChangesChanged -= new WorkspaceEventHandler(this.VCServer_PendingChangesChanged);

More specifically it holds the value for m_vcServer. Now if we go back to Visual Studio and hover at m_vcServer variable in the output of DumpIL command that we executed above, we will see that it is actually null.

image

At this point, if you are still reading this, let’s lean back and rethink. We have the Dispose method called by the Finalize method in a GC Finalizer Thread. Inside the Dispose method an object is being referenced and it is already null. Of course there is always possibility that someone just nulled this object explicitly. I do tend to believe that if this was the case, the programmer should also implemented the necessary null validation checks inside the Dispose method. Or may the object is not nulled explicitly but the GC already has collected it. I went ahead and refreshed my memory on Disposable pattern. And then it stoke me!

The IDisposable pattern is used to free managed and unmanaged resources. In order to allow code reuse a single method Dispose is usually implemented. The Dispose method gets invoked with a boolean parameter disposing to show if the method is called from Dispose directly or through Finalize method. If Dispose is called from Finalize, you are not allowed to call any managed objects as they might not longer exist. In our case the Dispose method neither has the parameter disposing implemented. As a result it ends up calling managed objects during finalization process and that generates the NullReferenceException.

Next steps – I am going to install the latest version available Team Foundation Server Power Tools August 2011 and will see if the Dispose pattern is implement correctly there. If not – I will use my MVP powers to reach the TFS product team at Microsoft and report the problem. I will keep you updated.

Tip: If you want to save the current program state and resume the debugging at some time later, you can do that by saving a memory dump file. Do that by opening Debug\Save Dump As… menu and select folder where to store the .dmp file. You can then open the saved .dmp file using Visual Studio 2010 File\Open menu. You will find out there is a nice Minidump File Summary window that was not available before. This type of debugging of the .dmp file is called postmortem debugging as opposed to the live debugging session that I did above. There are few things you should know about postmortem debugging like IL Interpreter for example. I will try to cover those in a separate post.

What does this mean to you as developer – read and learn the correct way of implementing IDisposable pattern. Go back to the books if you do not recall how it should be done. There is nothing wrong repeating what you have already learnt. The article DG Update: Dispose, Finalization, and Resource Management from Joe Duffy is the most comprehensive that I have read so far. It is great if you want to really go deep. If you just want to refresh your memory this blog post Notes on the CLR Garbage Collector from Vineet Gupta has a really nice recap of how GC works in CLR. It also have correct implementation of IDisposable pattern shown. At the end of the you will find tons (really tons) of resources on the memory management in CLR. Funny enough I even found IDisposable implementation in the Programming .NET Framework book that we did in Bulgarian back in 2005. The chapter of Memory and Resource Management was written by George Ivanov, fellow community lead who is now working at Microsoft Corp.

Rename TFS database

If you run in situation where you need to rename your TFS Team Collection database or TFS Configuration database, there are few simple steps how to do that.

First, why would you need to rename a TFS collection database? It happened to me, that my TFS upgrade from 2008 to 2010 failed at the first run so I ended up with a database name called Tfs_DefaultCollection_1. I really couldn’t live with that suffix in my database name, so I called TFS product group for help. Here are the steps:

Renaming Team Collection database:

1. Go to the admin console and take the collection Offline

2. Once the collection is offline proceed to SQL and rename the database. This may involve taking the database offline and back online in order to release any locks on the database.

3. Come back to the Admin console and select Edit Settings for that collection

4. In the edit settings dialog there is an entry for the database name

5. Change this to the new name

6. Wait for the action to complete (might take a while since this schedules a background activity to complete the operation)

Renaming TFS Configuration database:

1.  Close the admin console (we can't have UI open since everything is bound using the server model bound to the configdb - or at least it would add complexity)

2.  Stop the iis apppool (taking the server offline isn't enough because we're pulling the configdb out from under it and it's already loaded the connection string)

3.  Rename your config db.

4.  Go to tools and run tfsconfig help registerdb.  This ultimately just edits the web.config connection string but adds some checking for you around instanceid

Keep in mind that renaming any of the databases you need to plan some TFS downtime (around 3 min for renaming my team collection database).

September 09, 2011

DevReach 2011 Early Bird Offer


Специално издание на DotNetRocks, популярното интернет радио за .NET разработчици ще бъде записано на живо от DevReach 2011

Остават броени дни до края на ранната регистрация за DevReach 2011 на 15 септември 2011 г.

Шестото издание на DevReach, международната конференция за разработчици с Майкрософт технологии, ще се състои на 17 и 18 октомври 2011 г. в кино „Арена Младост” в София. Тазгодишното издание на DevReach привлича посетителите с горещи теми от света на технологиите (като Windows 8 например) и някои от най-популярните специалисти от света на .NET. Сред тях изпъква световно известния лектор и автор Скот Ханселман. Скот работи като главен програмен мениджър в Microsoft  и ще представи визията на Майкрософт относно инструментите, с които се изграждат уеб сайтове. Той ще открие DevReach и ще направи обзор на ASP.NET MVC 3 и 4, MVC Scaffolding, HTML5, NuGet, jQuery и др. Допълнително, ще представи и лекция за ASP.NET MVC Basic to Advanced. За почитателите на Скот, специално са подбрани и две лекции, в които гостите на конференцията ще могат да дискутират директно с него как да направят блога си по-добър и как да се справят с огромния обем информация, която заобикаля потребителите.

След Скот, Джеси Либерти – старши разработчик в екипа на Windows Phone, ще разкаже пред аудиторията за интригуващите функционалности в Windows Phone 7, които предстои да излязат съвсем скоро.

За първи път на DevReach ще гостуват и лекторите Гил Клиирън – Silverlight и Telerik MVP, Мартин Балиау – Windows Azure MVP, Charles Nurse – главен архитект в DotNetNuke. Лидерите на потребителски общности в Хърватска и Сърбия – Томислав Бронзин и Иван Павлович, ще представят лекции на тема Mobile BI и Behavior Driven Development.

Както всяка година, така и тази, DevReach ще включи интересни и лекции по най-важните и горещи теми – HTLM5, Windows 8 и създаване на естествен потребителски интерфейс. Това ще бъде направено от Тим Хъкаби, дългогодишен лектор на DevReach и Microsoft Regional Director, който ще открие втория ден на DevReach.

Силно застъпени ще бъдат и лекции на тема изграждане на модерен потребителски интерфейс с HTML5 и Silverlight, използване и приложение на ASP.NET MVC, оценка, проектиране и тестване на софтуерни проекти, както и уеб производителност.

За всички софтуерни предприемачи ще се състои специално издание на DotNetRocks, популярното интернет радио за .NET разработчици (www.dotnetrocks.com), което ще бъде записано на живо от DevReach 2011. За да бъдете част от тази жива дискусия е достатъчно само да посетите залата заедно с Ричард Кембъл,  Карл Франклин, Стивън Форте, Тим Хъкаби и Лино Тадрос.

Накратко за DevReach:
Сред лекторите са:
·              12 Microsoft MVPs
·              6 Регионални директори на Microsoft
·              6 INETA лектори (Speakers, Country and Regional Leaders)
·              10 автори на повече от 60 книги, публикувани от Microsoft Press, O'Reilly Media, Sams, Que, Wrox, Apress, Packt Publishing, Manning Publications и други 
·              Популярни лидери на потребителски общности с повече от 150 000 последователи

DevReach 2011 – технологии и теми
Сесиите са организирани в 5 основни потока (трака) с фокус върху:
·              Visual Studio 2010                           ASP.NET                               C#
·              Silverlight 5                                       HTML5                                Javascript
·              MS SQL Server                                  Linq                                      MS SharePoint 2010         
·              Windows Azure                                Cloud computing              Office365
·              Microsoft ALM Tools                       Agile                                      Behavior Driven Development
·              HTML 5 and Javascript – през перспективата на Windows 8
·              Ajax, CSS 3 and jQuery
·              HTML5 for Tablets & Mobile Devices
·              Разработка на мобилни приложения за iOS, Android and WP7

 Поглед към историята на DevReach
·              5 успешни издания, с нарастващ брой посетители всяка година
·              Повече от 550 експерти, посетили изданието DevReach 2010
·              Участници от повече 15 страни

За регистрация, моля посетете сайта на конференцията на адрес: www.devreach.com

За да получавате актуална информация за DevReach, следвайте ни във  Facebook, Twitter и LinkedIn.