In this post I will share a tip of how to install mstest in a standalone mode, i.e. without the Visual Studio.
Why would I want such a thing? Well, I am currently in the process of configuring our Continuous Integration (CI) server and I believe that such a server should have only the bare minimum necessary to do its job, nothing else. We develop our .NET code using Visual Studio 2008 Pro and use mstest as the unit test engine. So, these are the items I would like to have on my CI server:
- Windows Server 2003 – the OS
- Java, Tomcat, Hudson – my choice of the CI solution
- IIS with FrontPage server extensions – to perform unit tests of my WCF services hosted in IIS
- .NET v3.5
- msbuild
- mstest
- FxCop
Like I said, just the bare essentials. However, it turns out that mstest and Visual Studio are tightly coupled together. In fact, so tight, that googling for ways to break this coupling yielded no solutions. Not that other folks do not want to achieve the same goal as I do. The closest thing to qualify as the solution is posted by Christoph De Baene here. But, it requires us to install NUnit and Microsoft Team System NUnit Adapter from Exact Magic Software, which does the job of translating mstest attributes to nunit. Both tools are free, easy to install and NUnit is excellent, but, alas, we do not use it. (The reason is that it does not integrate with the Visual Studio, one needs TestDriven.NET for that, which would cost us more than $1000). So, I decided to divorce mstest from Visual Studio myself and the right tool for this is the excellent Process Monitor from SysInternals.
Without further ado I present my solution to how install mstest without the Visual Studio on a clean win2003 server.
Given:
- A win2003 server (the CIServer) with:
- IIS with FrontPage server extensions (irrelevant for this post)
- .NET framework v3.5
- My development machine (the DevMachine) with:
- Visual Studio 2008 Pro installed in C:\Program Files\Microsoft Visual Studio 9.0 (VSINSTALL)
- Microsoft SDK installed in C:\Program Files\Microsoft SDKs\Windows\v6.0A (MSSDK)
Isolating mstest from Visual Studio involves the following steps:
- In order to build .NET code the CIServer must have a few more files, which it currently lacks:
- sgen.exe – just to build any .NET code.
- al.exe – I use PostSharp, so this is needed as well.
All these files are found in the MSSDK directory on the DevMachine, so I just copied them to C:\WINDOWS\Microsoft.NET\Framework\v3.5 on the CIServer.
(Of course, you copy them with the respective config files – sgen.exe.config and al.exe.config) - Copy the TeamTest directory to C:\Program Files\MSBuild\Microsoft\VisualStudio\v9.0 on the CIServer.They are found in the same directory on the DevMachine.
This step is only needed if private accessors are used in the unit tests. - Put Microsoft.VisualStudio.QualityTools.Resource.dll and Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll in the GAC on the CIServer, because that is where they are on the DevMachine.
- Create a stub directory for Visual Studio files, I named it VS2008Stub and created it under C:\ on the CIServer. This directory is populated by some files from the respective locations in the VSINSTALL directory on the DevMachine, namely these:
- C:\VS2008Stub\Common7\IDE\MSTest.exe
- C:\VS2008Stub\Common7\IDE\MSTest.exe.config
- C:\VS2008Stub\Common7\IDE\VSTestHost.exe
- C:\VS2008Stub\Common7\IDE\VSTestHost.exe.config
- C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.AgentObject.dll
- C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.CommandLine.dll
- C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.Common.dll
- C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.ControllerObject.dll
- C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.ExecutionCommon.dll
- C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.HostAdapters.ASPNETAgent.dll
- C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.Tips.OrderedTest.ObjectModel.dll
- C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Adapter.dll
- C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.Tips.UnitTest.AssemblyResolver.dll
- C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.Tips.UnitTest.ObjectModel.dll
- C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.Tips.UnitTest.Tip.dll
- C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.QualityTools.TMI.dll
- C:\VS2008Stub\Common7\IDE\PrivateAssemblies\Microsoft.VisualStudio.SmartDevice.TestHostAdapter.dll
- C:\VS2008Stub\Common7\IDE\PublicAssemblies\Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
- C:\VS2008Stub\Common7\Packages\Debugger\msdia90.dll
- Setup the registry by running the attached registermstest file (it is zipped, so you will have to unzip it first). It was created by concatenating three different keys exported from the DevMachine registry:
- [HKLM\SOFTWARE\Microsoft\VisualStudio\9.0] – just the InstallDir value, which must be set to “C:\\VS2008Stub\\Common7\\IDE\\”
Without it msbuild.exe will not build the unit tests. (Of course, the DevMachine has this value refer the VSINSTALL directory) - [HKCR\CLSID\{4C41678E-887B-4365-A09E-925D28DB33C2}] – This is for the msdia90.dll loaded by mstest.exe
- [HKLM\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools] – needed by mstest.exe
- [HKLM\SOFTWARE\Microsoft\VisualStudio\9.0] – just the InstallDir value, which must be set to “C:\\VS2008Stub\\Common7\\IDE\\”
That’s it. One can now run msbuild and mstest without Visual Studio on the CIServer.
Although, I must admit that I briefly had a problem with mstest, which failed with a mysterious error: Can’t save results of the run since ResultFilePath is null. Nothing that I tried helped to resolve the issue. Finally, I decided to debug mstest (see how frustrated a man can be!) and so I installed the Remote Debugger for Visual Studio on the CIServer, which is a very light install (though it does create the “C:\Program Files\Microsoft Visual Studio 9.0″ directory). However, just installing the remote debugger had eliminated the problem! I guess, the install did something that fixed it. To test this assumption, I uninstalled it (which completely removed the “C:\Program Files\Microsoft Visual Studio 9.0″ directory). Guess what? It still works! So, it remains a mystery… I wish I could provide a more definite answer, but I cannot.
As I mentioned before, I used the Process Monitor to come up with the described procedure. It is possible, that some pieces are still missing and that my unit tests have not hit these missing areas yet. Well, when the time comes I will fire up the Process Monitor again and complete any missing pieces. That is the price to be paid for divorcing mstest from the Visual Studio.
IMPORTANT UPDATE:
We had to abandon mstest as our testing solution, because it lacks certain advanced features required to implement complex testing scenarios:
- Combinatorial tests
- Dynamic test cases
While combinatorial tests may be simulated using data sources, the question is what for? There are mature solutions, letting do it out of the box. So, we have switched to MbUnit 3 maintained by Gallio. Much recommended.
May 6th, 2009 at 9:19 am
Thanks very much for this, I used the info whilst setting up a automated functional test process : http://craigtech.wordpress.com/2009/04/28/automated-functional-testing-with-selenium-cruisecontrolnet-mstest/
One problem I did have initially was that I could only run MsTest specifying the /noisolation argument, not sure if this is something others are seeing. I was getting an error:
Failed to queue test run ‘cedmunds@ZEUS 2009-05-06 14:41:24′: Failed to get host
process location. Host process is not available.
It was because the key below did not update correctly when I ran the reg file, and after re-entering the key manually all worked fine
[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\VisualStudio\9.0\EnterpriseTools\QualityTools]
“HostProcessLocation”=”D:\ContinuousIntegration\System\VisualStudioTools\\Common7\\IDE\\vstesthost.exe”
June 12th, 2009 at 7:48 am
Thank you, but I cannot find Microsoft.VisualStudio.QualityTools.Resource.dll and Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll. I reached my whole machine, which installed vs2008, but cannot find these two files. Could you tell me where to find them? Thank you very much.
June 12th, 2009 at 3:59 pm
Hi Phoenix.
Have you checked the GAC?
Explore to C:\Windows\Assembly in the Windows Explorer and notice if the files are found there. They should, otherwise, you cannot run mstest unit tests within the Visual Studio itself.
Once you locate them there. you need to install them in the GAC on your CI server.
First copy them from GAC to another folder – do it from the command prompt. because I do not think you can from the Windows Explorer.
June 14th, 2009 at 11:20 pm
@Mark and Phoenix,
I actually couldn’t find the dlls easily either and then couldn’t find them in GAC either using command prompt. Then I found the path here in the GAC_MSIL directory rather then the GAC directory:
C:\Windows\assembly\GAC_MSIL\Microsoft.VisualStudio.QualityTools.Resource>
C:\Windows\assembly\GAC_MSIL\Microsoft.VisualStudio.QualityTools.UnitTestFramework>
Then just use the dir/w command to find the appropriate Assembly version and the copy command to get them out onto your drive somewhere.
June 15th, 2009 at 8:21 pm
thank you, it’s there. I did not know files in that directory cann’t be found with windows explorer.
But how to put it in gac? I use the command:
cd C:\WINDOWS\assembly\GAC_MSIL
mkdir Microsoft.VisualStudio.QualityTools.Resource
cd Microsoft.VisualStudio.QualityTools.Resource
mkdir 9.0.0.0__b03f5f7f11d50a3a
cd 9.0.0.0__b03f5f7f11d50a3a
copy d:\GAC\Microsoft.VisualStudio.QualityTools.Resource.dll Microsoft.VisualStudio.QualityTools.Resource.dll
is this right? Because when I run the test, it says
Loading test.webtest…
tesat.webtest
File extension specified ‘.webtest’ is not a valid test extension. I don’t know whether it is because I did not put the dll in GAC right.
Do you know how to resolve the problem?
June 17th, 2009 at 1:46 am
Thank you. Know I find the files and put them in destination machine using gacutil.exe. Yet I cann’t run the test. it says “.webtest” is not a valid extension. What I run is a web test, recorded whit fiddler. Do you know the solution?
June 17th, 2009 at 6:06 am
Hi folks.
GAC location is indeed C:\Windows\Assembly. Its internal structure is described in numerous articles on the net, for example this one – http://www.codeproject.com/KB/dotnet/demystifygac.aspx.
I should have elaborated in my post on how to actually copy stuff from GAC, because it is not obvious, but you have found it anyway. You are right about using the command prompt to locate the actual location of the files. Although, I personally, almost never use the Windows Explorer to browse my file system, but the Total Commander (http://www.ghisler.com/), which treats C:\Windows\Assembly as an ordinary folder and allows one to manipulate its contents in the usual manner (extra care must be exercised, of course).
June 20th, 2009 at 2:08 am
Hi Phoenix.
To say the truth I have never written a single bit of a web application, let alone unit tested one. But I will try to help you.
Can you create a trivial .webtest that works on your development machine
(the one with VS installed on it) and which requires no 3rd party dependencies? I would like to have it and if I can run it on my dev machine I believe I will be able to make it run on the CI server, without the VS installation.
I have a google email account as my name then dot then my surname (I do not write it in the open).
June 21st, 2009 at 10:14 pm
hi,Mark,Thank you very much! I send you the email.
July 19th, 2009 at 4:12 pm
You should read my post about using MSTest anywhere at: http://tikotal.blogspot.com/2009/01/running-mstest-without-visual-studio.html
July 20th, 2009 at 1:59 pm
Seems like an interesting alternative to my manual procedure. Although, I am not going to change the way our tests are run on the CI server at the moment, it may happen in the future. One of the possible triggers is the quality of the reports. Do you know what kinds of reports Gallio produces for data driven mstest unit tests? Because, right now, the mstest hudson plugin that we use (Hudson is our CI server) does not handle well results of a data driven mstest unit test – it does not allow to drill down the individual unit tests. I am wondering if we would still have had this problem had we chosen the Gallio way of running mstest unit tests.
July 22nd, 2009 at 12:06 pm
Seems like I’m also having problems running .webtest tests. Does anyone have a solution to this problem?
July 23rd, 2009 at 2:02 am
Hi Tim. The deal is simple. The procedure I have described works for neither .webtest nor .loadtest unit tests. The reason is that our company does not use the Team Suite and hence I cannot create or run such tests. So, naturally, I cannot know what they need in order to run without the whole Visual Studio installation.
But if you have a Team Suite at your organization, then you can repeat the steps I have taken.
Try these steps:
1. Run a .webtest on your developer machine with active Process Monitor. You only need to capture events for mstest.exe and vstesthost.exe (assuming they are used to run .webtest unit tests). Do not forget to detach from the event source once the test is over. Otherwise Process Monitor continues to consume resources and eventually will blow up your page file. You will need to save the PM log in both the native format to be able to filter it later and a plain text file. 2. Next try running the same test on your CI server, without the VS installed. Use Process Monitor in exactly the same way.
3. Analyze both logs. First have a look at the Process/Threads events. They tell which libraries were loaded into the process space of the run executables. Any coming from the VS directory should be found on the CI server.
4. Check the read registry settings. Again, you will likely need any coming from the VS registry location on the CI server as well.
5. Retry again on CI server.
Chances are after a couple of iterations you will have a working configuration on the CI server, without the whole VS bundle.
P.S.
I hope your license permits you installing another instance of Team Suite. If so, then you can view the process as though you have installed another Team Suite and manually removed most of it, leaving only the bear essentials for running the tests. Thinking of the process in this way eliminates any license doubts.
July 29th, 2009 at 7:52 pm
Learn how to run msTest without Visual Studio http://tinyurl.com/n39e4w
August 24th, 2009 at 11:32 am
Hey,
Everything worked perfect, but the problem is that when I’m trying to run a load test with a remote configuration settings,
it writes me an error
“The configuration settings associeted with this test are configured to be a remote test, for this you will need Visual Studio for Testers Edition.
Test can not be run remotely, running locally instead.”
(Those are not the exact words, but something like that)
When I do the same test on a computer with VSTS but through MSTest, it runs just fine.
What can I do?
Thanks ahead!!
November 4th, 2009 at 5:16 pm
Hi
Thanks very much for this information : it has been of great help to me. I know have mstest running standalone on our build server. I only have one limitation with the setup, and that is that mstest does not accept that /testlist and /testmetadata options. In fact invoking mstest /help does not list these two as recognised options at all, although they certainly exist in the mstest I have on my development machine with Visual Studio installed
Does anyone have any ideas what I can do to get these options working?
thanks
Mike
January 3rd, 2010 at 8:59 pm
Hey,
I followed all the steps and I am encountering the below blocking error. Can anybody please help me on this? Thanks in advance.
c:\VS2008Stub\Common7\IDE>mstest /noisolation /testcontainer:TestProject2.dll
Microsoft (R) Test Execution Command Line Tool Version 9.0.30729.1
Copyright (c) Microsoft Corporation. All rights reserved.
Loading TestProject2.dll…
HKEY_LOCAL_MACHINE\\EnterpriseTools\QualityTools\TestTypes Key cannot be found!
TestProject2.dll
File extension specified ‘.dll’ is not a valid test extension.
(I can run the same command without problem on the dev machine)
January 5th, 2010 at 3:18 am
Hi Eran.
Please, look at my previous reply to Tim dated 2009/07/23. Let me know if you have further questions.
January 5th, 2010 at 3:26 am
Hi MichaelCo.
You can do what I would have done in your place – trying to understand what mstest is missing in order to process these flags using the excellent ProcessMonitor utility. Please, look at my previous reply to Tim dated 2009/07/23, I describe the process there. Let me know if you have further questions.
January 5th, 2010 at 3:28 am
Hi Alien.
I think some of the steps failed. Make sure, all the registry settings were imported correctly.
January 18th, 2010 at 9:13 am
I have encountered the same issue as Alien, but only when installing on a 64bit Windows Server 2008 machine. On my 32bit Windows Server 2008 machine it all works fine.