Skip to content

Latest commit

 

History

History
203 lines (144 loc) · 7.11 KB

contribute.md

File metadata and controls

203 lines (144 loc) · 7.11 KB
Raw

Contribution Guide

This article will help you build, test and try out local builds of the VS test platform.

Prerequisites

Please ensure you have a .net 4.6.2 or higher installed on the machine.

Clone the repository to a local directory. Rest of this article assumes /src/vstest as the location of source enlistment.

> git clone https://github.com/Microsoft/vstest.git

If you're planning to use Visual Studio as development environment, please install VS 2017 RC 2 and the .NET Core and Docker (Preview) workload installed. See download link here.

If you're not planning to use Visual Studio and only use CLI. You will need to install .Net 46 targeting pack. The download link has two msis. Both needs to be installed. Otherwise build will fail asking to install net 46.

Rest of the article will provide steps for VS and CLI/Editors development.

Build

Building with Visual Studio

Open /src/vstest/TestPlatform.sln in VS.

Use Build Solution to build the source code.

Binaries for each assembly are produced in the artifacts/src/<Assembly>/bin/Debug directory.

Building with CLI, CI, Editors

To build the repository, run the following command:

> cd /src/vstest
> build.cmd

This command will fetch the latest dotnet-cli into /src/vstest/tools/dotnet directory. It will use the dotnet executable present there to build the source code. All the nuget required for build will be downloaded to /src/vstest/packages directory.

Build will produce following assets:

  • A portable vstest.console for desktop (net46 target) and xplat (netcoreapp) target
  • A visual studio extension Microsoft.TestPlatform.vsix with the test platform components required in VS test explorer
  • Test platform SDK packages for ObjectModel and TranslationLayer

We will discuss more on each assets are in the Deployment section below.

Binaries for each assembly is produced along side the source. E.g. ObjectModel assemblies can be found at /src/vstest/src/Microsoft.TestPlatform.ObjectModel/bin/Debug/net46/*.dll.

To build a particular configuration, use the -c option. E.g. to trigger a release build use

> build.cmd -c Release

For other options, check /src/vstest/scripts/build.ps1.

Test

There are following sets of tests:

  • Unit tests
    • Very fast tests primarily validating individual units
    • Named as <UnitUnderTest>.UnitTests where UnitUnderTest is any product assembly
  • Smoke tests
    • Slower end to end tests. Typically these cover P0 scenarios (99% of users will use these, if these are broken, PR will not be merged)
    • Driven by a real vstest.console executable
    • Named as Microsoft.TestPlatform.SmokeTests
  • End to end tests
    • Slower end to end tests for extensive coverage
    • Driven by a real vstest.console executable
    • Named as Microsoft.TestPlatform.AcceptanceTests

As a principle, most of tests are unit tests (~70-80%), few smoke tests (~20-15%), fewer acceptance tests (~10-5%).

Unit tests and smoke/acceptance tests are run with the vstest.console built by the build step above.

Running tests (Visual Studio)

RCurrently tests are run in VS 2017 RC using the test platform bits. You need to install the testplatform.vsix generated by build if you're using an older version of VS (< RC.3). See Deployment for installation steps.

Run the tests in Test Explorer. Use a search filter like project:"Unit" to run only unit tests. For running smoke tests, use the project:"Smoke" filter.

Running tests (CLI, CI, Editors)

To execute tests, run the following command:

> cd /src/vstest
> test.cmd

By default, only unit tests are run. To run the smoke tests, provide the -p option to test.cmd to set test assembly pattern:

> test.cmd -p smoke

The -p option can be used to run tests for any assembly as well. E.g. following command will run tests for datacollector:

> test.cmd -p datacollector

Tests for a particular configuration can be run with following command. By default, Debug configuration is run.

> test.cmd -c release

Deployment

This section will discuss the steps to use the vstest.console we've built using previous instructions.

Visual Studio

Visual Studio 2017 RC ships with the test platform vsix generated by build.cmd. To use the locally built version, use the following steps in a developer command prompt.

> vsixinstaller /src/vstest/artifacts/Debug/TestPlatform.vsix
# (replace Debug with Release as appropriate)

Command line (XPlat)

A netcoreapp target of vstest.console is dropped at /src/vstest/artifacts/<Configuration>/netcoreapp1.0/vstest.console.dll. It can be executed with any dotnet executable. You may choose to use the dotnet-cli we have in /src/vstest/tools/dotnet/dotnet as well :)

> /src/vstest/tools/dotnet/dotnet /src/vstest/artifacts/Debug/netcoreapp1.0/vstest.console.dll /?

Command line (Windows desktop)

A net46 target of vstest.console is dropped at src/vstest/artifacts/<Configuration>/net46/win7-x64. It can be run directly as follows:

> /src/vstest/artifacts/Debug/net46/win7-x64/vstest.console.exe /?

Diagnostics

Try to isolate the failure scenario. In the best case it's just a command line that can demonstrate a bug. For example, a bug in discovery of tests can show up in following command line:

> /src/vstest/artifacts/Debug/net46/win7-x64/vstest.console.exe mytest.dll /listTests /tests:*&&*#Ed

Next step is to enable verbose logging to understand details.

Another add a Debugger.Launch at the process launch points. E.g. testhost.exe or vstest.console.exe. Select the appropriate debugger (choose CoreCLR for netcoreapp scenario) and step through the code.

Running Tests with TPv2 using Test Explorer

Test Platform (TPv2) is packaged as a vsix with VS 2017 RC releases and lights up the .NET core and Live Unit Testing scenarios. It currently does not support UWP & data collector scenarios (code coverage & fakes). It is placed @ "%programfiles(x86)%\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform".

Desktop, UWP & Native unit testing continues to use the test platform (TPv1) located @ "%programfiles(x86)%\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TestWindow".

To use TPv2 for desktop - place testplatform.config @ "%programfiles(x86)%\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\Extensions\TestPlatform" with the contents below.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <appSettings>
    <add key="feature.net35" value="true" />
    <add key="feature.net40" value="true" />
    <add key="feature.datacollector" value="true"/>
  </appSettings>
</configuration>

If feature.net35 and feature.net40 are set to true, VS Test Explorer will use TPv2 for desktop flow for tests targetting net35 and net40 frameworks respectively. If feature.datacollector is set to true, VS Test Explorer will use TPv2 when data collectors are enabled.