diff --git a/docs/_config.yml b/docs/_config.yml deleted file mode 100644 index 3583dd630c8a5d3eebd9b63a2ac90343b0524266..0000000000000000000000000000000000000000 --- a/docs/_config.yml +++ /dev/null @@ -1,3 +0,0 @@ -title: Pairwise Testing -description: Combinatorial Test Case Generation -theme: jekyll-theme-slate diff --git a/docs/about.md b/docs/about.md deleted file mode 100644 index da069be24f835d798c15fde61713480f9b4a5305..0000000000000000000000000000000000000000 --- a/docs/about.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -layout: default ---- - -[(back)](./) - -Thanks for stopping by. My name is Jacek Czerwonka and I work for Microsoft. For a while now, in addition may day job, I have been involved in designing and implementing pairwise-related tools and evangelizing pairwise testing at Microsoft. This web site is a collection of links related to pairwise that I have accumulated in that period of time. I hope you find this information useful. - -If you come across something that should be included here, have a comment, or simply want to chat about pairwise testing, email me. - -jcz (at) ieee (dot) com diff --git a/docs/efficiency.md b/docs/efficiency.md deleted file mode 100644 index 9a05899047e95829a9531d8119209cdf4df48502..0000000000000000000000000000000000000000 --- a/docs/efficiency.md +++ /dev/null @@ -1,26 +0,0 @@ ---- -layout: default ---- - -[(back)](./) - -The basic measure of efficiency of a pairwise test generation tool is the number of tests a tool generates given some size of input. For example, when the input has four parameters with 3 values each, denoted 3â´ in the table below, tools create between 9 and 11 test cases. All of the test suites meet the pairwise testing criterion of covering each pair of values in at least one test case however some tools can pack all these combinations them into fewer tests. This may not matter for small problems, test suites for large test domains can exhibit bigger differences. - -Test generation efficiency is one aspect of a tool that a user will want to consider when choosing a tool to use, but numbers are easier to compare than less tangile aspects like "usability", so a standard set of benchmarks emerged over the years. The table below summarizes efficiencies for several tools that happened to publish their numbers. - -| # | Model | 3â´ | 3¹³ | 4¹ⵠ3¹ⷠ2²⹠| 4¹ 3³⹠2³ⵠ| 2¹â°â° | 10²Ⱐ| Source | -| :---- | :----: | ----: | ----: | ----: | ----: | ----: | ----: | :---- | -| 1 | AETG | 9 | 15 | 41 | 28 | 10 | 180 | Y. Lei and K. C. Tai [In-parameter-order: a test generation strategy for pairwise testing](http://www-cse.uta.edu/~ylei/paper/hase.pdf), p. 8. | -| 2 | IPO | 9 | 17 | 34 | 26 | 15 | 212 | K. C. Tai and Y. Lei [A Test Generation Strategy for Pairwise Testing](http://ranger.uta.edu/~ylei/paper/ipo-tse.pdf) p. 2. | -| 3 | TConfig | 9 | 15 | 40 | 30 | 14 | 231 | A. W. Williams [Determination of Test Configurations for Pair-wise Interaction Coverage](http://www.site.uottawa.ca/~awilliam/papers/Testcom2000.pdf), p. 15. | -| 4 | CTS | 9 | 15 | 39 | 29 | 10 | 210 | A. Hartman and L. Raskin [Problems and Algorithms for Covering Arrays](http://www.haifa.il.ibm.com/projects/verification/mdt/papers/AlgorithmsForCoveringArraysPublication191203.pdf), p. 11. | -| 5 | Jenny | 11 | 18 | 38 | 28 | 16 | 193 | Supplied by Bob Jenkins. | -| 6 | TestCover | 9 | 15 | 29 | 21 | 10 | 181 | Supplied by George Sherwood. | -| 7 | DDA | ? | 18 | 35 | 27 | 15 | 201 | C. J. Colbourn, M. B. Cohen, R. C. Turban [A Deterministic Density Algorithm for Pairwise Interaction Coverage](http://www.public.asu.edu/~rturban/dda.pdf), p. 6. | -| 8 | AllPairs [McDowell] | 9 | 17 | 34 | 26 | 14 | 197 | Supplied by Bob Jenkins. | -| 9 | PICT | 9 | 18 | 37 | 27 | 15 | 210 | Supplied by Jacek Czerwonka. | -| 10 | EXACT | 9 | 15 | ? | 21 | 10 | ? | J. Yan, J. Zhang [Backtracking Algorithms and Search Heuristics to Generate Test Suites for Combinatorial Testing](http://doi.ieeecomputersociety.org/10.1109/COMPSAC.2006.33), p. 8. | -| 11 | IPO-s | 9 | 17 | 32 | 23 | 10 | 220 | A. Calvagna, A. Gargantini [IPO-s: Incremental Generation of Combinatorial Interaction Test Data Based on Symmetries of Covering Arrays](http://www2.computer.org/portal/web/csdl/doi/10.1109/ICSTW.2009.7), p. 17. | -| 12 | ecFeed | 10 | 19 | 37 | 28 | 16 | 203 | Supplied by Patryk Chamuczynski. | -| 13 | JCUnit | 10 | 23 | 49 | 33 | 18 | 245 | Supplied by Hiroshi Ukai [link](https://github.com/dakusui/jcunit/blob/0.8.x-develop/src/test/java/com/github/dakusui/jcunit8/experiments/StandardFactorSpaces.java). | -| 14 | CoverTable | 9 | 17 | 34 | 26 | 12 | 195 | CoverTable's [webpage](https://github.com/walkframe/covertable). | diff --git a/docs/index.md b/docs/index.md deleted file mode 100644 index a9fb8e89a772353d1b83f7947c562838ab3a3109..0000000000000000000000000000000000000000 --- a/docs/index.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -layout: default ---- - -Pairwise testing is an effective test case generation technique that is based on the observation that most faults are caused by interactions of at most two factors. Pairwise-generated test suites cover all combinations of two therefore are much smaller than exhaustive ones yet still very effective in finding defects. - -### Resources -You can read more about this technique in many [papers](https://scholar.google.com/scholar?hl=en&as_sdt=0%2C48&q=pairwise+testing&btnG=) and [books](https://www.amazon.com/gp/search?ie=UTF8&tag=pairwise-20&linkCode=ur2&linkId=d32d92503bf7cc820739af70b6913097&camp=1789&creative=9325&index=books&keywords=software%20combinatorial%20testing) published over the years. - -### Tools -There are many [tools](./tools.md) that adopted and integrated pairwise testing as one of the test generation methods. - -### Links -* [Pairwise testing tools](./tools.md) -* [Comparison of tools efficiency](./efficiency.md) -* [PICT Pairwise Test Generation tool](https://github.com/microsoft/pict) -* [About this site](./about.md) - diff --git a/docs/tools.md b/docs/tools.md deleted file mode 100644 index eb731a5268ebd75bed6cdb93165f9d3b4760623f..0000000000000000000000000000000000000000 --- a/docs/tools.md +++ /dev/null @@ -1,65 +0,0 @@ ---- -layout: default ---- - -[(back)](./) - -# Available Tools - -| Tool | Authors | Notes | -| ---- | ---- | ---- | -| [CATS (Constrained Array Test System)](http://testcover.com/pub/background/index.php) | [Sherwood] Bell Labs. | | -| [OATS (Orthogonal Array Test System)](http://www.isixsigma.com/library/content/c030106a.asp) | [Phadke] ATT | | -| [AETG](http://aetgweb.argreenhouse.com/) | Telecordia | Web-based, commercial | -| [IPO (PairTest)](http://www-cse.uta.edu/~ylei/paper/hase.pdf) | [Tai/Lei] | | -| [TConfig](http://www.site.uottawa.ca/~awilliam/) | [Williams] | Java-applet | -| [TCG (Test Case Generator)](http://csmiss.jpl.nasa.gov/new/set/2000-prop1.doc) | NASA | | -| [AllPairs](http://www.satisfice.com/testmethod.shtml) | Satisfice | Perl script, free, GPL | -| [Pro-Test](http://www.sigmazone.com/protest.htm) | SigmaZone | GUI, commercial | -| [CTS (Combinatorial Test Services)](http://www.alphaworks.ibm.com/tech/cts) | IBM | Free for non-commercial use | -| [Jenny](http://burtleburtle.net/bob/math/jenny.html) | [Jenkins] | Command-line, free, public-domain | -| [ReduceArray2](http://www.sstc-online.org/proceedings/2004/PDFFiles/GTD683.pdf) | STSC, U.S. Air Force | Spreadsheet-based, free | -| [TestCover](http://www.testcover.com) | Testcover.com | Web-based, commercial | -| [DDA](http://www.public.asu.edu/~rturban/dda.pdf) | [Colburn/Cohen/Turban] | | -| [Test Vector Generator](http://sourceforge.net/projects/tvg/) | | GUI, free | -| [OA1](http://www.software-metrics.org/tools_testing.asp) | k sharp technology | | -| [TESTONA](https://www.assystem-germany.com/en/products/testona/) | Assystem Germany | GUI, free for non-comercial use | -| [AllPairs](http://www.mcdowella.demon.co.uk/allPairs.html) | [McDowell] | Command-line, free | -| [Intelligent Test Case Handler (replaces CTS)](http://alphaworks.ibm.com/tech/whitch) | IBM | Free for non-commercial use | -| [CaseMaker](http://www.casemakerinternational.com/) | DÃaz & Hilterscheid | GUI, commercial | -| [PICT](http://download.microsoft.com/download/f/5/5/f55484df-8494-48fa-8dbd-8c6f76cc014b/pict33.msi) | Microsoft Corp. | Command-line, open source at <a href="http://github.com/microsoft/pict">http://github.com/microsoft/pict</a> | -| [rdExpert](http://www.phadkeassociates.com/index_files/producthome.htm) | Phadke Associates, Inc. | | -| [OATSGen](http://delivery.acm.org/10.1145/1250000/1241582/p2-krishnan.pdf?key1=1241582&key2=7825558711&coll=ACM&dl=ACM&CFID=15151515&CFTOKEN=6184618) | Motorola | | -| [SmartTest](http://www.smartwaretechnologies.com/smarttestprod.htm) | Smartware Technologies Inc. | GUI, commercial | -| [EXACT](http://doi.ieeecomputersociety.org/10.1109/COMPSAC.2006.33 ) | [Yan/Zhang] | | -| [AllPairs](http://engineering.meta-comm.com/allpairs.aspx) | MetaCommunications | Free | -| [ATD](http://www.atyoursideconsulting.com/products/atd/atd_description.html) | AtYourSide Consulting | GUI, commercial | -| [ACTS [formerly: FireEye]](http://csrc.nist.gov/acts) | NIST | GUI | -| [Bender RBT Inc.](http://www.benderrbt.com/bendersoftware.htm) | BenderRBT | GUI, commercial | -| [Pairwise Test Case Generator](http://www.testersdesk.com/pairwse_testersdesk.html) | TestersDesk | Web-based | -| [Combo-Test](http://aehrc.com/research/health-data-management-and-semantics/combo-test) | The Australian eHealth Research Centre | Command-line, free | -| [IPO-s](http://www2.computer.org/portal/web/csdl/doi/10.1109/ICSTW.2009.7) | [Calvagna/Gargantini] | | -| [VPTAG ](http://sourceforge.net/projects/vptag/) | [Robert Vanderwall] | | -| [SpecExplorer](http://msdn.microsoft.com/en-us/library/ee620448.aspx) | Microsoft Corp. | GUI, free | -| [IBM Functional Coverage Unified Solution](http://researcher.ibm.com/project/1871) | IBM | GUI, commercial | -| [CombTestWeb ](http://alarcosj.esi.uclm.es/CombTestWeb/) | Universidad de Castilla-La Mancha | Web-based, free | -| [Hexawise](http://hexawise.com/) | Hexawise | Web-based, free & commercial | -| [PictMaster](http://en.sourceforge.jp/projects/pictmaster/) | IWATSU System & Software | Spreadsheet-based, free | -| [NTestCaseBuilder](https://www.nuget.org/packages/NTestCaseBuilder/) | [Murphy] | .NET library | -| [tcases](https://code.google.com/p/tcases/) | [Kimbrough] | Command-line, free | -| [Pairwiser](https://inductive.no/pairwiser/) | Inductive AS | Web-based, free & commercial | -| [NUnit](http://nunit.org/index.php?p=pairwise&r=2.6.4) | Poole et al | Unit test framework | -| [ecFeed](http://ecfeed.com/) | ecFeed AS | Standalone, Eclipse plug-in, and jUnit runner | -| [TechQA](http://pairwise.techqa.org:8080/) | | Web-based, free | -| [Pairwise Online Tool](https://pairwise.teremokgames.com/4s8/) | [Dementiev] | Web-based, free | -| [JCUnit](https://github.com/dakusui/jcunit) | [Ukai] | Unit test framework | -| [CAGen](https://matris.sba-research.org/tools/cagen/) | SBA Research | Web-based and command-line | -| [CTWedge](https://foselab.unibg.it/ctwedge/) | University of Bergamo | Web-based | -| [CAMetrics](https://matris.sba-research.org/tools/cametrics) | SBA Research | Web-based | -| [SQA Mate Tools: Pairwise](https://sqamate.com/tools/pairwise?rh=from.pairwise.org) | [Sotskov] | Web-based | -| [AllPairsPy](https://github.com/thombashi/allpairspy/) | [Hombashi] | Python library | -| [Pairwise Pict Online](https://pairwise.yuuniworks.com/) | [Tamura] | PICT on the web | -| [CoverTable](https://github.com/walkframe/covertable) | [Yasuyuki] | Python and TypeScript. Open source | - -# Links -* [Comparison of efficiency](./efficiency.md)