Nantes Université

Skip to content
Extraits de code Groupes Projets
Non vérifiée Valider 8038d659 rédigé par Jacek Czerwonka's avatar Jacek Czerwonka Validation de GitHub
Parcourir les fichiers

Refresh of the main doc (#102)

parent c5b19317
Aucune branche associée trouvée
Aucune étiquette associée trouvée
Aucune requête de fusion associée trouvée
...@@ -6,9 +6,9 @@ jacekcz@microsoft.com ...@@ -6,9 +6,9 @@ jacekcz@microsoft.com
PICT generates test cases and test configurations. With PICT, you can generate tests that are more effective than manually generated tests and in a fraction of the time required by hands-on test case design. PICT generates test cases and test configurations. With PICT, you can generate tests that are more effective than manually generated tests and in a fraction of the time required by hands-on test case design.
PICT runs as a command line tool. Prepare a model file detailing the parameters of the interface (or set of configurations, or data) you want to test. PICT generates a compact set of parameter value choices that represent the test cases you should use to get comprehensive combinatorial coverage of your parameters. PICT runs as a command line tool. Prepare a model file detailing the parameters of the interface (or a set of configurations, or data) you want to test and PICT generates a compact set of parameter value choices that represent the test cases you should use to get comprehensive combinatorial coverage of your parameters.
For instance, to create a test suite for disk partition creation, the domain can be described by the following parameters: **Type**, **Size**, **File system**, **Format method**, **Cluster size**, and **Compression**. Each parameter consists of a finite number of possible values. For example, **Compression** naturally can only be **On** or **Off**, other parameters are made finite with help of equivalence partitioning (e.g. **Size**). For instance, to create a test suite for disk partition creation, the domain can be described by the following parameters: ```Partition Type```, ```Partition Size```, ```File System```, ```Format Method```, ```Cluster Size```, and ```Compression```. Each parameter consists of a finite number of possible values. For example, ```Compression``` can only be ```On``` or ```Off```, other parameters are made finite with help of equivalence partitioning:
Type: Single, Span, Stripe, Mirror, RAID-5 Type: Single, Span, Stripe, Mirror, RAID-5
Size: 10, 100, 500, 1000, 5000, 10000, 40000 Size: 10, 100, 500, 1000, 5000, 10000, 40000
...@@ -17,7 +17,7 @@ For instance, to create a test suite for disk partition creation, the domain can ...@@ -17,7 +17,7 @@ For instance, to create a test suite for disk partition creation, the domain can
Cluster size: 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536 Cluster size: 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536
Compression: On, Off Compression: On, Off
For such a model, thousands of possible test cases can be generated. It would be difficult to test all of them in a reasonable amount of time. Instead of attempting to cover all possible combinations, we settle on testing all possible pairs of values. For example, **{Single, FAT}** is one pair, **{10, Slow}** is another. Consequently, one test case can cover many pairs. Research shows that testing all pairs is an effective alternative to exhaustive testing and much less costly. It will provide very good coverage and the number of test cases will remain manageable. For such a model, thousands of possible test cases can be generated. It would be difficult to test all of them in a reasonable amount of time. Instead of attempting to cover all possible combinations, we settle on testing all possible pairs of values. For example, ```{Type:Single, File system:FAT}``` is one such value pair, ```{Size:10, Format method:Slow}``` is another. Consequently, one test case can cover many pairs. Research shows that testing all pairs is an effective alternative to exhaustive testing and is much less costly.
# Usage # Usage
...@@ -37,13 +37,13 @@ PICT is a command-line tool that accepts a plain-text model file as an input and ...@@ -37,13 +37,13 @@ PICT is a command-line tool that accepts a plain-text model file as an input and
## Model File ## Model File
A model consists of the following sections: A model is a plain-text file consisting of the following sections:
parameter definitions parameter definitions
[sub-model definitions] [sub-model definitions]
[constraint definitions] [constraint definitions]
Model sections should always be specified in the order shown above and cannot overlap. All parameters should be defined first, then sub-models, and then constraints. The sub-models definition section is optional, as well as the constraints. Sections do not require any special separators between them. Empty lines can appear anywhere. Comments are also permitted and they should be prefixed with # character. The sections should always be specified in the order shown above. All parameters should be defined first, then sub-models, and then constraints. Parameter definitions are required, other parts are optional. Sections do not require any special separators between them. Empty lines can appear anywhere. Comments are permitted and they should be prefixed with # character.
To produce a basic model file, list the parameters—each in a separate line—with the respective values delimited by commas: To produce a basic model file, list the parameters—each in a separate line—with the respective values delimited by commas:
...@@ -52,7 +52,7 @@ To produce a basic model file, list the parameters—each in a separate line—w ...@@ -52,7 +52,7 @@ To produce a basic model file, list the parameters—each in a separate line—w
Example: Example:
# #
# This is a sample model for testing volume create/delete functions # This is a sample model for testing volume creation
# #
Type: Primary, Logical, Single, Span, Stripe, Mirror, RAID-5 Type: Primary, Logical, Single, Span, Stripe, Mirror, RAID-5
...@@ -62,19 +62,21 @@ Example: ...@@ -62,19 +62,21 @@ Example:
Cluster size: 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536 Cluster size: 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536
Compression: on, off Compression: on, off
A comma is the default separator but you can specify a different one using **/d** option. A comma is the default separator but you can specify a different one using ```/d``` option.
By default, PICT generates a pair-wise test suite (all pairs covered), but the order can be set by option **/o** to a value larger than two. For example, if **/o:3** is specified, the test suite will cover all triplets of values thereby producing a larger number of tests but potentially making the test suite even more effective. The maximum order for a simple model is equal to the number of parameters, which will result in an exhaustive test suite. Following the same principle, specifying **/o:1** will produce a test suite that merely covers all values (combinations of 1). By default, PICT generates a pair-wise test suite (all pairs covered), but the order can be set by option ```/o``` to a value larger than two. For example, when ```/o:3``` is specified, the test suite will cover all triplets of values. It will produce a larger number of tests but will potentially make the test suite even more effective. The maximum order for a model is equal to the number of parameters, which will result in an exhaustive test suite. Following the same principle, specifying ```/o:1``` will produce a test suite that merely covers all values (combinations of 1).
## Output Format ## Output Format
All errors, warning messages, and the randomization seed are printed to the error stream. The test cases are printed to the standard output stream. The first line of the output contains names of the parameters. Each of the following lines represents one generated test case. Values in each line are separated by a tab. This way redirecting the output to a file creates a tab-separated value format. The test cases are printed to the standard output stream. The first line of the output contains names of the parameters. Each of the following lines represents one generated test case. Values in each line are separated by a tab. This way redirecting the output to a file creates a tab-separated value format.
If a model and options given to the tool do not change, every run will result in the same output. However, the output can be randomized if **/r** option is used. A randomized generation prints out the seed used for that particular execution to the error output stream. Consequently, that seed can be fed into the tool with **/r:seed** option to replay a particular generation. If a model and options given to the tool do not change, every run will result in the same output. However, the output can be randomized if ```/r``` option is used. A randomized generation prints out the seed used for that particular execution to the error output stream. Consequently, that seed can be fed into the tool with ```/r:seed``` option to replay a particular generation.
All errors, warning messages, and other auxiliary information is printed to the error stream.
# Constraints # Constraints
Constraints allow you to specify limitations on the domain. In the example with partitions, one of the pairs that will occur in at least one test case is **{FAT, 5000}**. In reality, the FAT file system cannot be applied on volumes larger than 4,096 MB. Note that you cannot simply remove those violating test cases from the result because an offending test case may cover other, possibly valid, pairs that would not otherwise be tested. Instead of losing valid pairs, it is better to eliminate disallowed combinations during the generation process. In PICT, this can be done by specifying constraints, for example: Constraints express inherent limitations of the modelled domain. In the example above, one of the pairs that will appear in at least one test case is ```{File system:FAT, Size:5000}```. In practice, the FAT file system cannot be applied on volumes larger than 4,096 MB. Note that you cannot simply remove those violating test cases from the result because an offending test case may cover other, possibly valid, pairs that would not otherwise be tested. Instead of losing valid pairs, it is better to eliminate disallowed combinations during the generation process. In PICT, this can be done by specifying constraints, for example:
Type: Primary, Logical, Single, Span, Stripe, Mirror, RAID-5 Type: Primary, Logical, Single, Span, Stripe, Mirror, RAID-5
Size: 10, 100, 500, 1000, 5000, 10000, 40000 Size: 10, 100, 500, 1000, 5000, 10000, 40000
...@@ -88,18 +90,18 @@ Constraints allow you to specify limitations on the domain. In the example with ...@@ -88,18 +90,18 @@ Constraints allow you to specify limitations on the domain. In the example with
## Conditional Constraints ## Conditional Constraints
A term **[parameter] relation value** is an atomic part of a constraint expression. The following relations can be used: =, <>, >, >=, <, <=, and LIKE. LIKE is a wildcard-matching operator (* - any character, ? – one character). A term ```[parameter] relation value``` is an atomic part of a constraint expression. The following relations can be used: ```=```, ```<>```, ```>```, ```>=```, ```<```, ```<=```, and ```LIKE```. ```LIKE``` is a wildcard-matching string operator (```*``` - any character, ```?``` – one character).
[Size] < 10000 [Size] < 10000
[Compression] = "OFF" [Compression] = "OFF"
[File system] like "FAT*" [File system] LIKE "FAT*"
Operator IN allows specifying a set of values that satisfy the relation explicitly: Operator IN allows specifying a set of values:
IF [Cluster size] in {512, 1024, 2048} THEN [Compression] = "Off"; IF [Cluster size] IN {512, 1024, 2048} THEN [Compression] = "Off";
IF [File system] in {"FAT", "FAT32"} THEN [Compression] = "Off"; IF [File system] IN {"FAT", "FAT32"} THEN [Compression] = "Off";
The IF, THEN, and ELSE parts of an expression may contain multiple terms joined by logical operators: NOT, AND, and OR. Parentheses are also allowed in order to change the default operator priority: The ```IF```, ```THEN```, and ```ELSE``` parts of an expression may contain multiple terms joined by logical operators: ```NOT```, ```AND```, and ```OR```. Parentheses are allowed in order to change the default operator priority:
IF [File system] <> "NTFS" OR IF [File system] <> "NTFS" OR
( [File system] = "NTFS" AND [Cluster size] > 4096 ) ( [File system] = "NTFS" AND [Cluster size] > 4096 )
...@@ -109,7 +111,7 @@ The IF, THEN, and ELSE parts of an expression may contain multiple terms joined ...@@ -109,7 +111,7 @@ The IF, THEN, and ELSE parts of an expression may contain multiple terms joined
( [File system] = "NTFS" AND NOT [Cluster size] <= 4096 )) ( [File system] = "NTFS" AND NOT [Cluster size] <= 4096 ))
THEN [Compression] = "Off"; THEN [Compression] = "Off";
Parameters can also be compared to other parameters, like in this example: Parameters can be compared to other parameters, like in this example:
# #
# Machine 1 # Machine 1
...@@ -136,32 +138,34 @@ An invariant declares an always valid limitation of a domain: ...@@ -136,32 +138,34 @@ An invariant declares an always valid limitation of a domain:
# At least one parameter must be different to be a meaningful test case # At least one parameter must be different to be a meaningful test case
# #
[OS_1] <> [OS_2] or [SKU_1] <> [SKU_2] or [LANG_1] <> [LANG_2]; [OS_1] <> [OS_2] OR [SKU_1] <> [SKU_2] OR [LANG_1] <> [LANG_2];
# #
# All parameters must be different (we use AND operator) # All parameters must be different (we use AND operator)
# #
[OS_1] <> [OS_2] and [SKU_1] <> [SKU_2] and [LANG_1] <> [LANG_2]; [OS_1] <> [OS_2] AND [SKU_1] <> [SKU_2] AND [LANG_1] <> [LANG_2];
## Types ## Types
PICT uses the concept of a parameter type. There are two types of parameters: string and numeric. A parameter is considered numeric only when all its values are numeric. If a value has multiple names, only the first one counts. Types are only important when evaluating constraints. A numeric parameter is only comparable to a number, and a string parameter is only comparable to another string. For example: PICT has a simple type system. There are two types of parameters: a string and a numeric type. Types do not have to be expicitely declared. A parameter is considered numeric when all of its values can be converted to a number (an integer or a float).
Types are only important when evaluating constraints. A numeric parameter is only comparable to a number, and a string parameter is only comparable to another string. For example:
Size: 1, 2, 3, 4, 5 Size: 1, 2, 3, 4, 5
Value: a, b, c, d Value: a, b, c, d
IF [Size] > 3 THEN [Value] > "b"; IF [Size] > 3 THEN [Value] > "b";
String comparison is lexicographical and case-insensitive by default. Numerical values are compared as numbers. If a value has multiple names, only the first is considered when PICT detects the type.
## Case Sensitiveness ## Case Sensitiveness
By default, PICT does all its comparisons and checks case-insensitively. For instance, if there are two parameters defined: *OS* and *os*, a duplication of names will be detected (parameter names must be unique). Constraints are also resolved case-insensitively by default: By default, PICT does all its comparisons and checks case-insensitively. For instance, if there are two parameters defined: ```OS``` and ```os```, a duplication of names will be detected (parameter names must be unique). Constraints are also resolved case-insensitively by default:
IF [OS] = "Win10" THEN ... IF [OS] = "Win10" THEN ...
will match both *Win10* and *win10* values (values of a parameter are not required to be unique). Option **/c** however, makes the model evaluation fully case-sensitive. will match both ```Win10``` and ```win10``` values (values of a parameter are not required to be unique). Option ```/c``` however, makes the model evaluation fully case-sensitive.
# Advanced Modelling Features # Advanced Modelling Features
...@@ -183,7 +187,7 @@ Once a parameter is defined, it can help in defining other parameters. ...@@ -183,7 +187,7 @@ Once a parameter is defined, it can help in defining other parameters.
SKU_2: <SKU_1> SKU_2: <SKU_1>
LANG_2: <LANG_1>, Hindi LANG_2: <LANG_1>, Hindi
Less typing and better maintainability. Less typing and better for maintainability.
## Sub-Models ## Sub-Models
...@@ -217,25 +221,27 @@ The test generation for the above model would proceed as follows: ...@@ -217,25 +221,27 @@ The test generation for the above model would proceed as follows:
{ PLATFORM, CPUS, RAM, HDD } { OS, Browser } APP { PLATFORM, CPUS, RAM, HDD } { OS, Browser } APP
Notes: Notes:
1. You can define as many sub-models as you want; any parameter can belong to any number of sub-models. Model hierarchy can be just one level deep. 1. You can define as many sub-models as you want; any parameter can belong to any number of sub-models. However, the model hierarchy can be just one level deep.
2. The combinatory order of a sub-model cannot exceed the number of its parameters. In the example above, an order of the first sub-model can be any value between one and four. 2. The combinatory order of a sub-model cannot exceed the number of its parameters. In the example above, an order of the first sub-model can be any value between one and four.
3. If you do not specify the order for a sub-model, the default order, as specified by /o option, will be used. 3. If you do not specify the order for a sub-model, the order specified by ```/o``` option will be used.
## Aliasing ## Aliasing
Aliasing is a way of specifying multiple names for a single value. Multiple names do not change the combinatorial complexity of the model. No matter how many names a value has, it is treated as one entity. The only difference will be in the output; any test case that would normally have that one value will have one of its names instead. Names are rotated among the test cases. Specifying one value with two names will result in having them both show up in the output without additional test cases. Aliasing is a way of specifying multiple names for a single value. Multiple names do not change the combinatorial complexity of the model. No matter how many names a value has, they are treated as one entity. The only difference will be in the output; any test case that would normally have that one value will have one of its names instead. Names are rotated among the test cases.
By default, names should be separated by | character but this can be changed with option **/a**. By default, names should be separated by | character but this can be changed with option ```/a```.
OS_1: Win2008, Win2012, Win2016 OS_1: Win2008, Win2012, Win2016
SKU_1: Professional, Server | Datacenter SKU_1: Professional, Server | Datacenter
Note: Note:
When evaluating constraints, only the first name counts. For instance, **[SKU_1] = "Server"** will result in a match but **[SKU_1] = "Datacenter"** will not. Also, only the first name is used to determine whether a value is negative or a numeric type. 1. When evaluating constraints, only the first name counts. For instance, ```[SKU_1] = "Server"``` will result in a match but ```[SKU_1] = "Datacenter"``` will not. Also, only the first name is used to determine whether a value is "negative" or for the purpose of detecting its type.
## Negative Testing ## Negative Testing
In addition to testing valid combinations, referred to as “positive testing,” it is often desirable to test using values outside the allowable range to make sure the program handles errors properly. This “negative testing” should be conducted such that only one invalid value is present in any test case. This is due to the way in which typical applications are written: namely, to take some failure action upon the first error detected. For this reason, a problem known as input masking—in which one invalid input prevents another invalid input from being tested—can occur with negative testing. In addition to testing valid combinations, referred to as “positive testing,” it is often desirable to test using values outside the allowable range to make sure the program handles errors properly. This “negative testing” should be conducted such that only one out-of-range value is present in any test case. This is because of the way in which typical applications are written: namely, they often error out when the first error is detected.
"Input masking" is a type of problem in test design, in which one out-of-range input prevents another invalid input from being tested.
Consider the following routine, which takes two arguments: Consider the following routine, which takes two arguments:
...@@ -248,19 +254,18 @@ Consider the following routine, which takes two arguments: ...@@ -248,19 +254,18 @@ Consider the following routine, which takes two arguments:
}; };
Although the routine can be called with any numbers a or b, it only makes sense to do the calculation on non-negative numbers. For that reason by the way, the routine does verifications [1] and [2] on the arguments. Now, assume a test ( a= -1, b = -1 ) was used to test a negative case. Here, a = -1 actually masks b = -1 because check [2] never gets executed and it would go unnoticed if it didn’t exist. Although the routine can be called with any numbers for ```a``` or ```b```, it only makes sense to do the calculation on non-negative numbers. For that reason by the way, the routine does verifications [1] and [2] on the arguments. Now, assume a test ```( a: -1, b: -1 )``` was used to test that negative case. Here, ```a: -1``` "masks" ```b: -1``` because the check [2] never gets executed. If it did not exist at all, that problem in code would go unnoticed.
To prevent input masking issues, it is important that two out-of-range values do not appear in the same test case. Prefixing a value with ```~``` (tilde) marks it invalid or out-of-allowable-range. Option ```/n``` allows for specifying a different prefix.
To prevent input masking, it is important that two invalid values (of two different parameters) are not in the same test case. Prefixing any value with ~ (tilde) marks it invalid. Option **/n** allows for specifying a different prefix.
# #
# Trivial model for SumSquareRoots # SumSquareRoots model
# #
A: ~-1, 0, 1, 2 A: ~-1, 0, 1, 2
B: ~-1, 0, 1, 2 B: ~-1, 0, 1, 2
The tool guarantees that all possible pairs of valid values will be covered and all possible combinations of any invalid value will be paired with all positive values at least once. PICT guarantees that all possible pairs of valid (or in-range) values will be covered and all possible combinations of any out-of-range / invalid value will be paired only with all valid values.
A B A B
2 0 2 0
...@@ -280,19 +285,22 @@ The tool guarantees that all possible pairs of valid values will be covered and ...@@ -280,19 +285,22 @@ The tool guarantees that all possible pairs of valid values will be covered and
~-1 0 ~-1 0
Notes: Notes:
1. A prefix is not a part of a value when it comes to comparisons therefore in constraints it should appear without it. A valid constraint (although quite artificial in this example) would be: **if [A] = -1 then [B] = 0;**. Also checking for the type of a value is not affected by the prefix. For example, both parameters in the above example are numeric despite having non-numeric “~” in one of their values. The prefix however will show up in the output. 1. The ```~``` prefix is not a part of a value when it comes to type detection or constraint evaluation. A valid constraint for the above example would look like this: ```if [A] = -1 then [B] = 0;```. The ```~``` prefix however will show up in the output.
2. If a value has multiple names, only prefixing the first name will make the value negative. 2. If a value has multiple names, only prefixing the first name will make the value out-of-range.
### Excluding other parameters ## Excluding Entire Parameters
Sometimes we may want to ignore other parameters for certain values. A negative test is a typical scenario, as a failure may make any other parameters meaningless if it stops the application. In these cases we do not want to 'use up' any testable values for other parameters. A technique to handle this is be to add a dummy value to the parameters to be excluded. Since it is a dummy value, it must in turn be excluded from the other tests, which can be accomplished using the ELSE clause.
Some test suites may require that under centrain conditions entire parameters should be ignored. Negative testing could be such a scenario. There, an error or a failure may stop the application under test and make any other parameter choices meaningless. By design, PICT will try to pack as many combinations into fewest test cases possible, including those that contain values that will trigger failures. In these cases, we will see testable pairs esentially wasted.
PICT does not have any special handling for this scenario however one technique that could be useful is to add a dummy value to the parameters should be ignored.
Consider the following example where a `P1` value of `-1` is a negative test where we expect the application to fail. Consider the following example where a ```P1: -1``` is a value that triggers an application error.
P1: -1, 0, 1 P1: -1, 0, 1
P2: A, B, C P2: A, B, C
P3: X, Y, Z P3: X, Y, Z
We want to test this independently and not associate any `P2` and `P3` values with it, as it would needlessly increase the amount of tests and use up `P2`/`P3` combinations. So we add a dummy value `NA` to the `P2` and `P3` sets and exclude them from other cases with the ELSE clause: We want to test this condition independently and not associate any values of ```P2``` or ```P3``` with it. To accomplish that, we add a dummy value ```NA``` to parameters ```P2``` and ```P3```. That dummy value will indicate that the parameter should be ignored. We can then use constraints to make sure tests are generated correctly:
P1: -1, 0, 1 P1: -1, 0, 1
P2: A, B, C, NA P2: A, B, C, NA
...@@ -302,7 +310,7 @@ We want to test this independently and not associate any `P2` and `P3` values wi ...@@ -302,7 +310,7 @@ We want to test this independently and not associate any `P2` and `P3` values wi
THEN [P2] = "NA" AND [P3] = "NA" THEN [P2] = "NA" AND [P3] = "NA"
ELSE [P2] <> "NA" AND [P3] <> "NA"; ELSE [P2] <> "NA" AND [P3] <> "NA";
This will result in a single test for `P1 = -1` and no `NA`'s anywhere else; and no `P2` and `P3` combinations are used up for the -1 test: This will result in a single test for ```P1: -1``` with ```NA``` set for ```P2``` and ```P3```. No other combinations values of ```P2``` and ```P3``` are used up for the test containing ```P1: -1```:
P1 P2 P3 P1 P2 P3
0 C Z 0 C Z
...@@ -316,7 +324,7 @@ This will result in a single test for `P1 = -1` and no `NA`'s anywhere else; and ...@@ -316,7 +324,7 @@ This will result in a single test for `P1 = -1` and no `NA`'s anywhere else; and
0 B Y 0 B Y
1 A Z 1 A Z
However it will get progressively more complex if multiple layers of exclusions are needed. For example let's extend the above with an additional negative test where `P2` is `Null`: The downside of this method is that constraints will get progressively more complex. For example, when additional failure inducing values are added to the model e.g. ```P2: Null```:
P1: -1, 0, 1 P1: -1, 0, 1
P2: A, B, C, Null, NA P2: A, B, C, Null, NA
...@@ -338,7 +346,7 @@ Two additional tests will be added (not all output shown): ...@@ -338,7 +346,7 @@ Two additional tests will be added (not all output shown):
## Weighting ## Weighting
The generation mechanism can be forced to prefer certain values. This is done by means of weights. A weight can be any positive integer. If it is not specified explicitly it is assumed to be 1: Weights tell the generator to prefer certain parameter values over others. Weights are positive integers. When not explicitly specified, values have a weight of 1:
# #
# Let’s focus on primary partitions formatted with NTFS # Let’s focus on primary partitions formatted with NTFS
...@@ -348,55 +356,48 @@ The generation mechanism can be forced to prefer certain values. This is done by ...@@ -348,55 +356,48 @@ The generation mechanism can be forced to prefer certain values. This is done by
Format method: quick, slow Format method: quick, slow
File system: FAT, FAT32, NTFS (10) File system: FAT, FAT32, NTFS (10)
Note: Weights are not guarantees but merely opportunistic hints. You cannot be sure that the weights you specified will be honored at all.
Weight values have no intuitive meaning. For example, when a parameter is defined like this:
File system: FAT, FAT32, NTFS (10)
it does not mean that NTFS will appear in the output ten times more often than FAT or FAT32. Moreover, you cannot be sure that the weights you specified will be honored at all. PICT's main objective is to cover all combinations of values in the smallest number of test cases. Only when value choices do not affect that coverage criterion, weights will be used to determine the final result.
The reason for this is that we deal with two contradictory requirements: For that reason, weight values have no intuitive meaning. For example, when a parameter is defined like this:
1. To cover all combinations in the smallest number of test cases.
2. To choose values proportionally to their weights.
(1) will always take precedence over (2) and weights will only be honored when the choice of a value is not determined by a need of satisfying (1). More specifically, during the test case production, candidate values are evaluated and one that covers most of the still unused combinations is always picked. Sometimes there is a tie among candidate values and really no choice is better than another. In those cases, weights will be used to determine the final choice. File system: FAT, FAT32, NTFS (10)
The bottom-line is that you can use weights to attempt to shift the bias towards some values but whether or not to honor that and to what extent is determined by multiple factors, not weights alone. It does not mean that NTFS will appear in the output ten times more often than FAT or FAT32. It will merely be 10 times more likely to be chosen when the choice does not affect the coverage.
## Seeding ## Seeding
Seeding makes two scenarios possible: Seeding makes two scenarios possible:
1. Allows for specifying important e.g. regression-inducing combinations that should end up in any generated test suite. In this case, PICT will initialize the output with combinations you provide and will build up the rest of the suite on top of them, still making sure all n-order combinations get covered. 1. Allows for specifying important e.g. regression-inducing combinations that should end up in any generated test suite. In this case, PICT will initialize the output with combinations you provide and will build up the rest of the suite on top of those.
2. Let's you minimize changes to the output when your model is modified. Provide PICT with results of some previous generation and expect the tool to reuse the old test cases as much as possible. 2. Minimize changes to the output when your model is modified. You can feed PICT the results of a previous test generation and expect the tool to reuse the old test cases as much as possible.
Seeding rows must be defined in a separate file (a seeding file). Use new option **/e** to provide the tool with the file location: Seeding rows must be defined in a separate file. Use option ```/e``` to specify the seeding file's location:
pict.exe model.txt /e:seedrows.txt pict.exe model.txt /e:seedrows.txt
Seeding files have the same format as any PICT output. First line contains parameter names separated by tab characters and each following line contains one seeding row with values also separated by tabs. This format can easily be prepared from scratch (scenario 1) either in Notepad or in Excel and also allows for quick and direct reuse of any prior results (scenario 2). Seeding files have the same format as any PICT output. The first line contains parameter names separated by tab characters and each following line contains one test case with values also separated by tabs. This format can easily be prepared manually or prior output can be directly fed back to PICT:
Ver SKU Lang Arch Ver SKU Lang Arch
Win7 Pro EN x86 Win7 Pro EN x86
Win7 FR x86 Win7 FR x86
Win10 Pro EN x64 Win10 Pro EN x64
Any seeding row may be complete i.e. with values specified for all parameters, or partial, like the second seeding row above which does not have a value for the SKU parameter. In this case, the generation process will take care of choosing the best value for SKU. A seeding row may be complete i.e. have values specified for all parameters. It can also be partial, e.g. the second seeding row in the example above does not have a value for the ```SKU``` parameter. In this case, the generation process will take care of choosing the best value for ```SKU```.
Important notes:
There are a few rules of matching seeding rows with the current model: There are a few rules of matching seeding rows with the current model:
1. If a seeding file contains a parameter that is not in the current model, the entire column representing the parameter in the seeding file will be discarded. 1. If a seeding file contains a parameter that is not in the current model, the entire column representing the parameter in the seeding file will be ignored. Parameters are matched by name.
2. If a seeding row contains a value that is not in the current model, the value will be removed from the seeding row. The rest of the row however, if valid, will still be used. As a result of that, a complete row may become partial. 2. If a seeding row contains a value that is not in the current model, the value will be ignored. The rest of the row however, if valid, will still be used. As a result of that, a complete row may become a partial row.
3. If a seeding row violates any of the current constraints, it will be skipped entirely. 3. If a seeding row violates any of the current constraints, it will be skipped entirely.
PICT will issue warnings if (1) or (2) occurs. PICT will issue warnings if (1) or (2) occurs.
In addition, there are a few things that are normally allowed in PICT models but may lead to ambiguities when seeding is used: In addition, there are a few things that are normally allowed in PICT models but may lead to ambiguities when seeding is used:
1. Blank parameter and value names. 1. Blank parameter and value names.
2. Parameter and value names containing tab characters. 2. Parameter and value names containing tab characters.
3. Values and all their aliases not unique within a parameter. 3. Values and all their aliases not unique within a parameter.
When seeding is used, you will be warned if any of the above problems are detected in your model. These are best to avoid in any case. When seeding is used, you will be warned if any of the above problems are detected in your model.
# Constraints Grammar # Constraints Grammar
......
0% Chargement en cours ou .
You are about to add 0 people to the discussion. Proceed with caution.
Terminez d'abord l'édition de ce message.
Veuillez vous inscrire ou vous pour commenter