OpenCascade/occt
1
0
Fork 0
mirror of https://git.dev.opencascade.org/repos/occt.git synced 2025-09-08 14:17:06 +03:00
Code Activity

0026622: Tests - Avoid using dlog and decho commands in test scripts

Browse Source 
Added possibility to define patterns that must be present in the test log, so that test is considered OK only if all these patterns are found.
New REQUIRED statement is introduced for that, documented in dox/dev_guides/tests/tests.md.

Removed all uses of decho and dlog commands, added REQUIRED where necessary.
Command xdistcs is modified to output to Tcl instead of cout, and extended to report errors and warnings if distances are greater than tolerance (directly, instead of complex post-processing on Tcl level).

DEBUG mode for TODOs was removed (we should have no deviations in Debug mode).

Corrected indentation in DrawResources/TestCommands.tcl

HTML log will now highlight TODO statement causing IMPROVEMENT status, or REQUIRED statement causing FAIL, by corresponding color.
This commit is contained in:
abv
2015-10-09 20:15:11 +03:00
committed by bugmaster
parent ee9e67edc7
commit 6d3685029f
144 changed files with 1195 additions and 2642 deletions
Download Patch File Download Diff File Expand all files Collapse all files

71
dox/dev_guides/tests/tests.md
View File

@@ -7,7 +7,7 @@
This document provides OCCT developers and contributors with an overview and practical guidelines for work with OCCT automatic testing system.
Reading the Introduction is sufficient for OCCT developers to use the test system to control non-regression of the modifications they implement in OCCT. Other sections provide a more in-depth description of the test system, required for modifying the tests and adding new test cases.
Reading the Introduction should be sufficient for developers to use the test system to control non-regression of the modifications they implement in OCCT. Other sections provide a more in-depth description of the test system, required for modifying the tests and adding new test cases.
@subsection testmanual_1_1 Basic Information
@@ -28,10 +28,11 @@ Some tests involve data files (typically CAD models) which are located separatel
@subsection testmanual_1_2 Intended Use of Automatic Tests
Each modification made in OCCT code must be checked for non-regression
by running the whole set of tests. The developer who does the modification
by running the whole set of tests. The developer who makes the modification
is responsible for running and ensuring non-regression for the tests available to him.
Note that many tests are based on data files that are confidential and thus available only at OPEN CASCADE. Thus official certification testing of the changes before integration to master branch of official OCCT Git repository (and finally to the official release) is performed by OPEN CASCADE in any case.
Note that many tests are based on data files that are confidential and thus available only at OPEN CASCADE.
The official certification testing of each change before its integration to master branch of official OCCT Git repository (and finally to the official release) is performed by OPEN CASCADE to ensure non-regression on all existing test cases and supported platforms.
Each new non-trivial modification (improvement, bug fix, new feature) in OCCT should be accompanied by a relevant test case suitable for verifying that modification. This test case is to be added by the developer who provides the modification.
@@ -49,11 +50,10 @@ For this it is recommended to add a file *DrawAppliInit* in the directory which
Example (Windows)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~{.tcl}
~~~~~{.tcl}
set env(CSF_TestDataPath) $env(CSF_TestDataPath)\;d:/occt/test-data
return ;# this is to avoid an echo of the last command above in cout
~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Note that variable *CSF_TestDataPath* is set to default value at DRAW start, pointing at the folder <i>$CASROOT/data</i>.
In this example, subdirectory <i>d:/occt/test-data</i> is added to this path. Similar code could be used on Linux and Mac OS X except that on non-Windows platforms colon ":" should be used as path separator instead of semicolon ";".
@@ -169,17 +169,19 @@ test: Run specified test case
The detailed rules of creation of new tests are given in <a href="#testmanual_3">section 3</a>. The following short description covers the most typical situations:
Use prefix <i>bug</i> followed by Mantis issue ID and, if necessary, additional suffixes, for naming the test script and DRAW commands specific for this test case.
Use prefix <i>bug</i> followed by Mantis issue ID and, if necessary, additional suffixes, for naming the test script, data files, and DRAW commands specific for this test case.
1. If the test requires C++ code, add it as new DRAW command(s) in one of files in *QABugs* package. Note that this package defines macros *QVERIFY* and *QCOMPARE*, thus code created for QTest or GoogleTest frameworks can be used with minimal modifications.
2. Add script(s) for the test case in the subfolder corresponding to the relevant OCCT module of the group bugs <i>($CASROOT/tests/bugs)</i>. See <a href="#testmanual_5_2">the correspondence map</a>.
1. If the test requires C++ code, add it as new DRAW command(s) in one of files in *QABugs* package.
2. Add script(s) for the test case in the subfolder corresponding to the relevant OCCT module of the group *bugs* <i>($CASROOT/tests/bugs)</i>. See <a href="#testmanual_5_2">the correspondence map</a>.
3. In the test script:
* Load all necessary DRAW modules by command *pload*.
* Use command *locate_data_file* to get a path to data files used by test script. (Make sure to have this command not inside catch statement if it is used.)
* Use DRAW commands to reproduce the situation being tested.
* If test case is added to describe existing problem and the fix is not available, add TODO message for each error to mark it as known problem. The TODO statements must be specific so as to match the actually generated messages but not all similar errors.
* Make sure that in case of failure the test produces message containing word "Error" or other recognized by test system as error (see files parse.rules).
4. If the test case uses data file(s) not yet present in the test database, these can be put to subfolder data of the test grid, and integrated to Git along with the test case.
* Make sure that in case of failure the test produces message containing word "Error" or other recognized by test system as error (add new error patterns in file parse.rules if necessary).
* If test case reports error due to existing problem and the fix is not available, add @ref testmanual_3_6 "TODO" statement for each error to mark it as known problem. The TODO statements must be specific so as to match the actually generated messages but not all similar errors.
* To check expected output which should be obtained as a result of a test, add @ref testmanual_3_7 "REQUIRED" statement for each line of output to mark it as required.
* If test case produces error messages (contained in parse.rules) which are expected in that test and should not be considered as its failure (e.g. test for checkshape command), add REQUIRED statement for each error to mark it as required output.
4. If the test uses data file(s) not yet present in the test database, these can be put to (sub)directory pointed out by *CSF_TestDataPath* variable for running test. The files should be attached to Mantis issue corresponding to the modification being tested.
5. Check that the test case runs as expected (test for fix: OK with the fix, FAILED without the fix; test for existing problem: BAD), and integrate to Git branch created for the issue.
Example:
@@ -188,18 +190,18 @@ Example:
~~~~~
git status –short
A tests/bugs/heal/data/OCC210a.brep
A tests/bugs/heal/data/OCC210a.brep
A tests/bugs/heal/data/bug210_a.brep
A tests/bugs/heal/data/bug210_b.brep
A tests/bugs/heal/bug210_1
A tests/bugs/heal/bug210_2
~~~~~
* Test script
~~~~~
~~~~~{.tcl}
puts "OCC210 (case 1): Improve FixShape for touching wires"
restore [locate_data_file OCC210a.brep] a
restore [locate_data_file bug210_a.brep] a
fixshape result a 0.01 0.01
checkshape result
@@ -212,10 +214,10 @@ checkshape result
Standard OCCT tests are located in subdirectory tests of the OCCT root folder ($CASROOT).
Additional test folders can be added to the test system by defining environment variable *CSF_TestScriptsPath*. This should be list of paths separated by semicolons (*;*) on Windows
or colons (*:*) on Linux or Mac. Upon DRAW launch, path to tests subfolder of OCCT is added at the end of this variable automatically.
or colons (*:*) on Linux or Mac. Upon DRAW launch, path to *tests* subfolder of OCCT is added at the end of this variable automatically.
Each test folder is expected to contain:
* Optional file parse.rules defining patterns for interpretation of test results, common for all groups in this folder
* Optional file *parse.rules* defining patterns for interpretation of test results, common for all groups in this folder
* One or several test group directories.
Each group directory contains:
@@ -304,15 +306,15 @@ The test group may contain *parse.rules* file. This file defines patterns used f
Each line in the file should specify a status (single word), followed by a regular expression delimited by slashes (*/*) that will be matched against lines in the test output log to check if it corresponds to this status.
The regular expressions support a subset of the Perl *re* syntax. See also <a href="http://perldoc.perl.org/perlre.html">Perl regular expressions</a>.
The regular expressions should follow <a href="http://www.tcl.tk/man/tcl/TclCmd/re_syntax.htm">Tcl syntax</a>, with special exception that "\b" is considered as word limit (Perl-style), in addition to "\y" used in Tcl.
The rest of the line can contain a comment message, which will be added to the test report when this status is detected.
Example:
~~~~~
FAILED /\\b[Ee]xception\\b/ exception
FAILED /\\bError\\b/ error
FAILED /\b[Ee]xception\b/ exception
FAILED /\bError\b/ error
SKIPPED /Cannot open file for reading/ data file is missing
SKIPPED /Could not read file .*, abandon/ data file is missing
~~~~~
@@ -559,6 +561,7 @@ Other Tcl variables defined during the test execution are:
In order to ensure that the test works as expected in different environments, observe the following additional rules:
* Avoid using external commands such as *grep, rm,* etc., as these commands can be absent on another system (e.g. on Windows); use facilities provided by Tcl instead.
* Do not put call to *locate_data_file* in catch statement – this can prevent correct interpretation of the missing data file by the test system.
* Do not use commands *decho* and *dlog* in the test script, to avoid interference with use of these commands by the test system.
@subsection testmanual_3_5 Interpretation of test results
@@ -566,8 +569,8 @@ The result of the test is evaluated by checking its output against patterns defi
The OCCT test system recognizes five statuses of the test execution:
* SKIPPED: reported if a line matching SKIPPED pattern is found (prior to any FAILED pattern). This indicates that the test cannot be run in the current environment; the most typical case is the absence of the required data file.
* FAILED: reported if a line matching pattern with status FAILED is found (unless it is masked by the preceding IGNORE pattern or a TODO statement), or if message TEST COMPLETED is not found at the end. This indicates that the test has produced a bad or unexpected result, and usually means a regression.
* BAD: reported if the test script output contains one or several TODO statements and the corresponding number of matching lines in the log. This indicates a known problem . The lines matching TODO statements are not checked against other patterns and thus will not cause a FAILED status.
* FAILED: reported if a line matching pattern with status FAILED is found (unless it is masked by the preceding IGNORE pattern or a TODO or REQUIRED statement), or if message TEST COMPLETED or at least one of REQUIRED patterns is not found. This indicates that the test has produced a bad or unexpected result, and usually means a regression.
* BAD: reported if the test script output contains one or several TODO statements and the corresponding number of matching lines in the log. This indicates a known problem. The lines matching TODO statements are not checked against other patterns and thus will not cause a FAILED status.
* IMPROVEMENT: reported if the test script output contains a TODO statement for which no corresponding line is found. This is a possible indication of improvement (a known problem has disappeared).
* OK: reported if none of the above statuses have been assigned. This means that the test has passed without problems.
@@ -587,7 +590,7 @@ puts "TODO BugNumber ListOfPlatforms: RegularExpression"
Here:
* *BugNumber* is the bug ID in the tracker. For example: #12345.
* *ListOfPlatforms* is a list of platforms, at which the bug is reproduced (e.g. Mandriva2008, Windows or All). Note that the platform name is custom for the OCCT test system; it corresponds to the value of environment variable *os_type* defined in DRAW.
* *ListOfPlatforms* is a list of platforms, at which the bug is reproduced (Linux, Windows, MacOS, or All). Note that the platform name is custom for the OCCT test system; it corresponds to the value of environment variable *os_type* defined in DRAW.
Example:
~~~~~
@@ -616,7 +619,27 @@ puts "TODO OCC22817 All: \\*\\* Exception \\*\\*"
puts "TODO OCC22817 All: TEST INCOMPLETE"
~~~~~
@subsection testmanual_3_7 Marking required output
To check expected output which must be obtained as a result of a test for it to be considered correct, add REQUIRED statement for each specific message.
For that, the following statement should be added to such a test script:
~~~~~
puts "REQUIRED ListOfPlatforms: RegularExpression"
~~~~~
Here *ListOfPlatforms* and *RegularExpression* have the same meaning as in TODO statements described above.
The REQUIRED statament can also be used to mask message that would normally be interpreted as error (according to rules defined in *parse.rules*) but should not be considered as such within current test.
Example:
~~~~~
puts "TODO REQUIRED Linux: Faulty shapes in variables faulty_1 to faulty_5"
~~~~~
This statement notifies test system that errors reported by *checkshape* command are expected in that test case, and test should be considered as OK if this message appears, despite of presence of general rule stating that 'Faulty' signals failure.
If output does not contain required statement, test case will be marked as FAILED.
@section testmanual_4 Advanced Use