Table Of Contents

Testing

Both libmpdec and cdecimal have large test suites. Apart from IBM’s official test cases, many additional tests have been created, mostly by using Python’s decimal arithmetic module. Multi precision arithmetic is routinely checked against gmp and apfloat.

All tests run successfully under Valgrind.

Compilers

libmpdec and cdecimal have been tested on many operating systems and with several compilers.

Toolchain Bugs

  1. In December 2010 it was discovered that some gcc versions miscompile the inline assembly of the PPRO configuration. See also:

    The bug is only triggered in bignum arithmetic and will fail instantly with an assert in crt.c. The current configure script automatically detects the bug and enables a workaround (-fno-ipa-pure-const).

  2. gcc-4.6 exposes a bug in glibc, which had incorrect _FORTIFY_SOURCE wrappers for memmove and bcopy:

    If an unpatched glibc is detected, the current configure script automatically enables a workaround (-U_FORTIFY_SOURCE).

Tested Compilers

  gcc 4.1-4.4 gcc 4.5 gcc 4.6 [2] clang [3] icc [4] suncc [5] Visual Studio CompCert [7]
x64 ok ok ok ok ok ok ok n/a
uint128 ok ok ok n/a n/a n/a n/a n/a
ansi64 ok ok ok ok ok ok ok n/a
ppro ok ok [1] ok ok ok - [6] ok n/a
ansi32 ok ok ok ok ok ok ok n/a
ansi-legacy ok ok ok ok ok ok ok ok
[1]-fno-ipa-pure-const
[2]-U_FORTIFY_SOURCE
[3]Version >= 3.0
[4]Version >= 11.0
[5]Version >= 12.3
[6]register allocator fails
[7]Version >= 1.8.2

Users also have reported success with the xlc compiler.

libmpdec

Short Tests

It is highly recommended to run the short tests, which comprise IBM’s official tests as well as many additional tests that were generated with Python’s decimal.py module.

Warning

On systems with comparatively little memory (less than 1GB) the tests may produce false positives (Floating Point Exception). This is raised by the error handler of libmpdec if malloc fails.

Unix

If wget is installed, the tests are downloaded automatically. Otherwise, get the IBM tests and create this directory structure:

tests/testdata/abs.decTest
tests/testdata/add.decTest
...

Run the tests:

./configure
make
make check

Windows

In the vcbuild directory, run gettests.bat. This creates a directory testdata and copies additional tests into the directory. If wget is installed (Cygwin), the script tries to download IBM’s official test cases and copy them to testdata.

Otherwise, get the IBM tests as described above and create this directory structure:

vcbuild\tests\testdata\abs.decTest
vcbuild\tests\testdata\add.decTest
...
rem Windows 32-bit
vcbuild32.bat
runtest32.bat

rem Windows 64-bit
vcbuild64.bat
runtest64.bat

Long Tests

Starting from mpdecimal-2.4.0, the long tests are in a separate archive. Download mpdecimal-testit-2.4.1.tar.gz or mpdecimal-testit-2.4.1.zip.

Warning

Do not use mpdecimal-testit-2.4.1 for production. It contains additional code that enables 100% code coverage.

The tests are more useful if gmp (libgmp-dev) is installed. They include:

  • official tests
  • additional tests
  • coverage tests (100% code coverage if RAM is sufficient)
  • extensive tests against decNumber.
  • extensive bignum tests against gmp (if present)
  • test karatsuba multiplication against fnt multiplication
  • extensive locale and formatting tests (if python3.2 or python2.7 is installed)
# Unix:
cd libmpdec
./configure
make extended
cd tests && ./runalltests.sh

# Windows 32-bit: The tests are more useful if mpir (gmp) is installed.
# The gmp paths are hard-coded in the Makefile.
cd libmpdec
vcvarsall.bat x86
copy /y Makefile.vc Makefile
nmake MACHINE=ppro extended_gmp
cd tests && runalltests.bat

# Windows 64-bit: The tests are more useful if mpir (gmp) is installed.
# The gmp paths are hard-coded in the Makefile.
cd libmpdec
vcvarsall.bat x64
copy /y Makefile.vc Makefile
nmake MACHINE=x64 extended_gmp
cd tests && runalltests.bat

All Configurations

On a 64-bit platform, it is possible to run the long tests for all configurations if the 32-bit development tools are also installed:

# Linux
cd libmpdec/tests && ./runallconfigs.sh

# Linux, Valgrind
cd libmpdec/tests && ./runallconfigs.sh --valgrind
rem Windows
cd libmpdec\tests && runallconfigs.bat

Code Coverage

libmpdec is one of the few open source projects with 100% code coverage. Since libmpdec is an arbitrary precision library, full coverage can only be achieved if sufficient RAM is installed.

The numbers are 4-8GB for the 32-bit configuration and 1TB for the 64-bit configuration. In order to make 100% coverage feasible on cheap hardware, the configure script provides an option to compile a 64-bit library using the 32-bit configuration. This option should only be used for testing.

# Library coverage:
cd libmpdec
./configure
make coverage

# Get a coverage report:
python tests/covreport.py

# Do not use this library in production:
make distclean


# 100% library coverage (64-bit OS, 8GB RAM):
cd libmpdec
./configure MACHINE=full_coverage
patch < tests/fullcov_header.patch
make coverage

# Get a coverage report:
python tests/covreport.py

# Do not use this library in production:
make distclean

cdecimal

Short Tests

Download the official decimal.py tests (requires svn). It is vital to use the decimaltestdata directory from the py3k svn branch. Earlier versions (especially 2.5) of this directory may contain bugs, causing several tests to fail.

python setup.py build
cd python && ./gettests.sh # gettests.bat

# Official regression tests:
#
# Python-2.5.x: test_cdecimal2.5.py
# Python-2.6.x: test_cdecimal2.py
# Python-2.7.x: test_cdecimal2.py
# Python-3.1.x: test_cdecimal3.py
# Python-3.2.x: test_cdecimal3.py
python test_cdecimal2.py # test_cdecimal2.5.py test_cdecimal3.py


# Tests against decimal.py:
#
# Python-2.x.y: deccheck2.py, ctx-deccheck2.py
# Python-3.x.y: deccheck3.py
#
# Note for Python 2.5.0 through 2.5.4: decimal.py contains several bugs
# that make the tests impractical to run. It is recommended to upgrade
# to Python 2.5.5 or 2.6.6.

# Python-2.x.y:
python deccheck2.py
python ctx-deccheck2.py

# Python-3.x.y (context tests are integrated):
python deccheck3.py

Long Tests

These tests can take several days or weeks.

Test Failures

The most likely cause of failures are bugs in earlier versions of decimal.py or in decimaltestdata. Here is a short checklist:

  1. Make sure that you have ‘decimaltestdata’ from py3k svn.

  2. All tests are routinely run against these Python svn branches:

    • release25-maint
    • release26-maint
    • release27-maint
    • release31-maint
    • py3k

    Earlier micro versions (say 2.6.1 vs 2.6.6) often have bugs in decimal.py. deccheck tries to skip known issues, but the list might not be complete.

    Please consider testing against the latest micro release before reporting a bug.

Code Coverage

cdecimal currently has 85% code coverage. A higher percentage could only be achieved by simulating failures of Python API functions.

The current Makefile logic requires that both Python-2.x and Python-3.x are installed.

# Required Variables
PYTHON2INC=-I/usr/local/include/python2.7
PYTHON2EXEC=/usr/local/bin/python2.7
PYTHON3INC=-I/usr/local/include/python3.2m
PYTHON3EXEC=/usr/local/bin/python3.2m
export PYTHON2INC PYTHON2EXEC PYTHON3INC PYTHON3EXEC

# Module coverage:
./configure
make pycoverage

# Get a coverage report:
python tests/covreport.py

# Do not use the generated module:
make distclean