Table of Contents

Testing

libmpdec, libmpdec++ and cdecimal have large test suites. Apart from IBM’s official test cases, many additional tests have been created, mostly by using the pure Python version of Python’s decimal arithmetic module and decNumber. Multi precision arithmetic is routinely checked against gmp and apfloat.

All tests run successfully under Valgrind, ASan, UBSan and TSan.

Compilers

libmpdec and libmpdec++ have been tested in depth with a large variety of gcc versions, clang, icc, Visual Studio, xlc and ibm-clang.

Toolchain Restrictions

  1. The MACHINE=ppro configuration (default for x86) tests cannot be run under Valgrind. Valgrind does not support x87 80-bit arithmetic, which is used in umodarith.h, see:

Toolchain Bugs

  1. In December 2010 it was discovered that some gcc versions miscompile the inline assembly of the PPRO configuration. See also:

    The bug is only triggered in bignum arithmetic and will fail instantly with an assert in crt.c. The current configure script automatically detects the bug and enables a workaround (-fno-ipa-pure-const).

  2. gcc-4.6 exposes a bug in glibc, which had incorrect _FORTIFY_SOURCE wrappers for memmove and bcopy:

    If an unpatched glibc is detected, the current configure script automatically enables a workaround (-U_FORTIFY_SOURCE).

  3. The standalone LSAN sanitizer deallocates and re-uses the original pointer when realloc fails. This leads to memory corruption or incorrect results when running the libmpdec test suite (maxprec.decTest) under LSAN:

  4. The malloc implementation on OS X does not conform to the C standard. It emits implementation defined warnings (which are global side effects) when a large allocation fails:

    ./testdata/maxprec.decTest ...
    
    runtest(3706,0x7fff99302380) malloc: *** mach_vm_map(size=842105263157895168) failed (error code=3)
    *** error: can't allocate region
    *** set a breakpoint in malloc_error_break to debug
    

    Allocation failures are entirely well-defined by the standard, so make check disables these misguided “error” messages by setting the following environment variables:

    export MallocLogFile=/dev/null
    export MallocDebugReport=crash
    

    Note that FreeBSD, which has a similar malloc reporting mechanism, does not emit these messages.

libmpdec

Short Tests

It is highly recommended to run the short tests, which comprise IBM’s official tests as well as many additional tests that were generated with Python’s decimal.py module.

Warning

The bignum tests need around 200MB heap and 300K stack. The normal tests need around 10MB heap and 50K stack.

Unix

If wget is installed, the tests are downloaded automatically. Otherwise, get the IBM tests and create this directory structure:

tests/testdata/abs.decTest
tests/testdata/add.decTest
...

Run the tests:

./configure
make
make check

Windows

Powershell downloads the tests automatically. If that fails for some reason, get the IBM tests and create the same directory structure as described in the Unix section above.

rem Windows 64-bit
vcbuild64.bat
runshort.bat

rem Windows 32-bit
vcbuild32.bat
runshort.bat

Long Tests

Starting from mpdecimal-2.4, the long tests are in a separate archive. Download mpdecimal-testit-4.0.0.tar.gz.

Warning

Do not use mpdecimal-testit-4.0.0 for production. It contains additional code that enables 100% code coverage.

The long tests (runalltests.sh) have the following known issues:

  • runtest.c and cov.c will fail under low memory conditions. Please retry with at least 1GB of memory.

  • fntcov.c and test_transpose.c try to use as much memory as possible. Run under ulimit!

  • In very rare cases, deccheck.c has occasional ULP differences between decNumber and libmpdec. In these cases, check against _pydecimal.py, which (so far) always agrees with libmpdec (and gives the correct answer whenever such a case is investigated).

  • The Python generated format tests may produce false positives due to output encoding issues and incorrect locales. These false positives do not occur if the format strings are passed directly in a program rather than using a Unix pipe. The pipe is used because this setup allows running the format tests in allocation failure mode.

The tests are more useful if gmp (libgmp-dev) is installed. They include:

  • official tests

  • additional tests

  • coverage tests (100% code coverage if RAM is sufficient)

  • extensive tests against decNumber.

  • extensive bignum tests against gmp (if present)

  • test karatsuba multiplication against fnt multiplication

  • extensive locale and formatting tests (if python3.2 or python2.7 is installed)

Unix long tests

# Unix:
cd libmpdec
./configure
make extended
cd tests && ./runalltests.sh

Windows long tests

rem Windows 64-bit: The tests are more useful if mpir (gmp) is installed.
rem The gmp paths are hard-coded in the Makefile.
cd libmpdec
vcvarsall.bat x64
copy /y Makefile.vc Makefile
nmake MACHINE=x64 extended_gmp
cd tests && runalltests.bat

rem Windows 32-bit: The tests are more useful if mpir (gmp) is installed.
rem The gmp paths are hard-coded in the Makefile.
cd libmpdec
vcvarsall.bat x86
copy /y Makefile.vc Makefile
nmake MACHINE=ppro extended_gmp
cd tests && runalltests.bat

All Configurations

On a 64-bit platform, it is possible to run the long tests for all configurations if the 32-bit development tools are also installed:

# Linux
cd libmpdec/tests && ./runallconfigs.sh

# Linux, Valgrind
cd libmpdec/tests && ./runallconfigs.sh --valgrind
rem Windows
cd libmpdec\tests && runallconfigs.bat

Code Coverage

libmpdec is one of the few open source projects with 100% code coverage. Since libmpdec is an arbitrary precision library, full coverage can only be achieved if sufficient RAM is installed.

The numbers are 4-8GB for the 32-bit configuration and 1TB for the 64-bit configuration. In order to make 100% coverage feasible on cheap hardware, the configure script provides an option to compile a 64-bit library using the 32-bit configuration. This option should only be used for testing.

# Library coverage:
cd libmpdec
./configure
make coverage

# Get a coverage report:
python tests/covreport.py

# Do not use this library in production:
make distclean


# 100% library coverage (64-bit OS, 8GB RAM):
cd libmpdec
./configure MACHINE=full_coverage
patch < tests/fullcov_header.patch
patch < tests/fullcov.patch
make coverage

# Get a coverage report:
python tests/covreport.py

# Do not use this library in production:
make distclean

libmpdec++

Short Tests

The tests are run as part of the libmpdec tests, see above.

Long Tests

Additionally, libmpdec++ has been tested by replacing all libmpdec functions in Python’s decimal module with libmpdec++ functions.

The result passes Python’s unit tests as well as the long deccheck.py tests.

cdecimal

Short Tests

Download the official decimal.py tests (requires svn). It is vital to use the decimaltestdata directory from the py3k svn branch. Earlier versions (especially 2.5) of this directory may contain bugs, causing several tests to fail.

python setup.py build
cd python && ./gettests.sh # gettests.bat

# Official regression tests:
#
# Python-2.5.x: test_cdecimal2.5.py
# Python-2.6.x: test_cdecimal2.py
# Python-2.7.x: test_cdecimal2.py
# Python-3.1.x: test_cdecimal3.py
# Python-3.2.x: test_cdecimal3.py
python test_cdecimal2.py # test_cdecimal2.5.py test_cdecimal3.py


# Tests against decimal.py:
#
# Python-2.x.y: deccheck2.py, ctx-deccheck2.py
# Python-3.x.y: deccheck3.py
#
# Note for Python 2.5.0 through 2.5.4: decimal.py contains several bugs
# that make the tests impractical to run. It is recommended to upgrade
# to Python 2.5.5 or 2.6.6.

# Python-2.x.y:
python deccheck2.py
python ctx-deccheck2.py

# Python-3.x.y (context tests are integrated):
python deccheck3.py

Long Tests

These tests can take several days or weeks.

Test Failures

The most likely cause of failures are bugs in earlier versions of decimal.py or in decimaltestdata. Here is a short checklist:

  1. Make sure that you have ‘decimaltestdata’ from py3k svn.

  2. All tests are routinely run against these Python svn branches:

    • release25-maint

    • release26-maint

    • release27-maint

    • release31-maint

    • py3k

    Earlier micro versions (say 2.6.1 vs 2.6.6) often have bugs in decimal.py. deccheck tries to skip known issues, but the list might not be complete.

    Please consider testing against the latest micro release before reporting a bug.

Code Coverage

cdecimal currently has 85% code coverage. A higher percentage could only be achieved by simulating failures of Python API functions.

The current Makefile logic requires that both Python-2.x and Python-3.x are installed.

# Required Variables
PYTHON2INC=-I/usr/local/include/python2.7
PYTHON2EXEC=/usr/local/bin/python2.7
PYTHON3INC=-I/usr/local/include/python3.2m
PYTHON3EXEC=/usr/local/bin/python3.2m
export PYTHON2INC PYTHON2EXEC PYTHON3INC PYTHON3EXEC

# Module coverage:
./configure
make pycoverage

# Get a coverage report:
python tests/covreport.py

# Do not use the generated module:
make distclean