libmpdec, libmpdec++ and cdecimal have large test suites. Apart from IBM’s official test cases, many additional tests have been created, mostly by using the pure Python version of Python’s decimal arithmetic module and decNumber. Multi precision arithmetic is routinely checked against gmp and apfloat.
All tests run successfully under Valgrind, ASan, UBSan and TSan.
libmpdec and libmpdec++ have been tested in depth with a large variety of gcc versions, clang, icc, Visual Studio, xlc and ibm-clang.
The MACHINE=ppro configuration (default for x86) tests cannot be run under Valgrind. Valgrind does not support x87 80-bit arithmetic, which is used in umodarith.h, see:
In December 2010 it was discovered that some gcc versions miscompile the inline assembly of the PPRO configuration. See also:
The bug is only triggered in bignum arithmetic and will fail instantly with an assert in crt.c. The current configure script automatically detects the bug and enables a workaround (-fno-ipa-pure-const).
gcc-4.6 exposes a bug in glibc, which had incorrect _FORTIFY_SOURCE wrappers for memmove and bcopy:
If an unpatched glibc is detected, the current configure script automatically enables a workaround (-U_FORTIFY_SOURCE).
The standalone LSAN sanitizer deallocates and re-uses the original pointer when realloc fails. This leads to memory corruption or incorrect results when running the libmpdec test suite (maxprec.decTest) under LSAN:
The malloc implementation on OS X does not conform to the C standard. It emits implementation defined warnings (which are global side effects) when a large allocation fails:
./testdata/maxprec.decTest ...
runtest(3706,0x7fff99302380) malloc: *** mach_vm_map(size=842105263157895168) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
Allocation failures are entirely well-defined by the standard, so make check disables these misguided “error” messages by setting the following environment variables:
export MallocLogFile=/dev/null
export MallocDebugReport=crash
Note that FreeBSD, which has a similar malloc reporting mechanism, does not emit these messages.
It is highly recommended to run the short tests, which comprise IBM’s official tests as well as many additional tests that were generated with Python’s decimal.py module.
Warning
The bignum tests need around 200MB heap and 300K stack. The normal tests need around 10MB heap and 50K stack.
If wget is installed, the tests are downloaded automatically. Otherwise, get the IBM tests and create this directory structure:
tests/testdata/abs.decTest
tests/testdata/add.decTest
...
Run the tests:
./configure
make
make check
Powershell downloads the tests automatically. If that fails for some reason, get the IBM tests and create the same directory structure as described in the Unix section above.
rem Windows 64-bit
vcbuild64.bat
runshort.bat
rem Windows 32-bit
vcbuild32.bat
runshort.bat
Starting from mpdecimal-2.4, the long tests are in a separate archive. Download mpdecimal-testit-4.0.0.tar.gz.
Warning
Do not use mpdecimal-testit-4.0.0 for production. It contains additional code that enables 100% code coverage.
The long tests (runalltests.sh) have the following known issues:
runtest.c and cov.c will fail under low memory conditions. Please retry with at least 1GB of memory.
fntcov.c and test_transpose.c try to use as much memory as possible. Run under ulimit!
In very rare cases, deccheck.c has occasional ULP differences between decNumber and libmpdec. In these cases, check against _pydecimal.py, which (so far) always agrees with libmpdec (and gives the correct answer whenever such a case is investigated).
The Python generated format tests may produce false positives due to output encoding issues and incorrect locales. These false positives do not occur if the format strings are passed directly in a program rather than using a Unix pipe. The pipe is used because this setup allows running the format tests in allocation failure mode.
The tests are more useful if gmp (libgmp-dev) is installed. They include:
official tests
additional tests
coverage tests (100% code coverage if RAM is sufficient)
extensive tests against decNumber.
extensive bignum tests against gmp (if present)
test karatsuba multiplication against fnt multiplication
extensive locale and formatting tests (if python3.2 or python2.7 is installed)
# Unix:
cd libmpdec
./configure
make extended
cd tests && ./runalltests.sh
rem Windows 64-bit: The tests are more useful if mpir (gmp) is installed.
rem The gmp paths are hard-coded in the Makefile.
cd libmpdec
vcvarsall.bat x64
copy /y Makefile.vc Makefile
nmake MACHINE=x64 extended_gmp
cd tests && runalltests.bat
rem Windows 32-bit: The tests are more useful if mpir (gmp) is installed.
rem The gmp paths are hard-coded in the Makefile.
cd libmpdec
vcvarsall.bat x86
copy /y Makefile.vc Makefile
nmake MACHINE=ppro extended_gmp
cd tests && runalltests.bat
On a 64-bit platform, it is possible to run the long tests for all configurations if the 32-bit development tools are also installed:
# Linux
cd libmpdec/tests && ./runallconfigs.sh
# Linux, Valgrind
cd libmpdec/tests && ./runallconfigs.sh --valgrind
rem Windows
cd libmpdec\tests && runallconfigs.bat
libmpdec is one of the few open source projects with 100% code coverage. Since libmpdec is an arbitrary precision library, full coverage can only be achieved if sufficient RAM is installed.
The numbers are 4-8GB for the 32-bit configuration and 1TB for the 64-bit configuration. In order to make 100% coverage feasible on cheap hardware, the configure script provides an option to compile a 64-bit library using the 32-bit configuration. This option should only be used for testing.
# Library coverage:
cd libmpdec
./configure
make coverage
# Get a coverage report:
python tests/covreport.py
# Do not use this library in production:
make distclean
# 100% library coverage (64-bit OS, 8GB RAM):
cd libmpdec
./configure MACHINE=full_coverage
patch < tests/fullcov_header.patch
patch < tests/fullcov.patch
make coverage
# Get a coverage report:
python tests/covreport.py
# Do not use this library in production:
make distclean
The tests are run as part of the libmpdec tests, see above.
Additionally, libmpdec++ has been tested by replacing all libmpdec functions in Python’s decimal module with libmpdec++ functions.
The result passes Python’s unit tests as well as the long deccheck.py tests.
Download the official decimal.py tests (requires svn). It is vital to use the decimaltestdata directory from the py3k svn branch. Earlier versions (especially 2.5) of this directory may contain bugs, causing several tests to fail.
python setup.py build
cd python && ./gettests.sh # gettests.bat
# Official regression tests:
#
# Python-2.5.x: test_cdecimal2.5.py
# Python-2.6.x: test_cdecimal2.py
# Python-2.7.x: test_cdecimal2.py
# Python-3.1.x: test_cdecimal3.py
# Python-3.2.x: test_cdecimal3.py
python test_cdecimal2.py # test_cdecimal2.5.py test_cdecimal3.py
# Tests against decimal.py:
#
# Python-2.x.y: deccheck2.py, ctx-deccheck2.py
# Python-3.x.y: deccheck3.py
#
# Note for Python 2.5.0 through 2.5.4: decimal.py contains several bugs
# that make the tests impractical to run. It is recommended to upgrade
# to Python 2.5.5 or 2.6.6.
# Python-2.x.y:
python deccheck2.py
python ctx-deccheck2.py
# Python-3.x.y (context tests are integrated):
python deccheck3.py
These tests can take several days or weeks.
The most likely cause of failures are bugs in earlier versions of decimal.py or in decimaltestdata. Here is a short checklist:
Make sure that you have ‘decimaltestdata’ from py3k svn.
All tests are routinely run against these Python svn branches:
release25-maint
release26-maint
release27-maint
release31-maint
py3k
Earlier micro versions (say 2.6.1 vs 2.6.6) often have bugs in decimal.py. deccheck tries to skip known issues, but the list might not be complete.
Please consider testing against the latest micro release before reporting a bug.
cdecimal currently has 85% code coverage. A higher percentage could only be achieved by simulating failures of Python API functions.
The current Makefile logic requires that both Python-2.x and Python-3.x are installed.
# Required Variables
PYTHON2INC=-I/usr/local/include/python2.7
PYTHON2EXEC=/usr/local/bin/python2.7
PYTHON3INC=-I/usr/local/include/python3.2m
PYTHON3EXEC=/usr/local/bin/python3.2m
export PYTHON2INC PYTHON2EXEC PYTHON3INC PYTHON3EXEC
# Module coverage:
./configure
make pycoverage
# Get a coverage report:
python tests/covreport.py
# Do not use the generated module:
make distclean