Numerical accuracy

This document summarizes the accuracy of JAX mathematical functions.

Methodology and terminology

• Units in the Last Place (ULPs): Error is measured in ULPs relative to a higher-precision reference: for sub-64-bit types (bfloat16, float16, and float32), reference values are evaluated in double precision (float64); for float64, reference values are evaluated using mpmath with 100-bit precision. For a target format with \(p\) significand bits and normal exponent range \([e_{\min}, e_{\max}]\), \(\operatorname{ulp}(r) = 2^{\min(\max(\lfloor \log_2 |r| \rfloor, e_{\min}), e_{\max}) - (p - 1)}\). A bound of \(0.5\) ULP is a correctly rounded result.

  • Overflow and Infinity: Reference values at or beyond the round-to-nearest-even overflow threshold \(2^{e_{\max} + 1} - \frac{1}{2}\operatorname{ulp}(\text{max\_float})\) are treated as overflowing to \(\pm\infty\), so a computed \(\pm\infty\) has \(0\) ULP error. When the reference value is finite in or above the top normal binade (\(|r| \ge 2^{e_{\max}}\)) and the computed result has the same sign, rounding across the overflow threshold (either a computed \(\pm\infty\) just below the threshold, or a finite result such as \(\pm\text{max\_float}\) at or above the threshold) is measured continuously by placing \(\pm\infty\) at \(\pm 2^{e_{\max} + 1} = \pm(\text{max\_float} + 1\text{ ULP})\).

• Exhaustive vs. Sampled Testing:

  • For 16-bit types (bfloat16, float16) and unary operations on 32-bit types (float32), bounds are verified by exhaustive testing across all bit patterns (up to \(2^{32}\) inputs).

  • For binary operations on float32 and all operations on float64, bounds are estimated by testing a sample of inputs: values at and near special points (such as 0, \(\pm 1\), \(\pm\infty\), subnormal/normal boundaries, poles, and approximation thresholds) along with random floating-point values drawn across all exponents. Because sampling cannot guarantee hitting the worst-case input, these values are empirical bounds on the maximum error.

• Flush-To-Zero (FTZ) and Subnormals: Subnormal floating-point inputs and outputs (\(|v| < 2^{e_{\min}}\)) are typically flushed to zero by default, though some operations on types smaller than float32 (and float64 on GPU) preserve gradual underflow.

  • When gradual underflow is preserved (ftz=False), subnormal values are measured on the standard number line against the subnormal quantum \(\operatorname{ulp}_{\text{tiny}} = 2^{e_{\min} - (p - 1)}\).

  • When outputs flush to zero (ftz=True), any pair of values with both \(|c| < 2^{e_{\min}}\) and \(|r| < 2^{e_{\min}}\) has \(0\) ULP error, and distances that cross the subnormal region contract the unrepresentable interval \([-(2^{e_{\min}} - \operatorname{ulp}_{\text{tiny}}), +(2^{e_{\min}} - \operatorname{ulp}_{\text{tiny}})]\) to \(0\) so that the adjacent FTZ-representable values \((-2^{e_{\min}}, 0.0, +2^{e_{\min}})\) are spaced \(1\) ULP apart rather than incurring an artificial \((2^{p-1} - 1)\)-ULP gap.

• Hardware Platforms:

  • CPU: x86_64.

  • NVIDIA GPU: H100 and B200.

  • TPU: TPU v2–v5e, TPU v5p, TPU v6e, and TPU 7x.

---

bfloat16 accuracy

Maximum error in units in the last place (ULPs) for bfloat16:

Function	CPU	NVIDIA GPU	TPU v2–v5e	TPU v5p	TPU v6e	TPU 7x
acos()	1.5	0.5	0.5	0.5	0.5	0.5
acosh()	2.0	0.5	1.0	1.0	1.0	1.0
asin()	1.5	0.5	0.5	0.5	0.5	0.5
asinh()	1.5	0.5	1.0	1.0	1.0	1.0
atan()	0.5	0.5	0.5	0.5	0.5	0.5
atan2()	1.0	1.0	1.0	1.0	1.0	1.0
atanh()	1.5	0.5	1.0	0.5	0.5	0.5
bessel_i0e()	0.5	0.5	0.5	0.5	0.5	0.5
bessel_i1e()	0.5	0.5	0.5	0.5	0.5	0.5
cbrt()	0.5	0.5	0.5	0.5	0.5	0.5
cos()	0.5	0.5	0.5	0.5	0.5	0.5
cosh()	0.5	0.5	0.5	1.0	1.0	1.0
erf()	0.5	0.5	0.5	0.5	0.5	0.5
erf_inv()	0.5	0.5	0.5	0.5	0.5	0.5
erfc()	0.5	0.5	0.5	0.5	0.5	0.5
exp()	0.5	1.0	1.0	1.0	1.0	1.0
exp() (highest)	0.5	1.0	1.0	1.0	0.5	0.5
exp2()	101.0	101.0	44.0	44.0	44.0	75.0
expm1()	0.5	0.5	1.0	1.0	0.5	0.5
expm1() (highest)	0.5	0.5	0.5	0.5	0.5	0.5
log()	0.5	0.5	1.0	0.5	0.5	0.5
log1p()	0.5	0.5	1.0	0.5	0.5	0.5
log2()	2.0	2.0	1.0	1.0	1.0	1.0
logistic()	2.5	2.5	1.0	1.0	0.5	63.0
reciprocal()	0.5	0.5	1.0	1.0	0.5	0.5
rsqrt()	0.5	0.5	0.5	0.5	0.5	0.5
sin()	0.5	0.5	1.0	1.0	1.0	1.0
sinh()	0.5	0.5	1.0	1.0	1.0	1.0
sqrt()	0.5	0.5	0.5	0.5	0.5	0.5
square()	0.5	0.5	0.5	0.5	0.5	0.5
tan()	0.5	0.5	0.5	0.5	0.5	0.5
tanh()	0.5	0.5	0.5	0.5	0.5	0.5
deg2rad()	1.0	1.0	1.0	1.0	1.0	1.0
hypot()	1.5	1.5	1.0	1.0	0.5	1.0
log10()	2.0	2.0	1.0	1.0	1.0	1.0
rad2deg()	1.0	1.0	1.0	1.0	1.0	1.0
sinc()	0.5	0.5	0.5	0.5	0.5	0.5
erfcx()	0.5	0.5	0.5	0.5	0.5	0.5

---

float16 accuracy

Maximum error in units in the last place (ULPs) for float16:

Function	CPU	NVIDIA GPU	TPU v2–v5e	TPU v5p	TPU v6e	TPU 7x
acos()	1.5	1.0	1.0	1.0	1.0	1.0
acosh()	2.0	1.0	1.0	1.0	1.0	1.0
asin()	1.5	1.0	0.5	0.5	0.5	0.5
asinh()	1.5	1.0	1.0	1.0	1.0	1.0
atan()	1.0	1.0	1.0	1.0	1.0	1.0
atan2()	1.0	1.0	1.0	1.0	1.0	1.0
atanh()	1.5	0.5	1.0	1.0	0.5	0.5
bessel_i0e()	1.0	0.5	0.5	0.5	0.5	0.5
bessel_i1e()	1.0	1.0	1.0	1.0	1.0	1.0
cbrt()	1.0	1.0	1.0	1.0	0.5	0.5
cos()	1.0	1.0	1.0	1.0	1.0	1.0
cosh()	1.0	1.0	1.0	1.0	1.0	1.0
erf()	1.0	1.0	0.5	0.5	1.0	1.0
erf_inv()	1.0	1.0	1.0	1.0	1.0	1.0
erfc()	1.0	1.0	1.0	1.0	1.0	1.0
exp()	1.0	1.0	1.0	1.0	1.0	1.0
exp() (highest)	1.0	1.0	1.0	1.0	0.5	0.5
exp2()	14.0	14.0	7.5	7.5	7.5	7.5
expm1()	2.5	1.0	1.0	1.0	0.5	0.5
expm1() (highest)	2.5	1.0	1.0	1.0	0.5	0.5
log()	1.0	1.0	1.0	1.0	1.0	1.0
log1p()	1.0	1.0	1.0	1.0	1.0	1.0
log2()	2.0	2.0	1.5	1.0	1.0	1.0
logistic()	2.0	2.0	1.0	1.0	1.0	1.0
reciprocal()	0.5	0.5	0.5	0.5	0.5	0.5
rsqrt()	0.5	0.5	0.5	0.5	0.5	0.5
sin()	1.0	1.0	1.0	1.0	1.0	1.0
sinh()	1.0	0.5	1.0	1.0	1.0	1.0
sqrt()	0.5	0.5	1.0	1.0	1.0	1.0
square()	0.5	0.5	0.5	0.5	0.5	0.5
tan()	1.0	1.0	1.0	1.0	1.0	1.0
tanh()	0.5	0.5	1.0	1.0	0.5	0.5
deg2rad()	1.0	1.0	1.0	1.0	1.0	1.0
hypot()	1.5	1.5	1.0	1.0	1.0	1.0
log10()	1.5	1.5	1.0	1.0	1.0	1.0
rad2deg()	1.5	1.5	1.5	1.5	1.5	1.5
sinc()	0.5	0.5	0.5	0.5	0.5	0.5
erfcx()	1.0	1.0	1.0	1.0	1.0	1.0

---

float32 accuracy

Maximum error in units in the last place (ULPs) for float32:

Function	CPU	NVIDIA GPU	TPU v2–v5e	TPU v5p	TPU v6e	TPU 7x	Notes
acos()	1.5	1.5	5.0	5.0	4.0	5.0
acosh()	4.5	2.5	4031.0	61.0	4.5	5.0
asin()	2.0	1.5	5.0	5.0	4.5	4.5
asinh()	3.5	2.0	4034.5	62.5	3.5	3.5
atan()	4.0–5.5	1.5	2.5	2.5	2.5	2.5	CPU bound depends on AMD vs Intel
atan2()	0.5	3.0	3.5	3.5	3.5	3.5
atanh()	3.0	3.5	2183.5	43.0	3.5	3.5
bessel_i0e()	7.0	8.0	8.0	8.0	8.0	8.0
bessel_i1e()	11.0	15.5	15.5	15.5	15.5	15.5
cbrt()	0.5	1.5	4.5	4.5	1.5	1.5
cos()	0.5	2.0	3.5	3.5	3.5	3.0
cosh()	25.0	2.5	93.5	99.0	59.0	59.5
erf()	7.0	6.5	7.5	8.5	1.5	1.5
erf_inv()	65.0	65.0	427.0	65.5	65.0	65.5
erfc()	66.0	66.5	145.0	157.0	124.5	125.0
erfc() (highest)	4.0	5.0	5.5	5.0	5.0	5.0	accuracy=lax.AccuracyMode.HIGHEST
exp()	1.5	2.0	116.0	109.5	64.5	65.0
exp() (highest)	1.5	2.0	1.5	1.5	1.5	1.5	accuracy=lax.AccuracyMode.HIGHEST
exp2()	68.5	69.0	141.5	133.0	90.0	90.0
expm1()	6.5	1.5	1772.0	1357.5	3.5	3.5
expm1() (highest)	6.5	1.5	2.0	2.0	2.0	2.0	accuracy=lax.AccuracyMode.HIGHEST
log()	1.5	1.0	4030.5	62.0	2.5	2.5
log1p()	3.0	1.0	4034.0	63.0	3.0	3.0
log2()	2.5	2.0	5159.0	57.5	2.5	2.5
logistic()	2.5	4.0	243.0	124.0	65.5	64.0
reciprocal()	0.5	1.0	198.0	40.0	1.5	1.5
rsqrt()	1.0–2.0	2.0	2.5	2.5	1.5	1.0	CPU bound depends on AMD vs Intel
sin()	0.5	1.5	3.5	3.5	3.5	3.5
sinh()	25.0	3.0	1794.0	1332.5	59.0	59.5
sqrt()	0.5	1.0	3.0	3.0	2.0	2.0
square()	0.5	0.5	0.5	0.5	0.5	0.5
tan()	0.5	3.5	6.5	7.0	5.5	5.5
tanh()	5.0	5.5	1365.5	92.0	1.5	1.5
deg2rad()	1.0	1.0	1.0	1.0	1.0	1.0
hypot()	1.5	2.0	3.5	3.5	2.5	2.5
log10()	3.0	2.5	6213.0	57.0	3.0	3.0
rad2deg()	1.0	1.0	1.0	1.0	1.0	1.0
sinc()	2.5	3.5	4.0	4.0	4.0	3.5
erfcx()	3.5	4.0	4.0	4.0	4.5	4.5

---

float64 accuracy

Maximum error in units in the last place (ULPs) for float64.

Note

TPUs have no true float64 hardware. While bounds for 16-bit types and unary float32 operations are exhaustive, for binary float32 operations and float64 these are empirical bounds.

Function	CPU	NVIDIA GPU
acos()	1.5	1.5
acosh()	3.5	2.5
asin()	1.5	2.5
asinh()	2.5	2.5
atan()	3.5	2.5
atan2()	0.5	1.5
atanh()	2.5	3.5
bessel_i0e()	7.5	7.5
bessel_i1e()	10.5	7.5
cbrt()	0.5	1.5
cos()	0.5	1.5
cosh()	496.0	2.5
erf()	2.5	2.5
erf_inv()	500000.0	500000.0
erfc()	350.0	350.0
erfc() (highest)	2.5	2.5
exp()	2.0	1.5
exp2()	719.0	719.0
expm1()	4.5	1.5
log()	0.5	1.5
log1p()	2.0	1.5
log2()	1.5	1.5
logistic()	3.5	4.5
reciprocal()	0.5	0.5
rsqrt()	1.5	1.5
sin()	0.5	2.5
sinh()	496.0	2.5
sqrt()	0.5	0.5
square()	0.5	0.5
tan()	0.5	2.5
tanh()	7.0	3.5
deg2rad()	1.0	1.0
hypot()	1.5	1.5
log10()	2.0	2.5
rad2deg()	1.0	1.0
sinc()	2.0	2.0
erfcx()	3.0	3.0