I’ve been using property-based testing for quite some time now, and I find it particularly relevant now as an eval technique when using LLMs and, of course, to test my code.

Property-based testing doesn’t check by example; it checks rules (“properties”) that should always be true.

In a way, property-based testing is like generating proofs for your assumptions.

You define a law or invariant, and the test framework will then execute your function to see if it ever fails with the different input possibilities.

For example: Reversing a list twice should give you the original list.

In TypeScript, we could implement this test with the awesome fast-check lib.

npm install --save-dev fast-check

With the test consisting of:

import { testProp, fc } from 'fast-check';

testProp(
 'Reversing a list twice returns the original list',
 [fc.array(fc.integer())],
 (arr) => {
 expect(arr.slice().reverse().reverse()).toEqual(arr);
 }
);

• fc.array(fc.integer()) generates arrays of integers
• .slice() avoids mutating the original input
• testProp runs your property against generated inputs

I’ve also been practising this in Python using the excellent Hypothesis library.

pip install hypothesis

The above scenario looks like the following:

from hypothesis import given
import hypothesis.strategies as st

@given(st.lists(st.integers()))
def test_reverse_reverse(lst):
 assert lst[::-1][::-1] == lst

Even more concise 💁‍♂️

Hypothesis will generate random lists of integers and test your property against them, shrinking on failure.

One gotcha is that this technique works best with pure functions, anything with side effects would be handled by example based testing.

You also need to verify your properties, if these are wrong your tests could pass for the wrong reasons, also common in example based if you mess up your assertions!

This form of testing is applicable for: Library code. Data transformations. Validation logic. Business logic (when clearly defined and rule driven). Functional languages/paradigms. This works great with React given you’re doing functional. And of course anything with math or rules.

Your function is innocent until proven edge-cased.