Measure, Profile, and Benchmark

Measurement tells you what happened to an operation you care about. Profiling helps locate where time, allocation, or other resources were spent. Benchmarking compares an implementation under a specified workload and environment. Use them in that order of purpose: discover a meaningful problem, locate it, then evaluate a proposed improvement. 1

A method that looks inefficient may occupy a negligible part of a slow request. Making it twice as fast will not fix the database wait that dominates the user’s experience.

Measure one operation without timing its setup

Suppose an import repeatedly counts distinct identifiers. Prepare the input once, check the answer, then measure repeated executions. Do not include printing inside the timed loop: console output would add unrelated work.

The following is a small timing harness, not a statistically rigorous benchmarking framework. It performs one untimed warm-up and retains a checksum so the measured results are actually used.

public sealed record Measurement(double Milliseconds, long Checksum);

public static class Timing
{
    public static Measurement Measure(Func<int> work, int repetitions)
    {
        if (repetitions < 1) throw new ArgumentOutOfRangeException(nameof(repetitions));
        work(); // Untimed warm-up; not a guarantee of a fully warmed runtime.
        long checksum = 0;
        var timer = System.Diagnostics.Stopwatch.StartNew();
        for (int i = 0; i < repetitions; i++) checksum += work();
        timer.Stop();
        return new Measurement(timer.Elapsed.TotalMilliseconds, checksum);
    }
}

public static class Work
{
    public static int CountUnique(int[] values)
    {
        return new HashSet<int>(values).Count;
    }
}

int[] values = Enumerable.Range(0, 2000).Select(i => i % 100).ToArray();
if (Work.CountUnique(values) != 100) throw new Exception("Wrong result.");
for (int trial = 0; trial < 5; trial++)
{
    Measurement result = Timing.Measure(() => Work.CountUnique(values), 100);
    Console.WriteLine($"{result.Milliseconds:F3} ms; checksum {result.Checksum}");
}

from time import perf_counter


def measure(work, repetitions):
    if repetitions < 1:
        raise ValueError("Repetitions must be positive.")
    work()  # Untimed warm-up.
    checksum = 0
    started = perf_counter()
    for _ in range(repetitions):
        checksum += work()
    elapsed_ms = (perf_counter() - started) * 1000
    return elapsed_ms, checksum


def count_unique(values):
    return len(set(values))

values = [i % 100 for i in range(2000)]
assert count_unique(values) == 100
for _ in range(5):
    elapsed_ms, checksum = measure(lambda: count_unique(values), 100)
    print(f"{elapsed_ms:.3f} ms; checksum {checksum}")

Find the expensive part before changing it

If the full import is slow, use a profiler to distinguish repeated calls, expensive computation, allocation, and waiting. Python’s cProfile reports function-level activity; .NET diagnostic tools can expose CPU, allocation, and runtime behavior. A profiler’s instrumentation can add overhead, so its results locate candidates rather than replace an uninstrumented end-to-end measurement. 4

Then compare a candidate change using the same input distribution, output checks, runtime, build mode, and machine conditions. Include input sizes and edge cases that resemble actual use. A benchmark using an already populated cache answers a different question from a first request after startup.

Do not turn noise into a performance claim

The sample verifies that the expected work ran; it makes no assertion that one duration must be smaller than another. Background processes, compilation, scheduling, caches, and garbage collection can move short timings substantially. A fixed “must finish in one millisecond” unit test is often a test of the machine rather than the algorithm.

Repeat measurements, report the workload and variation, and check the full application after a local improvement. Retain a change when it improves the relevant outcome enough to justify its complexity—not merely because one run printed a pleasing number.

Sources

1. Sedgewick and Wayne — Analysis of Algorithms, on reproducible observations and cost models.
2. BenchmarkDotNet — How it works.
3. Python — timeit.
4. Python — Profilers and Microsoft — dotnet-counters.