Profiling: measure before you optimize

The #1 rule: don't guess. Human intuition about where Python code is slow is wrong ~half the time. Measure, then optimize the actual hotspot.

The two tools you'll reach for

timeit — micro-benchmarks

from timeit import timeit
timeit("[x*x for x in data]", setup="data = list(range(10_000))", number=1000)

Perfect for "which of these two expressions is faster?" It runs your snippet number times and reports total seconds. Divide by number for per-call.

cProfile — where does a real function spend time?

import cProfile
cProfile.run("my_function()", sort="cumulative")

Reports every function call: how many times, how long cumulatively, how long excluding sub-calls. Look at cumulative time first — that's where the wall-clock cost lives.

Reading a profile (what to look for)

• One function accounts for 60%+ of runtime → optimize that one first.
• Many small calls to the same function → cache with functools.lru_cache or restructure.
• All time spent in json.loads / re.match → the bottleneck is a library — either you're calling it in a loop or there's a faster alternative (orjson, precompiled regex).

Higher-signal tools

• line_profiler — line-by-line timings inside a function. Install: pip install line_profiler. Decorate with @profile.
• memory_profiler — same, but for RAM. Useful for "why does this eat 4GB?".
• py-spy — sampling profiler you can attach to a running process. Zero-overhead, no code change needed. Amazing for prod incidents.

The five shapes of real-world Python slowness

1. N+1 queries in DB/API code — one query per loop iteration. Fix: bulk fetch.
2. Loops that should be vectorized — pure Python over data → NumPy / pandas rewrites 20–200× faster.
3. Repeated JSON/regex — cache the parse or compile.
4. Excessive object creation — list of 10M small dicts blows past cache; use __slots__ or dataclass(slots=True).
5. GIL contention — CPU-bound work in threads. Fix: multiprocessing or a compiled extension.

Don't optimize what nobody notices

If a function runs once per hour and takes 500ms, it doesn't matter. Optimize the hot path — the code that runs per request, per row, per iteration.

Try it

1. Time sum(range(n)) vs n*(n-1)//2 for n = 10_000_000. How much faster is the closed-form?
2. Profile slow_pipeline and identify the biggest offender.