Profiling in Python Tutorial

Profiling in Python is another important parameter used by different programmers in performing different tasks. Learn more about it here.

Profiling in Python: %%timeit and %timeit in IPython

Profiling string concatenation:

In [1]: import string
In [2]: %%timeit s=""; long_list=list(string.ascii_letters)50 ….: for substring in long_list: ….: s+=substring ….: 1000 loops, best of 3: 570 us per loop In [3]: %%timeit long_list=list(string.ascii_letters)50
….: s="".join(long_list)
100000 loops, best of 3: 16.1 us per loop

Profiling loops over iterables and lists:

In [4]: %timeit for i in range(100000):pass
100 loops, best of 3: 2.82 ms per loop
In [5]: %timeit for i in list(range(100000)):pass
100 loops, best of 3: 3.95 ms per loop

Using cProfile (Preferred Profiler)

Python includes a profiler called cProfile. This is generally preferred over using timeit.

It breaks down your entire script and for each method in your script it tells you:

ncalls: The number of times a method was called

tottime: Total time spent in the given function (excluding time made in calls to sub-functions)

percall: Time spent per call. Or the quotient of tottime divided by ncalls

cumtime: The cumulative time spent in this and all subfunctions (from invocation till exit). This figure is accurate even for recursive functions.

percall: is the quotient of cumtime divided by primitive calls

filename:lineno(function): provides the respective data of each function

The cProfiler can be easily called on Command Line using:

$ python -m cProfile

To sort the returned list of profiled methods by the time taken in the method:

$ python -m cProfile -s time

Profiling in Python: timeit() function

Profiling repetition of elements in an array

import timeit

timeit.timeit('list(itertools.repeat("a", 100))', 'import itertools', number = 10000000) 10.997665435877963
timeit.timeit('["a"]*100', number = 10000000)

timeit command line

Profiling concatenation of numbers

python -m timeit "'-'.join(str(n) for n in range(100))"
10000 loops, best of 3: 29.2 usec per loop
python -m timeit "'-'.join(map(str,range(100)))"
100000 loops, best of 3: 19.4 usec per loop

Profiling in Python: line_profiler in command line

The source code with @profile directive before the function we want to profile:

import requests
def slow_func():
s = requests.session()
sum([pow(ord(x),3.1) for x in list(html)])
for i in range(50):

Using kernprof command to calculate profiling line by line

$ kernprof -lv
Wrote profile results to
Timer unit: 4.27654e-07 s
Total time: 22.6427 s
Function: slow_func at line 4

Line # Hits Time Per Hit % Time Line Contents

4 @profile
5 def slow_func():
6 50 20729 414.6 0.0 s = requests.session()
7 50 47618627 952372.5 89.9 html=s.get("").text
8 50 5306958 106139.2 10.0 sum([pow(ord(x),3.1) for x in list(html)])
Page request is almost always slower than any calculation based on the information on the page.

Learn More

Leave a Comment