The Python Standard Library Part – II | Python 3 Standard Library Tutorial

1. Output Format

The reprlib module provides a customized version of the repr() function for large or deeply nested container abbreviations :

>>> import reprlib
>>> reprlib.repr(set('supercalifragilisticexpialidocious'))
"set(['a', 'c', 'd', 'e', 'f', 'g', ...])"

The pprint module gives veterans an interpreter-readable way to gain deeper control over the printing of built-in and user-defined objects. When the output exceeds one line, “pretty printer” adds line breaks and identifiers, making the data structure appear clearer:

>>> import pprint
>>> t = [[[['black', 'cyan'], 'white', ['green', 'red']], [['magenta',
...     'yellow'], 'blue']]]
...
>>> pprint.pprint(t, width=30)
[[[['black', 'cyan'],
   'white',
   ['green', 'red']],
  [['magenta', 'yellow'],
   'blue']]]

The textwrap module formats paragraphs of text to fit the set screen width:

>>> import textwrap
>>> doc = """The wrap() method is just like fill() except that it returns
... a list of strings instead of one big string with newlines to separate
... the wrapped lines."""
...
>>> print(textwrap.fill(doc, width=40))
The wrap() method is just like fill()
except that it returns a list of strings
instead of one big string with newlines
to separate the wrapped lines.

The locale module accesses a pre-determined national information database. The locale’s set of formatting function properties provides a straightforward way to format numbers by grouping them:

>>> import locale
>>> locale.setlocale(locale.LC_ALL, 'English_United States.1252')
'English_United States.1252'
>>> conv = locale.localeconv()          # get a mapping of conventions
>>> x = 1234567.8
>>> locale.format("%d", x, grouping=True)
'1,234,567'
>>>.localformat_string("%s%.*f", (conv['currency_symbol'],
...                      conv['frac_digits'], x), grouping=True)
'$1,234,567.80'

2. Template

String provides a flexible template class Template , which allows end users to edit easily. This allows users to customize their applications without making changes.

Format $beginning with valid Python identification (numbers, letters and underlined) as a placeholder. The braces outside of placeholders allow it to be mixed with other characters without spaces. $$Create a separate one $:

>>> from string import Template
>>> t = Template('${village}folk send $$10 to $cause.')
>>> t.substitute(village='Nottingham', cause='the ditch fund')
'Nottinghamfolk send $10 to the ditch fund.'

When a placeholder is not provided in a dictionary or keyword argument, the substitute() method throws a KeyError exception. For mail-merge-style applications, the user-supplied data may not be complete, and it may be more appropriate to use the safe_substitute() method—if the data is incomplete, it will not change placeholders:

>>> t = Template('Return the $item to $owner.')
>>> d = dict(item='unladen swallow')
>>> t.substitute(d)
Traceback (most recent call last):
  ...
KeyError: 'owner'
>>> t.safe_substitute(d)
'Return the unladen swallow to $owner.'

Template subclasses can specify a custom separator. For example, the image viewer’s bulk renaming tool may choose to use the percent sign as a placeholder, like the current date, picture serial number, or file format:

>>> import time, os.path
>>> photofiles = ['img_1074.jpg', 'img_1076.jpg', 'img_1077.jpg']
>>> class BatchRename(Template):
...     delimiter = '%'
>>> fmt = input('Enter rename style (%d-date %n-seqnum %f-format):  ')
Enter rename style (%d-date %n-seqnum %f-format):  Ashley_%n%f

>>> t = BatchRename(fmt)
>>> date = time.strftime('%d%b%y')
>>> for i, filename in enumerate(photofiles):
...     base, ext = os.path.splitext(filename)
...     newname = t.substitute(d=date, n=i, f=ext)
...     print('{0} --> {1}'.format(filename, newname))

img_1074.jpg --> Ashley_0.jpg
img_1076.jpg --> Ashley_1.jpg
img_1077.jpg --> Ashley_2.jpg

Another application of the template is to classify the various output format details from the program logic. This makes XML files, plain text reports, and HTML WEB report custom templates possible.

Related Articles: Learn Python In One Day

3. Using Binary Data Record Layouts

The struct module provides pack() and unpack() functions for variable-length binary record formats . The following example shows how to iterate over the header information of a ZIP file without using the zipfile module. Compressed codes "H"and "I"respectively 2 and 4 byte unsigned numbers, "<"indicating that they are in accordance with standard size and little-endian byte ordering.

import struct

with open('myfile.zip', 'rb') as f:
    data = f.read()

start = 0
for i in range(3):                      # show the first 3 file headers
    start += 14
    fields = struct.unpack('<IIIHH', data[start:start+16])
    crc32, comp_size, uncomp_size, filenamesize, extra_size = fields

    start += 16
    filename = data[start:start+filenamesize]
    start += filenamesize
    extra = data[start:start+extra_size]
    print(filename, hex(crc32), comp_size, uncomp_size)

    start += extra_size + comp_size     # skip to the next header

4. Multithreading

Threading is a technique for separating tasks without sequence dependencies. Applications can become sluggish when certain tasks are running in the background, and threads can increase their speed. A related use is in I/O while other threads can compute in parallel.

The following code shows how the advanced module threading runs the task while the main program is running:

Related Article: Python Data Structures

import threading, zipfile

class AsyncZip(threading.Thread):
    def __init__(self, infile, outfile):
        threading.Thread.__init__(self)
        self.infile = infile
        self.outfile = outfile
    def run(self):
        f = zipfile.ZipFile(self.outfile, 'w', zipfile.ZIP_DEFLATED)
        f.write(self.infile)
        f.close()
        print('Finished background zip of:', self.infile)

background = AsyncZip('mydata.txt', 'myarchive.zip')
background.start()
print('The main program continues to run in foreground.')

background.join()    # Wait for the background task to finish
print('Main program waited until background was done.')

The main challenge for multi-threaded applications is to coordinate threads, such as sharing data or other resources among threads. To achieve that goal, the thread module provides many synchronized native support, including: locks, events, condition variables, and semaphores.

Although these tools are powerful, minor design errors can also cause irreparable failures. Therefore, the preferred method of task coordination is to centralize all access to a resource in a single thread, and then use the queue module to use that thread to service requests from other threads. Applications that use the Queue object for internal thread communication and coordination are easier to design, more readable, and more reliable.

Related Article: Python Modules

5. Logs

The logging module provides a complete and flexible logging system. Its simplest use is to record information and send it to a file or sys.stderr:

import logging
logging.debug('Debugging information')
logging.info('Informational message')
logging.warning('Warning:config file %s not found', 'server.conf')
logging.error('Error occurred')
logging.critical('Critical error -- shutting down')

Related Article: Python Input and Output

The output is as follows:

WARNING:root:Warning:config file server.conf not found
ERROR:root:Error occurred
CRITICAL:root:Critical error -- shutting down

Information and debug messages are captured by default and output is sent to the standard error stream. Other optional routing information is via email, data packets, sockets or HTTP Server. Based on the message attributes, new filters can be routed differently: DEBUG, INFO, WARNING, ERRORand CRITICAL.

The logging system can be customized directly in the Python code, or it can be loaded directly in a user-editable configuration file without going through the application.

Related Article: Python Errors and Exceptions

6. Weak References

Python automates memory management (reference counting and garbage collection for most objects – garbage collection – recycling) After the last reference disappears, memory is released quickly.

This way of working works well for most applications, but occasionally you need to track objects to do something. Unfortunately, simply creating a reference for tracking them will also make it long-term. 

Related Article: Python Classes and Objects

The weakref module provides a tracking object tool that does not create a reference. Once the object no longer exists, it automatically removes the weak reference list and triggers a callback. Typical applications include capturing difficult-to-construct objects:

>>> import weakref, gc
>>> class A:
...     def __init__(self, value):
...         self.value = value
...     def __repr__(self):
...         return str(self.value)
...
>>> a = A(10)                   # create a reference
>>> d = weakref.WeakValueDictionary()
>>> d['primary'] = a            # does not create a reference
>>> d['primary']                # fetch the object if it is still alive
10
>>> del a                       # remove the one reference
>>> gc.collect()                # run garbage collection right away
0
>>> d['primary']                # entry was automatically removed
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
    d['primary']                # entry was automatically removed
  File "C:/python34/lib/weakref.py", line 46, in __getitem__
    o = self.data[key]()
KeyError: 'primary'

7. List Tools

Many data structures may use built-in list types. However, it may sometimes be necessary to implement different performance costs.

The array module provides a list-like array() object that stores data only and is more compact. The following example demonstrates an array (type encoding "H") that stores two-byte unsigned integers rather than a normal regular list of 16-byte Python integer objects:

>>> from array import array
>>> a = array('H', [4000, 10, 700, 22222])
>>> sum(a)
26932
>>> a[1:3]
array('H', [10, 700])

The collections module provides a list-like deque() object that is faster to append and pop from the left, but is slower to query internally. These objects are more suitable for queue implementation and breadth-first tree search:

>>> from collections import deque
>>> d = deque(["task1", "task2", "task3"])
>>> d.append("task4")
>>> print("Handling", d.popleft())
Handling task1

unsearched = deque([starting_node])
def breadth_first_search(unsearched):
    node = unsearched.popleft()
    for m in gen_moves(node):
        if is_goal(m):
            return m
        unsearched.append(m)

In addition to the alternative implementation of the linked list, the library also provides bisect such modules to operate the storage chain list:

>>> import bisect
>>> scores = [(100, 'perl'), (200, 'tcl'), (400, 'lua'), (500, 'python')]
>>> bisect.insort(scores, (300, 'ruby'))
>>> scores
[(100, 'perl'), (200, 'tcl'), (300, 'ruby'), (400, 'lua'), (500, 'python')]

Heapq provides a heap implementation based on a normal linked list. The minimum value always stays at 0 o’clock. This is useful when you want to iterate through the smallest elements but do not want to perform a complete heap sort:

>>> from heapq import heapify, heappop, heappush
>>> data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0]
>>> heapify(data)                      # rearrange the list into heap order
>>> heappush(data, -5)                 # add a new entry
>>> [heappop(data) for i in range(3)]  # fetch the three smallest entries
[-5, 0, 1]

8. Decimal Floating Point Algorithm

The decimal module provides a Decimal data type for floating-point calculations. Float is implemented compared to the built-in binary float , this type helps

• Financial applications and other situations that require precise decimal representations

• control precision,

• Controlling rounding to suit legal or regulatory requirements,

• To ensure decimal digit accuracy,

  or

• The user wants the calculation results to match the hand calculation.

For example, to calculate the 5% tax calculation for a 70-point phone bill, the difference between the decimal floating-point number and the binary floating-point number calculation result is as follows. If rounding on points, this difference is important:

>>> from decimal import *
>>> round(Decimal('0.70') * Decimal('1.05'), 2)
Decimal('0.74')
>>> round(.70 * 1.05, 2)
0.73

Decimal ‘s result always has an ending 0, which automatically extends from two digits to four digits. Decimal reproduces manual mathematical operations. This ensures that binary floating-point numbers do not guarantee the accuracy of data.

High precision allows Decimal to perform modular operations and equivalent tests that cannot be performed with binary floating-point numbers:

>>> Decimal('1.00') % Decimal('.10')
Decimal('0.00')
>>> 1.00 % 0.10
0.09999999999999995

>>> sum([Decimal('0.1')]*10) == Decimal('1.0')
True
>>> sum([0.1]*10) == 1.0
False

Decimal provides the necessary high-precision algorithm:

>>> getcontext().prec = 36
>>> Decimal(1) / Decimal(7)
Dec