Tag Archives: Python 3

Python 3 – Unpacking Generalizations

Python 3.5 added more support for Unpacking Generalizations in PEP 448. According to the PEP, it added extended usages of the * iterable unpacking operator and ** dictionary unpacking operators to allow unpacking in more positions, an arbitrary number of times, and in additional circumstances. What this means is that we can now make calls to functions with an arbitrary number of unpackings. Let’s take a look at a dict() example:

>>> my_dict = {'1':'one', '2':'two'}
>>> dict(**my_dict, w=6)
{'1': 'one', '2': 'two', 'w': 6}
>>> dict(**my_dict, w='three', **{'4':'four'})
{'1': 'one', '2': 'two', 'w': 'three', '4': 'four'}

Interestingly, if the keys are something other then strings, the unpacking doesn’t work:

>>> my_dict = {1:'one', 2:'two'}
>>> dict(**my_dict)
Traceback (most recent call last):
  File "<pyshell#27>", line 1, in <module>
TypeError: keyword arguments must be strings

Update: One of my readers was quick to point out that the reason this doesn’t work is because I was trying to unpack into a function call (i.e. dict()). If I had done the unpacking using just dict syntax, the integer keys would have worked fine. Here’s what I’m talking about:

>>> {**{1: 'one', 2:'two'}, 3:'three'}
{1: 'one', 2: 'two', 3: 'three'}

One other interesting wrinkle to dict unpacking is that later values will always override earlier ones. There’s a good example in the PEP that demonstrates this:

>>> {**{'x': 2}, 'x': 1}
{'x': 1}

I thought that was pretty neat. You can do the same sort of thing with ChainMap from the collections module, but this is quite a bit simpler.

However this new unpacking also works for tuples and lists. Let’s try combining some items of different types into one list:

>>> my_tuple = (11, 12, 45)
>>> my_list = ['something', 'or', 'other']
>>> my_range = range(5)
>>> combo = [*my_tuple, *my_list, *my_range]
>>> combo
[11, 12, 45, 'something', 'or', 'other', 0, 1, 2, 3, 4]

Before this unpacking change, you would have had do something like this:

>>> combo = list(my_tuple) + list(my_list) + list(my_range)
[11, 12, 45, 'something', 'or', 'other', 0, 1, 2, 3, 4]

I think the new syntax is actually quite handy for these kinds of circumstances. I’ve actually run into this a time or two in Python 2 where this new enhancement would have been quite useful.

Wrapping Up

There are lots of other examples in PEP 448 that are quite interesting to read about and try in Python’s interpreter. I highly recommend checking it out and giving this feature a try. I am hoping to start using some of these features in my new code whenever we finally move to Python 3.

Python’s New secrets Module

Python 3.6 added a new module called secrets that is designed “to provide an obvious way to reliably generate cryptographically strong pseudo-random values suitable for managing secrets, such as account authentication, tokens, and similar”. Python’s random module was never designed for cryptographic use but for modeling and simulation. Of course, you could always use the urandom() function from Python’s os module:

>>> import os
>>> os.urandom(8)

But now that we have the secrets module, we can create our own “cryptographically strong pseudo-random values”. Here’s a simple example:

>>> import secrets
>>> import string
>>> characters = string.ascii_letters + string.digits
>>> bad_password = ''.join(secrets.choice(characters) for i in range(8))
>>> bad_password

In this example, we import the secrets and the string modules. Next we create a string of uppercase letters and integers. Finally we use the secrets module’s choice() method to choose characters randomly to generate a bad password. The reason I am calling this a bad password is because we aren’t adding symbols to the password. This is actually a fairly decent one compared with what a lot of people use. One of my readers pointed out that another reason this could be considered bad is that the user would probably just write it down on a piece of paper. While that may be true, using dictionary words is typically strongly discouraged so you should learn to use passwords like this or invest in a secure password manager.

Continue reading Python’s New secrets Module

What’s New in Python: Asynchronous Comprehensions / Generators

Python 3.6 added the ability to create Asynchronous Comprehensions and Asynchronous Generators. You can read about asynchronous comprehension in PEP 530 while the asynchronous generators are described in PEP 525. The documentation states that you can now create asynchronous list, set and dict comprehensions and generator expressions. Their example looks like this:

result = [i async for i in aiter() if i % 2]

Basically you just need to add Python’s new async keyword into your expression and call a callable that has implemented __aiter__. Trying to follow this syntax will actually result in a SyntaxError though:

>>> result = [i async for i in range(100) if i % 2]
  File "<stdin>", line 1
    result = [i async for i in range(100) if i % 2]
SyntaxError: invalid syntax

Continue reading What’s New in Python: Asynchronous Comprehensions / Generators

New in Python: Formatted string literals

Python 3.6 added yet another way to do string substitution that they are calling “Formatted String Literals”. You can read all about the concept in PEP 498. I take a bit of umbrage here in that the Zen of Python states that There should be one– and preferable only one –obvious way to do it. Now Python has three ways. Let’s take a walk down memory lane before we talk about the latest way to play with strings.

Ye Olde String Substitution

When Python first started, they followed C++ in the way they did string substitution by using %s, %i and the like. Here are a couple of examples:

>>> The %s fox jumps the %s' % ('quick', 'crevice')
'The quick fox jumps the crevice'
>>> foo = 'The total of your purchase is %.2f' % 10
>>> foo
'The total of your purchase is 10.00'

Continue reading New in Python: Formatted string literals

New in Python: Syntax for variable annotations

Python 3.6 added another interesting new feature that is known as Syntax for variable annotations. This new feature is outlined in PEP 526. The basic premise of this PEP is take the idea of Type Hinting (PEP 484) to its next logical step, which is basically adding option type definitions to Python variables, including class variables and instance variables. Please note that adding these annotations or definitions does not suddenly make Python a statically typed language. The interpreter still doesn’t care what type the variable is. However, a Python IDE or other utility like pylint could have an annotation checker added to them that could highlight when you use a variable that you have annotated as one type and then used incorrectly by changing its type mid-function.

Let’s look at a simple example so we can see how this works:

# annotate.py
name: str = 'Mike'

Continue reading New in Python: Syntax for variable annotations

New in Python: Underscores in Numeric Literals

Python 3.6 added some interesting new features. The one that we will be looking at in this article comes from PEP 515: Underscores in Numeric Literals. As the name of the PEP implies, this basically gives you the ability to write long numbers with underscores where the comma normally would be. In other words, 1000000 can now be written as 1_000_000. Let’s take a look at some simple examples:

>>> 1_234_567
>>> '{:_}'.format(1234567)

The first example just shows how Python interprets a large number with underscores in it. The second example demonstrates that we can now give Python a string formatter, the “_” (underscore), in place of a comma. The results speak for themselves.

The numeric literals that include underscores behave the same way as normal numeric literals when doing calculations:

>>> 120_000 + 30_000
>>> 120_000 - 30_000

The Python documentation and the PEP also mention that you can use the underscores after any base specifier. Here are a couple of examples taken from the PEP and the documentation:

>>> flags = 0b_0011_1111_0100_1110
>>> flags
>>> 0x_FF_FF_FF_FF
>>> flags = int('0b_1111_0000', 2)
>>> flags

There are some notes about the underscore that need to be mentioned:

  • You can only use one consecutive underscore and it has to be between digits and after any base specifier
  • Leading and trailing underscores are not allowed

This is kind of a fun new feature in Python. While I personally don’t have any use cases for this in my current job, hopefully you will have one at yours.

ANN: The Python by Example Udemy Course

I am happy to announce my first Udemy course on the Python programming language. It is called Python by Example and is a re-branding of my Python 101 Screencast series. I had originally hoped to keep the name, Python 101 on Udemy, but someone else had already taken it by the time I got my course started on there. Regardless, the course is made with the first 35 videos from the Python 101 Screencast. I plan to add the rest later this year.


What You Get

At the time of writing, you will receive 35 lectures or 6.5 hours of content. I have also included chapters from Python 101 that correspond with each lecture.

How to Purchase

To purchase Python by Example, just go to the following link: https://www.udemy.com/python-by-example/?couponCode=py25. The coupon code will give you 25% off.

Python 201: An Intro to importlib

Python provides the importlib package as part of its standard library of modules. Its purpose is to provide the implementation to Python’s import statement (and the __import__() function). In addition, importlib gives the programmer the ability to create their own custom objects (AKA an importer) that can be used in the import process.

What about imp?

There is another module called imp that provides an interface to the mechanisms behind Python’s import statement. This module was deprecated in Python 3.4. It is intended that importlib should be used in its place.

This module is pretty complicated, so we’ll be limiting the scope of this article to the following topics:

  • Dynamic imports
  • Checking is a module can be imported
  • Importing from the source file itself

Let’s get started by looking at dynamic imports!

Continue reading Python 201: An Intro to importlib

Python 3: An Intro to Encryption

Python 3 doesn’t have very much in its standard library that deals with encryption. Instead, you get hashing libraries. We’ll take a brief look at those in the chapter, but the primary focus will be on the following 3rd party packages: PyCrypto and cryptography. We will learn how to encrypt and decrypt strings with both of these libraries.


If you need secure hashes or message digest algorithms, then Python’s standard library has you covered in the hashlib module. It includes the FIPS secure hash algorithms SHA1, SHA224, SHA256, SHA384, and SHA512 as well as RSA’s MD5 algorithm. Python also supports the adler32 and crc32 hash functions, but those are in the zlib module.

One of the most popular uses of hashes is storing the hash of a password instead of the password itself. Of course, the hash has to be a good one or it can be decrypted. Another popular use case for hashes is to hash a file and then send the file and its hash separately. Then the person receiving the file can run a hash on the file to see if it matches the hash that was sent. If it does, then that means no one has changed the file in transit.

Continue reading Python 3: An Intro to Encryption

Python 201 – What is a ChainMap?

A ChainMap is a class from the collections module that provides the ability to link multiple mappings together such that they end up being a single unit. If you look at the documentation, you will notice that it accepts *maps, which means that a ChainMap will accept any number of mappings or dictionaries and turn them into a single view that you can update. Let’s look at an example so you can see how this works:

>>> from collections import ChainMap
>>> car_parts = {'hood': 500, 'engine': 5000, 'front_door': 750}
>>> car_options = {'A/C': 1000, 'Turbo': 2500, 'rollbar': 300}
>>> car_accessories = {'cover': 100, 'hood_ornament': 150, 'seat_cover': 99}
>>> car_pricing = ChainMap(car_accessories, car_options, car_parts)
>>> car_pricing['hood']

Here we import ChainMap from our collections module. Next we create three dictionaries. Then we create an instance of our ChainMap by passing in the three dictionaries that we just created. Finally, we try accessing one of the keys in our ChainMap. When we do this, the ChainMap will go through each map in order to see if that key exists and has a value. If it does, then the ChainMap will return the first value it finds that matches that key.

Continue reading Python 201 – What is a ChainMap?