New features and improvements in Python 3.11

In the last few years, we have seen a great evolution in the Python programming language as it has gained popularity to become one of the most widely used programming languages.

The involvement of the community with the development of the main implementation of the language (CPython), creating an annual release cycle, combined with the recent incorporation of the language’s creator (Guido Van Rosum) to Microsoft in the Faster CPython project, seem to indicate an even greater evolution in the coming years and an additional effort in improving the efficiency of CPython and the syntax of the language to adapt to increasingly complex projects.

In the following sections we explore the most important features added in the new version of Python 3.11 with an official release date of 24/10/2022.

Improved efficiency

Python is expected to increase its efficiency by between 10% and 60% with an average increase of 25%.

This is mainly due to the introduction of the specialized adaptive interpreter. The idea behind this implementation is that, although Python is a dynamically typed language, there are many areas of code where the types are stable. For that reason, the interpreter will try to adapt to the executed code by looking for patterns and type stability by substituting generic operations with operations specific to that type.

This specialization will occur in small code fragments executed frequently, thus avoiding the cost of adaptation for code that will only be executed once. For this reason, this improvement will have a much more significant impact on programs containing frequently executed blocks.

The interpreter can also be despecialized and revert to generic operations in case the use of the code changes or becomes a more dynamic fragment, thus adapting itself according to the needs of the executing program.

We will illustrate a case where this improvement is noticeable with the following code:

x = 1
for i in range(300):
   x += i

As we can see, we always work with integer types and, therefore, we have a stable type for which we can use optimised addition operations. Executing this fragment 1,000,000 times in Python 3.10 takes 8,822 seconds; whereas in version 3.11, it takes 7,110 seconds, a significant increase of 20%.

On the other hand, the time and memory efficiency for function calls at runtime is improved and the depth to which recursive functions in Python code can reach is increased.

Finally, another aspect to highlight is the increase of 10% to 15% in the initialization of programs due to the optimization of the loading times of the language’s base modules. This is especially relevant in the case of short-lived or frequently executed scripts.

Improvements in error reporting

A major improvement in the interpreter for debugging code is the flagging of errors by console messages. From version 3.11 onwards, the exact expression that caused the exception will be flagged instead of just the line.

For example, for the following code:

def division(x: int, y: int):
   return x / y


print(division(10, 0))

The following exception is displayed:

Improvements in exception management

The concept of Exception Groups has been added, allowing us to group multiple exceptions and use the new except* syntax to handle them.

This way, we can execute several except* blocks for a single try block, being able to manage multiple concurrent and even nested exceptions.

For example, in the following code you can see how two grouped exceptions are handled:

try:
   raise ExceptionGroup("group", [TypeError(), ValueError()])
except* ValueError:
   print("This will be handled first.")
except* TypeError:
   print("Then this will be handled.")

On the other hand, the possibility of adding notes to exceptions to increase their contextual information is also introduced. The new add_note() method is used for this purpose.

try:
   raise TypeError("Bad type")
except TypeError as e:
   e.add_note("Extra information")
   raise

Support for the TOML format

A new tomllib module has been added to parse the popular TOML format which has become the new metadata storage format for Python programs. It is as easy to use as:

import tomllib

with open("pyproject.toml","rb") as f:
   data = tomllib.load(f)

We can also work with python strings using the loads function:

import tomllib

data = """
python-version = "3.11.0"

dependencies = [
   "numpy>=1.23.0"
]
"""

tomllib.loads(data)

Improvements to the Typing module

Python is increasingly being used to build more complex software. Because of this, the typing module, which offers type suggestions, was introduced in version 3.5. In this new release, the typing module has been improved.

Support for variadic generics, that is, generic types containing an arbitrary number of types, has been added. This feature is especially useful when working with tensors or multidimensional arrays, as it can provide us with information about the type of the dimensions of our object or the type resulting from them when applying transformations.

The Self type is also introduced, allowing us to correctly annotate methods that return an instance of the same class, being especially useful for annotations in inheritance cases.

import abc
from typing import Self

class Shape(abc.ABC):
  def set_scale(self, scale: float) -> Self:
      self.scale = scale
      return self

class Circle(Shape):
   ...

Circle().set_scale(1.0) # The return type is Circle

Finally, it is worth noting the introduction of the new “LiteralString” type that allows us to identify literal strings. A possible use shown below is the protection against SQL injection vulnerabilities:

from typing import LiteralString

def execute_query(query: LiteralString):
   ...

execute_query("SELECT * FROM USERS") # Literal
execute_query(f"SELECT * FROM {table}") # Non literal

Other new features

Version 3.11 comes with many new features that are not possible to analyse in detail in a single post. However, it is important to mention some of the ones that might have more relevance:

A context manager has been added for the chdir method allowing to change directory only in the scope of the context manager.
New datetime.UTC alias for datetetime.timezone.UTC.
Support for atomic groups (?>…) and positive quantifiers (*+, ++, ?+, {m,n}+) in the re module.
Increased precision to nanosecond (if available on Unix system) for the time.sleep() method.
Addition of the TaskGroup class for the asyncio module allowing to group tasks and wait for their completion.

Conclusion

The improvements introduced in the typing module, as well as its popularity, and the improvement in exception handling show a tendency for the language to be used by increasingly complex software.

However, the focus of this update has been on speed improvement. Although its purpose is not and will not be to become a language whose main characteristic is efficiency, we see that the efforts of the community have the improvement of this as one of the main points to develop and this is reflected in version 3.11.

Possibly, in later versions, the increase in performance will continue to be one of the main objectives.

This is it! If you found this article interesting, we encourage you to visit the Software category to see all the related posts and to share it on social networks. See you soon!

programming, python