Frustrations in Python

Dark forces cast their wicked spells to leak into our realm of precious Python programs.

They spam their twisted magic uncontrollably and pollute our readable code.

Today I am going to reveal several chthonic creatures that might already live inside your codebase and accustom themselves enough to start making their own rules. We need a hero to protect our peaceful world from these evil entities. And you will be this hero to fight them!

All heroes need weapons enchanted with light magic to serve them well in their epic battles.

wemake-python-styleguide will be your sharp weapon and your best companion.

Let’s start our journey!

Space invaders

Not so long ago, space invaders were spotted in Python. They take bizarre forms.

  5:5      E225  missing whitespace around operator
  x -=- x
      ^

  5:5      WPS346 Found wrong operation sign
  x -=- x
      ^

  10:2     E225  missing whitespace around operator
  o+=+o
   ^

  14:10    E225  missing whitespace around operator
  print(3 --0-- 5 == 8)
           ^

  14:10    WPS346 Found wrong operation sign
  print(3 --0-- 5 == 8)
           ^

  14:11    WPS345 Found meaningless number operation
  print(3 --0-- 5 == 8)
            ^

  14:12    E226  missing whitespace around arithmetic operator
  print(3 --0-- 5 == 8)
             ^

  14:13    WPS346 Found wrong operation sign
  print(3 --0-- 5 == 8)
              ^

This is how our code base should look afterward:

x = 1
x += x

o = 2
o += o

print(3 + 5 == 8)

Readable and clean!

Mystical dots

Some citizens report that some strange codeglyphes are starting to appear. Look, here they are!

print(0..__eq__(0))
# => True

print(....__eq__(((...))))
# => True

What is going on here? Looks like a partial float and Ellipsis to me, but better be sure.

  21:7     WPS609 Found direct magic attribute usage: __eq__
  print(0..__eq__(0))
        ^

  21:7     WPS304 Found partial float: 0.
  print(0..__eq__(0))
        ^

  24:7     WPS609 Found direct magic attribute usage: __eq__
  print(....__eq__(((...))))
        ^

Ouch! Now we are sure. It is indeed the partial float with dot property access and Elipsis with the same dot access. Let’s reveal all the hidden things now:

print(0.0 == 0)
print(... == ...)

And still, it is better not to provoke wrath and not to compare constants in other places.

Misleading path

We have a new incident. Some values have never seen returned from a function. Let’s find out what is going on.

def some_func():
    try:
       return 'from_try'
    finally:
       return 'from_finally'

some_func()
# => 'from_finally'

We are missing 'from_try' due to a broken entity in our code, how this can be addressed?

 31:5 WPS419 Found `try`/`else`/`finally` with multiple return paths
 try:
 ^

Turns out wemake-python-styleguide knew it all along the way! It teaches us to never return from finally. Let’s obey it.

def some_func():
  try:
      return 'from_try'
  finally:
      print('now in finally')

The C-ursed legacy

Some ancient creature is awakening. It hasn’t been seen for decades. And now it has returned.

a = [(0, 'Hello'), (1, 'world')]
for ['>']['>'>'>'], x in a:
    print(x)

What is going on here? One can implicitly unpack values inside loops. And the target for unpacking might be almost any valid Python expression.

But, we should not do a lot of things from this example:

  44:1     WPS414 Found incorrect unpacking target
  for ['>']['>'>'>'], x in a:
  ^

  44:5     WPS405 Found wrong `for` loop variable definition
  for ['>']['>'>'>'], x in a:
      ^

  44:11    WPS308 Found constant compare
  for ['>']['>'>'>'], x in a:
            ^

  44:14    E225  missing whitespace around operator
  for ['>']['>'>'>'], x in a:
               ^

  44:21    WPS111 Found too short name: x
  for ['>']['>'>'>'], x in a:
                      ^

Looks like the ['>'\]['>'>'>'] is just ['>'\][0] because '>' > '>' is False.

This case is solved.

Signature of the black sorcerer

How complex can an expression be in Python? The Black Sorcerer leaves his complex mark on all classes he touches:

class _:
    # There are four of them, do you see it?
    _: [(),...,()] = {((),...,()): {(),...,()}}[((),...,())]

print(_._) # this operator also looks familiar 🤔
# => {(), Ellipsis}

How can this signature be read and evaluated? Looks like it consists of several parts:
– Declaration and type annotation: _: [(),...,()] =
– Dictionary definition with a set as a value: = { ((),...,()): {(),...,()} }
– Key access: [((),...,())]

While it does not make any sense to human beings from this world, it is still a valid Python code that can be used for something evil. Let’s remove it:

  55:5     WPS122 Found all unused variables definition: _
  _: [(),...,()] = {((),...,()): {(),...,()}}[((),...,())]
  ^

  55:5     WPS221 Found line with high Jones Complexity: 19
  _: [(),...,()] = {((),...,()): {(),...,()}}[((),...,())]
  ^

  55:36    WPS417 Found non-unique item in hash: ()
  _: [(),...,()] = {((),...,()): {(),...,()}}[((),...,())]
                                 ^

  57:7     WPS121 Found usage of a variable marked as unused: _
  print(_._)  # this operator also looks familiar 
        ^

And now this complex expression (with Jones Complexity rate of 19) is removed or refactored. Any the Signature of the Black Sourcerer is removed from this poor class. Let’s leave it in peace.

Metamagic

Our regular classes start to hang out with some shady types. We need to protect them from this bad influence.
Currently, their output really strange:

class Example(type((lambda: 0.)())):
 ...

print(Example(1) + Example(3))
# => 4.0

Why 1 + 3 is 4.0 and not 4? To find out, let’s unwrap the type((lambda: 0.)()) piece:
– (lambda: 0.)() is just 0. which is just 0.0.
– type(0.0) is float
– When we write Example(1) it is converted to Example(1.0) inside the class.
– Example(1.0) + Example(3.0) is Example(4.0)

Let’s be sure that our weapon is sharp as always:

  63:15    WPS606 Found incorrect base class
  class Example(type((lambda: 0.)())):
                ^

  63:21    WPS522 Found implicit primitive in a form of lambda
  class Example(type((lambda: 0.)())):
                      ^

  63:29    WPS304 Found partial float: 0.
  class Example(type((lambda: 0.)())):
                              ^

  64:5     WPS428 Found statement that has no effect
  ...
  ^

  64:5     WPS604 Found incorrect node inside `class` body
  ...
  ^

We have found all the possible issues here. Our classes are safe. Time to move on.

Regenerators

So similar and yet so different. Regenerator is found in our source code. It looks like an average generator expression, but it’s something totally different.

a = ['a', 'b']
print(set(x + '!' for x in a))
# => {'b!', 'a!'}

print(set((yield x + '!') for x in a))
# => {'b!', None, 'a!'}

This is a bug in Python — yes, they do exist. And since python3.8 is a SyntaxError, one should not use yield and yield from outside of generator functions.

Here’s our usual report about the incident:

  73:7     C401  Unnecessary generator - rewrite as a set comprehension.
  print(set(x + '!' for x in a))
        ^

  76:7     C401  Unnecessary generator - rewrite as a set comprehension.
  print(set((yield x + '!') for x in a))
        ^

  76:11    WPS416 Found `yield` inside comprehension
  print(set((yield x + '!') for x in a))

 

Also, let’s write comprehensions correctly as suggested.

print({x + '!' for x in a})

This was a hard one to solve. But in the end, Regenerator is gone and so are wrong comprehensions. What’s next?

Email evil clone

If one needs to write an email address, the string is used. Right? Wrong!

There are unusual ways to do regular things. And there are evil clones of regular datatypes.
We are going to discover them.

class G:
    def __init__(self, s):
        self.s = s
    def __getattr__(self, t):
        return G(self.s + '.' + str(t))
    def __rmatmul__(self, other):
        return other + '@' + self.s

username, example = 'username', G('example')
print([email protected])
# => [email protected]

How does it work?
– @ is an operator in Python, it’s behavior can be modified via __matmul__ and __rmatmul__ magic methods
– .com is an attribute com dot access, it can be modified via __getattr__

One big difference between this code and other examples is that this one is actually valid. Just unusual. We should probably not use it. But, let’s write this into our knowledge quest book.

Fallacy of the walrus

The darkness has fallen onto Python. The one that has split the friendly developer community, the one that brought the controversy.
You have gained the power to program in strings:

from math import radians
for angle in range(360):
    print(f'{angle=} {(th:=radians(angle))=:.3f}')
    print(th)

# => angle=0 (th:=radians(angle))=0.000
# => 0.0
# => angle=1 (th:=radians(angle))=0.017
# => 0.017453292519943295
# => angle=2 (th:=radians(angle))=0.035
# => 0.03490658503988659

What is going on here?
– f'{angle=} is a new (python3.8+) way to write f'angle={angle}
– (th:=radians(angle)) is an assignment expression, yes you can do assignments in strings now
– =:.3f is the formatting part, it returns the expression and its rounded result value
– print(th) works because (th:=radians(angle)) has the local scope effect

Should you use assignment expressions? Well, that’s up to you.
Should you assign values inside strings? Absolutely not.

And here’s a friendly reminder of things that you can (but also probably should not) do with f strings themselves:

print(f"{getattr(__import__('os'), 'eman'[None:None:-1])}")
# => posix

Just a regular module import inside a string — move on, nothing to see here.

Luckily, we are not allowed to write this line in our real code:

105:1    WPS221 Found line with high Jones Complexity: 16
  print(f"{getattr(__import__('os'), 'eman'[None:None:-1])}")
  ^

  105:7    WPS305 Found `f` string
  print(f"{getattr(__import__('os'), 'eman'[None:None:-1])}")
        ^

  105:18   WPS421 Found wrong function call: __import__
  print(f"{getattr(__import__('os'), 'eman'[None:None:-1])}")
                   ^

  105:36   WPS349 Found redundant subscript slice
  print(f"{getattr(__import__('os'), 'eman'[None:None:-1])}")
                                     ^

And one more thing: f strings cannot be used as docstrings:

def main():
    f"""My name is {__file__}/{__name__}!"""

print(main().__doc__)
# => None

Conclusion

We fought many ugly monsters that spawned inside our code and made Python land a better place to live. You should be proud of yourself, hero!

That was an epic journey. And I hope you learned something new: to be stronger for the next battles to come. The world needs you!

That’s it for today. Stay safe, traveler.

Useful links

• Python code disasters
• wtf, python?
• wemake-python-styleguide