dir(obj) introspection is slow

[raquo]

CircuitPython version and board name

- CircuitPython 9.2.8 @ Qt Py M0 (similar performance on 9.2.7)
- CircuitPython 9.2.8 @ ItsyBitsy M4 (have not tried other versions)

Code/REPL

import time
import supervisor

class Foo:
    pass

class Bar:
    a = 1
    b = 2
    c = 3
    d = 4
    e = 5
    f = 6
    g = 7
    h = 8
    i = 9
    j = 10

foo = Foo()

bar = Bar()

arr = [1, 2, 3, 4, 5]

dct = {"a": 1, "b": 2, "c": 3}

def measure(label, value):
    now = supervisor.ticks_ms()
    dir(value)
    diff = supervisor.ticks_ms() - now
    print("dir({}) time = {} ms".format(label, diff))

while True:
    time.sleep(1)
    print("\n")
    measure("123", 123)
    measure("\" \"", " ")
    measure("arr", arr)
    measure("dct", dct)
    measure("foo", foo)
    measure("Foo", Foo)
    measure("bar", bar)
    measure("Bar", Bar)

Behavior

CircuitPython 9.2.8 @ Qt Py M0:

dir(123) time = 7 ms
dir(" ") time = 10 ms
dir(arr) time = 9 ms
dir(dct) time = 9 ms
dir(foo) time = 31 ms
dir(Foo) time = 16 ms
dir(bar) time = 58 ms
dir(Bar) time = 30 ms

CircuitPython 9.2.8 @ ItsyBitsy M4:

dir(123) time = 4 ms
dir(" ") time = 9 ms
dir(arr) time = 6 ms
dir(dct) time = 7 ms
dir(foo) time = 14 ms
dir(Foo) time = 7 ms
dir(bar) time = 23 ms
dir(Bar) time = 12 ms

Description

Hello, I have a vague understanding that dir() needs to walk the class hierarchy to collect all the fields, but still I was not at all expecting it to take so long, for simple class instances like foo. I'm not sure if anything can be done to improve the speed, but figured I'd file the issue anyway in case it's a performance bug.

Additional information

No response

[Neradoc]

That would be a micropython core issue (I don't know if we have CP changes to it), but also dir is not really intended to be used outside of the REPL basically, so its performance doesn't seem to be relevant to me.

There are other reasons why you shouldn't use dir in scripts. Like the fact that it will evaluate properties, which could cause massive side effects, especially in Cicuitpython libraries where sensors often use computed properties that will cause an I2C transaction or something. Or raise "not implemented" exceptions that are sometimes used as placeholders (though they are silenced when calling dir in the REPL).

>>> class Thing:
...     @property
...     def foo(self):
...         print("This shouldn't happen")
...         
...         
... 
>>> t = Thing()
>>> dir(t)
This shouldn't happen
['__class__', '__module__', '__qualname__', '__dict__', 'foo']
>>> 

Quoting from this micropython#4546 (comment)

Since dir() (and tab completion, it works the same) are mainly for interactive use it was deemed more important to make them work well rather than efficiently. And by "work well" it means to list items from parent classes, handling multiple inheritance and without duplication.

[bablokb]

I don't agree, since dir() allows you to write very compact, dynamic code. I use it often in the context of configurations: with dir I can check if e.g. a user-level module overrides settings in a say global default module. For single attributes, I tend to use hasattr() or getattr(), for multiple attributes dir() works better.

As @Neradoc points out, you have to take care and you should know what you are doing (or should I say diring).

If your code is time-critical, introspection is never a good idea.

[raquo]

Thanks for the Micropython context, I didn't think to look there.

I ran into this issue as a sort of a "reverse heisenbug" – I used dir() in a runtime print statement to debug an unrelated issue, and this added an ~80ms delay on samd21. Even though my code can tolerate some delays, that was too much, causing the code to break in a similar yet different way to what the original unrelated issue was. It's hard to debug issues caused by debugging methods.

The issue you mentioned with dir() evaluating properties is surprising (and good to know), but in practice doesn't directly affect my code, because I only used dir() on my own classes, where I don't use @property-s.

I do wish that this issue you mentioned, and the dir() performance issue, was documented on a page like this somewhere (not sure if there is a similar page for CircuitPython). Although, maybe that's not feasible because there are too many such differences to document, I dunno.

Up to you guys if this issue should be closed or what, from the linked micropython issue, it kinda sounds like this may be very hard to fix, if it's even possible. For myself, I'll have to avoid using dir() at runtime.