Add single thread performance statistics for typechecking

[davidhalter]

In my experience it would be very interesting to have single threaded performance as well to see how

1. Type checkers perform on machines with fewer threads
2. How much Watt/type check they use to understand if a notebook will get warmer
3. How well a type checker scales with threads

https://python-type-checking.com/typecheck_benchmark/

For example, Zuban will most likely have a much better performance single core than Pyrefly/Ty, while on the other hand they scale better with threads (not sure how exactly though).

Alternatively it would also be an interesting metric to just have the used resources, e.g. Zuban will use significantly less CPU than Pyrefly/Ty at similar speeds. This is very relevant on notebooks for example, because they tend to get hot and slow their CPUs if you rerun type checking a few times.

[lolpack]

@davidhalter thanks for the feedback and feature idea. First off, well done on the memory design for Zuban! It's impressive to see how memory efficient it is. Second, thanks for your contributions to the community with Jedi - it shouldered the Python LSP workload for many many years :)

I'm open to changes to the benchmark or additional metrics/measurement approaches. Perhaps it sounds like a new page/benchmark called notebook_benchmark would be useful? I honestly don't know a ton about typical notebook environment setups, system requirements or type checking needs for that use case. If you want to propose and build a feature that show cases this use case along with the supporting methodology for why its relevant/high used, I am open to including it.

[davidhalter]

@lolpack Thanks! I don't mean notebooks as in "Jupyter notebooks", I mean it as in laptops/computers :)

I think the major issues notebooks have in comparison to servers is that they tend to get hot and slow everything massively. Just from anecdotal experience (see below) I can tell you that Pyrefly would run about 3 times slower when the computer got hot.

I think the most interesting graphic would not be "single-threaded" vs. "multi-threaded" (because who runs single threaded at this point), but accumulated CPU usage.

For example consider Pyrefly when my notebook is "cold":

$ /usr/bin/time -v pyrefly check mypy/
[...]
Command exited with non-zero status 1
	Command being timed: "pyrefly check mypy/"
	User time (seconds): 3.39
	System time (seconds): 0.19
	Percent of CPU this job got: 513%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.69

This is when the notebook gets hot:

Command exited with non-zero status 1
	Command being timed: "pyrefly check mypy/"
	User time (seconds): 9.54
	System time (seconds): 0.53
	Percent of CPU this job got: 550%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:01.83

This is just one example of why it is important to try to avoid using excessive CPU. Another one would be that if Pyrefly was able to the same work with 200% CPU, the processor wouldn't be as saturated and could be used for something else.

For example I'm sure that I could make Zuban slightly faster by using way more CPU and doing some calculations multiple times, but I want to avoid that as much as possible and I would like to see that tracked somewhere, ideally not just for Zuban. These regressions could be very annoying for users, especially in big projects.

In the above example if Pyrefly used 800% CPU instead of 513%, I would consider that a massive regression, because that would mean that the process uses much more power. In the world of Docker and virtualization there is also always a CPU that can use the CPU in the server space.

Is it more clear now? I'm really talking about a graph of "CPU time (user + system)" where "less used" is better.

[lolpack]

@davidhalter, thanks for the clarification on "notebooks" :). I see where you are coming from for measuring CPU time. I am open to adding the metric to the benchmark if there is a clear way to explain it to the user. I personally have not experienced throttling on my laptop, but I have heard user reports for the new checkers using a lot of CPU. Many of those I believed were bugs as the CPU spikes should last for a few seconds at most. However I think there is good reason to track CPU usage generally.

The good news is that on linux and mac (which I just discontinued from the GH runner along with Windows due to cost reasons), we can use the usr/bin/time command that is already run to parse out CPU (user + system). I will not be able to get around to this for a little while, but am happy to review a PR with the change.