Disable parallelism for profiling

[kxygk]

Maybe this is a misguided question..

But I've profiling with clj-async-profiler and I have some sections with TMD code called from within a complex Pathom resolver setup. My problem is that.. as far as I understand the callstacks, TMD seems to triggers execution on separate threads. My whole app is spending 2/3rds of the wall time parked and I'm trying to narrow down the cause.

In terms of profiling the issue is I have huge stacks on top of ForJoinTask - and hence I have no idea who triggered these calls :))

I also don't want to disable parallelism globally (b/c I think it's necessary for the Pathom setup I have - where it uses Promesa VThreads).

I think TMD doesn't use VThreads? so I'm implicitly using two threading models here which is probably not good. While TMD parallelism is a sensible default, my very-poorly-informed impression is that if you process multiple table on multiple threads you're gunna end up with a soup of threads.

I'll be honest, I'm a little out of my depths here. It's likely I'm "holding it wrong" or reading the stack-trace tea leaves wrong.

I'm guessing you guys profile TMD all the time so there must be some best-practices :)

EDIT:

To my best understanding, ds/row-map and ds/row-mapcat will under the hood call pmap-ds which calls indexed-map-reduce (in dtype-next/for.clj) . The docstring for indexed-map-reduce reads

" [...] If the current thread is already in the common pool, this function executes in the current thread"

At the top of indexed-map-reduce there is a check

     (if (or (< num-iters (* 2 parallelism))
             (ForkJoinTask/inForkJoinPool))
       (reduce-fn [(indexed-map-fn 0 num-iters)])
       ;; otherwise parallelize things

I don't really know enough to understand inForkJoinPool, but it's part of the Java API.
My best guess is that if you you use Promesa's vthread then this check fails.

There is no serial row-map so I think the only solution for the time being is to Promesa's thread (instead of vthread virtual threads) in Pathom instead. I think that would use the same thread pool and stops TMD from threading.

[harold]

My whole app is spending 2/3rds of the wall time parked and I'm trying to narrow down the cause.

Are you doing I/O inside of row-map?

[kxygk]

Pretty confident there is no IO.

The code is here if you're curious
https://github.com/kxygk/anemoi/blob/master/kxygk/anemoi/airport.clj
To be clear.. It's not imperative for me to get it running fast. + It's wrapped in Pathom resolvers so a bit hard to reason about, unless you use them regularly.. I was just playing around with clj-async-profiler and hit a weird situation.

My question was more broadly, concerning debuggability. I'm just wondering how you guys handle these situations. There is no flag to turn off auto-parallelization so the stacks just have dangling threads and it's unclear where they come from.

I have small suspicion the system clashes with VThreads in a way that's not outside the control of the user - but I could be wrong here. It's not super typical for libraries to do parallelization for the user unless specified. (b/c parallelization is usually better done higher in the stack)

EDIT: Reprofiling without VThreads (using normal threads instead) doesn't change the behavior. I still get huge stacks without any of my code :)). It's possible it's a Pathom issue.. I see Pathom code calling TMD code directly. I don't understand how it's possible. I must be missing something here

[kxygk]

I'm a bit slammed right now, but when I have some time I could try rerunning with Promesa threads and maybe if the issue persists I could make a minimal demo

EDIT: Changing to Promesa threads didn't fix the issue (see previous comment edits)

[harold]

row-map has not been a problem for debuggability, or performance testing for us.

It's not super typical for libraries to do parallelization for the user unless specified.

This is the precise use case of the JVM's commonPool that we use to do this parallelization. It's a mechanism in the JVM that all libraries can use to coordinate the use of extra hardware threads within the application for CPU intensive work.

I don't understand how it's possible. I must be missing something here

Perhaps that is the piece you're missing?

[kxygk]

Okay, if the comonPool is supposed to handle over-parallelizing then I need to debug further. Thanks for getting back to me :))

row-map has not been a problem for debuggability

Workflow-wise, how do you trace these call stacks though? .. since you've lost the original call site once its in the pool

[harold]

Not sure what you mean, like, you don't know where you called row-map?

[kxygk]

Right - your code-base has many row-map calls and in your performance profile you just see these bare threads - so you have no immediate indication where the hot-path is.

I'm just curious how you handle this.

You can of course remove row-map calls in your codebase or something like that and reprofile, but it seems a bit goofy. Maybe named lambda would work...? I will have to try.

My original issue was more a question of workflow and how you guys are profiling code. Disabling parallelism is the obvious solution, but that's not possible in the current setup.

[harold]

We profile with three tools mainly, clj-async-profilier, criterium, asprof+jfrconv. All of these work great with row-map and commonPool. In particular the -t option to asprof can be helpful to separate out which work is happening on which thread, when that matters.

If there are big stalls, a common cause is doing I/O or other blocking work on the CPU thread, which is why I asked you that first.

Speaking generally, 'disabling parallelism' doesn't sound right to me, good performance in these cases depends on doing things in parallel.

In extreme cases, such as large codebases we didn't write, we've used tricks like (with-out-str (.printStackTrace (Exception. "Native Java Trace"))) to figure out especially tricky codepaths, but that's rare.