Parallel processing fails

[evelinag]

I tried calling R from a parallel loop but it fails with Error: unprotect_ptr: pointer not found, the sequential version works without problems.

#r "packages/R.NET.Community/lib/net40/RDotNet.dll"
#r "packages/RProvider/lib/net40/RProvider.dll"
#r "packages/RProvider/lib/net40/RProvider.Runtime.dll"
open RProvider
open RProvider.stats
open RDotNet

// this works
[| 0..10 |]
|> Array.map (fun _ -> R.rnorm(10) )

// this fails
[| 0..10 |]
|> Array.Parallel.map (fun _ -> R.rnorm(10) )

[mae1st0rm]

Hi! I have the similar question: Is it possible to call RProvider functions in parallel? For example cal R.lm from different Threads/Tasks (PLINQ) simultaneously?

[tpetricek]

As far as I understand it, the underlying R engine can only be called from a single thread - so only a single thread can access it (and additionally, I think it also starts doing odd things when you call it from multiple threads, even when the calls are not concurrent).

In principle, I suspect that you can create multiple instances of the R.NET engine (@jmp75 would probably know), but this is not something that we are currently exposing in the R type provider (you cannot really currently initialize the R provider multiple times in a single process).

[AndrewIOM]

In the latest RProvider, the R engine runs in a single thread and is protected by a concurrent queue. This means that the parallel example above now runs without error. However, there could be contamination if shared state was used rather than a pure function as above. I've added a doc section for this: https://fslab.org/RProvider/expressions.html#Parallel-processing (part of #274 )

Closing as fixed.

#r "nuget: RProvider, 3.0.0"

open RProvider
open RProvider.stats

fsi.AddPrinter FSIPrinters.rValue

// this works
[| 0..10 |]
|> Array.map (fun _ -> R.rnorm(10) )

// this fails
[| 0..10 |]
|> Array.Parallel.map (fun _ -> R.rnorm(10) )

- // this works
- [| 0..10 |]
- |> Array.map (fun _ -> R.rnorm(10) )
- 
- ;;
val it: Abstractions.RExpr array =
  [|R semantic type <VectorType>

 [1]  1.4664090 -1.1928037  0.9373182 -1.4246276  0.1482254 -1.7768601
 [7] -1.8422563 -0.2156154  0.2309556 -0.4809747
;
    R semantic type <VectorType>

 [1] -0.07538752 -1.43483118  2.45089406  0.17223217  0.92043464 -0.20423681
 [7] -1.83782121  0.19536819 -1.80602446 -0.33764050
;
    R semantic type <VectorType>

 [1] -0.80089706  0.11380365 -0.97569420  1.45494553  0.20580912 -0.39291480
 [7]  1.45490642  0.03909807  0.62363923  0.56880720
;
    R semantic type <VectorType>

 [1] -0.51758893  1.14392196 -0.84797731 -0.08872545  2.48366224  1.05562365
 [7]  1.89353076 -0.32240527  0.69218519  0.59425063
;
    R semantic type <VectorType>

 [1] -1.82358199  0.33931304  1.10900782  1.32197938  0.05184075 -0.44528333
 [7] -0.72143762 -0.75861572  0.31894919 -0.19531565
;
    R semantic type <VectorType>

 [1]  3.00560590  0.84617807 -0.55241192 -0.03993405  0.30301718 -0.70964803
 [7] -1.20963216 -1.06976979 -1.93659382 -0.09173963
;
    R semantic type <VectorType>

 [1]  1.4622504 -0.5637009  0.4573032 -0.6885266  1.2912153  1.2140649
 [7]  0.1169715 -1.2814253  2.4764366 -0.8319336
;
    R semantic type <VectorType>

 [1] -1.6136560 -0.3965308 -0.2501449 -0.7073212 -0.5637366 -0.7660133
 [7]  0.1166676 -1.6804682 -0.5562971  0.2720669
;
    R semantic type <VectorType>

 [1]  0.58544595 -0.05955553 -0.44362748  1.78430051 -0.24246089 -0.39537149
 [7] -0.98709235 -1.15498701  0.48017433 -1.52747001
;
    R semantic type <VectorType>

 [1] -0.8169462 -0.1726986  1.2764338 -1.4211619  1.4113279 -0.6946599
 [7] -0.8976584 -0.2285615  0.9029203  0.5764646
;
    R semantic type <VectorType>

 [1]  0.5228729  1.2830871 -0.3807132  1.3740516  1.9683175 -0.7734014
 [7]  0.2142087 -0.2085771  0.6818469 -1.5679235
|]

> # 11 @"/Users/andrewmartin/Documents/GitHub Projects/RProvider/test.fsx"
- 
- // this fails
- [| 0..10 |]
- |> Array.Parallel.map (fun _ -> R.rnorm(10) )
- ;;
val it: Abstractions.RExpr array =
  [|R semantic type <VectorType>

 [1]  0.2771659  0.5794708 -1.1228358 -0.3739182  0.5535028 -0.2200564
 [7] -0.2445971  0.4502943  1.9809437 -1.1207714
;
    R semantic type <VectorType>

 [1]  0.2413230 -0.3114433  0.7176417 -0.5940656  0.9360953  0.3780308
 [7] -1.1713841  0.8546252 -0.1125551 -0.5902735
;
    R semantic type <VectorType>

 [1] -0.02497769 -0.57347560  0.98609640 -1.83566359 -2.09083559  1.33919560
 [7]  1.01243524 -2.34776825  1.48284965  1.39762811
;
    R semantic type <VectorType>

 [1] -0.03030485 -0.25567208  0.51854376 -0.27145648 -0.65031177  1.01174884
 [7]  0.53411024 -0.48475751 -0.94255861 -0.14596627
;
    R semantic type <VectorType>

 [1] -0.81609315 -0.58205503  0.57057747 -0.86149431 -1.22091277  0.07906066
 [7]  0.85345717 -1.32040658  0.21016734 -0.16759657
;
    R semantic type <VectorType>

 [1]  1.0227249  0.2122330  0.3172442  0.8866817  0.2638763 -0.3437931
 [7]  0.3144889 -0.4032902  0.2039647 -0.3438822
;
    R semantic type <VectorType>

 [1] -0.1401039  0.7436720  0.4362706 -1.4165923  1.7509426 -1.6200853
 [7]  0.4896085 -1.3861730  0.3483562  0.3454653
;
    R semantic type <VectorType>

 [1]  1.91253813  0.52877380  0.59389843  0.11633353  0.04181386 -0.89689920
 [7] -0.62217703 -0.86864581  0.44423882  0.56214917
;
    R semantic type <VectorType>

 [1]  0.69909172 -1.00086416 -0.09370275  0.33646114  1.24339935  0.42278147
 [7] -1.39095156 -0.23609874  0.08623617  0.54413063
;
    R semantic type <VectorType>

 [1] -0.22415228 -0.59413736 -0.23205490 -0.29544529 -0.49144120  0.13270147
 [7] -0.60590006 -1.33979886  0.09152349  1.62785890
;
    R semantic type <VectorType>

 [1] -1.42343459 -0.48727129 -0.34152586  1.13924767 -0.03530468 -0.26094098
 [7]  0.66339485 -2.39610094 -0.26880832  0.69468891
|]