SError when launching cedr on FPGA

[SwayYe]

Hello,
I am trying to run cedr on my zcu104 board. I followed the vivado and petalinux intrustions in your zcu102-2020.2 repository and adapted the vivado and petalinux setup to zcu104. Afterwards, I followed the tutorials to launch cedr on zcu104, but the terminal prompts some issues w.r.t. sem and shm:

~/CEDR/build-arm# ./cedr -c ./daemon_config.json -l NONE
sem_unlink failed in server initialization: No such file or directory
shm_unlink in server initializtion: No such file or directory

followed with crushes of the whole system:

[ 2412.278582] SError Interrupt on CPU0, code 0xbf000000 -- SError
[ 2412.278585] CPU: 0 PID: 769 Comm: cedr Tainted: G           O      5.4.0-xilinx-v2020.2 #1
[ 2412.278586] Hardware name: xlnx,zynqmp (DT)
[ 2412.278588] pstate: 60000000 (nZCv daif -PAN -UAO)
[ 2412.278589] pc : 0000ffffa66e5474
[ 2412.278590] lr : 0000ffffa66eaffc
[ 2412.278591] sp : 0000fffff0704000
[ 2412.278592] x29: 0000fffff0704000 x28: 0000000000000004
[ 2412.278595] x27: 0000ffffa6708000 x26: 0000aaaadc6463e0
[ 2412.278598] x25: 0000000000000000 x24: 0000000000000000
[ 2412.278601] x23: 0000fffff0707188 x22: 0000fffff0707158
[ 2412.278604] x21: 0000000000000005 x20: 0000ffffa3a5cd88
[ 2412.278607] x19: 0000000000028018 x18: 0000000000000000
[ 2412.278610] x17: 0000ffffa66eafb8 x16: 0000ffffa3a5cff8
[ 2412.278612] x15: 0000000000000080 x14: 0000000000000000
[ 2412.278615] x13: 0000000000000000 x12: 0000ffffa3a35750
[ 2412.278618] x11: 0000000000000000 x10: 0000000000000000
[ 2412.278620] x9 : 0000000000028308 x8 : 0000000000000039
[ 2412.278623] x7 : 0000ffffa66d8728 x6 : 0000ffffa66fdf10
[ 2412.278626] x5 : 0000ffffa3a5ce50 x4 : 0000ffffa3a35750
[ 2412.278629] x3 : 0000ffffa3a37f90 x2 : 0000ffffa3a37f48
[ 2412.278631] x1 : 0000000000000048 x0 : 0000aaaadc6460f0
[ 2412.278635] Kernel panic - not syncing: Asynchronous SError Interrupt
[ 2412.278637] CPU: 0 PID: 769 Comm: cedr Tainted: G           O      5.4.0-xilinx-v2020.2 #1
[ 2412.278638] Hardware name: xlnx,zynqmp (DT)
[ 2412.278639] Call trace:
[ 2412.278641]  dump_backtrace+0x0/0x140
[ 2412.278642]  show_stack+0x14/0x20
[ 2412.278643]  dump_stack+0xac/0xd0
[ 2412.278644]  panic+0x140/0x30c
[ 2412.278645]  __stack_chk_fail+0x0/0x18
[ 2412.278646]  arm64_serror_panic+0x74/0x80
[ 2412.278648]  do_serror+0x114/0x118
[ 2412.278649]  el0_error_naked+0x14/0x1c
[ 2412.278662] SMP: stopping secondary CPUs
[ 2412.278663] Kernel Offset: disabled
[ 2412.278664] CPU features: 0x0002,20002004
[ 2412.278665] Memory Limit: none

I checked the code, and found the prompts come from src-api/ipc.cpp. Is this issue caused by board compatibility problem, or caused by lack of certain package/setup during petalinux build? How can I solve this problem? Thanks.

[sahilhassanzee]

Hello,
Thanks for bringing up this issue. Given we don't have access to ZCU104, it may take a few iterations of trial and error before we could fix the issue. I can share a few insights for you to try.

By my best guess, SError can possibly come from an access attempt to an invalid memory address, especially on the PL side.
Please refer to the libdash/platform.h file, where first couple lines are defining different hardware platform settings (different board images, each with different accelerators on PL side, and different addresses). Here, the value of DASH_PLATFORM variable has to be set to the specific board that CEDR will be run on, which ensures CEDR has correct PL side device addresses to access.

For the image you used, please make sure it's set as #define DASH_PLATFORM DASH_ZCU102_2020_2 in the libdash/platform.h file- before compiling CEDR.
If despite that the problem isn't solved, I'd recommend matching the PL side IP addresses from the ZCU104 device tree, with the defined addresses for the DASH_ZCU102_2020_2 platform here.

Hope this helps. Please feel free to post follow ups.