propolis-standalone crashes on rep ins

[stsp]

Hi!

I tried to boot OVMF with propolis-standalone,
and got this:

ERRO Unhandled exit @rip:00859f60 InstEmul(InstEmul { inst_data: [243, 108, 235, 10, 227, 8, 236, 136, 7, 72, 255, 199, 226, 248, 76], len: 15 })

thread 'vcpu-0' panicked at bin/propolis-standalone/src/main.rs:729:25:
not yet implemented

After a LOT of debugging, I've traced
the problem to this code:
https://github.com/tianocore/edk2/blob/master/MdePkg/Library/BaseIoLibIntrinsic/X64/IoFifoSev.nasm#L87

[iximeow]

hum, that's odd! are you perhaps running propolis-standalone itself in an illumos VM? the exit (and that bit of edk2) don't tell what port was being accessed, but in this nested case I ran across a combination Linux/illumos pair of issues and worked around it with this patch: 7bb3e12 . this is not integrated because ... Linux probably should offer DecodeAssist to nested guests, which would have this handled by byhve, but I haven't tested nor checked the APM enough to be confident in offering a patch. and illumos should tolerate decoding port I/O without DecodeAssists (https://www.illumos.org/issues/17543) but the nested case isn't top of mind, so I haven't gotten to either of those.

if you're not in a propolis-on-illumos-on-linux situation: lots of questions, sorry :) what's the OVMF image you're booting, what's the CPU (and CPUID bits) you're working with, and is there anything interesting in the VM's config? I assume your debugging was probably going from the fault address of 859f60 to looking up where in the OVMF.fd you were, or matching up the instruction text in the output with somewhere in the file, and working back to source from there? in either case, you probably don't know which port was being read from?

[stsp]

hum, that's odd! are you perhaps running propolis-standalone itself in an illumos VM? the exit (and that bit of edk2)
don't tell what port was being accessed

That was an fwcfg port, reading of
the "QEMU" signature.

Yes, it IS a nested case.

With your patch I've got much further:

 Jul 10 08:36:19.107 ERRO Unhandled pio Out(IoPort { port: af00, bytes: 4 }, 0), rip: 8568659
 Jul 10 08:36:19.107 ERRO Unhandled pio Out(IoPort { port: af00, bytes: 4 }, 0), rip: 8568660
 Jul 10 08:36:19.107 ERRO Unhandled pio Out(IoPort { port: af05, bytes: 1 }, 0), rip: 8568665
 Jul 10 08:36:19.108 ERRO Unhandled pio In(IoPort { port: af00, bytes: 4 }), rip: 8568668
 Jul 10 08:36:19.123 ERRO Unhandled mmio Read(MmioReadReq { addr: fed40000, bytes: 1 }), rip: 536626571
 Jul 10 08:36:19.123 ERRO Unhandled mmio Read(MmioReadReq { addr: fed40000, bytes: 1 }), rip: 536626668

But it nevertheless booted to UEFI
shell, so I suppose it works.

I think you also need to enlarge the
size of bootrom to 4 or 8Mb. Otherwise
its very difficult to use modern builds of
edk2. I am using the build from this article:
https://artemis.sh/2022/03/14/propolis-oxide-at-home-pt1.html
but my own builds are much larger than that.

[stsp]

So I simply did this:

index cada3a60..845954ea 100644
--- a/bin/propolis-server/src/lib/initializer.rs
+++ b/bin/propolis-server/src/lib/initializer.rs
@@ -125,7 +125,7 @@ pub enum MachineInitError {
 }
 
 /// Arbitrary ROM limit for now
-const MAX_ROM_SIZE: usize = 0x20_0000;
+const MAX_ROM_SIZE: usize = 0x80_0000;
 
 /// End address of the 32-bit PCI MMIO window.
 ///
diff --git a/bin/propolis-standalone/src/main.rs b/bin/propolis-standalone/src/main.rs
index 3460bc77..6a9877d1 100644
--- a/bin/propolis-standalone/src/main.rs
+++ b/bin/propolis-standalone/src/main.rs
@@ -42,7 +42,7 @@ mod snapshot;
 
 const PAGE_OFFSET: u64 = 0xfff;
 // Arbitrary ROM limit for now
-const MAX_ROM_SIZE: usize = 0x20_0000;
+const MAX_ROM_SIZE: usize = 0x80_0000;
 
 /// End address of the 32-bit PCI MMIO window.

[iximeow]

going by

 Jul 10 08:36:19.107 ERRO Unhandled pio Out(IoPort { port: af00, bytes: 4 }, 0), rip: 8568659
 Jul 10 08:36:19.107 ERRO Unhandled pio Out(IoPort { port: af00, bytes: 4 }, 0), rip: 8568660
 Jul 10 08:36:19.107 ERRO Unhandled pio Out(IoPort { port: af05, bytes: 1 }, 0), rip: 8568665
 Jul 10 08:36:19.108 ERRO Unhandled pio In(IoPort { port: af00, bytes: 4 }), rip: 8568668
 Jul 10 08:36:19.123 ERRO Unhandled mmio Read(MmioReadReq { addr: fed40000, bytes: 1 }), rip: 536626571
 Jul 10 08:36:19.123 ERRO Unhandled mmio Read(MmioReadReq { addr: fed40000, bytes: 1 }), rip: 536626668

the port I/O is MaxCpuCountInitialization and the MMIO is for HPET. the first four are "normal": it's been a long while since I've looked at this, I don't recall why EDK2 is trying to do any of that when we don't even try to support CPU hotplug, but in all that happens on all boots for the time being.

the HPET bits are curious, I'm not sure why EDK2(?) would be trying to read that (as you see Propolis just doesn't implement it right now). does that possibly come from booting your build of EDK2, rather than the OVMF_CODE.fd out of oxide-omicron-build.s3.amazonaws.com? until very recently I'd expect newer EDK2 builds to outright not work, I can imagine unfortunate interactions between EDK2 changes in the last few years and what Propolis happens to present to guests today (also why the MAX_ROM_SIZE you see is a bit too small!)

[stsp]

does that possibly come from booting your build of EDK2, rather than the OVMF_CODE.fd out of
oxide-omicron-build.s3.amazonaws.com?

No, I checked and double-checked
now again, that this is exactly from
your OVMF_CODE.fd. The mistake on
my side is very unlikely, simply because
my own build does not boot to UEFI
shell. It dies earlier: just as you said,
newer EDKs do not work here. While
after exactly the above errors, I am
getting to UEFI shell.

also why the MAX_ROM_SIZE you see is a bit too small!

Why not to change that size then? :)
Granted the one can't boot fully, but
at least that helped me to use serial
debugging and to get anywhere at all.

Note that latest EDK2 does not even build.
I did this patch:
tianocore/edk2#12782
And the above amazon link to OVMF_CODE.fd
is difficult to find... Or am I missing some
README?