XQEMU is hard to run

[JayFoxRox]

Currently setting up XQEMU is a lengthy procedure, running it is complicated.

We should have:

• Open-Source replacement for the ridiculously hard to dump MCPX rom
• Open-Source replacement or patcher for the bios
• A legal Xbox homebrew tool to dump:
  • EEPROM
  • Harddisk
  • Kernel + Kernel parameters
  • Flash
  • Game discs
• Cross-platform script / tool to run XQEMU or custom GUI

I've worked on some of these tasks in the past. Please contact me if you want to help.
Feel free to split this into seperate issues too.

[mborgerson]

For anyone wanting to tackle the MCPX part this may be helpful... https://mborgerson.com/deconstructing-the-xbox-boot-rom

[LavenderMoon]

Would it be possible to use a custom dashboard by default in place of the official Xbox dashboard, without hampering compatibility (assuming such a dashboard were to be completed, of course)? If so, I'd definitely look into writing one without using the XDK.

[JayFoxRox]

You don't need a dashboard to run games. The kernel will either start the game OR dashboard (which is just a "pseudo-game" stored on the harddisk, ran if the DVD was not detected as a game). = If the game loads successfully you don't need a dashboard.

This issue is about the kernel and the bios (MCPX rom + 2bl).
Also see https://www.reddit.com/r/emulation/comments/6a958p/cxbx_reloaded_xbox_emulator_panzer_dragoon_orta/dherxa5/ and the surrounding discussion.

Eitherway: You'll need an Xbox to legally run xqemu (unless we find a way to get a usable kernel elsewhere). We can just try to make the dumping procedure as easy as possible.

This Xbox-requirement is also not an issue in my opinion; Citra has a similar requirement and it still grew. Currently we care about finding developers, and we have more cheap physical / dumpable Xboxes than developers (meaning devs will have / can get one).
We might find a way around this Xbox requirement for users later.
For now it's more important to legally preserve and document the hardware (which is a developer task / if some users benefit from it that is cool, but not necessary). Playable emulation will follow.

---

Aside from what @mborgerson linked:

• http://xboxdevwiki.net/BIOS
• http://xboxdevwiki.net/MCPX_ROM

Also there is my 2014 boot HLE in XQEMU. That should be split into a seperate bios file / project as QEMU is not designed to run guest code within it's source code ;) )

My approach back then was also too flexible. That is probably fine, but it's currently not necessary (and it adds a lot of points of failure + complexity to an otherwise small issue).
For now, we just need a standalone (= not in XQEMU) cromwell-like Xbox MCPX ROM + bios.

I also took shortcuts by disassembling the original files and just porting to C - the following code is probably fine legally, but still rather questionable. I also moved some values around from the MS disassembly and configured most things using header files (hence some files [*.fox] are still local).

• https://github.com/JayFoxRox/xqemu-jfr/blob/xbox_bootrom_hle/hw/xbox/mcpx_rom.c
• https://github.com/JayFoxRox/xqemu-jfr/blob/xbox_bootrom_hle/hw/xbox/bootloader.c

[JayFoxRox]

I looked into porting cromwell for XQEMU use (= load xboxkrnl.exe) now (Also see my updates to http://xboxdevwiki.net/Boot_Process#2BL).
I ran into a handful of issues, most of which I was able to resolve. However, I'm ultimately stuck now.

• I've created emuwell, based on cromwell, which is supposed to load xboxkrnl.exe into memory
• The first issue was: cromwell was hardcoded for 256 kiB flashes, however, xboxkrnl.exe is roughly 640 kiB for me and gzip compression + the high level cromwell code always made it larger than 256 kiB. emuwell now only supports 1 MiB images.
• Next I used a trimmed xboxkrnl.exe by accident which took me more than a day to realize..
• End-result: emuwell can load and boot a "clean" (= decrypted and decompressed) xboxkrnl.exe (!)

However, this is where the problems started: We can not dump a "clean" xboxkrnl.exe:

• Once loaded into memory, the .data section will be used. This is not too bad as we can just reload the .data section from flash (it keeps a duplicate there for quick-reboots)
• (Now the problem:) Once the boot animation after a cold-boot has finished, the INIT section is discarded (!) by freeing the pages. This means that homebrew can not dump the INIT section which contains the boot animation and kernel initialization. INIT, however, is critical to the boot process - it sets up the kernel data structures and plays the boot animation. It also contains the entry point to the kernel.

I've tried to find ways around this now:

• INIT is discarded after loading the XBE, as the very last thing before relocating the LaunchDataPage and the avSetSavedData; after those relocations the XBE is launched. As we are coming from a cold-boot, neither of those should be set, so they are essentially NOPs. Right after discarding the INIT, the XBE will be launched, so theoretically the pages for the INIT section should still contain the valid INIT data. Problem: If we don't cold-boot the pages for INIT will probably have been re-used. So by cold-booting a dumper XBE we could theoretically dump INIT. However: softmods don't ever cold-boot into a homebrew XBE. They boot into the dashboard (or games) which are then exploited. So this is likely a dead-end (for now)
• The routines to quick-reboot are not exported from the kernel. Manually rewriting a Kernel main() / reboot function to initialize the kernel without INIT sounds too much work
• We could probably create some kind of save-state dumper XBE which doesn't care about the INIT section or much else. Instead of running the MS INIT code, it would have to load all hardware registers manually (not necessarily in the right order etc.)

I've also checked if we could ditch the kernel dumping process altogether:

• The key used to decrypt the kernel is overwritten with zeros in the Kernel main(). So we can't just decrypt the kernel from flash either.

(What frustrates me most about the INIT situation, is that this seems unintentional by Microsoft. They probably wanted to save on RAM after bootup, however, for us this means lots of headaches)

[ripdajacker]

EEPROM dumping;
You can dump an EEPROMfrom an Xbox using your VGA/DVI/HDMI port's built-in i2c bus. I have done it with the standard i2c tools. in linux. I can write a guide for this if this can help the project.

Harddrive dumping:
Once you have the EEPROM you can dump the drive using the smartctl tools in the modern XboxHDM USB version. It would be very awesome if the Xbox-specific smartctl tools were easily compiled with a modern distro (I tried compiling the source from XboxHDM on an ubuntu 16.04 with little success).

[JayFoxRox]

@ripdajacker The issue of dumping has been solved for almost everything (including pure software tools = no need to open the Xbox or buy active components).
If you have working dumping methods, write a tutorial and link it from http://xboxdevwiki.net/ (put it in the article of the device you dump), if it's not already there.

What's missing, is MCPX / MCPX Key / 2BL / Kernel dumping. So we should focus our discussion on those. These are the parts which make XQEMU hard to run and have no known solution.