Over the last two weeks I have implemented most of the high-level emulation framework that I alluded to in my last post here as well as a large number of tracing wrappers for the original ROM calls. In the next stage I will start replacing some of those wrappers with re-implementations, starting with some easy ones.
It turns out I was somewhat optimistic; so far I have wrapped over 450 distinct ROM entry points (the actual current number of wrappers is 513 but there are some error catchers and possible duplicates). Creating the wrappers and writing and debugging the framework took more effort then I expected, but it was worth it: every call to a ROM entry point described or implied by the Green Book or OS-9 documentation is now wrapped with a high-level emulation function that so far does nothing except calling the original ROM routine and tracing its input/output register values.
Surely there aren't that many application-callable API functions, I can hear you think? Well actually there are, for sufficiently loose definitions of "application-callable". You see, the Green Book specifies CD-RTOS as being OS-9 and every "trick" normally allowed under OS-9 is theoretically legal in a CD-i title. That includes bypassing the OS-supplied file managers and directly calling device drivers; there are many CD-i titles that do some of this (the driver interfaces are specified by the Green Book). In particular, all titles using the Balboa library do this.
I wanted an emulation framework that could handle this so my framework is built around the idea of replacing the OS-9 module internals but retaining their interfaces, including all the documented (and possibly some undocumented) data structures. One of the nice features of this approach is that native ROM code can be replaced by high-level emulation on a routine-by-routine basis.
How does it really work? As a start, I've enhanced the 68000 emulation to possibly invoke emulation modules whenever an emulated instruction generates one of the following processor exceptions: trap, illegal instruction, line-A, line-F.
The emulation modules can operate in two modes: either copy an existing ROM module and wrap its entry points, or generate an entirely new memory module. In both cases, the emulation module will emit line-A instructions at the appropriate points. The emitted modules will go into a memory area appropriately called "emurom" that the OS-9 kernel scans for modules. Giving the emitted modules identical names but higher revision numbers then the ROM modules will cause the OS-9 kernel to use the emitted modules.
This approach works for every module except the kernel itself, because it is entered by the boot code before the memory scan for modules is even performed. The kernel emulation module will actually patch the ROM kernel entry point so that it jumps to the emitted kernel module.
The emitted line-A instructions are recognized by the emulator disassembler; they are called "modcall" instructions (module call). Each such instruction corresponds to a single emulation function; entry points into the function (described below) are indicated by the word immediately following it in memory. For example, the ROM routine that handles the F$CRC system call now disassembles like this:
modcall kernel:CRC:0
jsr XXX.l
modcall kernel:CRC:$
rts
Here the XXX is the absolute address of the original ROM routine for this system call; the two modcall instructions trace the input and output registers of this handler. If the system call were purely emulated (no fallback to the original ROM routine) it would look like this:
modcall kernel:CRC:0
modcall kernel:CRC:$
rts
Both modcall instructions remain, although technically the latter is now unnecessary, but the jsr instruction has disappeared. Technically, the rts instruction could also be eliminated but it looks more comprehensible this way.
One could view the approach as adding a very powerful "OS-9 coprocessor" to the system.
If an emulation function has to make inter-module calls, complications arise. High-level emulation context cannot cross module boundaries, because the called module may be native (and in many cases even intra-module calls can raise this issue). For this reason, emulation functions need additional entry points where the emulation can resume after making such a call. The machine language would like this, e.g. for the F$Open system call:
modcall kernel:Open:0
modcall kernel:Open:25
modcall kernel:Open:83
modcall kernel:Open:145
modcall kernel:Open:$
rts
The numbers following the colon are relative line numbers in the emulation function. When the emulation function needs to make a native call, it pushes the address of one such modcall instruction on the native stack, sets the PC register to the address it wants to call and resumes instruction emulation. When the native routine returns, it will return to the modcall instruction which will re-enter the emulation function at the appropriate point.
One would expect that emulation functions making native calls need to be coded very strangely: a big switch statement on the entry code (relative line number), followed by the appropriate code. However, a little feature of the C and C++ languages allows the switch statement to be mostly hidden. The languages allow the case labels of a switch statement to be nested arbitrarily deep into the statements inside the switch.
The entire contents of emulation functions are encapsulated inside a switch statement on the entry number (hidden by macros):
switch (entrynumber)
{
case 0:
...
}
On the initial call, zero is passed for entrynumber so the function body starts executing normally. Where a native call needs to be made, the processor registers are set up (more on this below) and a macro is invoked:
MOD_CALL(address);
This macro expands to something like this:
MOD_PARAMS.SetJumpAddress(address);
MOD_PARAMS.SetReturnLine(__LINE__);
return eMOD_CALL;
case __LINE__:
Because this is a macro expansion, both invokations of the __LINE__ macro will expand to the line number of the MOD_CALL macro invokation.
What this does is to save the target address and return line inside MOD_PARAMS and then return from the emulation function with value eMOD_CALL. This value causes the wrapper code to push the address of the appropriate modcall instruction and jump to the specified address. When that modcall instruction executes after the native call returns, it passes the return line to the emulation function as the entry number which will dutifully switch on it and control will resume directly after the MOD_CALL macro.
In reality, the code uses not __LINE__ but __LINE__ - MOD_BASELINE which will use relative line numbers instead of absolute ones; MOD_BASELINE is a constant defined as the value of __LINE__ at the start of the emulation function.
The procedure described above has one serious drawback: emulation functions cannot have "active" local variables at the point where native calls are made (the compiler will generate errors complaining that variable initialisations are being skipped). However, the emulated processor registers are available as temporaries (properly saved and restored on entry and exit of the emulation function if necessary) which should be good enough. Macros are defined to make accessing these registers easy.
When native calls need to be made, the registers must be set up properly. This would lead to constant "register juggling" before and after each call, which is error-prone and tedious. To avoid it, it is possible to use two new sets of registers: the parameter set and the results set. Before a call, the parameter registers must be set up properly; the call will then use these register values as inputs and the outputs will be stored in the results registers (register juggling will be done by the wrapper code). The parameter registers are initially set to the values of the emulated processor registers and also set from the results registers after each call.
The following OS-9 modules are currently wrapped:
kernel nrf nvdrv cdfm cddrv ucm vddrv ptdrv kbdrv pipe scf scdrv
The *drv modules are device drivers; their names must be set to match the ones used in the current system ROM in order to properly override those. The *.brd files in the sys directory have been extended to include this information like this:
** Driver names for ROM emulation.
set cddrv.name=cdapdriv
set vddrv.name=video
set ptdrv.name=pointer
set kbdrv.name=kb1driv
The kernel emulation module avoids knowledge of system call handler addresses inside the kernel by trapping the first "system call" so that it can hook all the handler addresses in the system and user mode dispatch tables to their proper emulation stubs. This first system call is normally the I$Open call for the console device.
File manager and driver emulation routines hook all the entry points by simply emitting a new entry point table and putting the offset to it in the module header. The offsets in the new table point to the entry point stubs (the addresses of the original ROM routines are obtained from the original entry point table).
The above works fine for most modules, but there was a problem with the video driver because it is larger then 64KB (the offsets in the entry point are 16-bit values relative to the start of the module). Luckily there is a text area near the beginning of the original module (it is actually just after the original entry point table) that can be used for a "jump table" so all entry point offsets fit into 16 bits. After this it should have worked, but it didn't because it turns out that UCM has a bug that requires the entry point table to *also* be in the first 64KB of the module (it ignores the upper 16-bits of the 32-bit offset to this table in the module header). This was fixed by simply reusing the original entry point table in this case.
One further complication arose because UCM requires the initialisation routines of drivers to also store the absolute addresses of their entry points in UCM variables. These addresses were "hooked" by adding code to the initialisation emulation routine that changes these addresses to point to the appropriate modcall instructions.
All file managers and drivers contain further dispatching for the SetStat and GetStat routines, based on the contents of one or two registers. Different values in these registers will invoke entirely separate functions with different register conventions; they really must be redirected to different emulation functions. This is achieved by lifting the dispatching to the emulation wrapper code (it is all table-driven).
Most of the above has been implemented, and CD-i emulator now traces all calls to ROM routines (when emurom is being used). A simple call to get pointing device coordinates would previously trace as follows (when trap tracing was turned on with the "et trp" command):
@00DF87E4(cdi_app) TRAP[5812] #0 I$GetStt <= d0.w=7 d1.w=SS_PT d2.w=PT_Coord
@00DF87E8(cdi_app) TRAP[5812] #0 I$GetStt => d0.w=$8000 d1.l=$1EF00FD
Here the input value d0.w=7 is the path number of the pointing device; the resulting mouse coordinates are in d1.l and correspond to (253,495),
When modcall tracing is turned on, this "simple" call will trace as follows:
@00DF87E4(cdi_app) TRAP[5812] #0 I$GetStt <= d0.w=7 d1.w=SS_PT d2.w=PT_Coord
@00F86EE0(kernel) MODCALL[16383] kernel:GetStt:0 <= d0.w=7 d1.w=$59 [Sys]
@00F86D10(kernel) MODCALL[16384] kernel:CCtl:0 <= d0.l=2 [NoTrap]
@00F86D1A(kernel) MODCALL[16384] kernel:CCtl:$ =>
@00F8A460(ucm) MODCALL[16385] ucm:GetPointer:0 <= u_d0.w=7 u_d2.w=0
@00FA10A4(pointer) MODCALL[16386] pointer:PtCoord:0 <= d0.w=7
@00FA10AE(pointer) MODCALL[16386] pointer:PtCoord:$ => d0.w=$8000 d1.l=$1EF00FD
@00F8A46A(ucm) MODCALL[16385] ucm:GetPointer:$ =>
@00F86D10(kernel) MODCALL[16387] kernel:CCtl:0 <= d0.l=5 [NoTrap]
@00F86D1A(kernel) MODCALL[16387] kernel:CCtl:$ =>
@00F86EEA(kernel) MODCALL[16383] kernel:GetStt:$ =>
@00DF87E8(cdi_app) TRAP[5812] #0 I$GetStt => d0.w=$8000 d1.l=$1EF00FD
You can see that the kernel dispatches this system call to kernel:GetStt, the handler for the I$GetStt system call. It starts by doing some cache control and then calls the GetStat entry point of the ucm modules, which dispatches it to its GetPointer routine. This routine in turn calls the GetStat routine of the pointer driver, which dispatches it to its PtCoord routine. This final routine performs the actual work and returns the results, which are then ultimately returned by the system call, after another bit of cache control.
The calls to ucm:GetStat and pointer:GetStat are no longer visible in the above trace as the emulation wrapper code directly dispatches them to ucm:GetPointer and pointer:PtCoord, respectively; it doesn't even trace the dispatching because this would result in another four lines of tracing output.
As a sidenote, all of the meticulous cache and address space control done by the kernel is really wasted, as CD-i systems do not need these. But the calls are still being made, which makes the kernel needlessly slow; one major reason why calling device drivers directly is often done. Newer versions of OS-9 eliminate these calls by using different kernel flavors for different processors and hardware configurations.
The massive amount of tracing needs to be curtailed somewhat before further work can productively be done; this is what I will start with next.
I have already generated fully documented stub functions for the OS-9 kernel from the OS-9 technical documentation; I will also need to generate for all file manager and driver calls, based on the digital Green Book.
It is perhaps noteworthy that some kernel calls are not described in any of the OS-9 version 2.4 documentation that I was able to find, but they *are* described in the online OS-9/68000 version 3.0 documentation.
Some calls made by the native ROMs remain undocumented but those mostly seem to be CD-i system-control (for example, one of them sets the front display text). Of the OS-9 kernel calls, only the following ones are currently undocumented:
F$AllRAM
F$FModul
F$POSK
Their existence was inferred by the appropriate constants existing in the compiler library files, but I have not seen any calls to them (yet).
Monday, November 8, 2010
Wednesday, October 27, 2010
CD-i Emulator Cookbook
Just a quick note that work on CD-i Emulator hasn't stopped.
I have some wild ideas about ROM-less emulation; this would basically mean re-implementing the CD-RTOS operating system. Somewhat daunting; it contains over 350 separate explicit APIs and callable entry points and many system data structures would need to be closely emulated. But it can be done, CD-ice proved it (although it took a number of shortcuts that I want to avoid).
I'm not going to tackle that by myself; my current thinking is to make a start by implementing a high-level emulation framework, tracing stubs for all the calls (luckily these can mostly be generated automatically from the digital Green Book and OS-9 manuals) and some scaffolding and samples.
One of the pieces of scaffolding would be a really simple CD-i player shell; one that just shows a big "Play CD-i" button and then starts the CD-i title :-)
For samples I'm thinking about a few easy system calls like F$CRC, F$SetCRC, F$SetSys, F$CmpNam, F$PrsNam, F$ID, F$SUser, F$Icpt, F$SigMask, F$STrap, F$Trans, F$Move, F$SSvc (I may not get through the entire list) and a new NVRAM File Manager (NRF).
It would be nice to do a minimal UCM with Video and Pointer driver so that the simple CD-i player shell would run, but that might be too much. We'll see.
However, it's the new NRF that would be the most immediately interesting for CD-i Emulator users. It would intercept NVRAM access at the file level and redirect it to the PC file system (probably to files in the the nvr directory). This would allow easy sharing of CD-i NVRAM files (e.g. game-saves) over player types or between CD-i emulator users.
To allow all of the above and clean up some dirty tricks that were needed for input playback and handling Quizard, I've done some internal restructuring of CD-i Emulator. In particular, I introduced a new "handler" class beneath the existing "device" and "memory" classes (which are now no longer derived from each other but from a common "component" base class). This restructuring isn't finished yet, but it will allow the input and Quizard stuff to become handlers instead of devices (the latter is improper because they shouldn't be visible on the CD-i system bus).
The new "module" class (a subclass of handler) will be used to add high-level emulation of OS-9 and CD-RTOS rom modules. I want to preserve the interfaces between the modules and the public data structures as much as possible, because it will allow a gradual transition from "real" to "emulated" modules.
To prepare for all of the above I had to do some fairly heavy design, which caused me to properly write down some of the design information and tradeoffs for the first time. This will be invaluable information for co-developers (if they ever materialize), hence the title "CD-i Emulator Cookbook". Well, at present it's more like a leaflet but I hope to expand it over time and also add some history.
Pieces of the cookbook will be added to the CD-i Emulator website if I feel they're ready.
I've also been giving some thought on a collaboration model for the ROM-less emulation. If there is interest I could do a partial source release that would allow other developers to work on the ROM-less emulation. This release would *not* contain any non-video chipset emulation but it would contain "generic" CD-i audio and video decoding. You would still need to use (part of) a system ROM (in particular the OS-9 kernel and file managers) until enough of the emulation is finished.
I'm still considering all this, but I wanted to get the word out to see if there is interest and to show that I haven't abandoned the project.
Potential co-developers should start boning up on OS-9 and CD-RTOS. All of the technical OS-9 documentation is online at ICDIA and links to the digital Green Book can also be found.
I have some wild ideas about ROM-less emulation; this would basically mean re-implementing the CD-RTOS operating system. Somewhat daunting; it contains over 350 separate explicit APIs and callable entry points and many system data structures would need to be closely emulated. But it can be done, CD-ice proved it (although it took a number of shortcuts that I want to avoid).
I'm not going to tackle that by myself; my current thinking is to make a start by implementing a high-level emulation framework, tracing stubs for all the calls (luckily these can mostly be generated automatically from the digital Green Book and OS-9 manuals) and some scaffolding and samples.
One of the pieces of scaffolding would be a really simple CD-i player shell; one that just shows a big "Play CD-i" button and then starts the CD-i title :-)
For samples I'm thinking about a few easy system calls like F$CRC, F$SetCRC, F$SetSys, F$CmpNam, F$PrsNam, F$ID, F$SUser, F$Icpt, F$SigMask, F$STrap, F$Trans, F$Move, F$SSvc (I may not get through the entire list) and a new NVRAM File Manager (NRF).
It would be nice to do a minimal UCM with Video and Pointer driver so that the simple CD-i player shell would run, but that might be too much. We'll see.
However, it's the new NRF that would be the most immediately interesting for CD-i Emulator users. It would intercept NVRAM access at the file level and redirect it to the PC file system (probably to files in the the nvr directory). This would allow easy sharing of CD-i NVRAM files (e.g. game-saves) over player types or between CD-i emulator users.
To allow all of the above and clean up some dirty tricks that were needed for input playback and handling Quizard, I've done some internal restructuring of CD-i Emulator. In particular, I introduced a new "handler" class beneath the existing "device" and "memory" classes (which are now no longer derived from each other but from a common "component" base class). This restructuring isn't finished yet, but it will allow the input and Quizard stuff to become handlers instead of devices (the latter is improper because they shouldn't be visible on the CD-i system bus).
The new "module" class (a subclass of handler) will be used to add high-level emulation of OS-9 and CD-RTOS rom modules. I want to preserve the interfaces between the modules and the public data structures as much as possible, because it will allow a gradual transition from "real" to "emulated" modules.
To prepare for all of the above I had to do some fairly heavy design, which caused me to properly write down some of the design information and tradeoffs for the first time. This will be invaluable information for co-developers (if they ever materialize), hence the title "CD-i Emulator Cookbook". Well, at present it's more like a leaflet but I hope to expand it over time and also add some history.
Pieces of the cookbook will be added to the CD-i Emulator website if I feel they're ready.
I've also been giving some thought on a collaboration model for the ROM-less emulation. If there is interest I could do a partial source release that would allow other developers to work on the ROM-less emulation. This release would *not* contain any non-video chipset emulation but it would contain "generic" CD-i audio and video decoding. You would still need to use (part of) a system ROM (in particular the OS-9 kernel and file managers) until enough of the emulation is finished.
I'm still considering all this, but I wanted to get the word out to see if there is interest and to show that I haven't abandoned the project.
Potential co-developers should start boning up on OS-9 and CD-RTOS. All of the technical OS-9 documentation is online at ICDIA and links to the digital Green Book can also be found.
Tuesday, October 12, 2010
Release blues and Quizard 1.7
Saturday I released CD-i Emulator 0.5.3-beta1. That was a big milestone for me.
The Internet didn't seem to feel that way. There was some interest but it was nothing compared to five years ago when the first version came out. Oh well...
Sunday a "disc activity" indicator was suggested by a MESS tester. It seemed cool so I added it by coloring the background of the "disc activity" textbox:
blue = reading TOC data
red = reading CD-audio data (mode 0)
yellow = reading CD-ROM data (mode 1)
green = reading CD-i data (mode 2)
I also wrote a proof-of-concept compatibility report table page for the web site so that I could view the reports entered, so far by a single user only (except myself). It needs more work, I won't tell you the URL yet :-)
A few days ago the MESS CD-i programmer requested help in figuring out why the Quizard 1.7 game didn't work and I provided some serial port help. In the process we deduced that the 1993-vintage 68070 datasheets on the Internet all have some of the UART command bits wrong. But I found a few datasheets dated April 1991 which have it right (and so does my 68070 User Manual, luckily, which is dated November 1991). With this fixed in MESS, both emulators produced the same effect: some loading activity and then remaining at a black screen.
Now, Quizard is one of those rare CD-i titles that require external hardware, in this case the so-called "CD-i Jamma" board that interfaces a CD-i player to the industry-standard (at one time) arcade enclosure interface. Bas wrote about this board in his Interactive Dreams blog here and here.
Part of the external interface is a "security" option that causes the game not to run unless the proper hardware is present, and this is what was causing Quizard to remain at a black screen.
Although it is certainly possible in principle to emulate the external board, including its security features, I was in the midst of releasing CD-i Emulator so I let it go for some time.
But today Just Deserts/Harmony/MooglyGuy (the main MESS CD-i developer) posted details about how to "patch" Quizard to avoid the security check and he also posted sufficient details to allow emulation of the arcade case buttons. So I dove in...
It took me an hour or two to create a special "quizard" device for CD-i Emulator that patches Quizard as needed and connects to the serial port to give the CD-i title the communications that it wants. For now I simply intercept PC keyboard digits 0 to 9 and pass them to Quizard as "button" information.
It works alright: Quizard plays its leader and starts "attract" mode and I can even play a game!
The following movie shows only the leader, however:
Note also the green flashing disc activity indicator!
The Internet didn't seem to feel that way. There was some interest but it was nothing compared to five years ago when the first version came out. Oh well...
Sunday a "disc activity" indicator was suggested by a MESS tester. It seemed cool so I added it by coloring the background of the "disc activity" textbox:
blue = reading TOC data
red = reading CD-audio data (mode 0)
yellow = reading CD-ROM data (mode 1)
green = reading CD-i data (mode 2)
I also wrote a proof-of-concept compatibility report table page for the web site so that I could view the reports entered, so far by a single user only (except myself). It needs more work, I won't tell you the URL yet :-)
A few days ago the MESS CD-i programmer requested help in figuring out why the Quizard 1.7 game didn't work and I provided some serial port help. In the process we deduced that the 1993-vintage 68070 datasheets on the Internet all have some of the UART command bits wrong. But I found a few datasheets dated April 1991 which have it right (and so does my 68070 User Manual, luckily, which is dated November 1991). With this fixed in MESS, both emulators produced the same effect: some loading activity and then remaining at a black screen.
Now, Quizard is one of those rare CD-i titles that require external hardware, in this case the so-called "CD-i Jamma" board that interfaces a CD-i player to the industry-standard (at one time) arcade enclosure interface. Bas wrote about this board in his Interactive Dreams blog here and here.
Part of the external interface is a "security" option that causes the game not to run unless the proper hardware is present, and this is what was causing Quizard to remain at a black screen.
Although it is certainly possible in principle to emulate the external board, including its security features, I was in the midst of releasing CD-i Emulator so I let it go for some time.
But today Just Deserts/Harmony/MooglyGuy (the main MESS CD-i developer) posted details about how to "patch" Quizard to avoid the security check and he also posted sufficient details to allow emulation of the arcade case buttons. So I dove in...
It took me an hour or two to create a special "quizard" device for CD-i Emulator that patches Quizard as needed and connects to the serial port to give the CD-i title the communications that it wants. For now I simply intercept PC keyboard digits 0 to 9 and pass them to Quizard as "button" information.
It works alright: Quizard plays its leader and starts "attract" mode and I can even play a game!
The following movie shows only the leader, however:
Note also the green flashing disc activity indicator!
Sunday, October 10, 2010
CD-i Emulator 0.5.3-beta1 released!
I have just released the new version of CD-i Emulator. It has received very limited pre-beta testing, but that's what public betas are for.
I've changed relatively little since the prebeta1 version.
The most import change is the addition of the Help | Report menu option that contains a link to the new Report section of the website and will automatically fill in most of the information fields on that form when clicked.
Unfortunately, to make this work properly I had to touch most files in the sys directory to add correct player model, extension and digital video cartridge identification.
I've also added checksum reporting of DV cartridges, let's find out how complete my current collection is. These checksums will not (yet) trigger an "Unknown ROM" dialog if unknown, but they are posted with compatibility reports.
The -writepng option also had to be fixed to support macros in its filename, otherwise each video frame would overwrite the previous one! The macros that I've decided to support for now are $seq$, which produces a 6-digit sequence number and $time$ which produces the 6-digit frame time (mmssff).
The Release Notes have been modified to include these changes.
Use of the input recording/playback functions revealed a number of bugs; I've fixed those and also added more player information recording so that recorded input files contain enough information for compatibility reports (this isn't used yet).
I've changed relatively little since the prebeta1 version.
The most import change is the addition of the Help | Report menu option that contains a link to the new Report section of the website and will automatically fill in most of the information fields on that form when clicked.
Unfortunately, to make this work properly I had to touch most files in the sys directory to add correct player model, extension and digital video cartridge identification.
I've also added checksum reporting of DV cartridges, let's find out how complete my current collection is. These checksums will not (yet) trigger an "Unknown ROM" dialog if unknown, but they are posted with compatibility reports.
The -writepng option also had to be fixed to support macros in its filename, otherwise each video frame would overwrite the previous one! The macros that I've decided to support for now are $seq$, which produces a 6-digit sequence number and $time$ which produces the 6-digit frame time (mmssff).
The Release Notes have been modified to include these changes.
Use of the input recording/playback functions revealed a number of bugs; I've fixed those and also added more player information recording so that recorded input files contain enough information for compatibility reports (this isn't used yet).
Monday, October 4, 2010
Website updates, bugfixes, compatibility reports
Today I updated the CD-i Emulator website for the upcoming beta release and fixed some remaining bugs, mainly with input recording/playback.
I updated the Release Notes section, put up the hopefully final version of the Full Release Notes, added more screenshots and YouTube embeds and did some miscellaneous text edits. I also updated the CD-i Types section and added a download for it.
It was my intention to release tonight, but that didn't happen because the closed pre-beta testers aren't ready yet. I'll give it a few more days.
My next project will be updating the Title Support section of the website; I want to replace the static table with a much more dynamic one that can be expanded and sorted in various ways, based on a database of actual compatibility reports for specific CD-i player ROMs and CD-i titles that could be entered online.
I might even accept MESS CD-i compatibility reports to provide a basis for comparison. Users would also be able to "me too" on reports.
I would also like to add a "Report Compatibility" option to CD-i Emulator; that would allow automatic filling in of most of the report fields so that entering a compatibility report becomes pretty much a question of clicking only a "compatibility class" button and possibly entering some notes (fields like Reporter could be remembered on the reporter's PC using cookies). It should be easy to add the CD-i Emulator part of this; I might even do it for the current beta if the testers take long enough :-)
For crashes or decoding issues, it would also be useful to allow uploading of input recordings. Actually, uploading a "good recording" would be nice even for successful reports. I could even add an "autorecord" setting to CD-i Emulator to make recording easy; the performance hit should be minimal.
This would also allow "batch" uploading of compatibility reports at some later time; input recordings contain all of the necessary information except for the compatibility class and notes and these could easily be added with input annotations, which I was planning to add anyway.
Unfortunately, uploading a file cannot be automated easily from within CD-i Emulator: the user will have to manually browse for the file. Another way to handle this would be to automatically generate an e-mail compatibility report...
I updated the Release Notes section, put up the hopefully final version of the Full Release Notes, added more screenshots and YouTube embeds and did some miscellaneous text edits. I also updated the CD-i Types section and added a download for it.
It was my intention to release tonight, but that didn't happen because the closed pre-beta testers aren't ready yet. I'll give it a few more days.
My next project will be updating the Title Support section of the website; I want to replace the static table with a much more dynamic one that can be expanded and sorted in various ways, based on a database of actual compatibility reports for specific CD-i player ROMs and CD-i titles that could be entered online.
I might even accept MESS CD-i compatibility reports to provide a basis for comparison. Users would also be able to "me too" on reports.
I would also like to add a "Report Compatibility" option to CD-i Emulator; that would allow automatic filling in of most of the report fields so that entering a compatibility report becomes pretty much a question of clicking only a "compatibility class" button and possibly entering some notes (fields like Reporter could be remembered on the reporter's PC using cookies). It should be easy to add the CD-i Emulator part of this; I might even do it for the current beta if the testers take long enough :-)
For crashes or decoding issues, it would also be useful to allow uploading of input recordings. Actually, uploading a "good recording" would be nice even for successful reports. I could even add an "autorecord" setting to CD-i Emulator to make recording easy; the performance hit should be minimal.
This would also allow "batch" uploading of compatibility reports at some later time; input recordings contain all of the necessary information except for the compatibility class and notes and these could easily be added with input annotations, which I was planning to add anyway.
Unfortunately, uploading a file cannot be automated easily from within CD-i Emulator: the user will have to manually browse for the file. Another way to handle this would be to automatically generate an e-mail compatibility report...
Saturday, October 2, 2010
Lucky Luke on YouTube
I have just posted a "proof of concept" demonstration of CD-i Emulator MPEG decoding on YouTube:
The Philips Media bumper animation is MPEG (audio and video) and so is the audio for the piano sequence. The in-game crash is due to an MPEG sound effect.
This is with a Gate Array MPEG AH01 cartridge (22ER9141 F1 AH01); as the cartridge types get newer the decoding starts working lesser and lesser. Later GMPEG cartridges will crash earlier in the gameplay, VMPEG and IMPEG will not play MPEG at all. This is simply because I haven't gotten around to fixing those things yet; there are still major bugs in the buffering of the MPEG data and while that remains unfixed there is no point.
The magenta border indicates "beta" quality; it is not that hard to display the proper color (black in this case), but this will remind everyone that it is a work in progress.
Many other DVC titles will play their Philips bumper but crash soon thereafter; some others won't even get that far. In some cases this is not even due to MPEG decoding issues but to various "other" reasons for crashing (The 7th Guest is a notorious example).
I have also put up a new draft of the upcoming version 0.5.3-beta1 Release Notes on the website.
The Philips Media bumper animation is MPEG (audio and video) and so is the audio for the piano sequence. The in-game crash is due to an MPEG sound effect.
This is with a Gate Array MPEG AH01 cartridge (22ER9141 F1 AH01); as the cartridge types get newer the decoding starts working lesser and lesser. Later GMPEG cartridges will crash earlier in the gameplay, VMPEG and IMPEG will not play MPEG at all. This is simply because I haven't gotten around to fixing those things yet; there are still major bugs in the buffering of the MPEG data and while that remains unfixed there is no point.
The magenta border indicates "beta" quality; it is not that hard to display the proper color (black in this case), but this will remind everyone that it is a work in progress.
Many other DVC titles will play their Philips bumper but crash soon thereafter; some others won't even get that far. In some cases this is not even due to MPEG decoding issues but to various "other" reasons for crashing (The 7th Guest is a notorious example).
I have also put up a new draft of the upcoming version 0.5.3-beta1 Release Notes on the website.
Release Notes
I have just put up a first draft of the Release Notes for the upcoming version 0.5.3-beta1 on the CD-i Emulator website.
More info coming...
More info coming...
Friday, October 1, 2010
CD-i Emulator pre-beta released
I have just released the pre-beta version of CD-i Emulator to my beta test group.
This version is intended for quick tests; if nothing drastically happens, the final beta1 release will happen in a few days.
In the meantime, I will be writing release notes and updating the CD-i Emulator web site. The final release notes will appear in the Release Notes section of the web site.
I managed to finish the input recording/playback code and also fix a few minor bugs.
Bug reports, comments or suggestions about this version should be posted on the closed CD-i Emulator beta Testing Forum.
This version is intended for quick tests; if nothing drastically happens, the final beta1 release will happen in a few days.
In the meantime, I will be writing release notes and updating the CD-i Emulator web site. The final release notes will appear in the Release Notes section of the web site.
I managed to finish the input recording/playback code and also fix a few minor bugs.
Bug reports, comments or suggestions about this version should be posted on the closed CD-i Emulator beta Testing Forum.
Thursday, September 30, 2010
Finalizing input record/playback
Today I resumed working on the input record/playback code to get it towards its final specification for this beta.
I want the record/playback code to be generic, by which I mean:
1. Recorded input files must be dumpable (e.g. by cdifile) without having to know intimate details of the recorded devices and messages. This can be achieved by labeling all input channels with name and type and recording the formats of all input message types, which is now mostly done.
2. Recorded input files must contain all the information required for faithful playback, which includes things like CD-i player model, DVC cartridge type, extension roms, PAL/NTSC and special startup options and emulated disc insertions.
The ideal is that "wcdiemu -playback" will just work without the need to fiddle with emulator options to reproduce the recording environment. This is now also mostly done except for the disc insertions (they are recorded but cannot yet be played back).
3. It must in principle be possible to play back on a different CD-i player model or with a different DVC cartridge. This means that the input channel matching must be somewhat "intelligent". The groundwork for this is in place but the actual channel matching is not yet there.
All of the above are needed to get the feature usable for extensive compatibility testing, which is a goal of this beta: it should be possible to exchange input recordings that reproduce crashes or rendering bugs, even when they require some time to reproduce. Emulator state snapshots would make this even better, but that is too much work for now.
Recorded input files are in IFF format with the following generic structure:
- each file contains a single FORM chunk of type INPT (recorded input)
- the first chunk inside the FORM is an IVER chunk (version information)
- the next chunk inside the FORM is an ICHN chunk (channel information)
- the final chunk inside the FORM is an IMSG chunk (input messages)
Input messages are recorded in binary format to keep their size small; the first time that a message type is used, the message format string is prepended.
The cdifile dumping code will not be released with this beta, there's too much unfinished code in there (i.e. the -dir[ectory] option to display the directory of a CD-i disc image file).
I will attempt to finish this tomorrow, but it's just barely achievable...
Until then, you'll have to do with this picture:
I want the record/playback code to be generic, by which I mean:
1. Recorded input files must be dumpable (e.g. by cdifile) without having to know intimate details of the recorded devices and messages. This can be achieved by labeling all input channels with name and type and recording the formats of all input message types, which is now mostly done.
2. Recorded input files must contain all the information required for faithful playback, which includes things like CD-i player model, DVC cartridge type, extension roms, PAL/NTSC and special startup options and emulated disc insertions.
The ideal is that "wcdiemu -playback" will just work without the need to fiddle with emulator options to reproduce the recording environment. This is now also mostly done except for the disc insertions (they are recorded but cannot yet be played back).
3. It must in principle be possible to play back on a different CD-i player model or with a different DVC cartridge. This means that the input channel matching must be somewhat "intelligent". The groundwork for this is in place but the actual channel matching is not yet there.
All of the above are needed to get the feature usable for extensive compatibility testing, which is a goal of this beta: it should be possible to exchange input recordings that reproduce crashes or rendering bugs, even when they require some time to reproduce. Emulator state snapshots would make this even better, but that is too much work for now.
Recorded input files are in IFF format with the following generic structure:
- each file contains a single FORM chunk of type INPT (recorded input)
- the first chunk inside the FORM is an IVER chunk (version information)
- the next chunk inside the FORM is an ICHN chunk (channel information)
- the final chunk inside the FORM is an IMSG chunk (input messages)
Input messages are recorded in binary format to keep their size small; the first time that a message type is used, the message format string is prepended.
The cdifile dumping code will not be released with this beta, there's too much unfinished code in there (i.e. the -dir[ectory] option to display the directory of a CD-i disc image file).
I will attempt to finish this tomorrow, but it's just barely achievable...
Until then, you'll have to do with this picture:
Tuesday, September 28, 2010
Housecleaning
Today I did some housecleaning for the upcoming beta, which I'm still hoping to release before the end of this month.
I decided to keep the current somewhat buggy MPEG decoding stuff in the beta, which meant that I had to clean up the MPEG ROM detection logic so that it actually works (until now I always used the -dvc option to explicitly force the DVC model).
The MPEG ROM detector will now only use the supplied "dummy" xmpeg.rom file if it cannot find another recognized ?mpeg*.rom file. If it does find such a file, the DVC type will now be displayed in the CD-i Emulator title bar (GMPEG, IMPEG, VMPEG). For GMPEG the AH0x number is also displayed, as the AH00/AH01 cartridges are very different from the AH02/AH03 ones.
Some little-known DVC tidbits:
- The Philips CD-i 370 uses VMPEG hardware but at a different memory address
- The Philips CD-i 615 uses IMPEG hardware but the ROM also contains non-FMV software
- The DVS VE-200 and LG GDI-700 use IMPEG hardware but at a different memory address
More details on the state of the MPEG emulation will be available in the beta release notes.
Earlier I had started using a manifest file to get a newer common controls DLL, thus modernizing the visual "look" of CD-i Emulator somewhat. However, this caused Windows Vista and Windows 7 to stop using "virtual store" redirection of the cdiemu.ini file which broke all settings stuff (including the recently used list).
Today I figured out that leaving out the UAC "requested level" entry from the manifest restores the "virtual store" redirection. This is good enough for this beta.
I also killed the "file registration" code because it would hijack any existing non-beta registration (this has nothing to do with licensing but with the way Windows finds executables to open certain files).
Finally, I added video decode skip validation. This gets a bit technical...
In CD-i players, video decoding steals bus cycles and thus slows down the machine. The amount of bus cycles stolen depends on the video mode, pixel repeat settings and, for run-length modes, on the actual data.
If the emulation starts to lag behind, CD-i Emulator starts skipping the decoding of video frames. Previously, I used a "worse-case" estimate for the number of stolen bus cycles in this case. However, this introduces problems with input recording and playback: if frame skipping differs (which is almost guaranteed to be the case), the playback will start to "drift" from the recording. Over many frames this will result in an incorrect playback.
I coded a "SkipLine" routine that does no actual video decoding like "DecodeLine" but it does decode just enough to accurately determine the number of stolen bus cycles. This routine is now used when the video frame is skipped.
The validation consists of running both routines when the video frame is not being skipped and ensuring that both "steal" the same number of bus cycles.
I decided to keep the current somewhat buggy MPEG decoding stuff in the beta, which meant that I had to clean up the MPEG ROM detection logic so that it actually works (until now I always used the -dvc option to explicitly force the DVC model).
The MPEG ROM detector will now only use the supplied "dummy" xmpeg.rom file if it cannot find another recognized ?mpeg*.rom file. If it does find such a file, the DVC type will now be displayed in the CD-i Emulator title bar (GMPEG, IMPEG, VMPEG). For GMPEG the AH0x number is also displayed, as the AH00/AH01 cartridges are very different from the AH02/AH03 ones.
Some little-known DVC tidbits:
- The Philips CD-i 370 uses VMPEG hardware but at a different memory address
- The Philips CD-i 615 uses IMPEG hardware but the ROM also contains non-FMV software
- The DVS VE-200 and LG GDI-700 use IMPEG hardware but at a different memory address
More details on the state of the MPEG emulation will be available in the beta release notes.
Earlier I had started using a manifest file to get a newer common controls DLL, thus modernizing the visual "look" of CD-i Emulator somewhat. However, this caused Windows Vista and Windows 7 to stop using "virtual store" redirection of the cdiemu.ini file which broke all settings stuff (including the recently used list).
Today I figured out that leaving out the UAC "requested level" entry from the manifest restores the "virtual store" redirection. This is good enough for this beta.
I also killed the "file registration" code because it would hijack any existing non-beta registration (this has nothing to do with licensing but with the way Windows finds executables to open certain files).
Finally, I added video decode skip validation. This gets a bit technical...
In CD-i players, video decoding steals bus cycles and thus slows down the machine. The amount of bus cycles stolen depends on the video mode, pixel repeat settings and, for run-length modes, on the actual data.
If the emulation starts to lag behind, CD-i Emulator starts skipping the decoding of video frames. Previously, I used a "worse-case" estimate for the number of stolen bus cycles in this case. However, this introduces problems with input recording and playback: if frame skipping differs (which is almost guaranteed to be the case), the playback will start to "drift" from the recording. Over many frames this will result in an incorrect playback.
I coded a "SkipLine" routine that does no actual video decoding like "DecodeLine" but it does decode just enough to accurately determine the number of stolen bus cycles. This routine is now used when the video frame is skipped.
The validation consists of running both routines when the video frame is not being skipped and ensuring that both "steal" the same number of bus cycles.
Sunday, August 1, 2010
Quick status update
It's been just over a half year since my last posting here. A quick status update...
In March I upgraded my trusty old desktop PC to a new laptop with a dual-core Pentium T4300 at 2.1 GHz. In the process of migrating my data I broke my old P-ATA hard disk :-(
The breakage was very minor (a broken connector lead) and seemed to be fixable with some soldering, but it took me nearly two months to get around to obtaining the proper cable and connector. Then I quickly made a full copy to a new S-ATA hard disk and breathed a sigh of relief.
I did have a backup but it was months old and did not include my most recent CD-i Emulator work, so I really wanted the up-to-date data.
However, as Charles commented on my previous entry, Visual Studio 6 has problems working under Windows 7. That demoralized me such that I left it hanging for a few months more. My older CD-i Emulator development builds didn't even work under Win7 :-(
Yesterday I finally got around to doing a full restore from the backup disk to my laptop and I also installed Visual Studio 2010 just to take a look at it.
Having done that, I realized that I could try building CD-i Emulator again!
I just finished doing that, and after finding the single source line that generated the 300+ errors I got on the first build, it appears that I now again have a working development build of the most recent CD-i Emulator, optimistically labeled v0.5.3-beta1. It seems to run like a charm, doing 50/50 frames on non-FMV titles and even 34/50 on my FMV test cases. Wow!
Using Visual Studio 2010 means that the minimum system requirement is now Windows XP, as earlier Windows versions are no longer supported as C++ build targets. But I don't think that should really be a problem.
At the end of next month it will be five years since the original CD-i Emulator v0.5.2 public release; I'm hoping to release 0.5.3-beta1 before that.
Please bear with me...
In March I upgraded my trusty old desktop PC to a new laptop with a dual-core Pentium T4300 at 2.1 GHz. In the process of migrating my data I broke my old P-ATA hard disk :-(
The breakage was very minor (a broken connector lead) and seemed to be fixable with some soldering, but it took me nearly two months to get around to obtaining the proper cable and connector. Then I quickly made a full copy to a new S-ATA hard disk and breathed a sigh of relief.
I did have a backup but it was months old and did not include my most recent CD-i Emulator work, so I really wanted the up-to-date data.
However, as Charles commented on my previous entry, Visual Studio 6 has problems working under Windows 7. That demoralized me such that I left it hanging for a few months more. My older CD-i Emulator development builds didn't even work under Win7 :-(
Yesterday I finally got around to doing a full restore from the backup disk to my laptop and I also installed Visual Studio 2010 just to take a look at it.
Having done that, I realized that I could try building CD-i Emulator again!
I just finished doing that, and after finding the single source line that generated the 300+ errors I got on the first build, it appears that I now again have a working development build of the most recent CD-i Emulator, optimistically labeled v0.5.3-beta1. It seems to run like a charm, doing 50/50 frames on non-FMV titles and even 34/50 on my FMV test cases. Wow!
Using Visual Studio 2010 means that the minimum system requirement is now Windows XP, as earlier Windows versions are no longer supported as C++ build targets. But I don't think that should really be a problem.
At the end of next month it will be five years since the original CD-i Emulator v0.5.2 public release; I'm hoping to release 0.5.3-beta1 before that.
Please bear with me...
Wednesday, January 20, 2010
Input record/playback, I2M emulation and a new start
It has been more then a month since my last blog entry. There are several reasons for this, among others the very busy month of december and a mild CD-i Emulator burn-out. This blog entry is part of an attempt to restart the process and hopefully arrive at a releasable beta version in the not too distant future.
Early december I did a little work on rewriting the input record/playback code to its final specifications, using the full IFF reading/writing code that I reported on in the previous entries. It was hairy work (refactoring often is) and I more or less threw in the towel at that point.
This was followed in mid december by an attempt to regain focus. Triggered by some forum discussions, I did some work on emulating the I2M Media Playback CD-i board for the PC (in a sense another completely different player generation).
This board uses a Motorola 68341 so-called "CD-i Engine" processor chip, which is a CPU32 processor core with some on-chip peripherals (a DMA controller, two different serial interfaces, a timer, etc), a VDSC video chip and a completely undocumented "host interface" to the PC bus. So far the board does not appear to have a separate CD/Audio interface, but it does have a VMPEG chip.
I had already implemented CPU32 emulation and partial emulation of its on-chip peripherals, but this needed to be extended a bit more. The main "problem" here appeared to be that these peripherals appear to ignore address bit 23.
I also had to reverse engineer the host interface, which amounted to some disassembly and tracing. In the process I also figured out that I had built a bad ROM image (the host software generates these on the fly from "ROM fragment" files and I had mis-interpreted the script file that tells it how to do this). I used the CD-i Playback 2.2.1 files for this.
The I2M board now successfully boots OS9, including displaying some tracing messages (it turns out that it can do this either via the host interface or via the serial interface, which was reason for lot's of head scratching until the "aha!" moment). The video display gets initialized (some kind of "blue" screen) and the then board "hangs" inside a watchdog process, presumably waiting on some signal from the host telling it what to do (it's not a crash but appears to be waiting for some host interrupt). Figuring this out requires disassembly of the watchdog process and that's where I stopped, having worked on it for about three evenings.
Somewhere in between (can't remember the exact timing) I also did a test build with Visual Studio 2008, which resulted in a few portability fixes but nothing dramatic. I had hoped this would make some speed difference but I didn't notice any. I've returned to my old trusty Visual Studio 6; it's smaller and faster and has everything I need.
That's more or less the current status; I hope to resume work on the input record/playback code soon which is the last thing holding up the current beta release.
Early december I did a little work on rewriting the input record/playback code to its final specifications, using the full IFF reading/writing code that I reported on in the previous entries. It was hairy work (refactoring often is) and I more or less threw in the towel at that point.
This was followed in mid december by an attempt to regain focus. Triggered by some forum discussions, I did some work on emulating the I2M Media Playback CD-i board for the PC (in a sense another completely different player generation).
This board uses a Motorola 68341 so-called "CD-i Engine" processor chip, which is a CPU32 processor core with some on-chip peripherals (a DMA controller, two different serial interfaces, a timer, etc), a VDSC video chip and a completely undocumented "host interface" to the PC bus. So far the board does not appear to have a separate CD/Audio interface, but it does have a VMPEG chip.
I had already implemented CPU32 emulation and partial emulation of its on-chip peripherals, but this needed to be extended a bit more. The main "problem" here appeared to be that these peripherals appear to ignore address bit 23.
I also had to reverse engineer the host interface, which amounted to some disassembly and tracing. In the process I also figured out that I had built a bad ROM image (the host software generates these on the fly from "ROM fragment" files and I had mis-interpreted the script file that tells it how to do this). I used the CD-i Playback 2.2.1 files for this.
The I2M board now successfully boots OS9, including displaying some tracing messages (it turns out that it can do this either via the host interface or via the serial interface, which was reason for lot's of head scratching until the "aha!" moment). The video display gets initialized (some kind of "blue" screen) and the then board "hangs" inside a watchdog process, presumably waiting on some signal from the host telling it what to do (it's not a crash but appears to be waiting for some host interrupt). Figuring this out requires disassembly of the watchdog process and that's where I stopped, having worked on it for about three evenings.
Somewhere in between (can't remember the exact timing) I also did a test build with Visual Studio 2008, which resulted in a few portability fixes but nothing dramatic. I had hoped this would make some speed difference but I didn't notice any. I've returned to my old trusty Visual Studio 6; it's smaller and faster and has everything I need.
That's more or less the current status; I hope to resume work on the input record/playback code soon which is the last thing holding up the current beta release.
Subscribe to:
Posts (Atom)