It's been just over a half year since my last posting here. A quick status update...
In March I upgraded my trusty old desktop PC to a new laptop with a dual-core Pentium T4300 at 2.1 GHz. In the process of migrating my data I broke my old P-ATA hard disk :-(
The breakage was very minor (a broken connector lead) and seemed to be fixable with some soldering, but it took me nearly two months to get around to obtaining the proper cable and connector. Then I quickly made a full copy to a new S-ATA hard disk and breathed a sigh of relief.
I did have a backup but it was months old and did not include my most recent CD-i Emulator work, so I really wanted the up-to-date data.
However, as Charles commented on my previous entry, Visual Studio 6 has problems working under Windows 7. That demoralized me such that I left it hanging for a few months more. My older CD-i Emulator development builds didn't even work under Win7 :-(
Yesterday I finally got around to doing a full restore from the backup disk to my laptop and I also installed Visual Studio 2010 just to take a look at it.
Having done that, I realized that I could try building CD-i Emulator again!
I just finished doing that, and after finding the single source line that generated the 300+ errors I got on the first build, it appears that I now again have a working development build of the most recent CD-i Emulator, optimistically labeled v0.5.3-beta1. It seems to run like a charm, doing 50/50 frames on non-FMV titles and even 34/50 on my FMV test cases. Wow!
Using Visual Studio 2010 means that the minimum system requirement is now Windows XP, as earlier Windows versions are no longer supported as C++ build targets. But I don't think that should really be a problem.
At the end of next month it will be five years since the original CD-i Emulator v0.5.2 public release; I'm hoping to release 0.5.3-beta1 before that.
Please bear with me...
Sunday, August 1, 2010
Wednesday, January 20, 2010
Input record/playback, I2M emulation and a new start
It has been more then a month since my last blog entry. There are several reasons for this, among others the very busy month of december and a mild CD-i Emulator burn-out. This blog entry is part of an attempt to restart the process and hopefully arrive at a releasable beta version in the not too distant future.
Early december I did a little work on rewriting the input record/playback code to its final specifications, using the full IFF reading/writing code that I reported on in the previous entries. It was hairy work (refactoring often is) and I more or less threw in the towel at that point.
This was followed in mid december by an attempt to regain focus. Triggered by some forum discussions, I did some work on emulating the I2M Media Playback CD-i board for the PC (in a sense another completely different player generation).
This board uses a Motorola 68341 so-called "CD-i Engine" processor chip, which is a CPU32 processor core with some on-chip peripherals (a DMA controller, two different serial interfaces, a timer, etc), a VDSC video chip and a completely undocumented "host interface" to the PC bus. So far the board does not appear to have a separate CD/Audio interface, but it does have a VMPEG chip.
I had already implemented CPU32 emulation and partial emulation of its on-chip peripherals, but this needed to be extended a bit more. The main "problem" here appeared to be that these peripherals appear to ignore address bit 23.
I also had to reverse engineer the host interface, which amounted to some disassembly and tracing. In the process I also figured out that I had built a bad ROM image (the host software generates these on the fly from "ROM fragment" files and I had mis-interpreted the script file that tells it how to do this). I used the CD-i Playback 2.2.1 files for this.
The I2M board now successfully boots OS9, including displaying some tracing messages (it turns out that it can do this either via the host interface or via the serial interface, which was reason for lot's of head scratching until the "aha!" moment). The video display gets initialized (some kind of "blue" screen) and the then board "hangs" inside a watchdog process, presumably waiting on some signal from the host telling it what to do (it's not a crash but appears to be waiting for some host interrupt). Figuring this out requires disassembly of the watchdog process and that's where I stopped, having worked on it for about three evenings.
Somewhere in between (can't remember the exact timing) I also did a test build with Visual Studio 2008, which resulted in a few portability fixes but nothing dramatic. I had hoped this would make some speed difference but I didn't notice any. I've returned to my old trusty Visual Studio 6; it's smaller and faster and has everything I need.
That's more or less the current status; I hope to resume work on the input record/playback code soon which is the last thing holding up the current beta release.
Early december I did a little work on rewriting the input record/playback code to its final specifications, using the full IFF reading/writing code that I reported on in the previous entries. It was hairy work (refactoring often is) and I more or less threw in the towel at that point.
This was followed in mid december by an attempt to regain focus. Triggered by some forum discussions, I did some work on emulating the I2M Media Playback CD-i board for the PC (in a sense another completely different player generation).
This board uses a Motorola 68341 so-called "CD-i Engine" processor chip, which is a CPU32 processor core with some on-chip peripherals (a DMA controller, two different serial interfaces, a timer, etc), a VDSC video chip and a completely undocumented "host interface" to the PC bus. So far the board does not appear to have a separate CD/Audio interface, but it does have a VMPEG chip.
I had already implemented CPU32 emulation and partial emulation of its on-chip peripherals, but this needed to be extended a bit more. The main "problem" here appeared to be that these peripherals appear to ignore address bit 23.
I also had to reverse engineer the host interface, which amounted to some disassembly and tracing. In the process I also figured out that I had built a bad ROM image (the host software generates these on the fly from "ROM fragment" files and I had mis-interpreted the script file that tells it how to do this). I used the CD-i Playback 2.2.1 files for this.
The I2M board now successfully boots OS9, including displaying some tracing messages (it turns out that it can do this either via the host interface or via the serial interface, which was reason for lot's of head scratching until the "aha!" moment). The video display gets initialized (some kind of "blue" screen) and the then board "hangs" inside a watchdog process, presumably waiting on some signal from the host telling it what to do (it's not a crash but appears to be waiting for some host interrupt). Figuring this out requires disassembly of the watchdog process and that's where I stopped, having worked on it for about three evenings.
Somewhere in between (can't remember the exact timing) I also did a test build with Visual Studio 2008, which resulted in a few portability fixes but nothing dramatic. I had hoped this would make some speed difference but I didn't notice any. I've returned to my old trusty Visual Studio 6; it's smaller and faster and has everything I need.
That's more or less the current status; I hope to resume work on the input record/playback code soon which is the last thing holding up the current beta release.
Tuesday, November 24, 2009
IFF reading and AVI writing bugfix
I've taken a few days off from CD-i work, mainly to catch some sleep. So there is only a small amount of progress to report today.
Yesterday I wrote and debugged proper IFF reading code. Well, it turns out the IFF reader itself was pretty much "obviously" correct but there were major bugs in the interaction of buffering and the SetPosition / SetLimit functions of my CArchive class.
Once finished, I added rudimentary IFF dump code (just chunk header info) to cdifile and proceeded to use that to test the IFF code against some "real" files (mainly WAVE and AVI).
Using my "bad" Burn:Cycle AVI from before I found the cause of the badness: the dwLength field of the audio 'strh' chunk was ridiculously large. Turns out I misread Media Player: the AVI claimed an audio length of 33 minutes although in reality it was only about 6. It had to be some bug in the aviio.c file that I've taken from MESS/MAME (this is allowed by the license for that file, which is now BSD).
I did come code reading and tried some "obvious" fixes, but not until I added tracing code today did I find the culprit: the total number of samples was being incremented way too much on each audio buffer flush. The fix was an obvious oneliner that I've also submitted to MAME and MESS.
You can download the fix here.
Yesterday I wrote and debugged proper IFF reading code. Well, it turns out the IFF reader itself was pretty much "obviously" correct but there were major bugs in the interaction of buffering and the SetPosition / SetLimit functions of my CArchive class.
Once finished, I added rudimentary IFF dump code (just chunk header info) to cdifile and proceeded to use that to test the IFF code against some "real" files (mainly WAVE and AVI).
Using my "bad" Burn:Cycle AVI from before I found the cause of the badness: the dwLength field of the audio 'strh' chunk was ridiculously large. Turns out I misread Media Player: the AVI claimed an audio length of 33 minutes although in reality it was only about 6. It had to be some bug in the aviio.c file that I've taken from MESS/MAME (this is allowed by the license for that file, which is now BSD).
I did come code reading and tried some "obvious" fixes, but not until I added tracing code today did I find the culprit: the total number of samples was being incremented way too much on each audio buffer flush. The fix was an obvious oneliner that I've also submitted to MAME and MESS.
You can download the fix here.
Thursday, November 19, 2009
Burn:Cycle on YouTube
It turns out that Ultra MP4 Video Converter could handle my "bad" AVI file, although the trial version only does the first five minutes. That's all right as the video contained no interesting stuff after that :-)
You can see it here.
You can see it here.
AVI writing fixed and some fast machine issues
In a new attempt to get a usable Burn:Cycle movie out of CD-i Emulator, I decided to add multiple PNG writing as a poor substitute for AVI writing. After all, if you have the images and audio, you can always postprocess them into a movie!
I hooked up the PNG writing code from MESS/MAME, which seemed simple enough. There was the small matter of adding 24 bits per pixel bitmap support (their code handles 16 and 32 but not 24 which is however the internal format of CD-i Emulator; I had already partially added such support for the AVI writing code).
However, the resulting PNG files would not display in Explorer or any other image tool, giving messages like 'Bad file format' and 'Cannot read'. In exploring the cause for this, I was looking at memory buffers being filled and suddenly noticed that they where being filled in little-endian byte order, which is definitely wrong as PNG is a big-endian byte order file format.
The source of the problem turned out the be a configuration error with the MESS/MAME code: it needs a macro LSB_FIRST so it knows that it's running on a little-endian machine, and I had forgotten to give it one. When I fixed this, the PNG files came out right. And even better, the AVI files suddenly also seemed correct (not surprisingly, as they use the same byte order flipping macros as the PNG writing code).
So now I have both PNG writing and AVI writing hooked up in CD-i Emulator. I proceeded to do some timing tests and in the process added a few new writing options. The -writesilent option writes AVI files without audio; the -writerate option sets the framerate for the AVI files.
Full framerate AVI files still play "jumpy" on my machine but that's caused by their uncompressedness (is that even a word?); if you convert them to MP4 for uploading they play perfectly.
I uploaded a first test to YouTube; you can see it here.
If you set the AVI framerate low enough, you can do real-time movie captures. On my slow home machine I can do real-time capturing with -writerate 2 or less (which gives very jumpy AVI files, but the audio is correct).
To do non-realtime movie rendering you need to use the input recording and playback options. I did that for one of my Burn:Cycle recordings (not a very good one, but it's hard to play on my machine as the framerate it really too low and as a result the mouse gets jerky, which makes you overshoot your targets, which results in easy death at the hands of the security people).
The rendering proceeded at about 15% real-time speed while I was typing the above and is now finished; it gave me an AVI file of 27 GB for a nearly 7 minute session. There's something wrong with that file; Media Player claims it's only 33 seconds but proceeds to play all 7 minutes of it. Huh?
I also have problems converting it to MP4, probably caused by the same problems. Currently testing conversion programs to find one that can handle it...
During all of the fast machine testing yesterday, I also noticed that after a few minutes the audio begins to lag behind when it shouldn't. This is probably caused by the audio not being decoded at exactly 44.1 kHz relative to the PC timers (or something like that). When audio is playing, it really needs to be taken as the master clock for emulation speed control but that is somewhat hard with the current speed control code. Not for this beta. To ameliorate the problem, I've added a "Flush Audio" option (keyboard shortcut Ctrl-F) that allows you to manually flush the lagging audio so that it regains emulation sync.
Yesterday I also started adding proper IFF reading code, as opposed to the basic version handling only a single large data chunk that is currently used by the input playback code. This is needed so I that can properly finish the recording and playback functionality.
The beta release seems to be coming nearer...
I hooked up the PNG writing code from MESS/MAME, which seemed simple enough. There was the small matter of adding 24 bits per pixel bitmap support (their code handles 16 and 32 but not 24 which is however the internal format of CD-i Emulator; I had already partially added such support for the AVI writing code).
However, the resulting PNG files would not display in Explorer or any other image tool, giving messages like 'Bad file format' and 'Cannot read'. In exploring the cause for this, I was looking at memory buffers being filled and suddenly noticed that they where being filled in little-endian byte order, which is definitely wrong as PNG is a big-endian byte order file format.
The source of the problem turned out the be a configuration error with the MESS/MAME code: it needs a macro LSB_FIRST so it knows that it's running on a little-endian machine, and I had forgotten to give it one. When I fixed this, the PNG files came out right. And even better, the AVI files suddenly also seemed correct (not surprisingly, as they use the same byte order flipping macros as the PNG writing code).
So now I have both PNG writing and AVI writing hooked up in CD-i Emulator. I proceeded to do some timing tests and in the process added a few new writing options. The -writesilent option writes AVI files without audio; the -writerate option sets the framerate for the AVI files.
Full framerate AVI files still play "jumpy" on my machine but that's caused by their uncompressedness (is that even a word?); if you convert them to MP4 for uploading they play perfectly.
I uploaded a first test to YouTube; you can see it here.
If you set the AVI framerate low enough, you can do real-time movie captures. On my slow home machine I can do real-time capturing with -writerate 2 or less (which gives very jumpy AVI files, but the audio is correct).
To do non-realtime movie rendering you need to use the input recording and playback options. I did that for one of my Burn:Cycle recordings (not a very good one, but it's hard to play on my machine as the framerate it really too low and as a result the mouse gets jerky, which makes you overshoot your targets, which results in easy death at the hands of the security people).
The rendering proceeded at about 15% real-time speed while I was typing the above and is now finished; it gave me an AVI file of 27 GB for a nearly 7 minute session. There's something wrong with that file; Media Player claims it's only 33 seconds but proceeds to play all 7 minutes of it. Huh?
I also have problems converting it to MP4, probably caused by the same problems. Currently testing conversion programs to find one that can handle it...
During all of the fast machine testing yesterday, I also noticed that after a few minutes the audio begins to lag behind when it shouldn't. This is probably caused by the audio not being decoded at exactly 44.1 kHz relative to the PC timers (or something like that). When audio is playing, it really needs to be taken as the master clock for emulation speed control but that is somewhat hard with the current speed control code. Not for this beta. To ameliorate the problem, I've added a "Flush Audio" option (keyboard shortcut Ctrl-F) that allows you to manually flush the lagging audio so that it regains emulation sync.
Yesterday I also started adding proper IFF reading code, as opposed to the basic version handling only a single large data chunk that is currently used by the input playback code. This is needed so I that can properly finish the recording and playback functionality.
The beta release seems to be coming nearer...
Tuesday, November 17, 2009
Fast machines, file writing and some bits
Today I used my work PC to examine CD-i Emulator on a fast machine.
It turns out that cpu throttling was already there; titles run at the right speed. However, the frame skip detection logic is too sensitive: it will skip frames even on fast hardware, just because of a momentary slowness (I think). I added an option -noskip to turn off frame skipping altogether; beter autodetection is too much for this beta.
Likewise, for people who always want to run full-speed, I've added an option -nowait to turn off cpu throttling. This can make your CD-i title unplayably fast and it will also cause the audio to lag behind if the title plays continuous audio (most games do).
On the AVI writing front I was less lucky. I cannot get decent write performance and this is not caused by the AVI writing code; just writing the raw data to a file will already cause very "skippy" emulation. Perhaps using a separate thread would solve this (especially on a multicore machine).
And what's worse, the resulting AVI files seem to have timecode problems because they are also skippy, which should not be the case. This is really weird as I've previously been able to write AVI files correctly. It needs more work, obviously.
On the other hand, the AVI files contain the correct audio data: if I extract the audio stream with avidemux, it's identical to a separately written WAV file.
In the process of testing all of this (with Burn:Cycle, what else?) I also fixed the audio crackling problem; it turns out that Burn:Cycle opens all four audio attenuators (L-R, L-L, R-L, R-R) and for this possibility I hadn't accounted in the mix scaling.
I also added the two keyboard emulations that I mentioned yesterday: Backspace for button 2 and Esc for button 3 (buttons 1 and 2 together). This works well for The Apprentice and Dimo's Quest and these are my reference titles anyway :-) I will ultimately add a UI for changing the keyboard map, of course.
It turns out that cpu throttling was already there; titles run at the right speed. However, the frame skip detection logic is too sensitive: it will skip frames even on fast hardware, just because of a momentary slowness (I think). I added an option -noskip to turn off frame skipping altogether; beter autodetection is too much for this beta.
Likewise, for people who always want to run full-speed, I've added an option -nowait to turn off cpu throttling. This can make your CD-i title unplayably fast and it will also cause the audio to lag behind if the title plays continuous audio (most games do).
On the AVI writing front I was less lucky. I cannot get decent write performance and this is not caused by the AVI writing code; just writing the raw data to a file will already cause very "skippy" emulation. Perhaps using a separate thread would solve this (especially on a multicore machine).
And what's worse, the resulting AVI files seem to have timecode problems because they are also skippy, which should not be the case. This is really weird as I've previously been able to write AVI files correctly. It needs more work, obviously.
On the other hand, the AVI files contain the correct audio data: if I extract the audio stream with avidemux, it's identical to a separately written WAV file.
In the process of testing all of this (with Burn:Cycle, what else?) I also fixed the audio crackling problem; it turns out that Burn:Cycle opens all four audio attenuators (L-R, L-L, R-L, R-R) and for this possibility I hadn't accounted in the mix scaling.
I also added the two keyboard emulations that I mentioned yesterday: Backspace for button 2 and Esc for button 3 (buttons 1 and 2 together). This works well for The Apprentice and Dimo's Quest and these are my reference titles anyway :-) I will ultimately add a UI for changing the keyboard map, of course.
Monday, November 16, 2009
Soundmap cleanup, file writing, beta preparation
Today I started with the planned cleanup of the revised soundmap playback code. This took some time but I had it finished my mid-afternoon (please note that I have small children around, which can sometimes make the progress quite slow).
By way of final test I did a 10 minute Burn:Cycle game session; there's still an audio decoding issue there because some parts of the music crackle heavily.
Then I continued working on the WAV / AVI writing front. I have this essentially working now; both types of files are correctly written (the AVI files now include both audio and video). There are new options -writewav and -writeavi to invoke these functions from the command line.
I used the WAV writing feature to check out the Burn:Cycle audio decoding issue. The audio samples go way out of range at the point of the crackles (they actually clamp against the minimum/maximum values). There is probably something wrong in the scaling or something.
One generic issue with WAV file writing is that silent periods (where the CD-i application doesn't play any audio) do not show up in the recorded audio. For the moment, I've decided that this is a feature :-)
There is also some kind of audio timing issue in AVI file writing; the recorded files sometimes sound "skippy". However, this may also be related to the speed of my PC; these are uncompressed AVI files which take 61.5 MB per second for the CD-i video data alone (50 x 3 x 768 x 560 bytes per second). Compared to this, audio is only a measly 172 KB per second :-)
I've found that real-time AVI writing is nearly impossible on my hardware; the framerate drops below 2/50 sometimes (I've lowered the previous 10/50 limit) which makes the titles unplayable. Tomorrow I'll try my work PC which is much faster. That should also give me an opportunity to add frame rate throttling; on fast PC's the emulator currently runs too fast.
However, non-realtime AVI writing also gives the "skippy" sound, so it may be unrelated. Sometimes the audio also gets way out of sync...
One worrying generic issue is that CD-i Emulator seems to have become slower as of late, for no good reason that I can think of. It may be that some debugging code somewhere is slowing things down, but I haven't found it yet. Or it could be that the titles I'm currently running are simple more demanding...
During testing, I also fixed a recent bug where mouse input and keyboard input interfere with each other. I also took out the use of the Shift key for button 2, as it is prone to generate false button presses when using Alt-Tab / Shift-Alt-Tab to switch between windows. You can still use the numeric "+" key for button 2, though, but that is too far away for generic use: I need an alternative closer to the space bar. Perhaps Backspace for button 2, and Esc for button 3 (buttons 1 and 2 together)?
Finally, I've started putting together the first v0.5.3 beta distribution. It will be mostly identical to a v0.5.2 one, with some updates in the sys directory, updated cdiroms.ini and cditypes.rul files and of course an updated executable which will be named wcdiemu-v053b1.exe (for beta 1) to avoid accidentally overwriting an existing v0.5.2 executable. I've also taken a first crack at a release notes document (very descriptively named BETA1).
By way of final test I did a 10 minute Burn:Cycle game session; there's still an audio decoding issue there because some parts of the music crackle heavily.
Then I continued working on the WAV / AVI writing front. I have this essentially working now; both types of files are correctly written (the AVI files now include both audio and video). There are new options -writewav and -writeavi to invoke these functions from the command line.
I used the WAV writing feature to check out the Burn:Cycle audio decoding issue. The audio samples go way out of range at the point of the crackles (they actually clamp against the minimum/maximum values). There is probably something wrong in the scaling or something.
One generic issue with WAV file writing is that silent periods (where the CD-i application doesn't play any audio) do not show up in the recorded audio. For the moment, I've decided that this is a feature :-)
There is also some kind of audio timing issue in AVI file writing; the recorded files sometimes sound "skippy". However, this may also be related to the speed of my PC; these are uncompressed AVI files which take 61.5 MB per second for the CD-i video data alone (50 x 3 x 768 x 560 bytes per second). Compared to this, audio is only a measly 172 KB per second :-)
I've found that real-time AVI writing is nearly impossible on my hardware; the framerate drops below 2/50 sometimes (I've lowered the previous 10/50 limit) which makes the titles unplayable. Tomorrow I'll try my work PC which is much faster. That should also give me an opportunity to add frame rate throttling; on fast PC's the emulator currently runs too fast.
However, non-realtime AVI writing also gives the "skippy" sound, so it may be unrelated. Sometimes the audio also gets way out of sync...
One worrying generic issue is that CD-i Emulator seems to have become slower as of late, for no good reason that I can think of. It may be that some debugging code somewhere is slowing things down, but I haven't found it yet. Or it could be that the titles I'm currently running are simple more demanding...
During testing, I also fixed a recent bug where mouse input and keyboard input interfere with each other. I also took out the use of the Shift key for button 2, as it is prone to generate false button presses when using Alt-Tab / Shift-Alt-Tab to switch between windows. You can still use the numeric "+" key for button 2, though, but that is too far away for generic use: I need an alternative closer to the space bar. Perhaps Backspace for button 2, and Esc for button 3 (buttons 1 and 2 together)?
Finally, I've started putting together the first v0.5.3 beta distribution. It will be mostly identical to a v0.5.2 one, with some updates in the sys directory, updated cdiroms.ini and cditypes.rul files and of course an updated executable which will be named wcdiemu-v053b1.exe (for beta 1) to avoid accidentally overwriting an existing v0.5.2 executable. I've also taken a first crack at a release notes document (very descriptively named BETA1).
Subscribe to:
Posts (Atom)