Monday, November 7, 2011

JNMS and Maxi-MMC updates

This weekend I fixed some disc emulation issues for the JNMS and Maxi-MMC boards. I had previously erroneously identified these two boards but they are different.

The JNMS board is the one in the CDI 180 player (also called the JNMS player). It is not used in any other player and contains a CDIC (CD Interface Controller) chip but no SLAVE processor.

The Maxi-MMC board is the one in the CDI 601 and 602 players. From the emulation point of view it is virtually identical to the Mini-MMC board used by the CDI 605 player, but it has a different CDIC chip version. Both boards contain a SLAVE processor.

The link between the JNMS and Maxi-MMC boards is the CDIC chip: both turn out to have the same older CDIC chip version that differs in a few details from the version used on the Mini-MMC and Mono-I boards players (I described these differences in the earlier “CD-i 180 disc playing” post).

I noticed the JNMS / Maxi-MMC link from the CD-i player type table in the July 1996 issue of The Interactive Engineer (it’s on the ICDIA site); turns out I had misinterpreted the Board column on page 4 (there’s also an error there: the 601/602 certainly do not have the 180 board!).

After noticing this I did some testing and it turns out that the CDIC modifications needed for the 180 also work for the 601, including the TOC reading problem.

I have yet to find a way to get chip version information from the CDIC chip itself, so for the time being I’ve keyed the differences on the SLAVE software version. The 180 has no such chip, the 601 has version 1.x where the 605 has version 3.x. For now I’ve assumed that version 2.x also uses the older CDIC chip, but that may be wrong (the 602 or 604 might be interesting test cases).

Having done that, I did some more digging into the TOC read issue. It turns out that the 601 ROM performs CRC validation on the subcode Q data from the lead-in area (which is where the TOC is stored), and CD-i Emulator didn’t provide a valid CRC (no other ROMs I’ve seen so far validate this in software). The ROM even has compensation for shifts of between 1 and 7 bits in the incoming subcode Q data, probably because of some hardware timing issue.

I also noticed a bug in the ROM: it always uses the first sector buffer because it takes the current buffer bit from the wrong memory location. Not that this really matters because the TOC data is repeated multiple times; half of the repetitions will always land in the first buffer anyway. The bug is fixed in the 605 ROM.

Generating a valid CRC turned out to be straightforward (it’s just a simple CRC-CCITT calculation), but the ROM wouldn’t recognize it! After some head scratching I focused on the ROXL instruction (Rotate Left with Extend) used in the validation code. It is quite an esoteric instruction; could it be that there was an emulation bug here? It turns out that there was indeed; during rotation the contents of the X flag where put in the wrong bit. After fixing this the ROM properly recognized the data and the TOC reads became just as quick as other player models.

In search of version information for the CDIC chip I looked at the emulations and found one potential point of interest: the release number displayed by the service shell. This is a special GUI shell that performs some service functions; you can get to it by inserting a specially wired test plug into input port 1.

After some digging I found that the service shell obtains this number from the SLAVE processor, so it probably does not directly correspond to a CDIC version. The number does appear to differ from other version numbers, though, at least on my two 605 players.

The service shell obtains this number using two special I$SetStt calls into the CDIC driver; extending CD-i Link to remotely perform these same calls was easy. The new -cds[tatus] option can now be used to make the special calls. Here's some representative output of the -cds A3 option:

CD status A3000000 -> A3320000

Extending CD-i Link with remote OS9 calls is actually a fairly easy way to perform some information and tracing actions; I will probably use it for sorting out other dangling issues in the near future. When possible, this technique avoids the problems of writing a fullblown memory-resident trace module.

A new public beta release of CD-i Emulator that has full JNMS and Maxi-MMC support (among other things) is scheduled before the end of this year; there are still a few other issues that need sorting out first. This release should also have better support for the PCDI board used by several portable players, including the CD-i 370.

The major player holes still remaining are the Sony IVO-10/11 players, the Kyocera player, the Bang&Olufsen TV/player combi and of course the I2M board. There is some perspective for all of these but they are not high priority; except for the latter I expect all of them to be minor hardware variations of existing boards.

The I2M board has the interesting feature that it has multiple "ROMs" downloaded from the PC software (which is available for download from ICDIA); it also has a very different way of reading from CD as this is handled by the PC. As a consequence of this, audio is probably also handled differently. I have this board booting to a blue screen where it hangs on host communication.

Tuesday, October 11, 2011

SCSI support and a big surprise

Last week I added SCSI disk support for the CD-i 60x extension board to CD-i Emulator. It took somewhat longer then I expected, though. This was mostly because the DP5380 SCSI controller chip exposes most low-level details of the SCSI protocol to the driver which means that all of these details have to be emulated.

The emulation ended up to be a more-or-less complete software implementation of the parallel SCSI-2 protocol, including most of the low-level signaling on the BSY, SEL, ATN, MSG, C/D-, I/O-, REQ and ACK lines. This is all implemented by the new CScsiBus class representing the SCSI bus that connects up to 16 instances of the CScsiPort class that each represent a single SCSI-2 bus interface. I was able to mostly avoid per-byte signaling of REQ and ACK if the target device implementation supports block transfers, a big performance win.

The new CCdiScsiDevice class emulates the DP5380 controller chip, working in conjunction with the CCdiScsiRamDevice and CCdiScsiDmaDevice classes that emulate the 32 KB of local extension SRAM and the discrete DMA logic around it that are included on the CD-i 60x extension board.

The CD-i 182 extension uses a compatible SCSI controller chip but a different DMA controller and has no local extension SRAM. I have not yet emulated these because I have almost no software to test it. 

The new CScsiDevice class implements a generic SCSI device emulating minimal versions of the four SCSI commands that are mandatory for all SCSI device types: TEST UNIT READY, REQUEST SENSE, INQUIRY and SEND DIAGNOSTIC. It implements most of the boiler-plate of low-level SCSI signaling for target devices and the full command and status phases of SCSI command processing, allowing subclasses to focus on implementing the content aspects of the data transfer phase.

The CScsiFile class emulates a SCSI device backed by a file on the host PC; it includes facilities for managing the SCSI block size and the transfer of block-sized data to and from the backing file.

The CScsiDisk and CScsiTape classes emulate a SCSI disk and tape device, respectively, currently supporting a block size of 512 bytes only. Instances of these classes are connected to the SCSI bus by using the new
-s[csi]d[isk][0-7] FILE and -s[csi]t[ape][0-7] FILE options of CD-i Emulator.

The CD-i 60x extension board normally uses SCSI id 5; the built-in ROM device descriptors for SCSI disks use SCSI ids starting at zero (/h0 /h1 /h2) while the built-in device descriptor for a SCSI tape uses SCSI id 4 (/mt0). This means that the useful options with the 60x are -scsidisk0, -scsidisk1, -scsidisk2 and -scsitape 4.

I've added the new dsk subdirectory to contain disk images; tape images have no standard location as they are mostly intended for bulk-transfer purposes (see below).

Inside the CD-i player this leads to the following response to the built-in inquire command:
$ inquire -i=0
vendor identification:"CDIFAN CDIEMU SCSIDISK "

$ inquire -i=4
vendor identification:"CDIFAN CDIEMU SCSITAPE "
where the "CDIFAN " part is the vendor name and the "CDIEMU SCSIXXXX " part is the product name.

In the previous post I described a 450 MB OS-9 hard disk image that I found on the Internet. After mounting it with
-scsidisk0 mw.dsk I got the following output:
$ free /h0
"MediaWorkshop" created on: Feb 17, 1994
Capacity: 1015812 sectors (512-byte sectors, 32-sector clusters)
674144 free sectors, largest block 655552 sectors
345161728 of 520095744 bytes (329.17 of 496.00 Mb) free on media (66%)
335642624 bytes (320.09 Mb) in largest free block

$ dir -d /h0

Directory of /h0 23:49:36
ASU/ AUDIO/ CDI_BASECASE/ CINERGY/ CMDS/
COPY/ CURSORS/ DEFS/ DEMOS/ ENET/
ETC/ FDRAW/ FONTS/ FontExample/ ISP/
LIB/ MAUI/ MAUIDEMO/ MENU/ MWOS/
NFS/ README_CIN README_MWS SCRIPT/ SHARE/
SHIP/ SYS/ T2D_RUNTIME/ TEMP/ TEMPMARK/
TEST/ USR/ VIDEO/ abstract.txt bibliographic.txt
bkgd.c8 bkgd.d cdb cdb1 cdb2
cdi_opt_install chris_test cin copyright.mws copyright.txt
csd_605 custominits_cin delme dos/ file
font8x8 get globs.mod go go.mkfont
inetdb ipstat kick1a_f.c8 kick2a_f.c8 mtitle
mws net new_shell new_shell.stb scratch
screen startup_cin thelist
You can see why thought it was a MediaWorkshop disc, but on closer inspection this turned out to something quite different. Some basic scrutiny lead to the hypothesis that this is probably a disk backup of someone from Microware working on early development of the DAVID (Digital Audio Video Interactive Decoder) platform. There are various surprises on the disk which I will describe below.

Anyway, I wanted to transfer the contents to the PC as a tar archive, similar to the procedure I used for my CD-i floppy collection. After starting CD-i Emulator with a -scsitape4 mw.tar option this was simply a matter of typing the following into the terminal window:
tar cb 1 /h0
This command runs the "tape archiver" program to create a tape with the contents of the /h0 directory, using a tape blocking size of 1 (necessary because my SCSI tape emulation doesn't yet support larger block sizes). The resulting mw.tar file on the PC is only 130 MB, not 450 MB which indicates that the disk is mostly empty. At some point I might use an OS-9 "undelete" program to find out if there are additional surprises.

Extracting the mw.tar file was now a simple matter of running the PC command
tar xvf mw.tar
This produced an exact copy of the OS-9 directory structure and files on the PC.

Many of the directories on the hard disk are clearly copies of various distribution media (e.g. CDI_BASECASE, CINERGY, CURSORS, ENET, FONTS, ISP, MWOS, NFS). The contents of the ENET, ISP and NFS directories at first appear to match some of my floppies, including version numbers, but on closer inspection the binaries are different. Running some of them produces "Illegal instruction" errors so I suspect that these are 68020 binaries.

The SHIP directory contains some prerelease RTNFM software; the readme talks about PES which is a type of MPEG-2 stream (Packetized Elementary Stream). Various asset directories contain versions of a "DAVID" logo.

The CMDS directory contains working versions of the Microware C compiler, identical to the ones I already had and also many other programs. It also contains some "cdb" files (configuration database?) that mention the 68340 processor.

The contents of the CMDS/BOOTOBJS directory produced a first surprise: it contains a subdirectory JNMS containing among others files named "rb1793" and "scsijnms". Could this be floppy and SCSI drivers for the CD-i 182 extension, as it contains with a 1793 floppy drive controller (the CD-i 60x uses a different one) and the player has a "JNMS" serial number?

Well, yes and no. Disassembly of the scsijnms file proved it to be compiled C code using an interface different from OS-9 2.4 drivers, so I suspect this is an OS-9 3.x driver. In any case, I cannot use it with the stock CD-i 180 player ROMs. Bummer...

And now for the big surprise: deeply hidden in a directory structure inside the innocently named COPY directory is the complete assembly source for the VMPEG video driver module "fmvdrv". At first glance it looked very familiar from my disassembly exercises on the identically-named Gate Array 2 MPEG driver module "fmvdrv", which is as expected because I had already noticed the large similarity between these two hardware generations.

The source calls the VMPEG hardware the "IC3" implementation, which matches CD-i digital video history as I know it. The Gate Array MPEG hardware would be "IC2" and the original prototype hardware would be "IC1". Furthermore, the sources contain three source files named fmvbugs1.a to fmvbugs3.a whose source file titles are "FMV first silicon bugs routines" to "FMV third silicon bugs routines". The supplied makefile currently uses only fmvbugs3.a as is to be expected for a VMPEG driver.

The fmvbugs1.a source contains some of the picture buffer manipulation logic that I've so far carefully avoided triggering because I couldn't understand it from my disassemblies, and this is now perfectly understandable: they are workarounds for hardware bugs!

As of two hours ago, I have verified that with a little tweaking and reconstruction of a single missing constants library file these sources produce the exact "fmvdrv" driver module contained in the vmpega.rom file directly obtained from my VMPEG cartridge.

In general these sources are very heavily commented, including numerous change management comments. They also include a full set of hardware register and bit names, although no comments directly describing the hardware. This should be of great help in finally getting the digital video emulation completely working.

All of the comments are English, although a few stray words and developer initials lead me to believe that the programmers were either Dutch or Belgian.

Disassembly comparisons lead me to the conclusion that careful undoing of numerous changes should result in exact sources for the GMPEGA2 driver module "fmvdrv" as well. I might even do it at some point, although this is not high priority for me.

The disk image containing all of these surprises is publicly available on the Internet since at least 2009, which is probably someone's mistake but one for which I'm very grateful at this point!

Wednesday, October 5, 2011

CD-i floppy inventory

Last weekend I future-proofed my CD-i floppy collection. A bit to my surprise, all floppies except one turned out to be perfectly readable (nearly twenty years after they were last written!). Luckily, the one exception was a backup copy so I didn’t lose any contents.

I had originally intended to use the borrowed CDI 182 unit for this (it has two floppy drives). The primary motivation for this was that my unstowed CDI 605 could not read beyond track zero of any floppy, but after giving the matter some thought I decided to try my other CDI 605 first, the primary motivation for this being speed (see below). It turned out that this 605 could read the floppies perfectly, including the three 38U0 format ones that gave problems on the 182 unit. Microware has defined a number of OS-9 disk formats for floppies, the 38U0 one supposedly being the “universal” 3.5" format (there is also a 58U0 “universal” 5¼" format).

The problem with the “universal” formats is that track zero can be (and on my floppies, is) in a different density which makes it a bad fit for most tools, both on CD-i and PC. It also means that only 79 tracks are used for data storage, giving a raw capacity of 79 × 2 × 16 × 256 = 632 KB. The 3803 format used by all my other CD-i floppies uses all 80 tracks and consequently has 8 KB more of raw storage for a total of 640 KB (these are both double-density, double-side formats (DS, DD) with 16 sectors of 256 bytes per track like nearly all OS-9 disk formats).

Before unstowing my other CDI 605 (it was nearly at the bottom of a 150 cm stowed equipment stack) I tried reading the floppies with my trusty old Windows 98 machine which still has floppy drives. I could not quickly find a DOS tool that handled the 256 byte sectors (not even raread and friends), although I suspect that Sydex’s TELEDISK product would have handled it just fine. I also tried Reischke’s OS9MAX which should handle all OS-9 formats under the sun according to its documentation. The demo version ran under MS-DOS and gave me working directory listings, even for the 38U0 floppies, but it does not support actually reading the files and I am somewhat doubtful about the current availability of the paid-for full version (even apart from cost concerns).

Why did I decide to use the 605? It was not a question of reading the disks (the 182 did this mostly fine) but of handling the data thus read. The 182 unit has a SCSI connector but I have no drivers for it (yet) and dumping my full floppy collection over the serial port did not really appeal to me for speed and reliability reasons (it could have been done, of course).

The 605 player has a SCSI connector and includes drivers for it so I could have just connected it to the SCSI disk in my E1 emulator and copied the floppies to hard disk (I would still have needed to transfer them to my laptop which would have been a two-step process via the Windows 98 PC as I have no SCSI connection on my laptop).

Instead I used the BNC network connector of the 605 to directly transfer floppy images to my laptop (it needs a network switch supporting both a BNC connector and the modern RJ45 connectors, but luckily I have two of those, even if they are only 10 Mbit/s). Starting up the network environment of the 605 took only two OS-9 commands at the command shell prompt:
ispmode /le0 addr=10.0.0.120
mbinstall
After this I could just ftp in to my laptop where I ran ftpdmin, a very minimal ftp server program, and transfer floppy disk images directly:
ftp 10.0.0.110
bin
put /d0@ floppy.dsk
(where /d0@ is the raw floppy device, for 38U0 I used /d0uv@, both are built-in for the 605).

The transfers ran at the maximum speed of the floppy drive (way below the 10 Mbit/s network speed), and the resulting .dsk files are perfectly readable using the –v option (virtual disk) of Carey Bloodworth’s os9.exe program even though that program was originally written for Tandy Color Computer OS9/6809 floppies (the floppy disk format was not changed for OS-9/68000 which is at the core of CD-i’s CD-RTOS operating system).

For easy access I also created a “tar” format archive of each floppy on a RAM disk:
chd /d0
tar cvf /r768/floppy.tar .
and ftp’d those to my laptop as well (the /r768 device is a 768 KB variation of the /r512 built-in 512 KB RAM disk device of the 605 player).

I ended up with the following collection of unique floppy disk images:
  • 605h3 - 605 H3 Driver Update (1 floppy)
  • 605upd - 605 Driver Update (1 floppy)
  • bcase - Basecase Tests (1 floppy)
  • eboot41 - Emulation Boot Diskette (1 floppy)
  • eburn41 - Emulation and CDD 521 Boot Diskette (1 floppy)
  • inet - CD-I Internet Installation Disk - V1.3 (1 floppy)
  • nfs - OS-9/68000 Network File System V.1.0 (1 floppy)
  • os9sys - OS-9 System Diskette (1 floppy)
  • pubsoft - OptImage Public Domain Software (2 floppies)
  • pvpak - OptImage Preview Pak Installation Disk (1 floppy)
  • ubridge - OS-9 UniBridge Resident Utilities (3 floppies)

The 605* and eb* floppies are mostly interesting for CD-i 605 or E1 emulator owners, but the bcase floppy contains a set of CD-i standard conformance test programs that.

The inet and nfs floppies contain a full set of Internet software including Telnet and FTP servers and clients and an NFS client (all except the latter are also in the 605 ROMs).

The os9sys floppy contains a full set of Professional OS-9 programs and is my original source for most of the OS-9 CD-i disc that I described earlier (most of these are not in ROM on any CD-i player that I’ve seen so far).

The pubsoft floppies contain miscellanous utilities such as bfed, du, kermit, umacs and vi, most of which can be obtained elsewhere, some CD-i specific utilities such as da (CD-i disk analyzer) and iffinfo (CD-i IFF file dumper) as well as library source files for the CD-i IFF file library.

The pvpak floppy contains preview software for CD-i images that will preview CD-i IFF files from an NFS-mounted host file system directory.

The ubridge floppies are the goldmine (and also the 38U0 format ones) as they contain a full set of native Microware C compiler/assembler/linker/debugger software for OS-9 complete with CD-i header files and libraries and C runtime startup sources. Both the srcdbg and sysdbg debuggers are included as well as the rdump utility for dumping ROFF (Relocatable Object File Format) files.

Unfortunately, most of the above software except for the pubsoft contents is copyrighted property of Microware (now Radisys) or OptImage (a former Philips/Microware joint venture) which means that I cannot distribute it, even though they could be very useful to CD-i homebrew developers. For that the hopefully soon-to-be available GCC cross-port will have to be enough...

While investigating all of the above I also stumbled upon a 450 MB OS-9 hard disk image for MediaWorkshop. The os9.exe program recognizes it just enough to say that it does not support it so I have no real idea about its contents except the obvious.

To remedy that problem I’m in the process of adding SCSI disk support to CD-i emulator so that I can use the SCSI support in the CD-i 605 ROMs to mount the disk image and look at it. This should also allow the CD-i 180 to boot from a SCSI disk if I ever find drivers for it (a possible path to that has just appeared, we’ll see...).

Friday, September 30, 2011

CD-i 180 experimentation

Early this week, CDinteractive.co.uk forum user Erroneous came by and we spent an interesting evening taking apart our CDI 18x units and figuring out serial ports.

Whereas my set consists of a CDI 180/37 and a CDI 181/37 unit, his set is the full 180/181/182 ensemble with the added bonus of supporting 220V power. I was not previously aware that such units even existed, but it turns out he has a 180/20 + 181/20 + 182/00 combination.

I’ve taken some photographs from his set, both intact and in various dismantled states, and these can be found here on the CD-i Emulator website. Nothing particularly surprising except for the small size ROM in the 182 unit, it’s only a pair of 27C512 chips which hold 32 KB each for a total of only 64 KB!

Erroneous sold me his spare CD-i 180 remote unit and serial port adapter so I now have a mostly functioning CD-i 180 set. Unfortunately, it turns out that my 180 CD drive unit has problems so I cannot play actual discs, but the set works fine using the E1 Emulator.

It turned out that his set, however, has some defect in the 181 MMC unit which prevents it from reading discs, either from the 180 CD drive or from the E1 Emulator. Using my 181 and his 180 and 182 units we managed to get a fully working set, albeit running on mixed 120V / 220V wall power!

Because at first we couldn’t get a working command prompt on the serial port of his 182 unit, he undertook to solder a spare DB9 connector to the 181/182 interconnection bus, based on the pinout of the serial adapter which attaches to that same bus (which matches the pinout I had previously figured out by tracing the circuit board). This gave output but not a working command prompt either.

It finally turned out to be a feature of the OS9 System Disk that we were using; it boots properly when you select “Floppy Application” from the “System” menu, but it’s final step starts a command prompt for the /term device and it turns out there is no such device in the 180 player. It has three (!) serial port devices but they are named /t0, /t01 and /t2 (see below) whereas the 60x players for which this disk was apparently intended do have a /term device. On the 180, avoiding the startup script by choosing “System” / “CD-RTOS” works fine however.

When we figured this out, we could get a command line prompt on either serial port, the ROMs are smart enough to select the device where a terminal is actually connected.

We confirmed that the serial I/O chip in the 182 unit is indeed an 68681 chip as I previously suspected, which supports two serial port devices of which only one has a connector on the outside of the unit. The connected device is supported with the /t0 device name, the unconnected one uses /t01. In addition to the 68070 built-in serial port this means that the 181+182 combination actually has three serial ports, but the usual hardware setup makes only one of them accessible at a time (connecting the units uses up the interconnection bus which means that the serial adapter cannot be connected at the same time).

At this point, it was getting late and Erroneous departed for home, graciously allowing me to temporarily borrow his 18x set for some more experimentation and dumping.

When the serial port allowed me to take look inside the running 180 player, it turned out that the four ROMs that I previously dumped were not in fact co-located in the address space. The “system” ROM pair lives at $180000 as expected but on Maxi-MMC it is only 256 KB; the other 256 KB ROM pair lives at $700000 (I’ve called it the “asset” ROM because it contains only a font and pictures). Leaving out the asset ROM inside CD-i Emulator gives a working player but without any background images or buttons, just the text over a black background. You can still start a CD-i application of play a CD-Audio disc, though...

Another small factoid is that the 182 ROM contains a single picture ps_child.dyuv that at first appears to be a revised version of the identically-named one in the 181 ROM, but both pictures are bitwise identical except for the module edition number and CRC. Weird...

Dumping the ROMs of Erroneous’s 181 set turned up nothing new; they are bitwise identical to the ones from my own unit (not really surprising as both units have big “1.1” stickers on the back which signifies the “final” ROM update that all Philips 18x players received shortly before the market introduction of CD-i).

Having the 182 unit ROMs I have now extended CD-i Emulator to also support the two additional serial ports, even though the second of these is not usable on the actual player! The floppy controller and the parallel and SCSI port remain for the future.

Later this week I also took apart my new CD-i 180 remote unit, which can be used over infrared but also supports a cable connection (I’ll need to make my own cable). Pictures of this are on my site here. I suspected that the interconnection would use the I2C protocol and this indeed turned out to be the case. The unit contains another 84C21 mask-programmable microprocessor labeled “REMOCON Ver. 2.0” and its I2C SDA and SCLK pins are more or less directly connected to the cable connector, which also has RESET, GND and +5V power connections. This should allow me to connect any home-brewn pointing device over I2C.

From a bit of running system and driver inspection I also found out some more details about the bus locations of the floppy and SCSI controller chips in the 182 unit. There are two surprisingly empty ROM sockets on the SCSI extension board that are probably intended for SCSI driver and application software; except for booting support the other ROMs contain none of this.

With the information learned so far I have expanded the cditypes.rul file with CD-i 180 ROM recognition and put it in the CD-i Types section of the site.

Having two working floppy drives also allowed me to review my CD-i floppy collection and most of those appear to be perfectly readable; they may yet turn out to contain something interesting.

Saturday, September 24, 2011

CD-i 180 disc playing

After a few hours of tweaking I’ve gotten CD-i 180 disc playing working in CD-i Emulator. Most of the time was spent on basic disassembly of the driver, which is very similar to the CD-i 605 version (but missing some features such as multi-session support and seek delay). Not very surprising as both players use the same CDIC chip, although they are supposedly different versions.

The problems were really quite trivial but there are of course complications...

The first problem turned out to be a memory map issue. On all CD-i players that I’ve encountered so far, a memory map chunk size of 128 KB works fine because memory from different devices (whether general purpose RAM or device-specific memory such as NVRAM and CDIC local memory) never lives inside the same 128 KB chunk of address space. On the CD-i 180 this is not true, as can be seen from the following fragment of maximmc.brd:
$00300000   cdic.dev     level=4
$00310000 nvr.dev
My code silently overwrote the CDIC mapping with the NVR mapping which meant that any access to CDIC local memory terminated in a bus error. Needless to say, this was not conducive to disc playback. Simply lowering the memory map chunk size to 64 KB fixed this problem.

Next it turned out that during initialization the cdap18x driver wants to do some kind of data transfer to address $320001, possibly DSP parameters or something, and there was no device there yet. This was easily fixed also.

At this point the disc play got started but the interrupt routine ignored all interrupts. After inspecting the driver it turned out that the CD-i 180 expects an additional status bit to be set in the DBUF register that was never used on other CD-i players with a CDIC. Still easy to fix.

Now the interrupt routine started trying to process the sector but it got bus errors in two places. It attempted to read the FILE selection register, which other CD-i players never did, and it took the buffer index bits (these specify the DMA buffer the hardware used from the sector) from a previously unimplemented register that I’ve dubbed DSEL for now. Easy fixes galore...

After this one sector got properly transferred to memory but an unending stream of interrupts hung up the player. It turns out that the CD-i 180 ROMs don’t read the CDIC XBUF register that CD-i Emulator used to clear the interrupt. Generating and clearing the interrupt from the DBUF register fixed that also, and now the disc actually started playing!

Of course I did some tests next and there are currently three known problems:

- Reads of the disc table-of-contents (TOC) take ages, probably because the ROMs are checking for or waiting on some flag or data byte that CD-i Emulator currently doesn’t provide. This should not be very hard to fix.

- Audio playback from memory doesn’t work, the symptoms are very similar to those of the audio sync issues that occurred during the MESS collaboration in 2009. This may be a hard problem to crack.

- And finally the most hairy issue: the CDIC emulation modifications break other players. The breakage is often subtle but very definitely there so I will have to do some careful juggling to get it working on all players, if necessary by explicit version checking but I hope to avoid that. The hardware should not be that dissimilar, really...

On the serial port and pointing device front, I’ve found someone who has a surplus serial port interface and an infrared remote. He also has a complete CD-i 180 set (including the CDI 182 unit) and I’m looking forward to extracting the ROMs from that one also.

He also took apart his mouse and it seems to be a fairly standard Amiga design which means that it should not be very hard to manufacture a replacement.

I’ve been looking into making a small PIC-based interface for PS2 and/or CD-i peripherals and it may still come to that but it seems to be not as necessary as I thought.

CD-i 180 internals

In the previous post I promised some ROM and chip finds. Well, here goes. To understand some of the details, you'll need some microprocessor and/or digital electronics knowledge, but even without that the gist of the text should be understandable.

The CDI 181 MMC unit contains the so-called Maxi-MMC board that is not used in any other CD-i player. Its closest cousin is the Mini-MMC board that is used in the CD-i 605 and CD-i 220 F1 players (a derivative of it is used in the CD-i 350/360 players).

The Mini-MMC board uses two 68HC05 slave processors for CD and pointing device control (they are usually called SERVO and SLAVE). The Maxi-MMC board does not have these chips, but it does have two PCF80C21 slave processors labeled RSX and TRANSDUCER that perform similar functions.

From their locations on the board I surmise that the RSX performs CD control functions; I know for sure that the TRANSDUCER performs only pointing device control. The latter is connected to the main 68070 processor via an I2C bus (I've actually traced the connections); I'm not completely sure yet about the RSX.

In order to emulate the pointing devices in CD-i Emulator, I had to reverse engineer the I2C protocol spoken by the TRANSDUCER chip; this was mostly a question of disassembling the "ceniic" and "periic" drivers in the ROM. The first of these is the low-level driver that serves as the common control point for the I2C bus; the second is the high-level driver that is instantiated separately for each type of pointing device. The ROMs support three such devices: /cdikeys, /ptr and /ptr2, corresponding to the player control keys and first and second pointing devices (the first pointing device is probably shared between the infrared remote sensor and the left pointing device port). Both pointing devices support absolute (e.g. touchpad) as well as relative (e.g. mouse) positioning.

Note that there is no built-in support for a CD-i keyboard or modem (you could use a serial port for this purpose).

However, knowing the I2C protocol does not tell me the exact protocol of the pointing devices, which therefore brings me no closer to constructing a "pointing device" that works with the two front panel MiniDIN-9 connectors. Note that these connectors are physically different from the MiniDIN 8 pointing device connectors used on most other CD-i players. According to the Philips flyers, these connectors have 6 (presumably digital) input signals and a "strobe" (STB) output signal. From the signal names I can make some educated guesses about the probable functions of the signals, but some quick tests with the BTN1 and BTN2 inputs did not pan out and it could be too complicated to figure out without measurement of a connected and working pointing device.

There is, however, also an infrared remote sensor that is supposed to expect the RC5 infrared signal protocol. This protocol supports only 2048 separate functions (32 groups of 64 each) so it should not be impossible to figure out, given a suitably programmable RC5 remote control or in the best case a PC RC5 adapter. I've been thinking about building one of the latter.

There is also a third possibility of getting a working pointing device. Although the case label of the front MiniDIN 8 connecter is "CONTROL", the Philips flyers label it "IIC" which is another way of writing "I2C", although they don't give a pinout of the port. It seems plausible that the connector is connected to the I2C bus of the 68070, although I haven't been able to verify that yet (the multimeter finds no direct connections except GND, so some buffering must be involved). If there is indeed a connection, I would be able to externally connect to that bus and send and receive the I2C bus commands that I've already reverse engineered.

If even this doesn't work, I can always connect directly to the internal I2C bus, but that involves running three wires from inside the player to outside and I'm not very keen on that (yet, anyway).

Now, about the (so far) missing serial port. There is a driver for the 68070 on-chip UART in the ROMs (the u68070 driver which is accessible via the /t2 device), and the boot code actually writes a boot message to it (CD-i Emulator output):
  PHILIPS CD-I 181 - ROM version 23rd January, 1992.
Using CD_RTOS kernel edition $53 revison $00
At first I thought that the UART would be connected to the "CONTROL" port on the front, but that does not appear to be the case. Tonight I verified (by tracing PCB connections with my multimeter) that the 68070 serial pins are connected to the PCB connector on the right side (they go through a pair of SN75188/SN75189 chips and some protection resistors; these chips are well-known RS232 line drivers/receivers). I even know the actual PCB pins, so if I can find a suitable 100-pins 0.01" spaced double edge print connector I can actually wire up the serial port.

Now for the bad news, however: the ROMs do not contain a serial port download routine. They contain a host of other goodies (more below) but not this particular beast. There is also no pointing device support for this port, contrary to all other players, so connecting up the serial port would not immediately gain me anything, I still need a working pointing device to actually start a CD-i disc…

There are no drivers for other serial ports in the ROMs, but the boot code does contain some support for a UART chip at address $340001 (probably a 68681 DUART included in the CDI 182 unit which I don't have). The support, however, is limited to the output of boot messages although the ROMs will actually prefer this port over the 68070 on-chip device if they find it.

As is to be expected from a development and test player, there is an elaborate set of boot options, but they can only be used if the ROMs contain the signature "IMS-TC" at byte offset $400 (the ROMs in my player contains FF bytes at these locations). And even then the options prompt will not appear unless you press the space bar on your serial terminal during player reset.

However, adding a -bootprompt option that handles both the signature and the space bar press to CD-i Emulator was not hard, and if you use that option with the 180 ROMs the following appears when resetting the player:
  PHILIPS CD-I 181 - ROM version 23rd January, 1992.

A-Z = change option : <BSP> = clear options : <RETURN> = Boot Now

Boot options:- BQRS
As specified, you can change the options by typing letters and pressing Enter will start the boot process with the specified options.

From disassembling the boot code of the ROMs I've so far found the following options:

D = Download/Debug
F = Boot from Floppy
L = Apply options and present another options prompt (Loop)
M = Set NTSC Monitor mode
P = Set PAL mode
S = Set NTSC/PAL mode from switch
T = Set NTSC mode
W = Boot from SCSI disk (Winchester)

It could be that there's also a C option, and I've as yet not found any implementations of the Q and R options that the ROMs include in the default, but they could be hidden in OS-9 drivers instead of the boot code.

Once set, the options are saved in NVRAM at address $313FE0 as default for prompts during subsequent reboots, they are not used for reboots where the option prompt is not invoked.

Options D, F and W look interesting, but further investigation leads to the conclusion that they are mostly useless without additional hardware.

Pressing lower-case D followed by Enter / Enter results in the following:
Boot options:- BQRSd
Boot options:- BDQRS
Enter size of download area in hex - just RETURN for none
called debugger

Rel: 00000000
Dn: 00000000 0000E430 0007000A 00000000 00000000 00000001 FFFFE000 00000000
An: 00180B84 00180570 00313FE0 00410000 00002500 00000500 00001500 000014B0
SR: 2704 (--S--7-----Z--) SSP: 000014B0 USP: 00000000
PC: 00180D2E - 08020016 btst #$0016,d2
debug:
One might think that entering a download size would perform some kind of download (hopefully via the serial port) but that is not the case. The "download" code just looks at location $2500 in RAM that's apparently supposed to be already filled (presumably via an In-Circuit Emulator or something like it).

However, invoking the debugger is interesting in itself. It looks like the Microware low-level RomBug debugger that is described in the Microware documentation, although I haven't found it in any other CD-i players. One could "download" data with the change command:
debug: c0
00000000 00 : 1
00000001 00 : 2
00000002 15 : 3
00000003 00 :
Not very userfriendly but it could be done. The immediate catch is that it doesn't work with unmodified ROMs because of the "IMS-TC" signature check!

Trying the F option results in the following:
Boot options:- BQRSf
Boot options:- BFQRS
Booting from Floppy (WD 179x controller) - Please wait
This, however, needs the hardware in the CDI 182 set (it lives at $330001). I could emulate that in CD-i Emulator of course, but there's no real point at this time. It is interesting to note that the floppy controller in the CD-i 605 (which I haven't emulated either at this point) is a DP8473 which is register compatible with the uPD765A used in the original IBM PC but requires a totally different driver (it also lives at a different memory address, namely $282001).

Finally, trying the W options gives this:
Boot options:- BQRSw
Boot options:- BQRSW
Booting from RODIME RO 650 disk drive (NCR 5380 SCSI) - Please wait
Exception Error, vector offset $0008 addr $00181908
Fatal System Error; rebooting system
The hardware is apparently supposed to live at $410000 and presumably emulatable; it's identical or at least similar to the DP5380 chip that is found on the CD-i 605 extension board where it lives at $AA0000).

Some other things that I've found out:

The CDI 181 unit has 8 KB of NVRAM but it does not use the M48T08 chip that's in all other Philips players, it's just a piece of RAM that lives at $310000 (even addresses only) and is supported by the "nvdrv" driver via the /nvr device.

In the CD-i 180 player the timekeeping functions are instead performed by a RICOH RP5C15 chip, the driver is appropriately called "rp5c15".

And there is a separate changeable battery inside the case; no "dead NVRAM" problems with this player! I don't know when the battery in my player was last changed but at the moment it's still functioning and had not lost the date/time when I first powered it on just over a week ago.

The IC CARD slot at the front of the player is handled like just another piece of NVRAM; it uses the same "nvdrv" driver but a different device: /icard. According to the device descriptor it can hold 32 KB of data, I would love to have one of those!

Monday, September 19, 2011

CD-i 180 adventures

Over the last week I have been playing with the CD-i 180 player set. There’s lots to tell about, so this will be a series of blog posts, this being the first installment.

The CD-i 180 is the original CD-i player, manufactured jointly by Philips and Sony/Matsushita, and for a score of years it was the development and “reference” player. The newer CD-i 605 player provided a more modern development option but it did not become the “reference” player for quite some years after its introduction.

The CD-i 180 set is quite bulky, as could be expected for first-generation hardware. I have added a picture of my set to the Hardware section of the CD-i Emulator website; more fotos can be found here on the DutchAudioClassics.nl website (it’s the same player, as evidenced by the serial numbers).

The full set consists of the CDI 180 CD-i Player module, the CDI 181 Multimedia Controller or MMC module and the CDI 182 Expansion module. The modules are normally stacked on top of each other and have mechanical interlocks so they can be moved as a unit. Unfortunately, I do not have the CDI 182 Expansion module nor any user manuals; Philips brochures for the set can be found here on the ICDIA website.

Why am I interested in this dinosaur? It’s the first mass-produced CD-i player (granted, for relatively small masses), although there were presumably some earlier prototype players. As such, it contains the “original” hardware of the CD-i platform, which is interesting from both a historical and an emulation point of view.

For emulation purposes I have been trying to get hold of CD-i 180 ROMs for some years, there are several people that still have fully operational sets, but it hasn’t panned out yet. So when I saw a basic set for sale on the CD-Interactive forum I couldn’t resist the temptation. After some discussion and a little bartering with the seller I finally ordered the set about 10 days ago. Unfortunately, this set does not include a CDI 182 module or pointing device.

I had some reservations about this being a fully working set, but I figured that at least the ROM chips would probably be okay, if nothing else that would allow me to add support for this player type to CD-i Emulator.

In old hardware the mechanical parts are usually the first to fail, this being the CDI 180 CD-i Player module (which is really just a CD drive with a 44.1 kHz digital output “DO” signal). A workaround for this would be using an E1 or E2 Emulator unit; these are basically CD drive simulators that on one side read a CD-i disc image from a connected SCSI hard disk and on the other side output the 44.1 kHz digital output “DO” signal. Both the CDI 180 and E1/E2 units are controlled via a 1200 baud RS232 serial input “RS” signal.

From my CD-i developer days I have two sets of both Emulator types so I started taking these out of storage. For practical reasons I decided to use an E1 unit because it has an internal SCSI hard disk and I did not have a spare one lying around. I also dug out an old Windows 98 PC, required because the Philips/OptImage emulation software doesn’t work under Windows XP and newer, and one of my 605 players (I also have two of those). Connecting everything took me a while but I had carefully stored all the required cables as well and after installing the software I had a working configuration after an hour or so. The entire configuration made quite a bit of mechanical and fan noise; I had forgotten this about older hardware!

I had selected the 605 unit with the Gate Array AH02 board because I was having emulation problems with that board, and I proceeded to do some MPEG tests on it. It turns out the hardware allows for some things that my emulator currently does not, which means that I need to do some rethinking. Anyway, on with the 180 story.

In preparation for the arrival of the 180 set I next prepared an disc image of the “OS-9 Disc” that I created in November 1993 while working as a CD-i developer. This disc contains all the OS-9 command-line programs from Professional OS-9, some OS-9 and CD-i utilities supplied by Philips and Microware and some homegrown ones as well. With this disc you can get a fully functional command-line prompt on any CD-i player with a serial port, which is very useful while researching a CD-i player’s internals.

The Philips/Optimage emulation software requires the disc image files to include the 2-second gap before logical block zero of the CD-i track, which is not usually included in the .bin or .iso files produced by CD image tools. So I modified the CD-i File program to convert my existing os9disc.bin file by prepending the 2-second gap, in the process also adding support for scrambling and unscrambling the sector data.

Scrambling is the process of XORing all data bytes in a CD-ROM or CD-i sector with a “scramble pattern” that is designed to avoid many contiguous identical data bytes which can supposedly confuse the tracking mechanism of CD drives (or so I’ve heard). It turned out that scrambling of the image data was not required but it did allow me to verify that the CD-I File converted image of a test disc is in fact identical to the one that the Philips/Optimage mastering tools produce, except for the ECC/EDC bytes of the gap sectors which CD-I File doesn’t know how to generate (yet). Fortunately this turned out not to be a problem, I could emulate the converted image just fine.

Last Thursday the 180 set arrived and in the evening I eagerly unpacked it. Everything appeared to be in tip-top shape, although the set had evidently seen use.

First disappointment: there is no serial port on the right side of 181 module. I remembered that this was actually an option on the module and I had not even bothered to ask the seller about it! This would make ROM extraction harder, but I was not completely without hope: the front has a Mini-DIN 8 connector marked “CONTROL” and I fully expected this to be a “standard” CD-i serial port because I seemed to remember that you could connect standard CD-i pointing devices to this port, especially a mouse. The built-in UART functions of the 68070 processor chip would have to be connected up somewhere, after all.

Second disappointment: the modules require 120V power, not the 220V we have here in Holland. I did not have a voltage converter handy so after some phone discussion with a hardware-knowledgeable friend we determined that powering up was not yet a safe option. He gave me some possible options depending on the internal configuration so I proceeded to open up the CDI 181 module, of course also motivated by curiosity.

The first thing I noticed was that there were some screws missing; obviously the module had been opened before and the person doing it had been somewhat careless. The internals also seemed somewhat familiar, especially the looks of the stickers on the ROM chips and the placement of some small yellow stickers on various other chips.

Proceeding to the primary reason for opening up the module, I next checked the power supply configuration. Alas, nothing reconfigurable for 220V, it is a fully discrete unit with the transformer actually soldered to circuit board on both input and output side. There are also surprisingly many connections to the actual MMC processor board and on close inspection weird voltages like –9V and +9V are printed near the power supply outputs, apart from the expected +5V and +/–12V, so connecting a different power supply would be a major undertaking also.

After some pondering of the internals I closed up the module again and proceeded to closely inspect the back side for serial numbers, notices, etc. They seemed somewhat familiar but that isn’t weird as numbers often do. Out of pure curiosity I surfed to the DutchAudioClassics.nl website to compare serial numbers, wanting to know the place of my set in the production runs.

Surprise: the serial numbers are identical! It appears that this exact set was previously owned by the owner of that website or perhaps he got the photographs from someone else. This also explained why the internals had seemed familiar: I had actually seen them before!

I verified with the seller of the set that he doesn’t know anything about the photographs; apparently my set has had at least four owners, assuming that the website owner wasn’t the original one.

On Friday I obtained a 120V converter (they were unexpectedly cheap) and that evening I proceeded to power up the 180 set. I got a nice main menu picture immediately so I proceeded to attempt to start a CD-i disc. It did not start automatically when I inserted it, which on second thought makes perfect sense because the 181 MMC module has no way to know that you’ve just inserted a disc: this information is not communicated over 180/181 interconnections. So I would need to click on the “CD-I” button to start a disc.

To click on a screen button you need a supported pointing device, so I proceeded to connect the trusty white professional CD-i mouse that belongs with my 605 players. It doesn’t work!

There are some mechanical issues which make it doubtful that the MiniDIN connector plugs connect properly, so I tried an expansion cable that fit better. Still no dice.

The next step was trying some other CD-i pointing devices, but none of them worked. No pointing devices came with the set, and the seller had advised me thus (they were presumable lost or sold separately by some previous owner). The only remaining option seemed to be the wireless remote control sensor which supposedly uses RC5.

I tried every remote in my home, including the CD-i ones, but none of them give any reaction. After some research into the RC5 protocol this is not surprising, the 180 set probably has a distinct system address code. Not having a programmable remote handy nor a PC capable of generating infrared signals (none of my PCs have IrDA) I am again stuck!

I spent some time surfing the Internet looking for RC5 remotes and PC interfaces that can generate RC5 signals. Programmable remotes requiring a learning stage are obviously not an option so it will have to be a fully PC-programmable remote which are somewhat expensive and I’m not convinced they would work. The PC interface seems the best option for now; I found some do-it-yourself circuits and kits but it is all quite involved. I’ve also given some thought to PIC kits which could in principle also support a standard CD-i or PC mouse or even a joystick, but I haven’t pursued these options much further yet.

Next I went looking for ways to at least get the contents of the ROM chips as I had determined that these were socketed inside the MMC module and could easily be removed. There are four 27C100 chips inside the module, each of which contains 128Kb of data for a total of 512Kb which is the same as for the CD-i 605 player (ignoring expansion and full-motion video ROMs). The regular way to do this involves using a ROM reading device, but I haven’t gotten one handy that supports this chip type and neither does the hardware friend I mentioned earlier.

I do have access to an old 8 bit Z80 hobbyist-built system capable of reading and writing up to 27512 chips which are 64Kb, it is possible to extend this to at least read the 27C100 chip type. This would require adapting the socket (the 27512 is 28 pins whereas the 27C100 has 32 pins) and adding one extra address bit, if nothing else with just a spare wire. But the Z80 system is not at my house and some hardware modifications to it would be required, for which I would have to inspect the system first and dig up the circuit diagrams; all quite disappointing.

While researching the chip pinouts I suddenly had an idea: what if I used the CD-i 605 Expansion board which also has ROM sockets? This seemed an option but with two kids running around I did not want to open up the set. That evening however I took the board out of the 605 (this is easily done as both player and board were designed for it) and found that this Expansion board contains two 27C020 chips, each containing 256Kb of data. These are also 32 pins but the pinouts are a little different, so a socket adapter would also be needed. I checked the 605 technical manual and it did not mention anything about configurable ROM chip types (it did mention configurable RAM chip types, though) so an adapter seemed the way to go. I collected some spare 40 pin sockets from storage (boy have I got much of that) and proceeded to open up the 180 set and take out the ROM chips.

When determining the mechanical fit of the two sockets for the adapter I noticed three jumpers adjacent to the ROM sockets of the expansion board and I wondered… Tracing of the board connections indicated that these jumpers were indeed connected to exactly the ROM socket pins differing between 27C100 and 27C020, and other connections indicated it at least plausible for these jumpers to be exactly made for the purpose.

So I changed the jumpers and inserted one 180 ROM. This would avoid OS-9 inadvertently using data from the ROM because only half of each 16-bit word would be present, thus ensuring that no module headers would be detected, and in the event of disaster I would lose only a single ROM chip (not that I expected that to be very likely, but you never know).

Powering up the player worked exactly as expected, no suspicious smoke or heat generation, so the next step was software. It turns out that CD-i Link already supports downloading of ROM data from specific memory addresses and I had already determined those addresses from the 605 technical manual. So I connected the CD-i 605 null-modem cable with my USB-to-Serial adapter between CD-i player and my laptop and fired off the command line:

cdilink –p 3 –a 50000 –s 256K –u u21.rom

(U21 being the socket number of the specific ROM I chose first).

After a minute I aborted the upload and checked the result, and lo and behold the u21.rom file looked like an even-byte-only ROM dump:
00000000  4a00 000b 0000 0000 0004 8000 0000 0000 J...............
00000010 0000 0000 0000 003a 0000 705f 6d6c 2e6f .......:..p_ml.o
00000020 7406 0c20 0000 0000 0101 0101 0101 0101 t.. ............
This was hopeful, so I restarted the upload again and waited some six minutes for it to complete. Just for sure I redid the upload from address 58000 and got an identical file, thus ruling out any flakey bits or timing problems (I had already checked that the access times on the 27C100 and 27C020 chips were identical, to say 150ns).

In an attempt to speed up the procedure, I next attempted to try two ROMs at once, using ones that I thought not to be a matched even/odd set. The 605 would not boot! It later turned out that the socket numbering did not correspond to the even/odd pairing as I expected so this was probably caused by the two ROMs being exactly a matched set and OS-9 getting confused as the result. But using a single ROM it worked fine.

I proceeded to repeat the following procedure for the next three ROMs: turn off the 605, remove the expansion board, unsocket the previous ROM chip, socket the next ROM chip, reinsert the expansion board, turn on the 605 and run CD-i Link twice. It took a while, all in all just under an hour.

While these uploads were running I wrote two small programs rsplit and rjoin to manipulate the ROM files into a correct 512Kb 180 ROM image. Around 00:30 I had a final cdi180b.rom file that looked good and I ran it through cditype –mod to verify that it indeed looked like a CD-I player ROM:
  Addr     Size      Owner    Perm Type Revs  Ed #  Crc   Module name
-------- -------- ----------- ---- ---- ---- ----- ------ ------------
0000509a 192 0.0 0003 Data 8001 1 fba055 copyright
0000515a 26650 0.0 0555 Sys a000 83 090798 kernel
0000b974 344 0.0 0555 Sys 8002 22 b20da9 init
0000bacc 2848 0.0 0555 Fman a00b 35 28611f ucm
0000c5ec 5592 0.0 0555 Fman a000 17 63023d nrf
0000dbc4 2270 0.0 0555 Fman a000 35 d6a976 pipeman
0000e4a2 774 0.0 0555 Driv a001 6 81a3e9 nvdrv
0000e7a8 356 0.0 0555 Sys a01e 15 e69105 rp5c15
0000e90c 136 0.0 0555 Desc 8000 1 f25f23 tim070
0000e994 420 0.0 0555 Driv a00c 6 7b3913 tim070driv
0000eb38 172 0.0 0555 Driv a000 1 407f81 null
0000ebe4 102 0.0 0555 Desc 8000 2 cf450e pipe
0000ec4a 94 0.0 0555 Desc 8000 1 f54010 nvr
0000eca8 96 0.0 0555 Desc 8000 1 17ec68 icard
0000ed08 1934 0.0 0555 Fman a000 31 b41f17 scf
0000f496 120 0.0 0555 Desc 8000 61 dd8776 t2
0000f50e 1578 0.0 0555 Driv a020 16 d0a854 u68070
0000fb38 176 0.1 0777 5 8001 1 a519f6 csd_mmc
0000fbe8 5026 0.0 0555 Sys a000 292 e33cc5 csdinit
00010f8a 136 0.0 0555 Desc 8000 6 041e2b iic
00011012 152 0.0 0555 Driv a02c 22 e29688 ceniic
000110aa 166 0.0 0555 Desc 8000 8 c5b823 ptr
00011150 196 0.0 0555 Desc 8000 8 a0e276 cdikeys
00011214 168 0.0 0555 Desc 8000 8 439a33 ptr2
000112bc 3134 0.0 0555 Driv a016 11 faf88d periic
00011efa 4510 0.0 0555 Fman a003 96 a4d145 cdfm
00013098 15222 0.0 0555 Driv a038 28 122c79 cdap18x
00016c0e 134 0.0 0555 Desc 8000 2 35f12f cd
00016c94 134 0.0 0555 Desc 8000 2 d2ce2f ap
00016d1a 130 0.0 0555 Desc 8000 1 1586c2 vid
00016d9c 18082 10.48 0555 Trap c00a 6 5f673d cio
0001b43e 7798 1.0 0555 Trap c001 13 46c5dc math
0001d2b4 2992 0.0 0555 Data 8020 1 191a59 FONT8X8
0001de64 134 0.0 0555 Desc 8000 2 c5ed0e dd
0001deea 66564 0.0 0555 Driv a012 48 660a91 video
0002e2ee 62622 0.1 0555 Prog 8008 20 ec5459 ps
0003d78c 4272 0.0 0003 Data 8001 1 9f3982 ps_medium.font
0003e83c 800 0.0 0003 Data 8002 1 c1ac25 ps_icons.clut
00040000 2976 0.0 0003 Data 8002 1 0a3b97 ps_small.font
00040ba0 7456 0.0 0003 Data 8002 1 764338 ps_icons.clu8
000428c0 107600 0.0 0003 Data 8002 1 7b9b4e ps_panel.dyuv
0005cd10 35360 0.0 0003 Data 8001 1 2a8fcd ps_girl.dyuv
00065730 35360 0.0 0003 Data 8002 1 e1bb6a ps_mesa.dyuv
0006e150 35360 0.0 0003 Data 8002 1 8e394b ps_map.dyuv
00076b70 35360 0.0 0003 Data 8002 1 c60e5e ps_kids.dyuv

File Size Type Description
------------ ------ ------------ ------------
cdi180b.rom 512K cdi000x.rom Unknown CD-i system ROM
cdi180b.rom 512K cdi000x.mdl Unknown CD-i player
cdi180b.rom 512K unknown.brd Unknown board
Of course cditype didn’t correctly detect the ROM, player and board type, but the list of modules looks exactly like a CD-i player system ROM. It is in fact very similar to the CD-i 605 system ROM, the major differences are the presence of the icard and *iic drivers, the absence of a slave module and the different player shell (ps module with separate ps_* data modules instead of a single play module).

It being quite late already, I resocketed all the ROMs in the proper places and closed up both players, after testing that they were both fully functional (insofar as I could test the 180 set), fully intending to clean up and go to bed. As an afterthought, I took a picture of the running 180 set and posted it on the CD-Interactive forums as the definitive answer to the 50/60 Hz power question I’d asked there earlier.

The CD-i Emulator urge started itching however, so I decided to give emulation of my new ROM file a quick go, fully intending to stop at any major problems. I didn’t encounter any of those, however, until I had a running CD-i 180 player three hours later. I reported the fact on the CDinteractive forum, noting that there was no pointing device or disc access yet, and went to a well-deserved sleep. Both of these issues are major ones and those I postponed for the next day.

To get the new player type up and running inside CD-i Emulater, I started by using the CD-i 605 F1 system specification files cdi605a.mdl and minimmc.brd as templates to create the new CD-i 180 F2 system files cdi180b.mdl and maximmc.brd. Next I fired up the emulator and was rewarded with bus errors. Not unexpected and a good indicator of where the problems are. Using the debugger and disassembler I quickly determined that the problems were, as expected, the presence of the VSR instead of VSD and the replacement of the SLAVE by something else. Straightening these out took a bit of time but it was not hard work and very similar to work I had done before on other player types.

This time at least the processor and most of the hardware was known and already emulated; for the Portable CD-i board (used by the CD-i 370, DVE200 and GDI700 players) both of these were not the case as they use the 68341 so-called integrated CD-i engine which in my opinion is sorely misnamed as there is nothing CD-i about the chip, it is just the Motorola version of an 68K processor with many on-chip peripherals in remarkably similar to the Philips 68070 in basic functionality.

Saturday was spent doing household chores with ROM research in between, looking for the way to get the pointing device working. It turned out to be quite involved but at the end of the day I had it sort of flakily working in a kludgy way; I’ll report the details in a next blog post.

Sunday I spent some time fixing the flakiness and thinking a lot about fixing the kludginess; this remains to be done. I also spent time making screenshots and writing this blog post.

So to finish up, there is now a series of 180 screenshots here on the CD-i Emulator website as reported in the What's New section. A very nice player shell, actually, especially for a first generation machine.

I will report some ROM and chip finds including new hopes for replacing the missing pointing device in a next blog post.

Sunday, September 4, 2011

MPEG decoding, state save/restore, NRF emulation, ...

It's been a while since I wrote anything here, but that doesn't mean that work on CD-i Emulator has stopped. On the contrary, a lot has happened in the last month and describing all of it will take a very long blog post. So here goes…

Last January an annoying date-checking bug was found which forced me to release beta2 somewhat earlier than anticipated. After that I did no further work on CD-i Emulator. There were various reasons for this, but the most import one was a very busy period at my day job.

After a well-earned vacation I resumed CD-i related work in early August. First I spent a few days on Walter Hunt's OS-9 port of gcc, the GNU C/C++ Compiler that I found in October of last year. Getting it working on a modern Cygwin installation was interesting and something very different from my usual line of work. The result could be useful for homebrew activities: it's a much more usable C compiler then the Microware OS-9 one and supports C++ as a bonus. I intend to use this for ROM-less emulation validation some day; see also below. The sources need to be released but I haven’t gotten to that stage yet.

After that I had another go at the Digital Video cartridge emulation. At the point where I left off last year the major stumbling block was the presumed picture / frame buffering logic of the MPEG video driver. When the appropriate interrupt status bits are set the driver starts copying a bulk of status information to an array of device registers and it will sometimes also read from those registers. This is all controlled by several status and timing registers that are also referenced elsewhere and I previously could not get a handle on it.

My first attempt this time was spending another few days staring at it and tracing it, but this did not gain me much new understanding. Finally I decided to just leave it for now and see how far I could get without understanding this part of the driver. I decided to once again attempt to get "CD-i Full Motion Video Technical Aspects" working.

This CD-i was produced by Philips to give future Full Motion Video (as the new MPEG playback functions were called at the time) developers a demonstration of the technical capabilities of the new hardware, at a time when this hardware was still in the early beta phase. The CD-i actually contains the compiler libraries necessary for making FMV calls from CD-i applications, as these had not previously been widely distributed.

It is not a very slick disc visually, being intended for developers, but it demonstrates a number of FMV techniques such as regular playback, playback control including pause, slow motion and single step, freeze frame and forward/backward scan, special effects like scrolling the FMV window, a seamless jump and a sample of overlay effects with the CD-i base case video planes.

I had previously tried to run this disc on CD-i Emulator, but it always crashed for an unknown reason that I attributed to MPEG device emulation problems. This time I traced back the crash and it turned out to have nothing at all to do with FMV playback but was instead caused by an incorrect emulation of the 68000 instruction "move ea,ccr" which is supposed to set the condition code register (ccr) to the value specified by the effective address (ea). In the processor manual this is classified as a word instruction and I had emulated it as such, which turned out to be wrong as it caused a word write to the full status register which should have been a byte write to the lower eight bits of it which hold the condition codes.

The problem manifested itself when the application calls the math trap handler for some mundane number calculations, which were naturally supposed to set the condition codes. The value written to the status register inadvertently changed the processor from user to system mode (and also scrambled the active interrupt masking level) which caused an instant stack switch that caused a bus error when the trap handler attempts to return to the application program (the cpu took the return address from the wrong stack and got garbage instead).

Most CD-i applications probably don't use the math trap handler so the problem went undetected for a long time. Now that it's fixed some other titles have probably started working but I haven't tested that.

After this, the FMV Technical Aspects application would get to its main menu screen, allowing me to start FMV playback operations. Regular playback worked fine until the end of the video clip, where there turned out to be status bit generation issues that prevented the application from properly detecting the end of video clip condition (the decoder is supposed to send a "buffer underflow" signal, among others, after the end of the MPEG data and my emulation didn't do that yet).

This was not very easy to fix because of the way that MPEG data buffering and decoding is handled inside CD-i Emulator, which I'll get into below. So it took me some time.

Regular play working fine, I started worrying about window control. This was the area where I feared the picture buffering stuff, but it turned out that this was easily bypassed. The horizontal / vertical scrolling functions were ideal to test this but it took me some time to get it working. There were bugs in several areas, including my integration of the MPEG video decoding code, which I took from the well-known mpeg2dec package. This code is written to decode a single video sequence and consequently did not handle image size changes without some re-initialization calls at the appropriate times. Failing that, it mostly just crashed (at the Windows application level) due to out-of-bounds video buffer accesses.

Another issue was the timing of device register updates for image size changes; I turned out to have the basic mechanism wrong and consequently the driver would keep modifying the window parameters to incorrect values.

Having all of the above fixed, I returned my attention to playback control. So far I can get the video playback properly paused, but I haven't been able to get it properly resumed. For some reason the application resumes the MPEG playback but it doesn't resume the disc playback. Since the driver waits for new data to arrive from disc before actually resuming MPEG playback nothing happens (this is documented as such). The application is presumably expecting some signal from the driver to get into the proper state for resuming disc playback, but I haven't found it yet.

At this point, it seemed promising to look at other CD-i titles using playback control and the Philips Video CD application is an obvious candidate. Again, regular playback appears to work fine, but playback control (including pause/resume) does not. It turns out that this application uses a different driver call (it uses MV_ChSpeed instead of MV_Pause, probably in preparation for possible slow motion or single step), which never completes successfully, probably again because of device status signaling. Similar issues appear to block playback control in a few other titles I tried.

I've given some thought to tracing driver calls and signals on an actual player to see what CD-i Emulator is doing wrong, and it appears to be relatively simple, there's just a bandwidth issue because all of the trace output will have to go out the serial port which can go no higher then 19200 baud. Some kind of data compression is obviously needed and I've determined a relatively simple scheme that should be enough (the CD-i player side will all need to be coded in 68000 machine language so simplicity is important!), but I haven't actually written any code for it yet.

I know there are issues with the proper timing of some video status signals. Things like start-of-sequence, end-of-sequence and start-of-picture-group should be delayed until display of the corresponding picture, at present they are delivered at decoding time, which can be a few pictures early. But that does not really affect the titles I've tried so far, because they do not attempt picture-synced operations. An application like The Lost Ride might be sensitive to thinks like this, though, and it needs to be fixed at some time. Similar issues are probably present with time code delivery. In addition, the last-picture-displayed and buffer-underflow signals are not always properly sent; I'm fixing these as I go along.

In the process, I decided that the magenta border was getting annoying and tried to fix it. That turned out to be harder then I thought. The MPEG chip has a special border color register that is written by the MV_BColor driver call and it seemed enough to just pass the color value to the MPEG window overlay routines. Well, not so. Again the issue turned out to be timing of decoder status signals, but of a different kind. The driver doesn't write the border color registers until it has seen some progress in certain timing registers related to the picture buffering thing, presumably to avoid visual flashes or something on the actual hardware. Fortunately, it turned out to be easy to simulate that progress, taking care not to trigger the complicated picture buffer code that I so far managed to keep dormant.

At some point, possibly related to slow motion or freeze frame, I might need to actually tackle that code but I hope to by that time have gained more understanding of the supposed workings of the MPEG chip.

Looking at the above, you might think that all of the difficulties are with the MPEG video decoding and that is indeed mostly true. I did have to fix something in the MPEG audio decoding, related to the pause/resume problems, and that was the updating of the audio decoder clock. When audio and video playback are synchronized the MPEG video driver uses the MPEG audio clock as it's timing reference, which means that it has to be stopped and restarted when video playback control operations occur. Since I had never before seriously tested this, the audio clock wasn't stopped at all and the video driver obligingly continued decoding and displaying pictures until it ran out of buffered data.

There is currently just one known problem with the MPEG audio decoding: the audio isn't properly attenuated as specified by the driver. This causes little audio distortions at some stream transitions and when buffers run out. There is also a problem with base case audio synchronization but that is hard to trigger and possibly even not audible in many titles so I'll worry about that much later.

Above I promised to get into the MPEG data buffering and decoding issue. The basic problem is one of conceptual mismatch: the CD-i decoding hardware gets data "pushed" into it (by DMA or direct device I/O) at the behest of the driver, whereas the MPEG decoding code (based on the publicly available mpeg2dec and musicout programs from the MPEG Software Simulation Group) expects to "pull" the data it needs during decoding. Things get messy when the decoding runs out of data, as the code does not expect to ever do so (it was originally written to decode from a disc file which of course never runs out of data until the end of the sequence). Some obvious solutions include putting the decoding in a separate thread (which given multi-core processors might be a good idea anyway from a performance perspective) and modifying it to become restartable at some previous sync point (most easily this would be the start of an audio frame or a picture or picture slice). Both options are somewhat involved although they have obvious benefits, and it may turn out that I will need to do one of them anyway at some point. For now I've avoided the problems by carefully timing calls into the MPEG decoding code so that enough data to decode a single audio frame or video picture should always be available; the MPEG data stream at the system level contains enough timestamp and buffering information to make this possible (in particular, it specifies the exact decoding time of every audio frame or video picture in relation to the timing of the data stream, thus making it possible to make those calls into the decoding code at a time that a valid MPEG data stream will have already filled the buffers far enough).

The approach depends on the timing of the MPEG data entering the decoder, which means that it does not handle buffer underflow conditions unless you add some kind of automatic decoding that continues even if no more MPEG data appears, and this is basically what I’ve done. In the end it was just relatively straightforward extension of the automatic decoding already there to handle the fact that MPEG audio streams do not have to explicitly timestamp every single audio frame (the CD-i Green Book does not even allow this unless you waste massive amounts of space in each MPEG audio data sector) and would have been needed anyway to correctly decode the last pictures of a sequence, but that had never been tested before.

For performance and possible patent reasons I have taken care to edit the MPEG decoding code (placing appropriate #ifdef lines at the right places) so that only MPEG 1 video and audio layer I/II decoding code is compiled into the CD-i Emulator executable. This is all that is needed for CD-i anyway and MPEG 2 video and audio layer III greatly complicate the decoding and thus significantly enlarge the compiled code.

Being somewhat stymied at the FMV front, I next decided to spend some time on another lingering issue. During testing, I often have to do the same exact sequence of mouse actions to get a CD-i application to a problem point and this is starting to be annoying. Input recording and playback are a partial solution to this but then you still have to wait while the application goes through it, which is also annoying and can sometimes take quite some time anyway. The obvious solution is a full emulation state save/restore feature, which I've given some thought and started implementing. It's nowhere near finished, though.

During the MESS collaboration I spent some time investigating the MESS save/restore mechanism. If at all possible I would love to be compatible for CD-i emulation states, but it turns out to be quite hard to do. The basic internal mechanism is quite similar in spirit to what I developed for CD-i Emulator, but it's the way the data is actually saved that makes compatibility very hard. Both approaches basically boil down to saving and restoring all the relevant emulation state variables, which includes easy things like the contents of cpu, memory and device registers but also internal device state variables. The latter are of course not identical between different emulators but they could probably be converted if some effort was thrown at it and for a typical device they aren't very complex anyway. The MESS implementation uses an initialization-time registration of all state variables; at save/restore time it just walks the registrations and saves or restores the binary contents of those variables. CD-i Emulator has a somewhat more flexible approach; at save/restore time it calls a device-specific serialize function to save or restore the contents of the state variables. The actual registration / serialization codes are structurally similar in the two emulators (a simple list of macro/function calls on the state variables) but the code runs at different times.

The real problem is that MESS includes very little meta information in the save files: only a single checksum of all the names and types of registered state variables in registration order. This is enough to validate the save data at restore time if the state variables of the saving emulator exactly match those of the restoring emulator, because there is no information to implement skipping or conversions. This holds between different versions or in some case even configurations of MESS emulators, but it holds even more so between MESS and CD-i Emulator! The meta information could of course be obtained from the MESS source code (relatively simply macro modifications could cause it to be written out) but that would require exact tracking of MESS versions because every version could have its own checksum corresponding to different meta information (in this case CD-i Emulator would need meta information sets for every MESS checksum value it wants to support).

I want CD-i Emulator to be more flexible, especially during development, so I decided to make full meta information an option in the save file. The saved state of every device is always versioned, which allows the save/restore code to implement explicit conversion where needed, but during development this isn't good enough. With full meta information turned on, the name and type of every state variable precedes the save data for that variable in the save file. This allows more-or-less automatic skipping of unknown state variables and when properly implemented the restore code can also handle variable reordering. At release time, I will fix the version numbers and save full metadata information sets for those version numbers so that the same automatic skipping and handling of reordering can be done even if the metadata isn't in the save file (it probably won't be because of file size considerations, although that may turn out to be a non-issue because save files need to include the full RAM contents anyway which is 1 MB of data in the simplest case without any compression, which is of course an option).

In addition to all of the above, I made some progress on the ROM-less emulation front. First I spent some time reading up on the internals of OS-9 file managers, because writing a replacement for the NRF file manager (NRF = Nonvolatile RAM File manager) seemed the logical next step. Actually writing it turned out not to be that hard, but there were of course bugs in the basic ROM emulation code. Most of them had to do with handlers not calling into the original ROM, which totally screwed up the tracing code. Some new functionality was also needed to properly read/write OS-9 data structures inside the emulated machine from the ROM emulation code; I wanted to implement this in such a way that compilation to "native" 68000 code remains a future option for ROM emulation modules. And of course the massive tracing described in the previous blog post had to be curtailed because it was impossible to see the relevant information in the morass of tracing output.

The new emulated NRF stores its files in the PC file system and it currently works fine when you start it with no stored files (i.e., the player will boot). In that case it will write out a proper "csd" (Configuration Status Descriptor) file. However, if this file already exists, the player crashes, although I have so far not found any fault in the NRF code. The origin of the problem probably lies elsewhere; I suspect it has to do with the hidden "player_shell_settings.prf" file. This file is read and written by the ROM bootstrap even before OS-9 is running; it does this by directly accessing the NVRAM memory (the file never changes size and is always the first one in NVRAM). Since the bootstrap accesses of this file do not go through the NRF file manager or even the NVRAM driver they are not redirected by the OS-9 emulation. However, later accesses by the player shell *are* redirected and the player shell does not seem able to handle the file not existing in the PC file system in the case where a csd file already exists. Solutions include extending the emulated NRF to always access this particular file from the NVRAM instead of the PC file system or somehow synchronizing the two locations for the file. The latter is probably the easiest route given the fixed location and size of the file, but the former is also useful as it would provide a full reimplementation of the original NRF that could in principle be compiled to native 68000 code to replace the "original" NRF in ROM (this is where gcc comes in as alluded to earlier, since all emulation code is written in C++).

In either case, I do not want the file manager to directly access emulated NVRAM although it could do so easily, as there is already an internal CNvramPort interface that provides just such access independent of the actual emulated NVRAM chip. The NRF file manager should instead call the NVRAM driver, which means that I need to implement cross-module calling first. It's not really hard in principle, the design has been done but there are a lot of little details to get right (the most obvious implementation uses at least 66 bytes of emulated stack space on each such call which I find excessive and might not even work; smarter implementations require some finicky register mask management or a "magic cookie" stacking approach, the latter having the best performance in the emulation case but being impossible in the native 68000 compilation case). When cross-module calling is working, I can also have the file manager allocate emulated memory and separate out the filename parsing functions by using the OS-9 system calls that provide these functions (the current emulated NRF does not allocate emulated memory which is arguably an emulation error and has the filename parsing coded out explicitly).

When everything works correctly with the emulated NRF, I have to find some way of integrating it in the user experience. You could always start over without any NVRAM files, but I'd like to have some way of migrating files between the two possible locations without having to run CD-i Emulator with weird options. Extending the CD-i File Extractor (cdifile) by incorporating (part of) the emulated NRF seems the obvious choice, which would also provide me with some impetus to finally integrate it with the CD-i File Viewer (wcdiview) program that's supposed to be a GUI version of cdifile but so far is just a very thin skeleton barely able to graphically display a single CD-i IFF image file passed on the command line (it doesn't even have a File Open menu) and will often crash. A proper implementation would look like Windows Explorer with a tree view on the left (CD-i file system, real-time channels and records, IFF chunk structure, etc) and a variable content display on the right (raw data view, decoded sector view, code disassembly view, graphical image view, audio playback, slideshow playback, decoded MPEG view, MPEG playback, etc).

That touches on another area in which I did some work last month: the saving of CD-i IFF image files for each emulated video frame. The motivation for this was to bring full-resolution real-time frame saving into the realm of the possible, as it would write only about 2 x (1024 + 280 x (384 + 32)) = 247 KB of raw CD-i video and DCP data per frame instead of 560 x 768 x 3 = 1260 KB of raw RGB. At least on my PC this has turned out not to be the case, however. The data is written out fine, which is not as easy as it sounds since video line data size can vary with each line because of pixel repeat and run-length encoding, but it's still too slow. That being so, I am not really very motivated to extend the CD-i IFF decoding implementation to actually decode this information. Some kind of compression could be an option, but that takes processor time and makes things even harder and possibly slower. Perhaps using another thread for this would be a solution, on a multi-core machine this should not greatly impact the basic emulation performance nor the debugging complexity as the compression code would be independent of the emulation itself.

So there is still a lot of work to be done, but it's all quite interesting and will provide for some entertaining evenings and weekends in the coming weeks or possibly months.