SCSI support and a big surprise

Last week I added SCSI disk support for the CD-i 60x extension board to CD-i Emulator. It took somewhat longer then I expected, though. This was mostly because the DP5380 SCSI controller chip exposes most low-level details of the SCSI protocol to the driver which means that all of these details have to be emulated.

The emulation ended up to be a more-or-less complete software implementation of the parallel SCSI-2 protocol, including most of the low-level signaling on the BSY, SEL, ATN, MSG, C/D-, I/O-, REQ and ACK lines. This is all implemented by the new CScsiBus class representing the SCSI bus that connects up to 16 instances of the CScsiPort class that each represent a single SCSI-2 bus interface. I was able to mostly avoid per-byte signaling of REQ and ACK if the target device implementation supports block transfers, a big performance win.

The new CCdiScsiDevice class emulates the DP5380 controller chip, working in conjunction with the CCdiScsiRamDevice and CCdiScsiDmaDevice classes that emulate the 32 KB of local extension SRAM and the discrete DMA logic around it that are included on the CD-i 60x extension board.

The CD-i 182 extension uses a compatible SCSI controller chip but a different DMA controller and has no local extension SRAM. I have not yet emulated these because I have almost no software to test it. 

The new CScsiDevice class implements a generic SCSI device emulating minimal versions of the four SCSI commands that are mandatory for all SCSI device types: TEST UNIT READY, REQUEST SENSE, INQUIRY and SEND DIAGNOSTIC. It implements most of the boiler-plate of low-level SCSI signaling for target devices and the full command and status phases of SCSI command processing, allowing subclasses to focus on implementing the content aspects of the data transfer phase.

The CScsiFile class emulates a SCSI device backed by a file on the host PC; it includes facilities for managing the SCSI block size and the transfer of block-sized data to and from the backing file.

The CScsiDisk and CScsiTape classes emulate a SCSI disk and tape device, respectively, currently supporting a block size of 512 bytes only. Instances of these classes are connected to the SCSI bus by using the new
-s[csi]d[isk][0-7] FILE and -s[csi]t[ape][0-7] FILE options of CD-i Emulator.

The CD-i 60x extension board normally uses SCSI id 5; the built-in ROM device descriptors for SCSI disks use SCSI ids starting at zero (/h0 /h1 /h2) while the built-in device descriptor for a SCSI tape uses SCSI id 4 (/mt0). This means that the useful options with the 60x are -scsidisk0, -scsidisk1, -scsidisk2 and -scsitape 4.

I've added the new dsk subdirectory to contain disk images; tape images have no standard location as they are mostly intended for bulk-transfer purposes (see below).

Inside the CD-i player this leads to the following response to the built-in inquire command:

$ inquire -i=0
vendor identification:"CDIFAN  CDIEMU SCSIDISK "
 
$ inquire -i=4
vendor identification:"CDIFAN  CDIEMU SCSITAPE "

where the "CDIFAN " part is the vendor name and the "CDIEMU SCSIXXXX " part is the product name.

In the previous post I described a 450 MB OS-9 hard disk image that I found on the Internet. After mounting it with
-scsidisk0 mw.dsk I got the following output:

$ free /h0
"MediaWorkshop" created on: Feb 17, 1994
Capacity: 1015812 sectors (512-byte sectors, 32-sector clusters)
674144 free sectors, largest block 655552 sectors
345161728 of 520095744 bytes (329.17 of 496.00 Mb) free on media (66%)
335642624 bytes (320.09 Mb) in largest free block
 
$ dir -d /h0

Directory of /h0 23:49:36
ASU/            AUDIO/          CDI_BASECASE/   CINERGY/        CMDS/
COPY/           CURSORS/        DEFS/           DEMOS/          ENET/
ETC/            FDRAW/          FONTS/          FontExample/    ISP/
LIB/            MAUI/           MAUIDEMO/       MENU/           MWOS/
NFS/            README_CIN      README_MWS      SCRIPT/         SHARE/
SHIP/           SYS/            T2D_RUNTIME/    TEMP/           TEMPMARK/
TEST/           USR/            VIDEO/          abstract.txt    bibliographic.txt
bkgd.c8         bkgd.d          cdb             cdb1            cdb2
cdi_opt_install chris_test      cin             copyright.mws   copyright.txt
csd_605         custominits_cin delme           dos/            file
font8x8         get             globs.mod       go              go.mkfont
inetdb          ipstat          kick1a_f.c8     kick2a_f.c8     mtitle
mws             net             new_shell       new_shell.stb   scratch
screen          startup_cin     thelist         

You can see why thought it was a MediaWorkshop disc, but on closer inspection this turned out to something quite different. Some basic scrutiny lead to the hypothesis that this is probably a disk backup of someone from Microware working on early development of the DAVID (Digital Audio Video Interactive Decoder) platform. There are various surprises on the disk which I will describe below.

Anyway, I wanted to transfer the contents to the PC as a tar archive, similar to the procedure I used for my CD-i floppy collection. After starting CD-i Emulator with a -scsitape4 mw.tar option this was simply a matter of typing the following into the terminal window:

tar cb 1 /h0

This command runs the "tape archiver" program to create a tape with the contents of the /h0 directory, using a tape blocking size of 1 (necessary because my SCSI tape emulation doesn't yet support larger block sizes). The resulting mw.tar file on the PC is only 130 MB, not 450 MB which indicates that the disk is mostly empty. At some point I might use an OS-9 "undelete" program to find out if there are additional surprises.

Extracting the mw.tar file was now a simple matter of running the PC command

tar xvf mw.tar

This produced an exact copy of the OS-9 directory structure and files on the PC.

Many of the directories on the hard disk are clearly copies of various distribution media (e.g. CDI_BASECASE, CINERGY, CURSORS, ENET, FONTS, ISP, MWOS, NFS). The contents of the ENET, ISP and NFS directories at first appear to match some of my floppies, including version numbers, but on closer inspection the binaries are different. Running some of them produces "Illegal instruction" errors so I suspect that these are 68020 binaries.

The SHIP directory contains some prerelease RTNFM software; the readme talks about PES which is a type of MPEG-2 stream (Packetized Elementary Stream). Various asset directories contain versions of a "DAVID" logo.

The CMDS directory contains working versions of the Microware C compiler, identical to the ones I already had and also many other programs. It also contains some "cdb" files (configuration database?) that mention the 68340 processor.

The contents of the CMDS/BOOTOBJS directory produced a first surprise: it contains a subdirectory JNMS containing among others files named "rb1793" and "scsijnms". Could this be floppy and SCSI drivers for the CD-i 182 extension, as it contains with a 1793 floppy drive controller (the CD-i 60x uses a different one) and the player has a "JNMS" serial number?

Well, yes and no. Disassembly of the scsijnms file proved it to be compiled C code using an interface different from OS-9 2.4 drivers, so I suspect this is an OS-9 3.x driver. In any case, I cannot use it with the stock CD-i 180 player ROMs. Bummer...

And now for the big surprise: deeply hidden in a directory structure inside the innocently named COPY directory is the complete assembly source for the VMPEG video driver module "fmvdrv". At first glance it looked very familiar from my disassembly exercises on the identically-named Gate Array 2 MPEG driver module "fmvdrv", which is as expected because I had already noticed the large similarity between these two hardware generations.

The source calls the VMPEG hardware the "IC3" implementation, which matches CD-i digital video history as I know it. The Gate Array MPEG hardware would be "IC2" and the original prototype hardware would be "IC1". Furthermore, the sources contain three source files named fmvbugs1.a to fmvbugs3.a whose source file titles are "FMV first silicon bugs routines" to "FMV third silicon bugs routines". The supplied makefile currently uses only fmvbugs3.a as is to be expected for a VMPEG driver.

The fmvbugs1.a source contains some of the picture buffer manipulation logic that I've so far carefully avoided triggering because I couldn't understand it from my disassemblies, and this is now perfectly understandable: they are workarounds for hardware bugs!

As of two hours ago, I have verified that with a little tweaking and reconstruction of a single missing constants library file these sources produce the exact "fmvdrv" driver module contained in the vmpega.rom file directly obtained from my VMPEG cartridge.

In general these sources are very heavily commented, including numerous change management comments. They also include a full set of hardware register and bit names, although no comments directly describing the hardware. This should be of great help in finally getting the digital video emulation completely working.

All of the comments are English, although a few stray words and developer initials lead me to believe that the programmers were either Dutch or Belgian.

Disassembly comparisons lead me to the conclusion that careful undoing of numerous changes should result in exact sources for the GMPEGA2 driver module "fmvdrv" as well. I might even do it at some point, although this is not high priority for me.

The disk image containing all of these surprises is publicly available on the Internet since at least 2009, which is probably someone's mistake but one for which I'm very grateful at this point!

CD-i floppy inventory

Last weekend I future-proofed my CD-i floppy collection. A bit to my surprise, all floppies except one turned out to be perfectly readable (nearly twenty years after they were last written!). Luckily, the one exception was a backup copy so I didn’t lose any contents.

I had originally intended to use the borrowed CDI 182 unit for this (it has two floppy drives). The primary motivation for this was that my unstowed CDI 605 could not read beyond track zero of any floppy, but after giving the matter some thought I decided to try my other CDI 605 first, the primary motivation for this being speed (see below). It turned out that this 605 could read the floppies perfectly, including the three 38U0 format ones that gave problems on the 182 unit. Microware has defined a number of OS-9 disk formats for floppies, the 38U0 one supposedly being the “universal” 3.5" format (there is also a 58U0 “universal” 5¼" format).

The problem with the “universal” formats is that track zero can be (and on my floppies, is) in a different density which makes it a bad fit for most tools, both on CD-i and PC. It also means that only 79 tracks are used for data storage, giving a raw capacity of 79 × 2 × 16 × 256 = 632 KB. The 3803 format used by all my other CD-i floppies uses all 80 tracks and consequently has 8 KB more of raw storage for a total of 640 KB (these are both double-density, double-side formats (DS, DD) with 16 sectors of 256 bytes per track like nearly all OS-9 disk formats).

Before unstowing my other CDI 605 (it was nearly at the bottom of a 150 cm stowed equipment stack) I tried reading the floppies with my trusty old Windows 98 machine which still has floppy drives. I could not quickly find a DOS tool that handled the 256 byte sectors (not even raread and friends), although I suspect that Sydex’s TELEDISK product would have handled it just fine. I also tried Reischke’s OS9MAX which should handle all OS-9 formats under the sun according to its documentation. The demo version ran under MS-DOS and gave me working directory listings, even for the 38U0 floppies, but it does not support actually reading the files and I am somewhat doubtful about the current availability of the paid-for full version (even apart from cost concerns).

Why did I decide to use the 605? It was not a question of reading the disks (the 182 did this mostly fine) but of handling the data thus read. The 182 unit has a SCSI connector but I have no drivers for it (yet) and dumping my full floppy collection over the serial port did not really appeal to me for speed and reliability reasons (it could have been done, of course).

The 605 player has a SCSI connector and includes drivers for it so I could have just connected it to the SCSI disk in my E1 emulator and copied the floppies to hard disk (I would still have needed to transfer them to my laptop which would have been a two-step process via the Windows 98 PC as I have no SCSI connection on my laptop).

Instead I used the BNC network connector of the 605 to directly transfer floppy images to my laptop (it needs a network switch supporting both a BNC connector and the modern RJ45 connectors, but luckily I have two of those, even if they are only 10 Mbit/s). Starting up the network environment of the 605 took only two OS-9 commands at the command shell prompt:

ispmode /le0 addr=10.0.0.120
mbinstall

After this I could just ftp in to my laptop where I ran ftpdmin, a very minimal ftp server program, and transfer floppy disk images directly:

ftp 10.0.0.110
bin
put /d0@ floppy.dsk

(where /d0@ is the raw floppy device, for 38U0 I used /d0uv@, both are built-in for the 605).

The transfers ran at the maximum speed of the floppy drive (way below the 10 Mbit/s network speed), and the resulting .dsk files are perfectly readable using the –v option (virtual disk) of Carey Bloodworth’s os9.exe program even though that program was originally written for Tandy Color Computer OS9/6809 floppies (the floppy disk format was not changed for OS-9/68000 which is at the core of CD-i’s CD-RTOS operating system).

For easy access I also created a “tar” format archive of each floppy on a RAM disk:

chd /d0
tar cvf /r768/floppy.tar .

and ftp’d those to my laptop as well (the /r768 device is a 768 KB variation of the /r512 built-in 512 KB RAM disk device of the 605 player).

I ended up with the following collection of unique floppy disk images:

• 605h3 - 605 H3 Driver Update (1 floppy)
  
• 605upd - 605 Driver Update (1 floppy)
  
• bcase - Basecase Tests (1 floppy)
  
• eboot41 - Emulation Boot Diskette (1 floppy)
  
• eburn41 - Emulation and CDD 521 Boot Diskette (1 floppy)
  
• inet - CD-I Internet Installation Disk - V1.3 (1 floppy)
  
• nfs - OS-9/68000 Network File System V.1.0 (1 floppy)
  
• os9sys - OS-9 System Diskette (1 floppy)
  
• pubsoft - OptImage Public Domain Software (2 floppies)
  
• pvpak - OptImage Preview Pak Installation Disk (1 floppy)
  
• ubridge - OS-9 UniBridge Resident Utilities (3 floppies)
  

The 605* and eb* floppies are mostly interesting for CD-i 605 or E1 emulator owners, but the bcase floppy contains a set of CD-i standard conformance test programs that.

The inet and nfs floppies contain a full set of Internet software including Telnet and FTP servers and clients and an NFS client (all except the latter are also in the 605 ROMs).

The os9sys floppy contains a full set of Professional OS-9 programs and is my original source for most of the OS-9 CD-i disc that I described earlier (most of these are not in ROM on any CD-i player that I’ve seen so far).

The pubsoft floppies contain miscellanous utilities such as bfed, du, kermit, umacs and vi, most of which can be obtained elsewhere, some CD-i specific utilities such as da (CD-i disk analyzer) and iffinfo (CD-i IFF file dumper) as well as library source files for the CD-i IFF file library.

The pvpak floppy contains preview software for CD-i images that will preview CD-i IFF files from an NFS-mounted host file system directory.

The ubridge floppies are the goldmine (and also the 38U0 format ones) as they contain a full set of native Microware C compiler/assembler/linker/debugger software for OS-9 complete with CD-i header files and libraries and C runtime startup sources. Both the srcdbg and sysdbg debuggers are included as well as the rdump utility for dumping ROFF (Relocatable Object File Format) files.

Unfortunately, most of the above software except for the pubsoft contents is copyrighted property of Microware (now Radisys) or OptImage (a former Philips/Microware joint venture) which means that I cannot distribute it, even though they could be very useful to CD-i homebrew developers. For that the hopefully soon-to-be available GCC cross-port will have to be enough...

While investigating all of the above I also stumbled upon a 450 MB OS-9 hard disk image for MediaWorkshop. The os9.exe program recognizes it just enough to say that it does not support it so I have no real idea about its contents except the obvious.

To remedy that problem I’m in the process of adding SCSI disk support to CD-i emulator so that I can use the SCSI support in the CD-i 605 ROMs to mount the disk image and look at it. This should also allow the CD-i 180 to boot from a SCSI disk if I ever find drivers for it (a possible path to that has just appeared, we’ll see...).