RK05 disk emulator development and discussion

[thunter0512]

I have added some very basic meta data:

Image Name (11 characters)
Description (200 characters)
Date & Time (20 characters)

All the field lengths include the zero terminator.

This makes it possible to show the Image Name as the "disk label" like in the photo below "OS/8 v3d":

 

[thunter0512]

hmmm, interesting idea. You're thinking a value that would be unique to determine that a file is different from any other file that's not produced via a direct copy? Could be a hash of the current date, time, and user ID, something like that?

Maybe Dare was thinking about a Unix style "magic number" which can be used by the "file" command and other utilities to determine the file type.

 

[Dare]

Maybe Dare was thinking about a Unix style "magic number" which can be used by the "file" command and other utilities to determine the file type.

Exactly. Although I think a hash of the header data, including the magic number, could be good as well.

I bring this up because at the moment I'm busy adding support for various virtual disk container formats to my retro-fuse project. This is happening as part of my effort to support mounting Xenix filesystems. But the work will make it easy to support different container types for any supported filesystem. With a magic number in the header, I should be able to autodetect one of these emulated RK05 files and automatically mount it (in the case where it contains a Unix filesystem).

 

[AK6DN]

Have you guys not followed the fiasco that happened with SIMH around adding/extending metadata within legacy disk images?

Upshot is fooling around with the disk data image, either prepending or appending arbitrary metadata within the file itself is highly frowned up.

Add a separate companion file that contains anything you want in whatever format you need (json, xml, raw bits) and maybe even the name of the companion data file.

But don't change the concept that the data file is just a pure data image of size N blocks from blocks 0 to N-1.

Just my 2c, but the arguments for/against this concept blew up the old SIMH project into the old and new forks.

And yes I realize also there were some personality issues involved.

 

[djg]

Have you guys not followed the fiasco that happened with SIMH around adding/extending metadata within legacy disk images?

From previous postings the emulator doesn't directly work with SIMH images but has a program to convert so don't think this is an issue. Original SIMH format files won't be modified. I do something similar with my MFM disk emulator.

 

[BitWiz]

hmmm, interesting idea. You're thinking a value that would be unique to determine that a file is different from any other file that's not produced via a direct copy? Could be a hash of the current date, time, and user ID, something like that

Rather than some kind of hash, why don't you use something like a CRC or an MD5. That way it can be used for both identification and validation.

 

[thunter0512]

Exactly. Although I think a hash of the header data, including the magic number, could be good as well.

I bring this up because at the moment I'm busy adding support for various virtual disk container formats to my retro-fuse project. This is happening as part of my effort to support mounting Xenix filesystems. But the work will make it easy to support different container types for any supported filesystem. With a magic number in the header, I should be able to autodetect one of these emulated RK05 files and automatically mount it (in the case where it contains a Unix filesystem).

Who assigns magic numbers and how big are they? Adding one would be trivial.

 

[thunter0512]

Have you guys not followed the fiasco that happened with SIMH around adding/extending metadata within legacy disk images?

Upshot is fooling around with the disk data image, either prepending or appending arbitrary metadata within the file itself is highly frowned up.

Add a separate companion file that contains anything you want in whatever format you need (json, xml, raw bits) and maybe even the name of the companion data file.

But don't change the concept that the data file is just a pure data image of size N blocks from blocks 0 to N-1.

Just my 2c, but the arguments for/against this concept blew up the old SIMH project into the old and new forks.

And yes I realize also there were some personality issues involved.

I did follow the drama around SIMH adding meta-data to existing images. Here we talk about something different.

There is no intention of touching the SIMH disk image format other than using SIMH image files as an input into a converter for the RK05 emulator's image container.

The main reason for not using the SIMH disk format directly is that the RK05 emulator is storing the sector header, sector data and sector CRC in the image. This header and CRC are required by real disk controllers interfacing to the RK05 emulator.

Also the header and CRC provide a certain level of end-to-end data integrity assurance that is missing in the SIMH format. With modern SSD disk bit rot this is a good thing.

With the conversion utilities it is trivial to convert from one format to the other. Additionally the meta-data can nicely describe the RK05 emulator image.

 

[AK6DN]

I did follow the drama around SIMH adding meta-data to existing images. Here we talk about something different.
There is no intention of touching the SIMH disk image format other than using SIMH image files as an input into a converter for the RK05 emulator's image container.

Got it. You are making a tradeoff in the design of your controller to simplify it at the expense of preprocessing disk images for use with it.

The main reason for not using the SIMH disk format directly is that the RK05 emulator is storing the sector header, sector data and sector CRC in the image. This header and CRC are required by real disk controllers interfacing to the RK05 emulator.

But you could compute the track/sector numbers from the position of the data block in the file, correct?
And vice versa to go from a track/sector address to an offset into the data file?
And you must be able to generate a new sector CRC in your controller anyway if you are able to write new sectors.

Also the header and CRC provide a certain level of end-to-end data integrity assurance that is missing in the SIMH format. With modern SSD disk bit rot this is a good thing.

LOL really? Then I hope you have at least parity or even better ECC and have used TMR techniques to protect all the internal memories and logic in your controller FPGA.
Those stray cosmic rays can be so problematic. SIMH seems to work just fine using the SIMH disk format. Don't know why your controller would be different.

With the conversion utilities it is trivial to convert from one format to the other. Additionally the meta-data can nicely describe the RK05 emulator image.

But if all the inputs are coming from SIMH disk images (most likely) there won't be any metadata except what gets created in your conversion operation.
Unless you are proposing all the existing SIMH RK05 disk images switch to your container format? Color me skeptical.

 

[thunter0512]

Got it. You are making a tradeoff in the design of your controller to simplify it at the expense of preprocessing disk images for use with it.

But you could compute the track/sector numbers from the position of the data block in the file, correct?
And vice versa to go from a track/sector address to an offset into the data file?
And you must be able to generate a new sector CRC in your controller anyway if you are able to write new sectors.

Yes the RK05 could compute RK8-E CRCs and headers "on-the-fly", but that is not what a real RK05 does. It faithfully stores the controller supplied sector header, data and CRC on a surface of spinning rust as a series of magnetic flux transitions.

The RK05 emulator does exactly the same. It does not know anything about the data. During a read it transparently presents to the controller what it has recorded previously and during a write it transparently records what is supplied by the controller.

Note that a RK05 disk drive (and this emulator) can work with non-DEC controllers which may implement a completely different disk format including different headers and CRC algorithms.

This emulator does not emulate the controller - nor should it.

LOL really? Then I hope you have at least parity or even better ECC and have used TMR techniques to protect all the internal memories and logic in your controller FPGA.
Those stray cosmic rays can be so problematic. SIMH seems to work just fine using the SIMH disk format. Don't know why your controller would be different.

The quoted data integrity was just a benefit of replicating what a real RK05 disk drive does not the reason for the chosen approach. BTW I have seen corrupted files on my old ACER laptop. I can't prove it, but I blame the SSD.

But if all the inputs are coming from SIMH disk images (most likely) there won't be any metadata except what gets created in your conversion operation.
Unless you are proposing all the existing SIMH RK05 disk images switch to your container format? Color me skeptical.

A disk image is not a static thing. Some RK05 emulator disk images will originally be converted from SIMH, others may come from other sources. The important feature is that the emulator is controller agnostic and won't just work with an RK8-E controller. With the (RK8-E compatible) conversion utility you can easily move between real RK8-E controller based environments and SIMH based environments.

 

[Dare]

Who assigns magic numbers and how big are they? Adding one would be trivial.

You get to do that. All you need is a fixed, unique sequence of bytes, preferably placed at a known location in the file (right at the beginning is good). An 8 byte sequence is the minimum I would use in this case, since you're not really space constrained. But 4 would work.

It doesn't matter what the sequence is, as long as it is unique. I often use a random number generator to generate the byte sequence. E.g.:

$ dd bs=1 count=8 if=/dev/random 2>/dev/null | od -A none -t x1
5a 12 ce d1 d0 6c 86 56

For extra credit, you can add a pattern line to /etc/magic so that the file(1) command recognizes your emulator files:

0 string \x5a\x12\xce\xd1\xd0\x6c\x86\x56 Emulated RK05 disk data file

 

[gwiley]

So, the emulator not performing any controller functions is really the main reason for the present format being what it is, with JSON metadata followed by the disk sector data written on the RK05 platter. Mostly, the JSON metadata describes physical and timing attributes. The new fields being discussed: Image Name, Description, Date & Time, magic-number-or-otherwise-unique-value, are likely very helpful to the user but don't affect the functionality of the emulator.

There are still a few basic assumptions about how attached controllers will work: FM recording, there will be an all-zero preamble followed by a sync bit and an all-zero postamble. But for the most part, the file describes the recording on the surface and properties of the sector and index marks.

Adding header fields that don't affect the emulator function doesn't deviate from the original philosophy. It's like sticking informative labels on the outside of the pack... useful for humans but doesn't affect operation of the system.

 

[gwiley]

Currently, there is some granularity of supported bit rates, only 1.44 (RK8-E & RK11-D), 1.545 (RK11-E), and 1.600 Mbps (Xerox Alto) are supported.
These have been verified using the RK05 Tester but so far, only the RK8-E properties have been checked using a "real computer".

 

[PDP11GY]

In my RL, MFM and RK05 disk emulator I used the .DEC/MFM data Format for drive specific data and the .DSK Format for a direct interface to SIMH. My emulators can read and write both file format. It's open Sourcecode on GitHub or for the RK05 emulator on my Homepage.

 

[DougIngraham]

Do any of these emulators emulate seek and rotational latency?

I have one RK05 drive and a controller set for my 8/e but I have not gotten around to working on that.

 

[Al Kossow]

I suggest including a magic number at the beginning of the file to make automatic identification easier.

A magic number is rule #1 in creating a container format.
Too bad nothing in SIMH followed that rule.

Designing File Formats

 

[thunter0512]

A magic number is rule #1 in creating a container format.
Too bad nothing in SIMH followed that rule.

Designing File Formats

Thanks for the link. There are a lot of good suggestions in the article.

 

[Dare]

A magic number is rule #1 in creating a container format.
Too bad nothing in SIMH followed that rule.

Designing File Formats

Wow. I'd completely forgotten that Andy wrote that.

 

[gwiley]

Do any of these emulators emulate seek and rotational latency?

Rotational latency, yes. The interface signals pretty much require that. I suppose it might be possible to spin the virtual surface faster until the controller decides to read or write a particular sector, but I was too afraid to deviate from the normal operational characteristics. I wanted to emulate the real drive as much as possible. The emulator outputs index and sector pulses at the same rate and width as an RK05.

Seek latency, no, or not yet, anyway. The emulator seeks instantly to any cylinder in the valid range. I'd like to maybe add an option bit to seek at the same speed as a real RK05. It would require knowing the not ready time as a function of change in cylinder address. There's a rough idea of that function in the maintenance manual but it would be interesting to see some actual data from a real RK05 of the pulse width of the BUS_RWS_RDY_L signal for different delta cylinder address values. There's a good amount of information about delta cylinder address to how hard the voice coil is driven, but the ultimate response of that system translated to settling time is something I don't know yet. With actual settling time data from a real drive, I think it's something that could be coded into the FPGA logic fairly easily. The result would be the seek time would be like the real drive, but slower than it is now.

I have one RK05 drive and a controller set for my 8/e but I have not gotten around to working on that.

Maybe this project configured as a drive tester would be helpful to debug the drive. With different Pico software and FPGA code the device becomes a tester. I think the emulator by @PDP11GY can also become a tester.

 

[gwiley]

Just a quick update… at the end of last year after a good amount of testing with RK8-E controllers, I updated all of the PCBs: emulator main board, front panel electronics, microSD adapter, M993 clone (paddle card that connects from flat cables to RK05 bus), tester adapter paddle card, and a power supply to power 3 emulators.

I was trying to send them to JLCPCB to be finished ahead of the lunar new year but didn't quite succeed. The PCBs were made and SMT assembly was almost complete, but the final QA inspection happened after everyone returned after their holiday.

I just received the boards (10 of everything) but haven't built them up yet with the through-hole parts. I'm still waiting for a couple of parts from China. I've started to assemble 10 kits of parts and want to build the first updated emulator from one of the kits to confirm that the parts kits are organized properly.

A few things are on github but there’s a lot more that still needs to be uploaded. The plan is to make everything available… Gerbers, BOM, pick and place file, schematics, priced parts list with all of the sources, FPGA code, software, etc. I just hesitate to put files there that until they've been rigorously debugged and tested.

Main board v1:


Had JLCPCB install every SMT part. There is a missing IC that's an alternate part in case the new reset chip has an issue. Some of the updates are: there's a power connector, added two additional SN75452 drivers so the board has 20 outputs and 20 input so that a board configured as a tester can drive every input signal of an emulator for the i/o test, swapped out all FPC connectors for 0.1" headers (the one that's still there is a spare to use the old microSD boards). A ton of traces in the middle had to move to fit in the larger connectors. Added a lot of silkscreen text to assist with assembly.

Front panel board v3:

Replaced the FPC connector with a 0.1" header. Added a lot of silkscreen text to assist with assembly.

microSD adapter card:

A strip of microSD adapters... replaced the FPC connector with a 0.1" header. There are millions of microSD adapters in the world already, but this one has mounting holes and board outline exactly right for this application.