Michael Robinson
Member
Greetings!
I have been enjoying using Joerg Hoppe's Unibone on my PDP-11/45 for a few years now. If you have a Unibus PDP-11, I highly recommend getting one!
It occurred to me last week that I ought to be able to use the Unibone to snapshot the entire PDP-11 memory. To my delight Joerg's
I have watched several operating systems boot using this technique. Ultrix-11 is the most exciting of the bunch, because it is quite involved. (For comparison, RT-11 boots so fast that I basically can't capture it! Well, I can see the first and second stage bootloader zip past, but the drivers load so fast that there isn't much to see.)
TL;DR: Here's the result:
The interpretation of the memory display in the upper right corner is that the horizontal axis is for addresses, while each column is a histogram of byte values aggregated over swaths of 1024 bytes. The vertical axis has 0 at the bottom and 255 at the top; brightness indicates how frequently that byte value was observed in the corresponding swath. PDP-11 opcodes are not uniformly distributed in byte values (nor is human-readable ASCII), so there is a definite visual "texture" that can help identify when sections of memory are copied. This happens early in the boot process. What's kind of striking to me is how huge Ultrix is and how it appears to basically reserve the lower half of addressable memory. You can see the small-ish, swappable processes slotting in and out above that once timesharing starts.
It is worth noting that although the video is a composition of two video streams, I didn't adjust the timing between them. The console and the live memory display are just as they appeared on my workstation, in sync with the front panel. There is a noticeable lag of about a second between when something happens on the front panel and/or console and when it appears in the memory display window. This is a result of the 1/2 second update from the Unibone, the network transmission delay, and a not-insignificant delay for the image viewer.
How does this work: Unibone scripting
The basic idea is that if you're on the Unibone in the
to save the contents of memory to a file
The
to the end of an otherwise working command file. Now the
How does this work: Linux workstation scripting
Once the Unibone is busy snapshotting the memory, I have a script on my Linux workstation that copies the
The memory graphic is produced by a three-step process: (1) aggregate byte histograms over a sliding window, (2) render the sliding window into a PNG, and (3) display the PNG.
Item (1) is done using a
https://github.com/kb1dds/bytewise-stats
(Build instructions are not included because all that's required is a single call to
Item (2) is done in R, but could have just as easily been done using Python, C, or whatever. As an example of what's involved,
https://github.com/kb1dds/bytewise-stats/blob/main/examples/ultrix11_booting.R
I made a few modifications from the above R script for the one used in video:
Unfortunately, as packaged with Ubuntu (and Debian),
Closing thoughts:
It's been a dream of mine to peer inside the workings of a "live" machine for a long time. Although the front panel of a "proper" minicomputer (or mainframe) gives a good picture, it's really only a tiny window in time and address space. The fact that the Unibone is fast, scriptable, and has plenty of storage space opens up lots of possibilities for visualization experimentation!
Enjoy!
Michael
I have been enjoying using Joerg Hoppe's Unibone on my PDP-11/45 for a few years now. If you have a Unibus PDP-11, I highly recommend getting one!
It occurred to me last week that I ought to be able to use the Unibone to snapshot the entire PDP-11 memory. To my delight Joerg's
demo program already has this capability. With some truly hackish scripting, I set the Unibone to snapshot the entire 248k of memory every 1/2 second and ship the resulting memory.dump file over to my Linux workstation for a real-time display. The Linux side uses even more hackish scripting; a mixture of C for preprocessing and R for rendering.I have watched several operating systems boot using this technique. Ultrix-11 is the most exciting of the bunch, because it is quite involved. (For comparison, RT-11 boots so fast that I basically can't capture it! Well, I can see the first and second stage bootloader zip past, but the drivers load so fast that there isn't much to see.)
TL;DR: Here's the result:
The interpretation of the memory display in the upper right corner is that the horizontal axis is for addresses, while each column is a histogram of byte values aggregated over swaths of 1024 bytes. The vertical axis has 0 at the bottom and 255 at the top; brightness indicates how frequently that byte value was observed in the corresponding swath. PDP-11 opcodes are not uniformly distributed in byte values (nor is human-readable ASCII), so there is a definite visual "texture" that can help identify when sections of memory are copied. This happens early in the boot process. What's kind of striking to me is how huge Ultrix is and how it appears to basically reserve the lower half of addressable memory. You can see the small-ish, swappable processes slotting in and out above that once timesharing starts.
It is worth noting that although the video is a composition of two video streams, I didn't adjust the timing between them. The console and the live memory display are just as they appeared on my workstation, in sync with the front panel. There is a noticeable lag of about a second between when something happens on the front panel and/or console and when it appears in the memory display window. This is a result of the 1/2 second update from the Unibone, the network transmission delay, and a not-insignificant delay for the image viewer.
How does this work: Unibone scripting
The basic idea is that if you're on the Unibone in the
demo program under the device emulation menu, you can say
Code:
m dmemory.dump on the Unibone's SD card. The
demo program can read command files, e.g. Joerg's pre-packaged boot scripts, though these command files do not support looping or branches. However, you can insert a delay if you like... so all one really needs to do is append about 1000 or so copies of
Code:
m d
.wait 500memory.dump will be updated every half second for about eight minutes. (You could try a shorter delay between updates. This hogs the Unibus, making the already-long boot time of Ultrix much longer...)How does this work: Linux workstation scripting
Once the Unibone is busy snapshotting the memory, I have a script on my Linux workstation that copies the
memory.dump to a local working directory every second or so. Because my day job involves statistical analysis, it was quickest for me to throw together a script in R since that's where I do my rendering. Bash would have been just fine too. I use passwordless scp to do the copy, since it scripts easily. The memory graphic is produced by a three-step process: (1) aggregate byte histograms over a sliding window, (2) render the sliding window into a PNG, and (3) display the PNG.
Item (1) is done using a
window_distribution program I wrote in C. You can get the source from https://github.com/kb1dds/bytewise-stats
(Build instructions are not included because all that's required is a single call to
gcc.)Item (2) is done in R, but could have just as easily been done using Python, C, or whatever. As an example of what's involved,
https://github.com/kb1dds/bytewise-stats/blob/main/examples/ultrix11_booting.R
I made a few modifications from the above R script for the one used in video:
- Filename paths are adjusted accordingly, obviously.
- It's on an infinite loop.
- It handles the
scpand execution ofwindow_distributionusing the dreaded-but-usefulsystem()function. - It doesn't color the memory locations by k-means clusters, since it's annoying (but not impossible) to stabilize clusters across memory snapshots.
- It plots to
memory1.pngand then renames this tomemory.png. This way, the time during which an invalidmemory.pngfile is present on the disk is minimized. Most image viewers will happily open a half-written PNG file but then promptly complain that it's broken.
eog image viewer will redraw on file update by default. This works tolerably well, but the redraw involves blanking the image first. This makes it hard to see small memory updates. Instead, I used a different image viewer that accepts an explicit reload option
Code:
feh --reload 1 -g 640x480 -. memory.pngfeh cannot update faster than once per second. (Apparently, if you build from source, you can have it trigger an update from file modification. I didn't try that.)Closing thoughts:
It's been a dream of mine to peer inside the workings of a "live" machine for a long time. Although the front panel of a "proper" minicomputer (or mainframe) gives a good picture, it's really only a tiny window in time and address space. The fact that the Unibone is fast, scriptable, and has plenty of storage space opens up lots of possibilities for visualization experimentation!
Enjoy!
Michael
