Finished most of my CP/M compatible OS outline today. :) Looking for suggestions.

[cj7hawk]

Thought I'd share with the group and thank the group for it's support and help with my project - The Open Spectrum Project.

I've been working on building the Sinclair Loki as a long-term project. It was touted in 1985 as a machine that would have similar hardware-accelerated graphics to the Amiga, Better sound than the Atari, and cost only GBP200 - around USD400... Which at the time would have been amazing.... In fact it was widely thought to be impossible by everyone to complete at that price. So Last year I did a university assignment looking into whether it was possible and discovered that it was actually plausible, and I've been slowly working to making one using only the technology of the era that was available and common in 1985. Long Story Short, since I haven't had time to revisit the hardware, I've been working on the OS ( Loki OS )which was supposed to be based on CP/M and customised for the Spectrum.

A key element, and the reason I've been learning CP/M is that it would have run CP/M as a base OS, but would have needed to be Case Insensitive as the original Spectrum it was to replace needs to support both lower and uppercase filenames, so I built an emulator, and an assembler ( because at the time I couldn't find a suitable alternative for Windows 10 ) and then began writing CP/M from scratch, much the same way as MS-DOS was created - difference being I needed mine to be compatible with CP/M and to be able to run CP/M programs, which was a Loki specification.

And then I discovered just how difficult a task that is to complete.

Today, I finished the initial build of my CCP, My BDOS is mostly functional ( still a few bugs and missing functions, but supports random read and write and sequential read and write and follows CP/M format for disks and storage ) and my BIOS is skeletal, but sufficient for testing. I finally reached the point where I can load up a CP/M file and execute it and it runs OK, so things like BASIC work well on it even before I work on control codes. I've run a couple of flavors of microsoft basic, nevada basic and a few others, and they all seem to work OK now.

It's not as cleanly written as CP/M was - writing an OS from scratch is something I'm not an expert at, but it works, and the BIOS, Stack and BDOS are about 4K combined, and the CCP is another 4K, though the current CCP also includes a monitor and debugger and has extra functions for disk handling like Copy ( DOS style - which works really well with CP/M ) and all of the other functions expected ( DIR, ERA, SAVE, RENAME etc ) work OK. Since the CCP will be stored in a ROM, reloading post execution of a COM won't be a problem or a delay.

I've learned a lot writing an OS. I tried to better understand the original by reading the source, but that didn't help, so my code structure is very very different, though even in writing it differently, I've learned a lot about why CP/M was the way it was. I push the stack a lot less than CP/M does and make extensive use of in-code variables and both index pointers (eg, with FCBs), though I've tried to avoid using the shadow registers.

I'm also thinking of including a small assembler and debugger ( both very rudimentary ) in the monitor, and was curious as to what other small routines anyone might think would be worthwhile including in a CCP? Such as Format, Assembler, Disassembler etc and other common functions, assuming that there's about 12K of reasonable space for an over-stuffed CCP, allowing still that replacement routines can be run from disk in any event, as could a complete native CP/M if the user wanted, so I'm not too worried about cloning CP/M as much as just being compatible.
( 12K is kind of the practical limit, given the CCP also uses 16K for copying extents and that allows CCP functions without obliterating existing user code within the lower 32K of space - and it doesn't do anything like paging at the moment, just one big contiguous memory space ).

And likewise, what functions would you definitely avoid putting into a CCP? The sort that should only even run off of a disk? ( Assuming there are functions like that ).

I'm putting my code down for a couple of weeks to finish other projects, but wanted to ask the combined wisdom here for suggestions, thoughts and anything else that might come up in the discussion.

Also a big thanks to Mike, who helped me get my Amstrad working and talked me through my early figuring out CP/M disk formats, and Doug, whose emulator made it possible for me to figure out why mine wasn't working right, but thanks also to everyone who has helped so far. It's been amazingly fun, though at times frustrating, writing an OS from scrt

David

 

[Chuck(G)]

Well, there are CCPs that do nothing but load executable binaries. No intrinsic commands, aside from perhaps providing memory examination and modification and a few very low-level commands.

 

[cj7hawk]

Thanks for the thoughts Chuck - Do you mind if I ask what is the most complex CCP you've encountered and what functionality did it have? At the moment, I have only added a file copy command ( I used COPY instead of PIP, so it won't clash with PIP later ) and a monitor that can access both memory and disk contents. However I was thinking to add a basic disassembler and assembler to the monitor ( like DOS' debug command ) to round it out.
Since I am planning on having the CCP on a memory disk in the final architecture, reloading it isn't a problem for me, since the boot rom will double as a drive, and I'm placing a non-exclusive BIOS/BDOS/CCP on it.

I also miss having a rudimentary format commant at times, so thought that might also be useful, as it shouldn't be a complicated thing to add.

I can't imagine that much is required, since all other commands can be loaded later, but there are times, when switching disks, that I desire a function that I can't use without a specific disk, and it's not practical to copy those functions to every possible disk I format, so I figure those are the kinds of applications I want to embed in the CCP.

Also, I'm not sure if there's a good reason not to make a fat CCP given it's entirely sacrificial to applications - so I was thinking of extending the command loader to the BDOS, or at least a part of it, allowing a command to take up the entire TPA, then reload the CCP on return or reset.

Though this is only my current thinking - I don't have the background with CP/M to know if this is a bad idea for any specific reason - Hence I am realy looking for some help thinking around this problem by those more familiar with what problems this might cause.
The Assembler functions I mention are intended to be very rudimentary - Like Debug. And would work under the monitor directly, rather than being intended for any serious work. I should probably add that I'm working in z80 and not 8080 in case that makes a difference.

David.

 

[durgadas311]

There have been programs that make assumptions about the size of the BDOS and CCP. The DRI 2.2 BDOS was 3.5K and the CCP was 2K. However, that's probably not something you need to worry about (I think most of those programs got exiled a long time ago).

CP/M 3 went to running the CCP at 0100h and using an RSX (resident system extension) in high memory to load the program - allowing programs to also optionally use it to chain other programs.

Note, CP/M program semantics already allow for a program to use the entire TPA. The CCP (and often BDOS) is reloaded on warm boot. However, such programs cannot "return" to the CCP - they must know to do a warm boot (jump to 0000). Some programs would "compute" (assume) the location of the CCP and then use that to decide if they had overwritten it. One place where the size of the CCP mattered.

 

[Chuck(G)]

When you think about size, consider that a DOS/360 OS could occupy as little as 6KB of memory.

Another old guy story: When Neil Lincoln (CDC/ETA) was pitching his "Super X by God" machine to the government, he ran the design highlights past me for comments. Rather than use the variable allocation system that we're used to divide user- from privileged-memory, he simplified things by saying "The OS has 16KW (64 bit words) and it's hard-wired. Any instructions executed in this space will be privileged--anything outside will be user mode". When I mentioned my reservations about the memory limitation, his response was "If you can't fit an operating system in 16K, you have no right to be in this business". Would that we could say that to our bloatware promulgators today! FWIW, the machine was never built.

Now, get off my lawn!

 

[cj7hawk]

Thanks Chuck, That's a interesting story and retelling is appreciated. I was just a small kid when all of this was going on, living in the most remote major city the world where technology took years to trickle down. Most of what happened I had to read about much later in book compiling stories from the era - it wasn't until 1982 that I got to try a home computer hands-on and after that, spent the remainder of my childhood trying to work out how it worked and add functionality to it. OK, I'll get off your lawn now

Still, I still can't get over the fact that even the basic I now use requires >20K just to 'print "hello"'. No windows or anything else, just a straight text interface. Full of bloat

eg;

D:\microcontrollers\basic>type hello.bas
print "hello"
D:\microcontrollers\basic>dir hello.exe
 Volume in drive D is New Volume
 Volume Serial Number is 4E27-8CF7

 Directory of D:\microcontrollers\basic

29/12/2022  08:33 AM            25,088 hello.exe
               1 File(s)         25,088 bytes
               0 Dir(s)  1,417,220,141,056 bytes free

D:\microcontrollers\basic>

as
From the conversation and Doug's comments also, it seems like the key theme is to keep the BDOS/BIOS tight, while the CCP is just a convenience and can be reloaded all the time with the right architecture when there's not a disk system involved, or more than a "track" to read, and it still meets all CP/M requirements since it's entirely sacrificial command-to-command.

Which answers my original second question.

Only the first remains then - It seems a monitor is a good addition to a bloaty CCP, and I think "COPY" is useful too. What are the other common CP/M functions that were usually only available as disk commands would be nice to carry with the OS? I've been trying to think of a few - stat came to mind, which should be tight code given the AVtables are already stored, as does a way to see file size. Were there times you booted off of a bare system disk and then realized a command you really needed wasn't on it? I've noticed a lot of early CP/M disks were fairly small as capacity goes, and lack directories, so system files seem to get quickly abandoned to make room for the disk's primary application.

The other think that came to mind was a very small text editor/creation facility - eg, with DOS it was "copy con: filename.txt" which created a quick file.

The current list in mind then is;

monitor - including simple label-free assembler and disassembler, disk reader/editor, hex editor and execution capabilities.
micro-editor - One line at a time, no backward editing and Ctrl-Z to exit, written to a filename as text.
upgraded dir function to show remaining disk space and file size of the files listed ( as a group ) - n Kb files listed, n Kb disk used, n Kb disk free.
format - A way to format a disk, to at least hold data.
copy - basic file copying but at up to an extent at a time.
All of the usual CCP functions.

And also basic IP (via SLIP) which I'll shoehorn into either the BIOS or the BDOS with a primitive, possibly along with UDP support allowing for windowless transfer since memory is a little tight. Was there ever a standard for that for CP/M?

I went looking for other people who have written BDOS / CCP before and found a few, but they were all in LBR format, which is a pain to decompress, so I need to write a windows-10 function LBR decompressor as well so I can see what others did in the past.

Thanks again for the thoughts so far -
David

 

[Al Kossow]

"If you can't fit an operating system in 16K, you have no right to be in this business".

You can't even fit the exception table of a modern processor in 16K

 

[durgadas311]

Note that STAT.COM is a rather large "swiss army knife". You probably don't want to include all of that in your CCP. I assume you really mean a "long directory listing".

The original 2.2 CCP had DIR, ERA, TYPE, SAVE, REN, USER. I used a CCP that added PRNT ("type" a file to the printer, with FF every 62 lines and expanded TABs), RES to reset disks (without ^C or OS diskette in A: ), and CLS to clear screen (not practical unless you only have one terminal type ever connected). This was done under the constraint of keeping the CCP at 2K in size.

One of the motivations for DRI having built-in commands in the CCP was for convenience/necessity when (temporarily) having a diskette in A: that did not have an OS on it (was not SYSGENed). This included single-drive systems, which were common at the start. With a non-system diskette in A:, you could not ^C to reset the drives nor run any program that exited via warm boot (even if the program was on the current diskette).

As far as networking, DRI created CP/NET which was a way to "network" drives or the LST: device (also CON:, but that was abysmal). IP wasn't that well-established yet and also the size of such a network stack on even a Z80 was prohibitive. Even CP/NET cuts into your TPA a bit.

You have some pretty loft goals here, it will be interesting to see whether you can reach them.

 

[whartung]

Today a "do nothing" CCP may be more viable, i.e. one that has little capability beyond running commands, or perhaps, several commands ("BAT" files, etc.).

The reason for this is simply the system are much faster.

Having to load DIR, scan the directory, and then reload the CCP on a 2-4MHz 8080/Z80 with ancient floppies just tries everyones patience.

Also there's the overhead of single use commands in terms of disk space and directory space that makes it more difficult as well. On original CP/M that, too, would be quite costly.

 

[Svenska]

I would not include FORMAT functionality, as it is specific to the underlying disk and file systems (it would be a waste of address space on CP/NET, for example).

Generally, DIR (plus file sizes and pagination), COPY (including from the keyboard), TYPE (including pagination and to LST/PRN), ERA/DEL (with confirmation), REN and possibly SAVE are sufficient in my opinion. Any remaining space should be used for batch files, line editing and command history, not intrinsic commands of dubious value.

Keep in mind that 8-bit systems in today's world are not constrained by memory, but address space instead. In the past, it was both plus high-latency, low-throughput disk access. Reloading CCP is very common, so it needs to be fast.

 

[Chuck(G)]

Take a look at the distribution PRLs with MP/M II:

ABORT.PRL     DSKRESET.PRL  MPMSTAT.PRL  REN.PRL    SPOOL.PRL     TYPE.PRL
ASM.PRL       DUMP.PRL      PIP.PRL      SCHED.PRL  STAT.PRL      USER.PRL
CONSOLE.PRL   ED.PRL        PRINTER.PRL  SDIR.PRL   STOPSPLR.PRL
DIR.PRL       ERA.PRL       PRLCOM.PRL   SET.PRL    SUBMIT.PRL
DISKSTAT.PRL  ERAQ.PRL      RDT.PRL      SHOW.PRL   TOD.PRL

Notice anything? All of the "builtin" commands are separate page-relocatable binaries, right down to ERA, TYPE and REN. In other words, if you typed "DIR" it was loaded from disk and executed. It makes sense, if you have a low-latency storage medium with enough space. Essentially a do-little CCP.

 

[durgadas311]

Well, MP/M did not have a CCP - it had a CLI that had to be re-entrant. It would have been too costly (in RAM) to have built-in commands. There are a couple optional "builtins" for MP/M, like MPMSTAT and ABORT, but those are a different thing. Not really a good comparison, MP/M is a different "beast" with different goals and requirements.

 

[Chuck(G)]

I was pointing out that a massive CCP is completely unnecessary and, in the case of today's technology, probably irrelevant. I recall that ISIS II didn't have very many internal commands; DIR, COPY and RENAME were not internal commands, but loaded on demand. Same author, floppy disk system, runs in 32K.
Externalizing the commands also allows for the user substituting his own versions.
On DX85 (floppy based again), we went with the simple loader with a couple of debug commands (load, dump, execute) to enable live patching of binaries. Everything else was up to the user. The most common configuration was to pass control to a BASIC program, which would then present the user with a menu-driven interface. But all utility commands (DIR, DELETE, COPY, etc.) were external CPU was a 3.5MHz 8085. The interesting thing was that the ISAM file management was resident, because it was used by many applications--we had a multi-user system.

 

[cj7hawk]

You have some pretty loft goals here, it will be interesting to see whether you can reach them.

At the moment it's only simulated and running in the emulator, so I still have to build the hardware as well as finish the software - Lofty goals are good - What they say about aim for the moon, and even if you miss you'll be up amongst the stars.

Though I think I'd still be working on getting the CCP right if not for your emulator, so the path is not without it's difficulties.

Thanks
David

 

[usotsuki]

The funny thing is the Unix equivalents of "dir", "era", "type" and "ren" (ls, rm, cat, mv resp.) are all external - generally with the Bourne shell, anything that doesn't absolutely *have* to be implemented as part of the shell in order to be implemented isn't.

 

[Chuck(G)]

If I were still using CP/M routinely, I'd like to see a better version of SUBMIT--preferably one that doesn't do a disk write every time a line is processed.

 

[cj7hawk]

If I were still using CP/M routinely, I'd like to see a better version of SUBMIT--preferably one that doesn't do a disk write every time a line is processed.

OK, this one has me scratching my head - and I was not aware of this behavior. Why does Submit do a disk write each time a line is processed? Or alternately, if the reason is not known, where does it write to and what does it write?

 

[durgadas311]

Submit (2.2) uses an obscure "feature" of BDOS file I/O, and truncates one record off the file with each command executed. The temp file is created backwards, first command at the end. CP/M 3 submit is a bit more sophisticated.

 

[Chuck(G)]

It's been years, but here's what I remember. SUBMIT creates a file on the boot volume called $$$.SUB and copies the orignal .SUB file, one statement per 128 byte sector in reverse order. Then, as it executes statements from the $$$.SUB file, it truncates the file one sector per statement. If your boot volume is write-protected and you have a $$$.SUB file on it, CP/M will execute the statements in the file with every reboot.
XSUB adds another layer of strangeness to the production...

There were turnkey applications that, as part of the boot process, placed a $$$.SUB file on the boot volume, so that the user never was able to get manual command-line control of the system.

The operation of SUBMIT is why there's no way to jump backwards in a SUB script and why statements don't use substitutions within statements.

 

[JohnElliott]

OK, this one has me scratching my head - and I was not aware of this behavior. Why does Submit do a disk write each time a line is processed? Or alternately, if the reason is not known, where does it write to and what does it write?

CP/M 2 SUBMIT parses the .SUB file and writes a temporary file called $$$.SUB with one line per record, in reverse order. The CCP checks for this file, loads the last record into memory, shortens the file by one record, and executes the command. By default SUBMIT.COM writes $$$.SUB either to the current drive or to drive A: (depending on whether an official patch has been applied).

SUBMIT was redesigned for CP/M 3 to share code with GET (the keyboard input redirection facility) but the old $$$.SUB method is supported for backward compatibility. CP/M 3 also allows the drive for temporary files to be reconfigured, so $$$.SUB can be put on a fast drive (eg a RAMdisc) if your computer has one.