Choosing MP/M and CP/M disk sizes, vs. remaining RAM

[tomjennings]

I'm working at compromises regarding disk size for my CP/M and now MP/M machine. In the old days this wasn't much of an issue, even with early hard disks. Today I've got drives emulated on SDcards. The SD cards are infinitely large. It's tempting to made drives "as large as possible" but of course that shrinks TPA. And what was OK on CP/M is less so on MP/M.

There's not a lot to discuss but I'm curious what sort of compromises and decisions folks have made. I'm using the largest block size, the BLS, of (16384 bytes), and fiddling with number of blocks (DKS). The well known effect is that ALVx, the allocation bit map I assume, needs a bit per block. For my CP/M I have eight 32MB drives fixed two removable, that's 3K of disk buffer, for a 59K CP/M. For MP/M that leaves 47K bank size, not enough.

The MP/M manual states is supports drives up to 25x megabytes. Assuming 16384 byte allocation units, that's 16384 blocks, or a bit map of 2048 bytes, rather large. I suppose if it's your only drive...

What are folks doing here?

 

[durgadas311]

There are practical limits to how large a CP/M drive can be. One is the directory size, at least without CP/M 3's directory hash you incur a lot of overhead searching a directory large enough to support all those blocks. As you point out, the ALV gets outrageous too. Practically speaking, how much data can you really USE in CP/M? While it's compelling to try and use more of the SDCard, it may not really get you anything - and you even end up into diminishing returns.

Each OS has it's sweet spot. CP/M 2 was cheap and easy to port, and so fit well in the low-end of the market. CP/M 3 is higher-end for applications demanding higher performance (when I switched to editing source and building my BIOS on CP/M 3 it was like a speed daemon). MP/M would be where you need to have multiple users in a shared environment - although CP/NET allowed that at a higher price but much better performance for each user. There may have been some MP/M installations that leveraged multiple processes (single user console) in which case there was no alternative - but TPA was less of an issue.

I tend to setup 8 x 8MB partitions on my disk images for CP/M 3 on the Heathkit computers. Even that's probably overkill. I'm using a 4K block size and ~2000 blocks.

Block size really is dependent on what size files you'll be using. If a disk is used for a database or something else where the files are large, a large block size makes sense. If you're doing something with lots of small files, a large block size hurts. Finding a general-purpose compromise isn't always easy. And total number of blocks informs directory size, if most files use one block then you'll need the same number of directory entries as block to be able to use the whole disk. Large files means less directory entries - but a large file still uses multiple directory entries, influenced by the EXM which determines how many 16K extents can be represented by a single directory entry. Lots of inter-dependent variables and considerations. If you want to dive into that, we can go into more detail. But one of the most common mistakes made when "designing" a DPB for a disk is not considering whether the the entire disk can actually be used - especially in the target application environment.

 

[cj7hawk]

So you can fit 128K of blocks into a byte at maximum. If you're not planning on too many files, that works pretty well and should work out to around 192 or so bytes for an allocation vector for around 25Mb of disk.

If you reserve 256 bytes for allocation vectors, I wonder if you can just rewrite the bios to page in different allocation vectors based on the SELDSK call so that all DPH point to the same block of memory for the AVs but you just swap for the current AV when the disk is selected?

Actually for that matter, I wonder if you could just save the AVs to the DISK itself somewhere in the reserved boot track and read them in like a pseudo-FAT every time you select the disk.

That would allow you to have 16 AVs for 16 25Mb partitions without too much overhead.

Then you would only need to write a CHKDSK for CP/M to recover lost allocations It would save time having to build the allocation vectors each time you relog the disk too

 

[tomjennings]

Each OS has it's sweet spot. CP/M 2 was cheap and easy to port, and so fit well in the low-end of the market. CP/M 3 is higher-end for applications demanding higher performance (when I switched to editing source and building my BIOS on CP/M 3 it was like a speed daemon). MP/M would be where you need to have multiple users in a shared environment - although CP/NET allowed that at a higher price but much better performance for each user. There may have been some MP/M installations that leveraged multiple processes (single user console) in which case there was no alternative - but TPA was less of an issue.

CP/M makes it easy -- everything's so small that even the 3K of alloc able for eight fixed/2 removable disks leaves a roomy TPA. I know how the calcs work (for xP/M, DOS FAT, ext3, etc) and starting out (OK again, decades later) I just went for maximum. This was fine for CP/M. MP/M, not so much.

"Most files" are one block, 16K, a few are 2 blocks, so the current setup, 32 MB drives, 16384 BLS, provides 2048 blocks. 1000 files on a CP/M disk will consume slightly more than half the space. A thousand CP/M files is a lot of files! Even with MP/M making user numbers useful. But maintenance, moving files, backup, etc requires at least one additional drive, and then there's removables.

Reducing to two fixed, two removable, 32 MB each is 1K of buffers plus 256 for the removable dirs, less than half of the foolishness currently. "Many" things run fine in the 47K TBA/bank, another couple K should be fine.

It boots into CP/M, and run MPMLDR to start MP/M, I will not have it boot into MP/M directly. So I can, and may, leave 8 drives for CP/M, where the TPA is fine, and have only the first two be MP/M drives. Weird but functional. If I got fancy I could make MPMLDR ask "2 drives or 4?" and boot the small bank/TPA 8-fixed XIOS for system tomfoolery, and 2-fixed for actual work.

So you can fit 128K of blocks into a byte at maximum. If you're not planning on too many files, that works pretty well and should work out to around 192 or so bytes for an allocation vector for around 25Mb of disk.

Yes, it's so nicely linear... one page (256 bytes) for 32MB! It's so easy to work with!

If you reserve 256 bytes for allocation vectors, I wonder if you can just rewrite the bios to page in different allocation vectors based on the SELDSK call so that all DPH point to the same block of memory for the AVs but you just swap for the current AV when the disk is selected?

I was wondering why DRI didn't put the allocation vectors down in bank 0, but there must have been some reason, such as BANKBDOS not being the entity that needs access to it. It is tempting to think up some novel scheme! I technically could, in my emulating host, trap AV addresses, look at the current disk, and swap in the corresponding table and.... greatly increase complexity, and probably Have Less Fun coding it all up! lol

But yeah, it's fun to think about. The address space is just so small! But they've done such great things with so little resources, I'm gonna work within what they've given us.

(I wrote a simple BLS/AV table calculator in a spreadsheet, so it was reason to learn SuperCalc2... man is that a nice program, even by today's standards.)

 

[cj7hawk]

We never found out how far you can push CP/M... History ended that experiment far too soon

 

[Svenska]

I've built a CP/M 2.2 system with SD storage, and decided to only expose four drives of four MB each to reduce memory cost. That is still plenty of storage and switching "virtual disks" provides extensibility for little memory (never implemented switching, though).

I do not see the appeal of having eight or more drives, my brain can barely keep track of three. I feel the same about 8 MB drives, most files are quite small and dealing with many files becomes tedious. While USER areas are not supported too well in CP/M and its utilities, I use them to keep things separate - otherwise even 4 MB drives are too much for me.

The TPA cost boils down to (number of drives) x (size of drives). Limiting both helps a lot.

 

[norwestrzh]

I've always used sixteen 8 MB "drives" (4k BLS) on the compact flash I use. That gives me about 55k TPA. Maybe I'm doing something I shouldn't do? The downside to larger drives is the time needed to scan the directory when the "drive" is logged. There are well over 1,000 files on some of my CP/M SBCs! It surprises me that so many CP/M applications exit by jumping to 0h (warm start) or making the "system reset" system call (function 0). That un-logs all "drives". I always try to save the stack passed to the application, and then restore it on exit and do a RET. That usually saves constantly re-logging in "drives". Again, maybe I'm doing something wrong?

Roger

 

[durgadas311]

That is where CP/M 3 really makes an improvement. With directory hashing, and relaxing the impact of "jmp 0", the speed is quite impressive. and the higher TPA due to much more being kept in banked memory makes the experience great.

I recall the company I worked for being one of the first to actually enable all 16 drives (8 SASI HDD partitions plus lots of floppies). At a West Coast Computer Fair, DRI was sending all their employees to our booth to see 16 drives in real life, saying "it really works!".

 

[durgadas311]

There is a "bug" in RMAC/MAC that took drive P: to mean the printer, so when trying to compile things using drive P: it failed. Oddly, LINK did not have the problem. There are patches to RMAC/MAC to fix this, aligning them to LINK's definition of X:, Y:, Z:.

 

[tomjennings]

I've always used sixteen 8 MB "drives" (4k BLS) on the compact flash I use.

OK for CP/M that's OK, though it means you have 4K in AV buffer. With a modest BIOS I can see 55K TPA, and that is very good.

MP/M though, that much AV space would push the Common boundary very low. I've got 3K of AV and dir buffer already, and the Common boundary is 47K, a small bank/TPA. That's the problem I'm trying to solve.

But your idea of "many" small drives I think is the sensible path.

That is where CP/M 3 really makes an improvement.

I was trying to avoid CPM/3, for no real reason, but I think I need to revisit that decision...

There is a "bug" in RMAC/MAC that took drive P: to mean the printer, so when trying to compile things using drive P: it failed.

Ugh. Supports my using 8 * 16 MB drives. Not that a patch would be too hard.

 

[durgadas311]

The drive P: issue is really only a problem if you're using drive P: as part of you MAC/RMAC commands (redirecting input/output files to P:, or if the default drive is 'P:'). I seem to recall running into it on CP/NET, where it is convenient to assign a network drive to P: - something "out of the way".

I have patched versions here (among other stuff): https://github.com/durgadas311/MmsCpm3/tree/master/dist/cmd/bin. Basically, there are compare immediate instructions that checked for 0FH (drive `P:`) and now check for 18H(`Y:`).

 

[johnx993]

I find smaller block sizes better on machines where I do a lot of 8080/Z80 assembly work in CP/M.
Just hate writing a thousand byte program and it eating a whopping 4K on the disk.
But then... maybe that's why we ended up with larger multi-function utilities instead of lots of little ones?

 

[Koolstar42]

I am using space for 3 8Mb disks which are loaded from an SD card. They are 2kb block size and 512 dir entries. This uses 1.4 kb of buffer space in CP/M 2. The trick is that on each of those disks one of the 256 available a 8Mb volumes on the sd card can be (re)mounted. This gives 58 kb TPA in CP/M 2 and 61 kb TPA in CP/M 3. Basically a system with swappable hard drives.

 

[cj7hawk]

Large blocks are a problem, but did anyone ever create an algorythm that got put into actual file systems so that small files didn't occupy large blocks?

 

[durgadas311]

I am using space for 3 8Mb disks which are loaded from an SD card. They are 2kb block size and 512 dir entries. This uses 1.4 kb of buffer space in CP/M 2. The trick is that on each of those disks one of the 256 available a 8Mb volumes on the sd card can be (re)mounted. This gives 58 kb TPA in CP/M 2 and 61 kb TPA in CP/M 3. Basically a system with swappable hard drives.

That's an idea I always wanted to explore, but never took the time to develop. Since DRI provided no standard framework for "mount"/"unmount", you'd have to make something up and each vendor would probably have done it differently. Too bad nothing was standardized. If you made it something formal like mount/unmount, you can get away with no checksum vector since the media can't be changed unexpectedly, like a floppy can. This mount/umount action would have to be sure to reset the disk being swapped, but that's a simple matter of a BDOS call. We used to do that when formatting floppies, too. When I did think about this feature, I considered putting some sort of "volume catalog" on the SDCard such that mount/unmount/etc could report a name/label and the user could use that label to select the drive they wanted to mount. Keeping track of what is in the "Nth" drive without a label would be cumbersome, probably requiring a list kept on paper. Having that on the media and accessible by the utilities would be a great help. I would submit that such a "catalog" should include the LBAs of each drive (so they can be made any size, customized) and also the CP/M DPB (so a user can customize the driver "geometry" as well). If such a thing were standardized, then SDCards (and images) could be exchanged and used on different systems, and even accessed from PC using a wrapper around cpmtools.

One note, 512 directory entries is not enough to access the entire 4096 blocks on an 8M disk, unless EVERY directory is completely full. In practice, you'll run out of directory entries before consuming even half the storage.

Back when HDDs were first being used on CP/M, the size was usually 5MB *total* space, and if you had deep pockets 10MB (and then 12MB). These drives were typically divided up into partitions, so nothing close to 8MB (max for 2.2) CP/M drives were typically being used (even the Kaypro 10 split that 10MB into two drives).

 

[Svenska]

That's an idea I always wanted to explore, but never took the time to develop.

As I wrote earlier, I thought about this to save TPA and never ended up needing it.

Since DRI provided no standard framework for "mount"/"unmount", you'd have to make something up and each vendor would probably have done it differently. Too bad nothing was standardized.

That is true, but with the powers of future, hindsight, homebrew and emulation, we can standardize now. RomWBW is fairly widely supported and supports slices and drive mappings for them.

Large blocks are a problem, but did anyone ever create an algorythm that got put into actual file systems so that small files didn't occupy large blocks?

How do you imagine that would work?

 

[krebizfan]

The old Unix File System and compressed drives under MS-DOS could share a block between multiple small files. Nothing inherently prevents CP/M from doing the same.

I know I couldn't write code to do that at a reasonable speed while fitting in a few hundred bytes.

 

[cj7hawk]

How do you imagine that would work?

I have no idea... hence why the question occurred to me.

I suppose in practice though it would have to work a bit like how a word processor keeps a document in memory - sometimes it would be up to the write process to make the space and clean things up that it never wrote itself.

But archive files, TAP files and the like? They do achieve that. And I'd imagine some ROM based file systems might work like that also.

 

[Svenska]

The smallest reasonable block size is the physical sector size, but larger block sizes reduce the amount of bookkeeping needed. Bookkeeping costs resources, which in CP/M are memory (reducing TPA), storage (the directory size must cover enough entries to actually fill the disk) and processor time (searching large directories and allocation vectors is slow).

Trying to use partial blocks in CP/M kills the main reason for having larger blocks in the first place. (The second reason for larger blocks is avoiding 16-bit integer overflows in the bookkeeping code, but that could have been with 32-bit math at a performance cost.)

What you can do is replace the file system and related interfaces altogether. But that's not an algorithm, that's a total replacement, and it wouldn't be CP/M. DOS and Unix did that, but they targeted much larger machines and could spare substantial amounts of memory to make it efficient. CP/M simply can't do that.

 

[johnx993]

The smallest reasonable block size is the physical sector size, but larger block sizes reduce the amount of bookkeeping needed. Bookkeeping costs resources, which in CP/M are memory (reducing TPA), storage (the directory size must cover enough entries to actually fill the disk) and processor time (searching large directories and allocation vectors is slow).

Trying to use partial blocks in CP/M kills the main reason for having larger blocks in the first place. (The second reason for larger blocks is avoiding 16-bit integer overflows in the bookkeeping code, but that could have been with 32-bit math at a performance cost.)

What you can do is replace the file system and related interfaces altogether. But that's not an algorithm, that's a total replacement, and it wouldn't be CP/M. DOS and Unix did that, but they targeted much larger machines and could spare substantial amounts of memory to make it efficient. CP/M simply can't do that.

To overcome this, maybe use banked memory ?