Implementing YModem - in 2026!?

[voidstar78]

I'm aware of Kermit and ZModem, but I decided to take a deeper look specifically into YModem Batch.

SyncTerm kept crashing when I tried its YModem. And I found a bug in ZOC's YModem when using it at low speeds (which the author has since fixed now in their version 9).

But experimenting with Telix, QModem, Procomm, TeleMate, Conex, and a few other terminals - each one is slightly different in how it handles YModem. For example, TeleMate uploaded via YModem fine, but on download is padded the received file to multiples of 128 byte block sizes. Procomm's YModem was actually kind of broke. And many of these terminals didn't support actively selecting multiple files (one of the benefits of using YModem Batch) -- you had to shell to DOS and write down the filenames. Telix was mostly OK, except on receiving a 0-byte file, it wouldn't actually save it to a file. So little inconsistencies like that (relative to receiving the file from a x64 Wintel system and ZOC). I tried szsr under Cygiwn or MSYS2 but not much success (but didn't try too hard).


The "YModem Spec" document floating around from 1988 was also still unclear on a few things. So, I just recorded a working ymodem transfer, and dug through the interactions. That analysis is attached, and some other test artifacts are archived here:

GitHub - voidstar78/ymodem: Notes on a modern implementation of YModem Batch for RS-232 ports

Notes on a modern implementation of YModem Batch for RS-232 ports - voidstar78/ymodem

github.com

I see how today we could make an improved protocol - but lots of existing legacy software is out there that supports ymodem (well at least for MS-DOS).


Piece parts to a work-in-progress project.

 

[MiaM]

Cool!

"Fun fact": (sorry if I'm Captain Obvious)
In case you wonder, I think that the general padding thing used in some older modem protocols kind of originates from CP/M (and maybe other OSes too?) not having variable length files. CP/M files always have a length evenly divisible by the disk sector size, hence that the end-of-file character was important to keep well defined for text files.
Thus to transfer files without needing to interpret different file types, you would send all disk space allocated to a file, even if the last part of the last sector were unused and contained zeros or garbage.

 

[voidstar78]

It may be even older than that - it may be from when 7-track and 9-track tape were the main means of storage.

I view it like buckets. To transfer water from one pool to another, you'd use buckets from Lowe's - and they'd all be the same size. And that last bucket most likely won't be full. And the first bucket contains a little letter, detailing like "type of water" (filename) and other meta data (why they used octal for the timestamp is beyond me, when they used ASCII for the filesize and other attributes -- I know octal was more "in fashion" then, but the inconsistency in the header still seems odd; maybe the API to query time on most Unix systems was already natively in octal?).

On the earliest IBM 8" disk drives, I believe from the get-go they used 512-byte clusters. And on some DC300 tapes I've seen 512-byte clusters used there as well. I only used one 9-track Harris tape system, but dunno what the typical cluster size was -- but I do know you had to "mark" your files first, to kind of pre-partition and decide how many files and their assigned sizes upfront. But we overlook this often today - that your filesize is really the ceiling of a cluster size (in terms of that's how much effort will be involved, whether it's 0, 1, a few, or a full cluster size). And reality is, you can "hide" data past the "end of your file"


In early systems, particularly microcomputers, even 512 bytes RAM could be a lot. So a 128-byte block size is a reasonable choice for the time. You'd have to place a block into RAM, to check the CRC before committing it back to a local file. With pristine quality null modem cables, a CRC error is less likely these days - of course over copper phone lines, there was always the chance someone picked up the line on either end for al ittle bit. I guess if I prep this test again, I should try a long transfer and just yanking the null modem cable part way through.


In the original YModel "spec" doc from 1988, I think it also talks about "if the file is growing while it is being transferred" -- implying you might not really know the total filesize. So the header size could be "0" or could be "99999999999" and that could be interpreted in different ways (as like an "infinite file") Imagine rendering a video, and then transferring it basically as it is being encoded or transcoded. The those oldays, maybe the data was a minute-by-minute weather logger, and trying to stream that log file as its being appended? Or maybe another use-case was they did have tarballs then, so imaging sending the compressed tarball of a set of files while they are being compressed or at least concatenated?


BTW, the "error" or bug in ZOC was really just a choice in how patient to be in waiting for a ACK or NAK, or how many re-tries. Their implementation of the protocol doesn't expose all those dials - and at slow 300 baud, they just didn't want long enough for that first header block to be completed, and gave up on it too soon.

 

[voidstar78]

REV3, some minor updates and clarifications.

Also at github

ymodem/NOTES at main · voidstar78/ymodem

Notes on a modern implementation of YModem Batch for RS-232 ports - voidstar78/ymodem

github.com

 

[voidstar78]

Here is my code for YModem in BASIC, C, and prog8 (C64/X16 6502 specialized language).

The BASIC version is slow (even at 8MHz X16), we're talking a throughput of barely 110 baud regardless of the actual baud setting (300, 1200, etc). Mainly this is because the CRC is calculated inline with the received data. It uses CTS/RTS and throttles the FIFO setting down to 8 bytes. If there is a hiccup from the sender, this BASIC version doesn't handle NAK. For small files (say under 33KB), this has been instructive as insight into the flow of the protocol. NOTE: BASLOAD is basically CBM BASIC without line numbers (and the long variable names get translated when BASLOAD generates the actual tokenized BASIC version of the listing). Meaning, it's fairly easy to back-port this to a "normal" BASIC (the port numbers to the UART will probably be different, and the Divider values).


The C version uses a lookup table for the 16-bit CRC, and on the X16 was substantially faster - we're able to get 39KB/sec (or a throughput of *almost* 400,000 baud, this was with cc65/ca65). Note that the received blocks are just stored into a BANK RAM slot, not actually written to file yet (to keep the code example shorter). This C version does handle NAK's correctly (timeout for dropped characters, mismatched CRC, or mismatched block count). Again, your Dividers would probably be different (the X16 is using a 14-ish MHz UART).


The prog8 is essentially an exact port of the C version. prog8 doesn't support unsigned long, nor arrays longer than 256 indexes. So those adjustments were made, and it will receive 128-byte or 1K blocks, and batched sets of files (just like the C version). prog8 uses 64tass, but the overall throughput was the same in the resulting PRG.


NOTE: None of these support cancel either (intended for 8-bit so I don't poll the keyboard for ESC or anything - a "real" version would check for this then send CAN CAN to indicate to the uploader to abort the transfer).



A question I have is, I've been doing overnight test-transfers of a large 2GB file. I'm getting an average of 1 error about every 500MB. This error gets corrected, since the CRC and NAK stuff is all working and the last good-block gets resent. So, the transfer is still successful. We've wondered if it's bus errors (on the X16), but that doesn't really seem realistic. I added SEI/CLI interrupt blocks around the main receive loop -- and that did seem to make a difference (prior to that, I was getting 1 error per about 150MB). I've not explicitly checked UART overflow bits and such. Again, we reduced the FIFO down to 8 bytes (so 8-byte runway for the CTS/RTS transition) instead of the 14-byte with 2 byte runway. I still think this is UART-overflow related more than bus-hardware ringing or similar kind of issue.

I'm going to test more with more traditional systems (a late 90's laptop with a super IO that can still handle 480Kbps), to see if it gets any errors over a long sustained transfer with YModem. In the 90s, I don't recall ever using X/Y/Zmodem for such a large transfer, so never really scrutinized the "natural" error rates (of either bus or UART related, not folks picking up the phone).



"Sector not acknowledged" means we didn't send the ACK within the timeout ZOC expected (which is about 15-25 seconds). The transfer didn't abort, just the last packet got re-sent and the protocol re-synced - so not a total show stopper, but curious how on a 12ft wired connection, this hiccup still happened (albeit quite rarely, since >500MB is fairly massive for most 8-bits).

 

[ECNeilson]

This is very interesting. I have found YModem to be one of the better protocols for 8-bit computer file transfers (when it works between computers). I get get higher speeds with YModem than I do with ZModem and still get the batch downloads unlike XModem. I've been meaning to make a matrix of which computers and modem software work interoperate with YModem. Now you've inspired to get that done.

ECNeilson

 

[voidstar78]

This is very interesting. I have found YModem to be one of the better protocols for 8-bit computer file transfers (when it works between computers). I get get higher speeds with YModem than I do with ZModem and still get the batch downloads unlike XModem. I've been meaning to make a matrix of which computers and modem software work interoperate with YModem. Now you've inspired to get that done.

ECNeilson

This might not be quite the same as what you had in mind, but I did test some serial exchanges last year and made this report:

[ which yes, we found that across a wireless connection -- like two WiModems -- that YModem ended up substantially faster; ironic since per my understanding, YModem was based on YAM which was developed for radio packet comms ]

YModem is fickle though, which is why I think it gets a bad rep. Reading between the lines in the 1988 "Chuck standard" doc for YModem, ymodem got a little misinterpreted in the first year or two of its introduction. Then came YModem-G, or how is YModem different than YModem Batch, etc. So in 1987-1990 terminal software, it be a little hit or miss on its correctness. Tentatively, modern day TeraTerm struggled for me with YModem - but I need to go back and re-test it more fairly. I found ZOC works for me for modern 64-bit app and YModem (there were some bugs with its ymodem at low speeds, but I got those fixed now in ZOC9).

The matrix of this would be complicated, by not just systems involved, but specific serial H/W spec and terminal software involved.





I'm running YModem across ZOC9 and WinXP HyperTerminal now, with steady 9.2KB/sec (old P3 laptop's UART sadly only goes up to 115200; I have a faster PCMCIA serial card, but left it at another location and will try with that again later). I'm using the same exact file as in prior testing, so we'll see in a day or two if any "natural" block resends happen under this setup also.

This test file is "so large" the Windows HyperTerminal shows the size as -1829079K. But, the transfer is proceeding just fine. Technically it's true, most YModem implementations used an unsigned long for the file size, artificially limiting it to 2GB files.

 

[voidstar78]

So even on a regular PC to PC connection @ 115200 baud, still got a "natural" retry (Short packet) after about 150MB xfer. Despite CTS/RTS, overflows can still happen ("naturally" meaning not due to pulling cable, picking up phone, etc).


Also, here is a YModem demo across the X16, where we assume the uploaded file is an exact VERA VRAM dump. We resend blocks if one ends up short, and 2min into this video shows the result (pixel data gets repeated so the image is skewed, but not missing).

 

[voidstar78]

Wanted to archive/note an experiment I ran - I decided to try transferring my "normal large test file" (>2GB) with YModem on "modern equipment" (ZOC terminal under Win11/i7 PC to a late 1990s P3 laptop at 600MHz w/ WinXP). And I still got a couple "Sector not acknowledge" errors that average to 1 per 500MB. What's interesting is that's the same error rate I observed on an 8-bit X16 platform also using 16650 but even higher speed (since using a ~14MHz clocked UART). Meaning, I think this is a normal/natural error rate for this kind of framed serial - even when using CTS/RTS and reducing FIFO so there is a runway to handle in lag in responding to flow control flags, we still occasionally get an overflow such that a character gets dropped - hence "Sector not acknowledged" (it didn't get the ACK from the receiver after the block, either because it got dropped or the sender didn't xmit a full block and the receiver was still waiting for that and never sent the ACK within a timeout).

Tedious testing, but it tells me that CRC checks wasn't essential just for the "someone else picked up a phone on the line" or other "line scrambled temporarily" reasons - but there are also natural edge cases where hardware flow control still has mishaps now and then. The CRC is important for line corruption (bit-flips), but in this case whole byte gets dropped from the FIFO buffer.

(and this is all speculation on my part -- but for sure, I am observing these Sector not acknowledged errors at this rate; which just means the block is resent, costing efficiency, but not catastrophic to the transfer)

 

[Istarian]

No support for canceling a data transfer probably wasn't an issue when memory and storage were small and even a big file was likely to be less than 1 MB.

Why would you even care about a 2 GB file cap given the relatively slow rate of transferring data?

 

[voidstar78]

Issue here wasn't the size/length of the transfer. It was that during a sustained exchange (whether a file transfer or long stream of text), there are still corner cases where we still end up with dropped characters despite the flow control mechanisms. (in this case, it gets detected and manifest into resending a block - so not a big deal; but if the data is streamed straight to video RAM, it explains why the images get skewed once in awhile)


Like in multi-line BBS chat that uses exotic ANSI stream sequences, every once in awhile those go awry also (if a classic UART is involved; probably not an issue for telnets). And this is evidence as to why, even at a modest xfer rate. Or it's evidence that even 16-byte FIFOs aren't quite enough at 115200 (in my test cases, I dropped the buffer to 4-byte to give plenty of runway for the equipment to respond to the CTS/RTS switches since I was using 921Kbps -- and these about-once-per-500MB drops still happen).


EDIT: But yes, probing into late 1980s YModem code, they tend to use signed long type for the file length. So they artificially limited themselves to 2.1GB, even though that's not really a protocol limitation. But again the file length wasn't the focus here, it was just a practical way to force a sustained stream through the UART (but with the periodic CRC checks and ACK exchange to flag if any hiccup has happened)

 

[voidstar78]

YModem video and BASIC/C code samples, and then exercised for testing using a variety of legacy systems.

Note, the X16 serial solution has a UART that features auto-CTS/RTS (so the BASIC code doesn't have to deal with that). I see now for systems without hardware flow control, you just have to severely reduce the baud rate.

Then I got into the weeds of the MS-DOS FOSSIL driver (or OS2 SIO driver), that basically wraps the UART with a virtual buffer - which is essentially one of the big differences between Win95/XP serial driver than Windows 3.1 original (I got curious on it since I noticed in Win95/XP you can adjust the FIFO buffer length for input vs output, but can't do so in Win3.1).

The other habit hole is the crossover point between polling and interrupts. Some systems, interrupts have a heavier cost - since the handler code has to evaluate what caused the interrupt.

 

[fritzeflink]

And I found a bug in ZOC's YModem when using it at low speeds (which the author has since fixed now in their version 9)

Thanks for your post.
I use ZOC since I got the OS/2 Version I think it was ZOC 2.xxx in 1994(5). I use ZOC 4.15 for OS/2 today even 9.xx for Windows.

Sometimes I had to restart my Win10 PC as the ZOC GUI didn't reply if I change baud rate or disconnect and reconnect the RS232 / USB adapter.
Mostly I'm a big fan of ZOC for the great terminal emulations that makes it possible to connect a lot of my CP/M systems with the nativ terminal emulation of the past.
In ZOC 9 there was I small text bug in the GUI which is corrected in the actual version. (picture just for info below).

Mostly I use Kermit protocol for easy file transfer - of course Y or Z-modem are mostly faster.
Sadly I habe no idea to help with you Y-modem problem.

 

[voidstar78]

I like ZOC too, it's very well made and been around for a very long time - plus its extensive scripting, you can essentially host a BBS directly from ZOC.

The only problem in ZOC8 was basically they had a 20 second timeout in YModem waiting for the ACK after a block was sent. This was ok for small files. But at 300 baud: 1024 characters / 30 cps = 34 seconds. So it would start to timeout and give up after 7 retries. Not a protocol issue, other than the "max wait times" and "number of retries" isn't part of the protocol spec, so each terminal can choose differently on that. My understanding is rather than adjusting the timeout, instead they changed it so that rates below 9600 always just use 128 byte blocks.


Something not in the video: the first mention of YModem that I came across wasn't until an issue of BYTE around mid-1986 (where some terminal software started advertising support for it). But my understanding is the protocol was used as early as 1985, just it was called YAM (and used on CP/M systems), until Ward suggested the name YModem (as a natural follow-on to XModem).

Although Chuck posted a "formal spec" of YModem in 1988 to try to unify things, it seems many pre-1991 terminal software (or BBS host software) get the implementation wrong. Like there is no "formal" YModem-1K spec -- except the ZOC issue points out that it is a choice on when to use 1KB blocks, and some early systems might struggle to handle that (like something with only 4KB of RAM). So does YModem-1K mean even the header block is 1K, and even the tail/ending block? Does it imply that "YModem" then never uses 1K blocks, and only 128-byte blocks? So it's hit-or-miss if the "YModem" between two systems will actually be compatible.

 

[Istarian]

YMODEM - Wikipedia

en.wikipedia.org

Whoever wrote and/or modified the wikipedia article would seem to be suggesting that (a) many mutually incompatible implementations of "YMODEM" were produced and that a non-trivial number did not implement fall backs to smaller block sizes (<1K) or any checks other than CRC-16.

 

[voidstar78]

I see a Sept. 1982 YAM.DOC in this archive: https://archive.org/details/yam-pkt-82.10.19.vjw
I don't see things like "header block" or "inverse block count" described in there. To me, YAM looks more like a variant of FTP or KERMIT (since it describes various system commands, like a dir listing). But YAM is in the context of CP/M systems, and mentions
"...currently supports the Z89 (with aux board), Cromemco TUART, TRS-80 II, and Apple (currently with Z19 console)."

Z89 = some Z80 based system by Zenith Data
Cromemco TUART = Cromemco System One?? (68K based system?) TUART is Two or dual UART thing.
TRS-80 Model II we all know
I'm not sure what Apple / Z19 refers to (like Apple with a CP/M card? a different Zenith console?)


The earliest mention of the "what-became-YModem" header block is in a post by Chuck himself from Dec. 1983: (search for "octal" to find the function that packs that header)

rb and sb sources (YAM)

The code references V7 and longer-filenames coming in BSD 4.2


So I came up with this kind of time-line of protocols (not all of them, but some of the main ones)

1977  XMODEM-CHECKSUM  NAK --> SOH BLK ~BLK 128-DATA CHECKSUM (8-BIT) CP/M .ASM
1979  MODEM7           FILENAME CHAR-BY-CHAR HANDSHAKE, THEN DO XMODEM
1981  KERMIT           MARK LEN SEQ TYPE DATA CHECK (VARIABLE-LENGTH PACKETS)
      XMODEM-CRC       "C" --> SOH BLK ~BLK 128-DATA CRC-16 (NO HEADER BLOCK)
      XMODEM-1K        "C" --> SOH|STX BLK ~BLK 128|1024-DATA CRC-16 (NO HEADER BLOCK)
1982  (YAM as "proto-YMODEM" on CP/M systems)
      (as described in Chuck's 1988 "True YModem" spec document,
      Ward likely proposed the "enhanced XModem's" to be called YModem,
      not the header-block variants, aka "YModem Batch")
1983  sb/rb "YMODEM"   "C" --> SOH|STX BLK ~BLK 128|1024-DATA CRC-16, BLOCK 0 HEADER (unix)
1984  PUNTER           C64 OPTIMIZED (MULTI-BLOCK WINDOWED TRANSFER)
      SEAlink          XMODEM WITH SLIDING ACK
1985  YMODEM-G         "G" --> SOH|STX BLK ~BLK 128|1024-DATA CRC-16, STREAMING (NO ACK)
                       (i.e. undocumented "experimental" variants of YModem)
1986  YMODEM           First mentioned in July BYTE magazine (for PC's)
      ZMODEM           "**" --> STREAMING FRAMES, CRC32 DATA / CRC16 HEADER, WINDOWED ACK
1988                   Chuck's "True YModem" spec. document
                       (I don't really see any pre-1990 PC terminal
                       that gets YModem Batch right; earliest I find is
                       LYNC or PROCOMM+ from ~1990)
1989  BIMODEM          FULL-DUPLEX BI-DIRECTIONAL TRANSFERS + CHAT
1991  YMODEM-G         First mentioned in April Dr. Dobbs magazine

For old modem stuff, yeah ZModem is the way to go. But for null-modem wired connections, or for slower 8-bit systems, YModem might still have some merit. In my case, it was a useful way to queue up a set of files to send to the X16. And since it is a reliable short distance connection, we can do YModem-G type things (well, if we have a multi-byte UART and a working CTS/RTS flow control - which not all 8-bits did).

I'm also finding that ZModem is kind of a "waste" on WiFi Modems that wrap the transfer over TCP/IP. That's inherently a variable speed connection and ZModems timing stuff ends up kind of chasing its tail sometimes. So in like near-field wireless connections or even Bluetooth, YModem again might be the faster way to go (when interoperating with legacy systems {or even modern bootloader things} that don't have their own native TCP/IP).

 

[geneb]

The Z-89 is effectively the Zenith version of the Heathkit H-89. It ran either CP/M 2.2 or HDOS.

Great job on the YMODEM video!

g.

 

[davidrg]

Mostly I use Kermit protocol for easy file transfer - of course Y or Z-modem are mostly faster

Kermit protocol performance can depend quite a lot on the particular implementations in use. Long Packets, Sliding Windows (like zmodem) and Streaming (like ymodem-g) aren't mandatory features so not all implementations support them, possibly because too old/too resource constrained/can't be bothered.

A few of the 3rd-party implementations I've looked at (eg, Tera Term/HyperTerm) don't provide much in the way of configuration options for the protocol either, which may have an impact on performance or reliability I guess. For a long time C-Kermits default configuration was for reliability over speed, but at least you could change it if needed.