Stone
10k Member
I like U - N E T II for a Server/Workstation based network. It can use anything from a PC on up through 486 and PS/2. And it's parallel port so no NICs are required. DOS 3.3 or higher is recommended.
- VCF South West - June 14 - 16, Davidson-Gundy Alumni Center at University of Texas at Dallas
- VCF West - Aug 2 - 3, Computer History Museum, Mountain View, CA
- VCF Midwest - Sept 7 - 8 2024, Schaumburg, IL
- VCF SoCal - Mid February 2025, Location TBD, Southern CA
- VCF East - April 2025, Infoage Museum, Wall NJ
-
Please review our updated Terms and Rules here
EtherDFS - an ethernet drive for DOS
- Thread starter Pepinno
- Start date
archeocomp
Experienced Member
Stone would you mind posting a link? I can not find anything with that name.
Trixter
Veteran Member
Unfortunately my initial tests with 8088 systems (which work 100% with mTCP) have not been very promising. On a PCjr with a Xircom PE3, the driver appears to load, but it occasionally truncates a byte from the stream, which means I can't rely on it for anything. When I copy a file to the local drive, then copy it again, then run a file compare, they don't match up. Example:
https://goo.gl/photos/55Zhg6UBdceihdga8
Looking at PROTOCOL.TXT, it appears that etherdfs's protocol does not contain any error-checking/checksum whatsoever, so until this is added, etherdfs is unusable for me. And, honestly, I don't think it should be relied on for anything other than tinkering until this is added. No network is 100% perfect.
On a real IBM PC 5160 with an Intel Etherexpress 8/16, the driver hangs while initializing. This might be because the NIC is connected to a wireless adapter, but again it works with mTCP perfectly so I'm not sure where the trouble is. I've double-checked the MAC is correct, but it just sits there:
https://goo.gl/photos/ZjHbgZWuczGAVWUX9
The author already stated no DOS server will be written by him. Also, the DOS network redirector interface wasn't really usable until 4.x, so you should be using 4.x or higher with this.
My suggestions for any features or enhancements:
- Add a checksum and retransmit if it fails. At a bare minimum, using the UDP 16-bit checksum should be enough, and it is simple enough to be optimized easily in assembly.
- Rewrite the client in assembly. Not necessarily to reduce its size, but rather to make it faster and easier to read and maintain without C or compiler/stack tricks.
Last edited:
eeguru
Veteran Member
A more immediate compromise maybe to recompile the server for modern Windows under Cygwin or MinGW32. That is something that could be done and tested by someone other than the original author. It's been a while since I last looked at networking under DJGPP. But depending on what is there, that might be a possibility as well for 32-bit DPMI DOS.
Redirectors prior to v5.x are troublesome. 3.3 initially added redirector support and while it is very similar to 5.x support, there is enough difference that chasing quirky bugs is way more than a trivial effort. 4.x redirector support uses an entirely different interrupt and calling convention. It's so dissimilar that you might as well write an entire different implementation just for 4.x.
Stone
10k Member
It doesn't seem to be out in the wild so I'll send you a link.
keenerb
Veteran Member
I haven't seen any data integrity issues yet. I wonder if it's more or less likely to lose data than say FTP?
Your photo isn't loading for me, also.
Ethernet frames are protected by a CRC that the hardware computes and checks. In theory the packet driver will not ever see an Ethernet frame that was corrupted in flight because it will fail the CRC check. So for a local network your biggest danger is packet loss, not packet corruption. Assuming everything is working correctly.
FTP is going to be far more reliable than just blasting raw packets onto the wire, but at the expense of more computational overhead. In addition to the Ethernet CRC you get an IP header checksum and a TCP checksum that covers parts of the IP header, the full TCP header, and the full payload.
You can get really good throughput with this technique, assuming no packet drops. To make it more robust you have to start adding sequence numbers, re-transmission, and error detection. That leads to dead time on the wire, which then leads to implementing sliding windows for the packets. By the time you get to that you might as well have just implemented TCP/IP which features all of those things and is a universal standard.
Side note: For UDP data transfers the checksum is optional. You still get the IP header checksum, but the UDP packet does not need to have a checksum. Nobody uses that feature because it is a terrible idea.
FTP is going to be far more reliable than just blasting raw packets onto the wire, but at the expense of more computational overhead. In addition to the Ethernet CRC you get an IP header checksum and a TCP checksum that covers parts of the IP header, the full TCP header, and the full payload.
You can get really good throughput with this technique, assuming no packet drops. To make it more robust you have to start adding sequence numbers, re-transmission, and error detection. That leads to dead time on the wire, which then leads to implementing sliding windows for the packets. By the time you get to that you might as well have just implemented TCP/IP which features all of those things and is a universal standard.
Side note: For UDP data transfers the checksum is optional. You still get the IP header checksum, but the UDP packet does not need to have a checksum. Nobody uses that feature because it is a terrible idea.
mateuszviste
Experienced Member
That's quite a bold claim. Don't you know that every Ethernet frame is protected by a hardware CRC? Of course it's possible that your PE3 adapter doesn't bother computing and/or checking this mandatory CRC field, but then I guess you should complain to Xircom about that. I don't have any PE3 so I cannot test, though. I do, however, use EtherDFS on my 80C86 laptop through a PLIP link with 100% success.
That being said, I do plan to add a checksum to EtherDFS before v1.0 (for fun, mostly), but it's really low priority, since it shouldn't ever be required anyway.
Sadly I am unable to open your photos. I understand you pass the MAC address (argument) in its full form to EtherDFS - in such case, the hang is surely not network-related, because in such situation EtherDFS doesn't send nor expect any packet. What's the exact version of DOS you are running there? Perhaps this version does things slightly differently... I'd be willing to test it, but I need to know what is the exact OS. Also, do you you have any other INT 2F TSRs running there? (like CDROM, networking drive, or similar). If you'd be willing to perform a few tests, I could prepare a few debug versions of EtherDFS so we could, perhaps, at least pin-point the issue (if not solving it).
I don't think the author ever said that. He did say, however, that he don't do Windows, but that's another story. I might port ethersrv to DOS at some point, but cannot say for sure - it's not a trivial thing to do, even though I'd really like it, and my hobby time is very limited.
That's very true indeed. I target MS-DOS 5.0+ for this reason exactly - covering versions below becomes very messy, very fast. Is there any reason why one would want to use MS-DOS 3.x instead of 5.x?
It would hardly make EtherDFS faster, and probably not smaller either. It's perhaps a personal thing, but reading and maintaining C always has been significantly easier than assembly to me. Also, the amount of bugs and amount of time needed for development is very much in favor of developing in C. Shortly said - sorry, won't happen. My project, my time, my rules. Feel free to fork an assembly version, though - I'd very much welcome such initiative, and would be happy to assist during the struggle.
mateuszviste
Experienced Member
That's correct. One could argue that the Ethernet CRC could still fail detecting some corruptions because of CRC collisions, but I believe this to be anecdotal at best.
Especially true when running over wifi (which I do myself - my 8086 PC is linked through a PLIP link to a wifi AP). EtherDFS handles packet loss without any troubles, and performs retransmissions whenever necessary, in a way that is totally transparent both to the user and his applications.
The above is perfectly true, but I have to state that performance is not part of EtherDFS' goals. I use raw Ethernet frames mostly for simplicity, and also for being able to share the packet driver with other networking tools. EtherDFS doesn't come with sliding windows - in this regard it's more like TFTP than FTP (ie "send single frame, expect single ACK"). Sliding windows are great when there is latency, but on a LAN (where EtherDFS is supposed to be used), the latency should not be that important.
keenerb
Veteran Member
I ran a few tests last night with my 1000SL and Etherlink III card.
I copied a 13k file from remote to local 5,000 times and computed a checksum after each copy.
There were no data errors or mismatched md5sums for the entire time period. That's probably good enough for me. I really wouldn't want to try this over anything wireless or with any packet loss, though...
Also, according to the protocol.txt file, the packet does include a sequence number:
In the event a packet is lost, the client should simply re-send the request right?
I copied a 13k file from remote to local 5,000 times and computed a checksum after each copy.
There were no data errors or mismatched md5sums for the entire time period. That's probably good enough for me. I really wouldn't want to try this over anything wireless or with any packet loss, though...
Also, according to the protocol.txt file, the packet does include a sequence number:
Code:
57 | S | a single byte with a "sequence" value. Each query is supposed to
| | use a different sequence, to avoid the client getting confused if
| | it receives an answer relating to a different query than it
| | expects.In the event a packet is lost, the client should simply re-send the request right?
While the Ethernet CRC should protect you, it is still inadequate.
I've helped people debug quite a few old machines where the packet driver presented corrupted data to the application. Being that it was somebody else's machine I could not do a root cause analysis, but it was very clear that they were dealing with bad hardware. What you really need is end-to-end protection. The TCP/IP checksums get you closer to that by ensuring that not only is the Ethernet CRC working, but the interface to the host machine and the host machine itself are working properly. A real end-to-end solution is using MD5 to compare checksums after it is written to disk. Zip files are great for data transfer in this regard because their CRC will detect pretty much any error in the entire process.
Trixter might have bad hardware, but given that FTP is working well it can't be that bad. Even with good hardware you are going to have errors at various stages of the data transfer, which is why it is advisable not to rely on just the Ethernet CRC. You have to assume the hardware is trying to sabotage you.
I've helped people debug quite a few old machines where the packet driver presented corrupted data to the application. Being that it was somebody else's machine I could not do a root cause analysis, but it was very clear that they were dealing with bad hardware. What you really need is end-to-end protection. The TCP/IP checksums get you closer to that by ensuring that not only is the Ethernet CRC working, but the interface to the host machine and the host machine itself are working properly. A real end-to-end solution is using MD5 to compare checksums after it is written to disk. Zip files are great for data transfer in this regard because their CRC will detect pretty much any error in the entire process.
Trixter might have bad hardware, but given that FTP is working well it can't be that bad. Even with good hardware you are going to have errors at various stages of the data transfer, which is why it is advisable not to rely on just the Ethernet CRC. You have to assume the hardware is trying to sabotage you.
mateuszviste
Experienced Member
I totally agree with you. I also had the occasion to observe a (high-end, corporate) switch that was corrupting Ethernet frames, and forwarding them with a *recomputed* (hence seemingly valid) CRC32. But this, along with the case that you mention, is something I consider odd-ball situations. They are also the secondary reason why I do plan to implement an (optional, ie. that can be disabled) CRC-like mechanism in EtherDFS.
In trixter's case, I wonder if the problem comes from the PE3 thing that do not bother validating the CRC, or perhaps his parallel port is misbehaving and dropping some bits here and there. In any case, the next version of EtherDFS will be able to catch such misbehaving hardware. It's worth noting though, that there is no silver bullet for bad hardware - should a single RAM cell go bad and 'forget' to flip from time to time, transmissions will misbehave no matter how many CRCs there are on the wire.
Trixter
Veteran Member
I corrected the photos in my previous post.
So there are two concerns:
This https://goo.gl/photos/fRUzQKaEG5PWqMny6 shows that sometimes a byte is silently lost. The photo shows the same 512K file transferred only twice. If you want to argue that my hardware is bad, then that's the end of that conversation, although I'm not sure how I'm expected to fix it.
My second concern is that the driver completely hangs on initialization on a real PC with a "real" ethernet card (Intel Etherexpress 8/16). That system is not running any other network redirectors at all (no other network sharing, no mscdex, no cdrom drive, no parallel-port devices), and it runs IBM PC DOS 2000 (ie. IBM PC DOS 7.00, revision 1). The cards in that system are an IBM CGA, Intel NIC, Sound Blaster Pro, ADP-50 IDE hard drive adapter, floppy controller, Central Point option Board. I will trace through driver initialization to see if I can pinpoint where it is locking up.
On the system where it is hanging, I have the NIC connected to a wireless bridge. I will test connecting it wired to see if that fixes the problem.
EDIT: I found on my PC DOS 2000 system, it was hanging on exit (on return to the OS). I then booted completely clean with F5 and found that it no longer hung on return to the OS, but the drive that was created shows "invalid directory" when queried. I will try with MS-DOS 6.22 and see if I get the same results.
EDIT2: Same result with MS-DOS 6.22. I'll try a wired connection next.
EDIT3: A wired connection worked, but the resulting network drive created appears to be read-only. On the other system where it is working with corruption, the drive is working correctly as read-write. I'll try going back to PC-DOS 7 to see if I can narrow down what TSR or configuration is causing trouble with etherdfs.
So there are two concerns:
This https://goo.gl/photos/fRUzQKaEG5PWqMny6 shows that sometimes a byte is silently lost. The photo shows the same 512K file transferred only twice. If you want to argue that my hardware is bad, then that's the end of that conversation, although I'm not sure how I'm expected to fix it.
My second concern is that the driver completely hangs on initialization on a real PC with a "real" ethernet card (Intel Etherexpress 8/16). That system is not running any other network redirectors at all (no other network sharing, no mscdex, no cdrom drive, no parallel-port devices), and it runs IBM PC DOS 2000 (ie. IBM PC DOS 7.00, revision 1). The cards in that system are an IBM CGA, Intel NIC, Sound Blaster Pro, ADP-50 IDE hard drive adapter, floppy controller, Central Point option Board. I will trace through driver initialization to see if I can pinpoint where it is locking up.
On the system where it is hanging, I have the NIC connected to a wireless bridge. I will test connecting it wired to see if that fixes the problem.
EDIT: I found on my PC DOS 2000 system, it was hanging on exit (on return to the OS). I then booted completely clean with F5 and found that it no longer hung on return to the OS, but the drive that was created shows "invalid directory" when queried. I will try with MS-DOS 6.22 and see if I get the same results.
EDIT2: Same result with MS-DOS 6.22. I'll try a wired connection next.
EDIT3: A wired connection worked, but the resulting network drive created appears to be read-only. On the other system where it is working with corruption, the drive is working correctly as read-write. I'll try going back to PC-DOS 7 to see if I can narrow down what TSR or configuration is causing trouble with etherdfs.
Last edited:
I wouldn't look at the PE3 too closely; I'm sure it is validating the CRC. Just like they don't let you set the CRC, they also take responsibility to enforce the CRC.
I would be more inclined to blame an occasional bit flip on the rest of the hardware, especially if a parallel port is involved. However, even with a 5 nines correctness rate (99.999%) per packet you will still get 1 packet in 100,000 that is corrupted. On the conservative side that would mean a bad bit every 150MB at best, assuming no protocol overhead and full packets. (100,000 * 1500). If you are only sending 512 bytes per packet because you are emulating a block device at a low level that would leave you exposed to corruption every 50MB or less.
It is not software's job to fix bad hardware. It is software's job to work around it. Which is why multiple layers of error detection are so important. Especially on older hardware that is well past its expected operating lifetime. For older hardware assume that it is out to kill you and code defensively.
I would be more inclined to blame an occasional bit flip on the rest of the hardware, especially if a parallel port is involved. However, even with a 5 nines correctness rate (99.999%) per packet you will still get 1 packet in 100,000 that is corrupted. On the conservative side that would mean a bad bit every 150MB at best, assuming no protocol overhead and full packets. (100,000 * 1500). If you are only sending 512 bytes per packet because you are emulating a block device at a low level that would leave you exposed to corruption every 50MB or less.
It is not software's job to fix bad hardware. It is software's job to work around it. Which is why multiple layers of error detection are so important. Especially on older hardware that is well past its expected operating lifetime. For older hardware assume that it is out to kill you and code defensively.
keenerb
Veteran Member
Is it reproducible reliably?
I have a few PE3 adapters, I could fire one up on a non-Tandy box and see what
mateuszviste
Experienced Member
Thanks for the additional information! I absolutely don't want to argue about bad hardware, nor anything, really. Still, I would be surprised if EtherDFS was loosing bytes in such manner (because it works very well for many people), and I'd be also surprised if the problem was coming from the network (because of the Ethernet CRC32 and because "loosing a byte" is a somewhat unusual kind of network corruption, and I've seen many). The most probable culprit in my opinion is between your NIC and EtherDFS - perhaps your parallel port, perhaps something else. Or perhaps I'm wrong about everything. Doesn't matter to me really. The important fact is that having some kind of CRC on EtherDFS frames might solve (or mask at least) the problem. I will do some prototype very soon - would be great if you could test it out to see if it makes things better for you.
I didn't test PC DOS at all - the only OSes I tested EtherDFS on were FreeDOS and MS-DOS (v5.x, v6.x and v7.x), hence it may be very possible that PC-DOS does something slightly differently. I will grab a copy of PC DOS 2000 and try to reproduce the problem. But first I need to close the CRC thing.
Trixter
Veteran Member
I've never accused EtherDFS of losing bytes. I'm sure something about my hardware is losing bytes, and I also think this is normal and expected with the type of hardware that EtherDFS targets, so that's why I was arguing for some form of error detection beyond layer 2. The TCP and UDP RFCs were created around the same time as this hardware, and they felt it was necessary to have their own protocol checksum...
I am happy to test any prototype. I will be busy this weekend at a gaming convention, but after that I am available to test a CRC'd version on my dodgy hardware. I have a strong, vested interest in seeing this get better and succeed, as it solves a lot of problems for myself and my colleagues.
I tested with both PC DOS 7 and MS-DOS 6.22 with a clean boot, and I directly connected that system to the network with a cable, and now it doesn't hang on exit to DOS. Unfortunately, the resulting drive is not fully functional; the resulting network drive created appears to be read-only. Any attempt to change an existing file, or create a new one, results in odd errors from DOS such as "not enough space" (there is) or some other unspecified error that I don't get when using etherdfs on my other system. I will try to find out exactly what in my autoexec/config.sys was causing it to hang on exit and report back.
Trixter
Veteran Member
Update: I found what was causing the hang on my 5160 system: 4DOS. (I actually use an older version branded by Symantec called NDOS.) Switching to stock COMMAND.COM fixed the problem and etherdfs now returns to DOS and appears to run.
However, it is still not fully functional. Here is the behavior I'm seeing on the 5160 with an Intel EtherExpress 8/16 card:
Here is a quick video of me reproducing the bugs: ("sled" is a text editor I've used on this system for 30 years, I know it isn't producing the extra characters)
However, it is still not fully functional. Here is the behavior I'm seeing on the 5160 with an Intel EtherExpress 8/16 card:
- Trying to create a text file on the network drive at the command line with "copy con test.txt" results in "Insufficient disk space" when there is plenty of free disk space.
- Editing a text file on the network drive and saving it appears to make a copy of the last character in the file, and also appends a junk character. This is 100% reproducible. The source filesystem on the ethersvr is an MS-DOS filesystem.
Here is a quick video of me reproducing the bugs: ("sled" is a text editor I've used on this system for 30 years, I know it isn't producing the extra characters)
evildragon
Veteran Member
I think I might have been one of those people with such hardware, my SMC card, because I remember trying to troubleshoot that with you, and ever since switching to a Xircom PE-3, everything's been great.
mateuszviste
Experienced Member
Trixter, could you please test this version of EtherDFS on your "corrupting" PC?
http://mateusz.viste.fr/temp/etherdfs-test/etherdfs081beta.zip
The above version comes with a checksum implementation on each frame.
Note, that for this version to work, you need to upgrade ethersrv-linux to version 20170406.
About the "read-only" and "additional char" troubles you have on the other PC: could you compile ethersrv-linux in debug mode, and provide me with the console output of ethersrv-linux when the problem occurs? To enable debug mode on ethersrv-linux, you would need to edit the file debug.h and set the "DEBUG" definition to "1" (and then run "make" to recompile the program).
http://mateusz.viste.fr/temp/etherdfs-test/etherdfs081beta.zip
The above version comes with a checksum implementation on each frame.
Note, that for this version to work, you need to upgrade ethersrv-linux to version 20170406.
About the "read-only" and "additional char" troubles you have on the other PC: could you compile ethersrv-linux in debug mode, and provide me with the console output of ethersrv-linux when the problem occurs? To enable debug mode on ethersrv-linux, you would need to edit the file debug.h and set the "DEBUG" definition to "1" (and then run "make" to recompile the program).