• Please review our updated Terms and Rules here

mTCP Backup Program/Script Generator (v 0.01)

This is not perfectly polished code, but it illustrates how to use _dos_findfirst and _dos_findnext (equivalent to DOS int 21/4E and 4F) to do a directory walk without using recursion.

It takes one command line argument - the starting directory. It should end with a / or you will get bad results. This code works in DOSBox and under DOS 6, and probably any version of DOS that has the 4E and 4F interrupts.

I would use something like this as the outer loop of a program that does puts against each file it comes up with. The error conditions you are looking for are a bad return code from the server when you do a put or a broken socket.

I suspect this works because DOS is using part of the file_ structure as a "cursor" so that it remembers where to start the search the next time that _dos_findnext is used. But I did not dig far enough into the reserved parts of the data structure to prove that.


-Mike


Code:
#include <dos.h>
#include <malloc.h>
#include <stdio.h>
#include <string.h>


// These two arrays will serve as our stack of fileinfo structures and paths.
// You can save some space by just maintaining one path string and growing it
// and shrinking it as you add or remove directories instead of what we do here,
// which is lazy.
//
struct find_t fileInfos[32];
char *paths[32];


// This is just a temporary variable.
//
char filespec[128];


int main( int argc, char *argv[] ) {

  int filesFound = 0;
  int dirsFound = 0;

  int stackPos = 0;


  // Prime the pump - do the first findfirst on the root directory.

  paths[stackPos] = strdup( argv[1] );
  sprintf( filespec, "%s*.*", paths[stackPos] );
  unsigned int rc = _dos_findfirst( filespec, (_A_NORMAL | _A_SUBDIR), &fileInfos[stackPos] );

  while ( 1 ) {

    if ( rc == 0 ) {

      if ( fileInfos[stackPos].attrib & _A_SUBDIR ) {

        dirsFound++;

        // Exclude . and .. from the directory walk, or we will get nowhere fast.

        if ( strcmp(fileInfos[stackPos].name, ".") != 0 && strcmp(fileInfos[stackPos].name, ".." ) != 0 ) {

          int newPathLength = strlen( paths[stackPos] ) + strlen( fileInfos[stackPos].name ) + 2;
          paths[stackPos+1] = (char *)malloc( newPathLength );
          sprintf( paths[stackPos+1], "%s%s/", paths[stackPos], fileInfos[stackPos].name );

          stackPos++;


          // 32 might not be correct ..  it might be closer to 40 based on the maximum allowable
          // DOS path length.  This is close enough for demo code.

          if ( stackPos == 32 ) {
            puts( "Too many levels of directories" );
            break;
          }

          sprintf( filespec, "%s*.*", paths[stackPos] );
          rc = _dos_findfirst( filespec, (_A_NORMAL | _A_SUBDIR), &fileInfos[stackPos] );
          continue;

        }

      }

      else if ( ((fileInfos[stackPos].attrib & _A_NORMAL) == _A_NORMAL) || ((fileInfos[stackPos].attrib & _A_RDONLY) == _A_RDONLY) ) {
        filesFound++;
        printf( "%s%s\n", paths[stackPos], fileInfos[stackPos].name );
      }

      rc = _dos_findnext( &fileInfos[stackPos] );

    }

    else {

      free( paths[stackPos] );

      if ( stackPos == 0 ) {
        break;
      }

      --stackPos;

      rc = _dos_findnext( &fileInfos[stackPos] );

    }

  }

  printf( "Total subdirectories: %u, Total files: %u\n", dirsFound, filesFound );
  return 0;
}
 
For those following this thread and tried mbbrutman's code, compile with /Za99 on OpenWatcom or move the "unsigned int rc" declaration just below "int stackPos = 0;". Otherwise the DOS C compiler may complain.

https://github.com/cr1901/nwbackup
Consult DIR.C and DIRTEST.C for my implementation. It's probably overkill, but it will correctly handle _dos_findfirst (which returns an open directory and files in the same type of struct) and POSIX opendir (where directories and files have different types- in the case of WATCOM, however, DIR and struct dirent have the same type, presumably because it wraps _dos_findfirst). The reason source files are divided and typedefs are used all over is that- depending on how this initial implementation goes- there is a nontrivial chance I will port it to another platform (without a C++ compiler- so classes aren't an option), and the typedefs will make it easier to change code around.

Code:
// These two arrays will serve as our stack of fileinfo structures and paths.
// You can save some space by just maintaining one path string and growing it
// and shrinking it as you add or remove directories instead of what we do here,
// which is lazy.
This is exactly what I did. I cannot see any harm in this, and my stack routines check to make sure that the substring matches the new string before replacement (you need to allocate two or more different stacks if you want to traverse two noncontiguous portions of the file system). Additionally, my stack doesn't free() memory until the end, under the assumption that it will need to be allocated again.
 
No extra options are needed - just compile it as a C++ program:

wcl -0 findtest.cpp

The -0 says to create code for 8088 processors. When this is done you will get findtest.exe.

I'm not a big fan of sprinkling the code with #defines to handle multiple different compilers. After a while it becomes much nicer to create some #defines that look like functions in a new .h file. Your code can use those new #defines and let them hide the compiler specific ugliness in one file.



For those of you who might still be reading and wondering what we're talking about ...

It is not terribly difficult to get started programming for DOS using C. The compiler that I use for mTCP is called OpenWatcom and it can be found here: http://www.openwatcom.org/

This compiler runs under DOS, Windows and Linux, and OS/2. I am using it on a Windows 7 machine and previously I have used it on Windows XP. The compiler can generate code for DOS and Windows, so I do my development on the nice Windows 7 machine and then test using DOS elsewhere. For code that is not dependent upon DOS you can just compile it as a native Windows program and test right on the same machine in a command window.

For testing my DOS executables I use DOSBox and VirtualBox. Windows has been dropping support for 16 bit environments so if you want to test 16 bit DOS code DOSBox works well. DOSBox is only a DOS emulator though - it's not a full version of DOS. If you want to test under real DOS you can move your code to a real machine running DOS or test on a virtual machine that has DOS installed.

(It is probably time I made a page for this with all of the details in it ...)
 
Re: the backup program discussion

I think the first thing I am going to do is improve the code in the mTCP FTP client so that it does not care how many files are in a directory on an MPUT operation. The current code scans the directory and makes a list of files, and that list has to fit within 8KB. There really is no reason to make the list ahead of time; the dos_findfirst and dos_findnext functions can be used instead. (The same is not true for MGET; you have to do a directory list from the server and hold it in memory so that you know what GET requests to make.)

The second thing that can be done is a new command can be added to do a recursive MPUT. The code I provided above is basically all that you need for that new command. I need to add some error checking code for the input and some error handling code in case the server reports an error. But otherwise, that code can call a lower level PUT function in much the same way the existing MPUT does now.

Adding this does not make a full-fledged backup program. A backup program would have logging, much better error handling, the ability to filter on what files should be backed up, possibly compression, etc. But this extra function for the FTP client seems useful.


Mike
 
For logging, chances are I'm going to use libjsmn (pronounced "Jasmine"), which is a json parser that can be compiled without an existing C standard library. I already compiled it successfully for DOS.

Another issue with logging in code pages... IDEALLY, it would be nice to take the current code page and convert it to UTF-8 to store on a server... but there is no iconv() implementation for DOS, so I'm probably going to just use a JSON field to store the encoding, the true file name, and convert non-ASCII filenames to ASCII... but this is later of course :p.
 
We might be talking about two different things here.

A good backup program writes a log file about what it backed up, and what it got errors on. This really should be no more complicated than writing output to a file.

Why does this need json?
 
We might be talking about two different things here.

A good backup program writes a log file about what it backed up, and what it got errors on. This really should be no more complicated than writing output to a file.

Why does this need json?
So it can send a control file to the server about each file's parameters, such as last access time, attributes, and whatnot... but yes, we're talking about different things. I'm referring to the analogue of the CONTROL file from Microsoft's BACKUP/RESTORE program.

EDIT: Additionally, suppose we have to restore after a complete failure... how do we know what directories are on the server to restore without a control file? MGET and NLST might work to some extent, but there should be a place where all the information is kept in one place.
 
Last edited:
Mike B., I'm having a difficult time integrating my program logic into your FTP state machine without crashes, so I think I'll need to write my own very small client...

Your ftp client's state machine can get away with just responding to a number of return codes by just returning to the command line state (such as "mkdir failed"), The "feedback loop" between the server and program logic includes a human operator and visual response codes. If a command fails, the operator can just read stdout from CON to decide what to do next, and your program just goes back to the command line state to wait for the next action.

In the case of my program however, since everythin's automated, all expected server return codes need to handled with a state corresponding to the return code, so that the program logic knows when a response code means the action succeeded, should be tried again (transient data socket close), the error is harmless (remote directory already exists, in some cases) or the action shouldn't be retried and we should quit (bad password). This entailed adding a number of states to ClientState_t, and it may just be better to write the minimum components of what I need.

I understand how most of your ftp code works, except a few things pertaining to the file send and receive loop:
  • What is the purpose of the following data structure- does TCP require a specific format, or just a specific number of data bytes:
    Code:
    typedef struct {
      TcpBuffer b;
      uint8_t data[1460];
    } DataBuf;
  • What does dataSocket->enqueue do, ditto with PACKET_PROCESS_MULT(5);, which mentions ACK packets (I think that's a TCP thing, not an FTP thing?), and TcpBuffer::getXmitBuf( )/returnXmitBuf(). The whole send/receive routine seems a bit lower-level than the remaining portions of the program program.
  • PACKET_PROCESS_MULT(5); (any reason for 5) is included twice in the send function- once with DriveArp()/DrivePackets() and once with only DrivePackets(). From what I can gather, one call is for sending data and the other for receiving/flushing receive buffers?
 
The FTP program was written from the ground up as an interactive client, so there is always the assumption that when in doubt, you can break the data connection and go back to the command line. The code would be more usable if it were structured a little differently - the individual FTP commands should be functions that return error codes, and it should be packaged as a library separate from the user interface. That would make your job a lot easier, but up until now there has been no demand for it so I have not done the restructuring. I would think you would have an easier time if you did the refactoring/restructuring first.

In response to your specific questions:

The data structure represents the buffer used for outgoing packets. TcpBuffer is the meta data for the packet, the Ethernet headers, IP headers and TCP headers. The meta data includes when was it sent, how many retries have there been, etc. The data part of the packet is the actual packet payload of the packet, excluding the Ethernet, IP and TCP headers. If you look at how the data is laid out, the meta data is first and then the Ethernet headers, IP headers, TCP headers and TCP payload are laid out in order as it would be transmitted on the wire. To transmit you just tell the packet driver to start at an offset into the data structure that skips the meta data.

Enqueue is touched on in the Wiki: https://code.google.com/p/mtcp/wiki/TCP_Design . Your code never sends a packet directly; instead it adds the packet to an outgoing queue. This is a lower level interface than calling "send", but it also avoids an extra memcpy which is important for performance. Most applications will use "send" instead and let send worry about the TcpBuffers and the enqueue call. The comments in the TCP library code are pretty clear about how enqueue differs from send.

PACKET_PROCESS_MULT is effectively the same as calling PACKET_PROCESS_SINGLE several times. Incoming packets arrive on a ring buffer from the packet driver, and these two calls process them. Any time I expect new packets to have arrived I call these functions. This is especially true when receiving data because you don't want the incoming packet ring buffer to fill up; that causes packet loss when the packet driver has no place to put incoming packets.

When you call Tcp::DrivePackets the outgoing queue gets processed and sent. DrivePackets also handles the timeout and retransmit function for lost packets. DriveArp is similar and in theory it is needed less often but it does not hurt to have it there.

Every TCP packet that comes in has to be ACKed at some point. The quicker you get the ACK packets out to the sender, the quicker they will start sending the next packet. So minimizing the time before sending ACK packets is important.
 
The data structure represents the buffer used for outgoing packets. TcpBuffer is the meta data for the packet, the Ethernet headers, IP headers and TCP headers. The meta data includes when was it sent, how many retries have there been, etc. The data part of the packet is the actual packet payload of the packet, excluding the Ethernet, IP and TCP headers. If you look at how the data is laid out, the meta data is first and then the Ethernet headers, IP headers, TCP headers and TCP payload are laid out in order as it would be transmitted on the wire. To transmit you just tell the packet driver to start at an offset into the data structure that skips the meta data.
I should've prefixed this with the reason I asked in the first place. Without knowing mTCP's internal implementation (I do not have the time to learn the entire TCP/IP suite), this struct requires struct packing to be byte-based, correct (one field right after another)? This is a portability question. Presumably, you can use memcpy() to copy the entire struct to an outgoing buffer.

One issue I see with separating each command into a function is the case where the function (for example, MKD) times out while waiting for a response code- you can always return a timeout code, but it's very possible that the response packet from the FTP server is sent after the function returns, and you either have to ignore it (by doing a dummy read from the input buffer), or have the next function call (for example, PASV) deal with an unexpected server message (example, "550" for a failed "MKD" wouldn't make sense if the PASV function saw it when it's waiting for a return code).

This is not as easy as I thought it was going to be :p... my sympathies for you while you wrote it. Should I go ahead and fork your FTP client from google code and start refactoring it?
 
Last edited:
If you are interested in modifying the code to add new function or restructuring it then you are probably going to have to get your fingers dirty and look at the library. One of the horrors of programming for small/slow machines is that you can choose to have pretty code or you can choose to have fast code. I try to strike a balance between the two, but getting the speed requires some ugliness. Memcpy is especially slow on an 8088, so any time you can avoid copying data you are helping your performance.

People with faster machines don't have this problem. Which is why there are two ways to send data - the low level "enqueue" method and the higher level "send" method. Send is easier for beginners, but there is an extra memcpy.

Another key problem that you have observed is the asynchronous nature of communications programming. Packets get delayed. Socket connections get broken. Error messages from the other side sometimes come at bizarre times. You just need to plan on everything going wrong all of the time. Reasonable timeout periods help deal with late error messages, but sometimes you are just going to get weird stuff on the line that you did not anticipate and you will need to figure out how to get back to a good known state.

The code is public; you can fork it, restructure it, play with it, enhance it, etc. It is GPL3 code, so you have to preserve that license. I have this refactoring on my todo list so that other people can more easily reuse the FTP functions, but like I said there has not been a lot of demand. You might also consider just starting from scratch - my FTP client implementation is complex because it supports active and passive connections. If you assumed just passive connections and got rid of functions that you did not need you could cut out a lot of code.
 
I can probably get away with using send instead of enqueue, because a backup program really is not a speed-critical program.

I DO have my own passive implementation working... so I could've plugged that in, and I'd have a working backup program. However, it's NOT pretty by ANY stretch of the imagination, and it's become unmaintainable in its current form. So another rewrite is in order... I didn't fail- I just found 2 ways NOT to write an automated FTP client.

I thought Active and Classic were the original FTP modes? I suppose this doesn't really make much of a difference though... your ftp server supports PASV, and is the last (only?) FTP server for DOS that I'm aware of which is actively maintained. I can always just require that the server support PASV connections, and that should satisfy 90% of users. It would be nice (and retro :p) if my client could connect and backup to an ancient server- ya know, such as your PDP-11 running System V UNIX FTP server- as a bonus feature XD.
 
Another way to do this is to not do file level backups, but to do drive or partition level backups instead. I recently hacked up the netcat program to read raw sectors from the drive using BIOS and send the data to a Linux machine. That allowed me to do a full image backup of the drive. On the target machine I "imported" the raw image into Virtual Box and was able to mount the result in Virtual Box and see the data.

Something I forgot to mention- I did try something like this using DOS int 25h, which encompasses any type of DOS block device, instead of BIOS int 13h, which is limited to floppies and hard drives. The problem I ran into with int 25h is that DOS doesn't give you a way to specify "end of drive" without looking at offsets within the first logical sector of each logical drive- i.e. the bootsector, which chances are varies on a device-type basis (RAMdisk, Bernoulli Drive, hard drive etc). With my method of file backup, I can encompass nearly the same amount of block devices as int 25h without the "end of partition" problems by examining the file system instead (which has a well-defined terminating condition).
 
Mike B:

I'm very close to getting this right, but I do have a question that's bothering me...
WgF2JYC

The attached picture represents one particular run of my automated FTP test program... more or less, everything checks out, except for the ABOR command I send to the server after opening a data socket to do a (currently fake) file transfer. Although I send an ABOR before closing the data socket, the ABOR isn't acknowledged until after the server responds with 226 Transfer Complete.

The CHDIR error code 17 is also significant- to ease synchronization, each function polls the control socket for 300 ms at the beginning to check if the buffers are empty... if they are not, the function assumes synchronization with the server is lost- SYNC_ERROR in my enum.

Since the ABOR isn't processed until after the socket closes, this tells me that the control socket will not send data while the data socket is open, and queues data until the socket is closed. But RFC 959 explicitly says that an ABOR command can follow a command which requires an open data socket such as STOR. Is this expect behavior for mTCP? And if so, is it a packet driver, DOS, or mTCP limitation?

EDIT: All basic commands have been implemented, and my test program (FTPTEST) works on both my PC AT and 5150. Check github for current source. I have attached an exe for convenience.

Question #2...
PACKET_PROCESS_MULT is effectively the same as calling PACKET_PROCESS_SINGLE several times. Incoming packets arrive on a ring buffer from the packet driver, and these two calls process them. Any time I expect new packets to have arrived I call these functions. This is especially true when receiving data because you don't want the incoming packet ring buffer to fill up; that causes packet loss when the packet driver has no place to put incoming packets.
By "lost" packets, you mean packets which must be resent, correct? I don't understand TCP well, but I seem to recall that packets aren't actually ever lost- just need to be resent.

Additionally, shouldn't TCP flow control kick in BEFORE packets start being lost (again, my TCP knowledge is limited :D!)?
 

Attachments

  • ftp_abor2.jpg
    ftp_abor2.jpg
    22.3 KB · Views: 1
  • ftptest.zip
    80.1 KB · Views: 1
Last edited:
My memory is a bit hazy on the ABOR command. As far as I know there is no code that prevents sending the ABOR command while a data socket is open, but in general that should be the only command that would ever be sent on the control socket while the data socket is open. If you think about the interactive version of the program, a user doesn't type an ABOR command - they hit "Ctrl-C" and that during a data transfer is what starts the process to break the data transfer. It assumes keyboard input and sets a global variable so that any routine sending or receiving data knows to drop off. Are you doing something similar?

From the perspective of the networking layer, TCP packets get lost all of the time. The application is given the illusion of never losing packets because the retransmit support hides the lost packets.

Flow control is different. Flow control has to do with the amount of data in flight and the size of the receiver's window.
 
If you think about the interactive version of the program, a user doesn't type an ABOR command - they hit "Ctrl-C" and that during a data transfer is what starts the process to break the data transfer. It assumes keyboard input and sets a global variable so that any routine sending or receiving data knows to drop off. Are you doing something similar?
Not implemented yet, but yes. My ABOR function was coded up so that the server knows "stop trying to write and delete the fake test file" before I close the data socket. I wanted the server to not place an empty file in response to sending 0 bytes over the data socket, but I suppose I could just delete the file afterward if an xfer abort.
 
First off Mike (Brutman), I wanted to thank you for your help over AIM tonight... I was able to tweak performance and get good numbers, but a few things are still bothering me. Mainly I was getting decreasing transfer speeds when I started using a large number of characters to read per fread() call (8*BUFSIZ vs BUFSIZ). I'd start at 13kB/sec (continuous fprintf- slows the xfer down, but I can visually see the performance cripple), and decrease to nearly 512 BYTES/second before canceling a 14MB transfer- at 5MB. Perhaps you might have some idea of what happened? EDIT: It's a bit more complicated than I made it out to be, because I mixed up buffer names in my files. I was attempting to fill a 8*BUFSIZ long buffer with 8*BUFSIZ characters before sending the packet off, every loop that the file send routine ran (draining the buffer before beginning the next loop). I'll figure out why that crippled performance eventually, but ignore everything after the first sentence of this paragraph for now.


Is it related to the enqueue size? Note that setvbuf was/is still disabled during these tests, and the program in it's various optimized forms is available in previous revisions of the git repo (sorry :p).

The lack of updates was due to a memory corruption bug. This nasty bug was rather difficult for me to fix, and it would cause the program to randomly crash if I attempted to send data too quickly (essentially, I needed stall the transfer every iteration with fprintf to slow the xfer down) and/or attempted to jump to a certain memory location(s) during a function call. I'm fairly convinced the speed at which I sent data and the time at which I called a function are related. Even printing output changed the position of where the crash happened. But tonight I was able to fix it to my satisfaction with other pairs of eyes looking over the code. I still have yet to figure out what truly fixed the bug causing the memory corruption, but I know what I can try (git add -p). That way, I can figure out once and for all what the problem REALLY was.

With that said, I now have a preliminary working backup program! I was able to get 40kB/sec sending a 14MB file when using the program... and about 62kB/sec when sending the same file with Mike's FTP client. Note this is on a real machine only- my program chokes in a DOSBOX environment- 13kB/sec vs 252kb/sec for Mike's FTP client- I still haven't figured out why. But it's a start :D!
 
Last edited:
Some numbers, for those who are interested... these are the maximum speeds I can generate from my backup program vs sending a file over mTCP's FTP client with a reasonably long (100-300kB) file, and the performance differences. Percent increase is from NWBACKUP to MTCP FTP:

NameNWBACKUPMTCP FTP%increase
PC AT50.8kB/sec62.7kB/sec23.4%
IBM PC16.6kB/sec20.9kB/sec25.9%
DOSBOX (3000 cycles)70.3kB/sec222.5kB/sec216.5%

I should note that the best performance from the AT used different optimization settings than what I currently tested against. That version of the program can get about 54000 bytes/sec (52.7kB/sec) at max, and the percentage difference falls to 19.2%. The IBM PC has an ST-506 in it as well (apparently the average seek time is 170ms- about the speed of a floppy), so that is the bottleneck there :p.

So... I wonder if the Pareto Principle applies here? I'm about 80% of the way there :p, and you can still back up a 20MB drive in less than 8 minutes :D!

PC AT full backup was successful, but in IBM PC's backup, a bug in what I presume is the packet driver caused the NIC to stop sending data and have nearly constantly-filled data buffers. The driver wasn't even sending the data when a free data buffer was available, according to my debug output. My program has a timer that successfully aborted the program after three failed attempts to send the file (as opposed to crashing ;)...) after the packet driver went haywire. Jury's out on why DOSBOX performance is so jarringly slow compared to FTP, but I'll figure it out at some point, even if it involves adding logic to detect DOSBOX :/.

So yea, sorry for the rambling, but this is a bit exciting for me :D. I finally did it- it took me over a year (on and off, obviously), but I finally did it!
 
It's fun when you can get it working well enough to start measuring performance.

Keep your changes checked into the repo and I'll pull it down as I have time to look for bugs or possible performance improvements.
 
It's fun when you can get it working well enough to start measuring performance.

Keep your changes checked into the repo and I'll pull it down as I have time to look for bugs or possible performance improvements.
dirStack now allocates all memory at once rather than on-demand. I reduced the stack depth to 64 (I think that's the max possible with the path \A\B\C\D\E...) Not sure how much this'll actually improve performance, but I'll make a small library that incorporates Michael Abrash's Zen Timer and find out on real hardware (since as Trixter pointed out, DOSBOX cannot be used for reliable timing).
 
Back
Top