IBM 5160 memory management - C code compiling with Open Watcom.

[ThePresi]

I am programming a simple database in C for the IBM 5160, which has 640KB of RAM. I am compiling it with Open Watcom using a large memory model. I am having a hard time keeping track of the memory available/used by the records in my database, as I want it to grow as much as the total installed memory allows. My understanding is that there is no straightforward way to accomplish this. Therefore, I created a small function that dynamically allocates chunks of 512 bytes of RAM, cumulatively, into an array of pointers until an allocation error occurs. I then multiply the total number of allocated pointers by 512 to calculate the available RAM. Naturally, I then free the array and proceed. The result of this method shows a very low amount of memory, around 30K. Even if I compile just the function alone, without the rest of the code, the result does not change. I was under the impression that Open Watcom, set on large mode, would automatically scale up the malloc command by adding the offset part at the memory address to span across the segment.

What am I doing wrong?

 

[mbbrutman]

It would be easier to see what is wrong if you posted a slimmed down version of the code that has the problem. Include your compiler command line too.

If you are using the large memory model then you should be getting the far version of malloc automatically. That version returns far pointers to you, which consist of a segment and an offset. It sounds like you are using the one of the smaller memory models that only allows for 64KB of data.

 

[ThePresi]

Here is the test code. I do not launch the compiler via command line but within the IDE... I keep everything default. The only compiler switch I check are the large memory model and target 8086.
(I did some tests at one point and I indeed verified that by compiling in large model I get 16 bit addresses and 8 bit with the small model).
Also, the MEM command from DOS prompt gives me more than 600KB free for programs and data.

int main() {
    long memoryAvailable;
    int SAMPLE_SIZE = 512;  /* 512 Bytes */
    void **memoryBlock; /* Array of pointers */
    int i = 0, j = 0;
    int maxBlocks = 1000; /* for a max of about 512KB */

    /* Allocate memory for pointers */
    memoryBlock = malloc(maxBlocks * sizeof(void*));
    if (memoryBlock == NULL) {
        /* Handle allocation failure of memoryBlock */
        return -1;
    }
    /* Allocate 512 chunks */
    for (i = 0; i < maxBlocks; i++) {
        memoryBlock = malloc(SAMPLE_SIZE);
        if (memoryBlock == NULL) {
            break; /* end of available memory reached */
        }
    }

    /* Calculate total memory allocated successfully */
    memoryAvailable = SAMPLE_SIZE * i;

    cprintf("Memory available: %d", memoryAvailable);
 
    /* Free everything */
    for (j = 0; j < i; j++) {
        free(memoryBlock[j]); /* Free each allocated block */
        memoryBlock[j] = NULL;
    }
    /* Free the array of pointers itself */
    free(memoryBlock);
    memoryBlock = NULL;

    return 1;
}

Thanks!

 

[mbbrutman]

Welcome to the wonderful world of x86 memory models ...

Your cprintf can handle an int, not a long. You need to change the %d to %ld. And since it will never be negative, just go to %lu.

Your allocation loop has a bug - memoryBlock = malloc(SAMPLE_SIZE) keeps updating the first element of memoryBlock. I think you mean memoryBlock[j] = malloc(SAMPLE_SIZE). When the program ends it gets cleaned up, but your free loop is possibly trying to run free against pointers that were never written.

When I make the corrections and run it under DOSBox I get 512000, which is expected.

 

[ThePresi]

Thanks @mbbrutman !
The missing index was a silly oversight. What you pointed out makes sense, but the program kept spitting out crazy numbers. After conducting some additional tests, I figured out why: At the end of the FOR loop, the index displayed a valid number: 1158 (I had increased the maxBlocks to 1250 for testing), and the SAMPLE_SIZE remained the same (512). However, the result of their multiplication was incorrect...
Then it hit me: I was multiplying two integers and storing the result in a long variable (memoryAvailable = SAMPLE_SIZE * i). I changed both the index and the constant SAMPLE_SIZE to long, and now it works.
I had no idea the compiler wouldn't handle the conversion automatically... perhaps there's an optimization switch involved?
I was losing my mind over this one!
Thanks again!

 

[kerravon]

Then it hit me: I was multiplying two integers and storing the result in a long variable (memoryAvailable = SAMPLE_SIZE * i). I changed both the index and the constant SAMPLE_SIZE to long, and now it works.
I had no idea the compiler wouldn't handle the conversion automatically... perhaps there's an optimization switch involved?

It did do the conversion automatically.

Multiplying two ints together gives an int result - as expected. (and truncates - as you didn't want).

Putting the int into a long automatically does the conversion of the (truncated) result from int to long.

Note that if you had multiplied a long by an int, the int would be converted to long as you wish, so you wouldn't have got a truncation.

Also note that if you want pointers to adjust the segment to go above 64k, you need to make them huge pointers. You can get all pointers to be huge by default by selecting the huge memory model. In practice that adds something like 4% overhead compared to large (figure provided a year ago or something in this exact message area by someone who did real world testing).

However, even then you can't do malloc() to get more than 64k, because size_t is only 16-bits in the C library. So you either need to use farmalloc(), or use a different C library (still using the Watcom compiler), that makes size_t 32 bits (even on the 8086).

 

[ThePresi]

Thanks @kerravon. When you say:

Also note that if you want pointers to adjust the segment to go above 64k, you need to make them huge pointers. You can get all pointers to be huge by default by selecting the huge memory model. In practice that adds something like 4% overhead compared to large (figure provided a year ago or something in this exact message area by someone who did real world testing).

However, even then you can't do malloc() to get more than 64k, because size_t is only 16-bits in the C library. So you either need to use farmalloc(), or use a different C library (still using the Watcom compiler), that makes size_t 32 bits (even on the 8086).

You mean, using farmalloc and the huge model in the case where I need to allocate a single block of consecutive memory larger than 64K, correct? From what I understand and verified with the (fixed) test above, the runtime should allocate chunks of memory < 64KB across segments with no issues under the large model.

In my case, the database grows by one record at a time, and each will be well under the 64KB limit, so the large model should suffice.

This is my first program, so I'm constantly encountering pitfalls and mistakes due to lack of experience. All things considered, it's progressing fairly well, although I would appreciate finding some solid references on how to implement databases for the 8088. Does anyone know of a good book or documentation on databases coded in C for the 8088?

Thanks!

 

[mbbrutman]

Well the general advice is going to be, you are doing it wrong. ;-0

I would change your approach. Don't insist that everything be in memory. Start instead with concentrating on storing your records in a file, and then using fseek, fread, and fwrite to locate, read and write records. Make that reliable.

Then start by adding a simple indexing system on top of that if you need to sort on a specific column. Indexes are great because instead of physically moving records around to get the sort order you need, you just move a list of record indexes around. Which allows you to have more than one index active at a time.

If you need performance then you can start experimenting with allocating memory for records and leaving them in memory. Reads are easy, while writes should always update the on-disk copy first and then the in memory copy. It's basically implementing a write-through cache for your database records.

 

[kerravon]

You mean, using farmalloc and the huge model in the case where I need to allocate a single block of consecutive memory larger than 64K, correct?

Sort of, yes. You don't necessarily need the huge memory model, you can use a single huge pointer (using the "huge" keyword) as required. I personally don't like using non-C90 anything, so I prefer to eliminate both the huge keyword and the use of farmalloc. And am happy to have any performance hit as a result. But I don't think most people like that approach.

From what I understand and verified with the (fixed) test above, the runtime should allocate chunks of memory < 64KB across segments with no issues under the large model.

I don't understand the term "across segments".

In my case, the database grows by one record at a time, and each will be well under the 64KB limit, so the large model should suffice.

Yes, or compact would be better if your program isn't that big.

This is my first program, so I'm constantly encountering pitfalls and mistakes due to lack of experience. All things considered, it's progressing fairly well, although I would appreciate finding some solid references on how to implement databases for the 8088. Does anyone know of a good book or documentation on databases coded in C for the 8088?

I think I saw some btree code posted once back in the 80s, called btree.c. Can't remember if it was public domain or copyrighted freeware or commercial.

 

[ThePresi]

Sort of, yes. You don't necessarily need the huge memory model, you can use a single huge pointer (using the "huge" keyword) as required. I personally don't like using non-C90 anything, so I prefer to eliminate both the huge keyword and the use of farmalloc. And am happy to have any performance hit as a result. But I don't think most people like that approach.



I don't understand the term "across segments".



Yes, or compact would be better if your program isn't that big.



I think I saw some btree code posted once back in the 80s, called btree.c. Can't remember if it was public domain or copyrighted freeware or commercial.

With 'across segment' I meant that basically I won't have to worry about my dynamic allocation to be restricted within a segment of 64KB memory when compiling in Large Mode.

I will definitely try using Compact mode once the program is complete. It's getting large, but after the final optimization pass, if it fits, I will opt for it.

Thanks!

 

[ThePresi]

Well the general advice is going to be, you are doing it wrong. ;-0

I would change your approach. Don't insist that everything be in memory. Start instead with concentrating on storing your records in a file, and then using fseek, fread, and fwrite to locate, read and write records. Make that reliable.

Then start by adding a simple indexing system on top of that if you need to sort on a specific column. Indexes are great because instead of physically moving records around to get the sort order you need, you just move a list of record indexes around. Which allows you to have more than one index active at a time.

If you need performance then you can start experimenting with allocating memory for records and leaving them in memory. Reads are easy, while writes should always update the on-disk copy first and then the in memory copy. It's basically implementing a write-through cache for your database records.

Thanks @mbbrutman.

I opted for a full database in memory due to the scope of the project and its ultimate use. The program is designed for personal use, capable of handling a few hundred text records at most, which should not exceed 300-400 KB in size. Database performance in retrieving the necessary records is crucial, hence keeping everything in memory is ideal. The only challenge was determining how much memory the database would need for operational management and storage of each record. The revised version of the above function, optimized with a binary search-like approach, enables me to do that.

I chose a self-balancing binary AVL tree structure since the records consist of fields with a key and a value. Preliminary tests demonstrated high retrieval speeds, even on an IBM 5160!

Additionally, I incorporated a linear array of pointers to the records within the data structure to facilitate easy serialization during saving/loading and to allow for a linear, bidirectional display of the records. Achieving this with only an AVL structure was somewhat inefficient.

So, it's been a lot of fun...

Thank you for the advice!

 

[doshea]

Also note that if you want pointers to adjust the segment to go above 64k, you need to make them huge pointers. You can get all pointers to be huge by default by selecting the huge memory model. In practice that adds something like 4% overhead compared to large (figure provided a year ago or something in this exact message area by someone who did real world testing).

I could be wrong, but I think huge pointers are only required if you want two pointers to the same memory location to always have the same value, rather than having the possibility of for example one pointer having segment:offset 1234:0010 and another being 1235:0000 - those pointers wouldn't compare equal using == but point to the same memory location. The overhead (for anyone who isn't aware) is due to normalisation of the pointers.

It sounded like ThePresi's code doesn't work in a way where such pointer normalisation is required. It might be required if they had a pointer to one record and then wanted to increment that pointer to point to the next one and compare the resulting pointer against some other pointer, but since the allocations are all separate, no such assumptions can be made about the memory being adjacent anyway.

However, even then you can't do malloc() to get more than 64k, because size_t is only 16-bits in the C library. So you either need to use farmalloc(), or use a different C library (still using the Watcom compiler), that makes size_t 32 bits (even on the 8086).

I guess the thing people might not understand about this is that yes, you can't do malloc() to get more than 64k in a single call to that function regardless of memory model, but whether or not you can call malloc() to get more than 64k across all calls to malloc() depends on the memory model.

I don't have time to look further right now, but it seems that OpenWatcom 1.9 doesn't have farmalloc(), but instead has _fmalloc() which still takes a size_t. Is farmalloc() just hidden away somewhere? Turbo C++ 3 seems to have farmalloc() accepting an unsigned long or something though.

 

[ThePresi]

I could be wrong, but I think huge pointers are only required if you want two pointers to the same memory location to always have the same value, rather than having the possibility of for example one pointer having segment:offset 1234:0010 and another being 1235:0000 - those pointers wouldn't compare equal using == but point to the same memory location. The overhead (for anyone who isn't aware) is due to normalisation of the pointers.

It sounded like ThePresi's code doesn't work in a way where such pointer normalisation is required. It might be required if they had a pointer to one record and then wanted to increment that pointer to point to the next one and compare the resulting pointer against some other pointer, but since the allocations are all separate, no such assumptions can be made about the memory being adjacent anyway.


I guess the thing people might not understand about this is that yes, you can't do malloc() to get more than 64k in a single call to that function regardless of memory model, but whether or not you can call malloc() to get more than 64k across all calls to malloc() depends on the memory model.

I don't have time to look further right now, but it seems that OpenWatcom 1.9 doesn't have farmalloc(), but instead has _fmalloc() which still takes a size_t. Is farmalloc() just hidden away somewhere? Turbo C++ 3 seems to have farmalloc() accepting an unsigned long or something though.

Thanks, @doshea, for clarifying. Real mode memory handling can be tricky. In my case, unless I have misunderstood, I do not think I will need to manually manipulate specific memory addresses, as I have all my data organized in arrays of pointers and the AVL tree data structure.

 

[kerravon]

I could be wrong, but I think huge pointers are only required if you want two pointers to the same memory location to always have the same value, rather than having the possibility of for example one pointer having segment:offset 1234:0010 and another being 1235:0000 - those pointers wouldn't compare equal using == but point to the same memory location. The overhead (for anyone who isn't aware) is due to normalisation of the pointers.

Huge pointers are required if you want to allocate a block bigger than 64k, and be able to traverse that whole block using normal pointer arithmetic, e.g. p++. If your pointers are only far, regardless of whether you manually defined that in any memory model, or whether you didn't use the far keyword, but simply used the compact or large memory model, then the pointer will not be able to access more than 64k (using normal arithmetic) because the offset will wrap to 0 without appropriate manipulating the segment.

I guess the thing people might not understand about this is that yes, you can't do malloc() to get more than 64k in a single call to that function regardless of memory model

Unless you change to a better C library, ie PDPCLIB or Smaller C (Smaller C requires an 80386 though - but still using standard real mode and 640k).