I have done a lot of testing, and expanded my list of things to look at. Last night I was even unable to get the CPU to run properly anymore.
I am retracing my steps and restoring the initial situation. I can't exclude that there may be some marginal problem in one of the CPLDs even since they are recycled chips. Also I will do more work to test all the ICs once more to make sure no ICs are defective. I need to make sure of a lot of things since I can't get any normal POST and CPU operation going right now.
If someone is reading this thread who knows about the operation of Quartus in compiling a CPLD design, what I still am not sure of is the output pin operation when we want a bidirectional function. Like for example XA0 and XBHE. They are always inputs and controlled by the CPU, except those times when the AT is doing DMA operations, which will have the CPU in hold status and the A0 and BHE are floating. So in certain situations when there is a DMA operation the CPLD will need to generate XA0 and XBHE to control the high and low byte transceivers during the operations, including when there is high to low byte conversion needed. So in all situations the XA0 and XBHE should not be an output, except when it is necessary for DMA operations. So I wired these lines to the proper circuits for decoding their states, and connected a "TRI" element to these pins to control them during the DMA operations. The conditions are known and fed into the tri state control pins of these TRI elements. And I entered the pins XA0 and XBHE as "bidirectional" type pins in the quartus design. So in theory it should be right, but it's hard to actually confirm the tri state operation since I can't know how quartus deals with this. This information I could not find anywhere which confirms that type of pin operation. The documentation focuses more on the complex type of CPLD programming like bus operations, ALU type functions etc.
What's also strange and unclear is the fact that quartus appears to assign a "open drain" type of pin configuration to XA0, but not to XBHE. The open drain type of output mode would suggest that there would be a pull-up resistor needed in that case. So why we have this at XA0 and not at XBHE. I looked everywhere but it seems that the user cannot define this but rather at compile time this is determined by quartus itself. When opening the "Technology Map Viewer" this can be observed when looking at both these bidirectional pins. There is no setting to be found anywhere, and I didn't define any either. Weird stuff. And of course I need to convert the design from POF to JED where we have another setting for "open collector" which I left at "auto" in the hope that this can identify the pins which need it and apply the correct output pin programming in the JED file.
Any situation where you have more than one open variables will reach a level of complexity which makes it harder to identify problems and issues. So as discussed I tried to prevent and exclude what I can before hand and even during the test work, but now at the test phase there are always in my experience some practical issues introduced which cannot be see when purely looking at schematics. So when errors occur, I try to make very sure of as many factors as possible, that's the only way to approach the matter since I don't have advanced equipment which allow me to analyse multiple signals simultaneously. In fact I have no professional equipment to speak of so the process is made more difficult by this fact. If all fails I may slow down the clock pulse considerably and see if I can identify problems in that way. But first I prefer to look at the system under normal intended operation since that is what I usually do when debugging prototypes. This AT is much more complex to deal with than 8 bit systems and it is definitely showing up a lot in the work I am doing these days.
Right now I am working to restore everything to the initial known functional test situation and after I can confirm functionality I can proceed again with testing and debugging from there. But first now I need to retrace.
What I also want to mention is that I am suspecting one of the reasons for the random power up situations is that possibly certain clock signals need to be running in synchronized state. This is also why I modified the DMA clock frequency to 4Mhz for the time being, just in case the BIOS is doing something with the DMA controllers which needs to happen in sync in order to pass the POST operations and checks. So I am matching the DMA clock to the original 5170 4Mhz frequency and circuits later. But for now I am first restoring the original programming of the CPLDs to attempt to get the CPU operational. This is made more difficult by the fact that I need at least 20-40 power cycles before I get a working state of the CPU and system.