I don't think that prevents any OS developer from writing a compatibility/emulation layer in software to handle 16-bit software.
One of the reasons to migrate to amd64 is to drop all the compromises gathered along the way, sourced from the lineage of first PC being a really really simple and cheap machine. The protected mode builds on 8086 because it has to, Intel tried a more "pure" mode with 286 and failed hard. It doesn't include VM86 because it's a worthwhile feature, it is a baseline thing coming from the primary requirement of being fully compatible with 8086 from the get go. 20 years ahead, that stuff became an impediment for the evolution of the platform and it was dropped. The new platform architecture was created, that can accomodate the old way, the new way, but cannot mix them, so the new way isn't hampered by compatibility burden, not only in realization but in CPU design.
"Native" software that does not directly access the hardware does not exist, that's the point of it. A memory operation is direct hardware access. In x86 we have segment : offset. In amd64 long mode memory is flat and segment registers are not used for the same purpose. In protected mode, opcodes that set up <1MB memory r/w hit a segmentation fault. In long mode, the same opcodes cannot set up the <1MB access through seg : off. No instructions can. Therefore the code is not able to perform one of its primary functions.
Addressing is only one problem there are still paging and cache coherency issues which are much more complex.
The bottom line is CPU in long mode cannot run 8086 code directly. In protected mode it runs directly, but hits fences. VM86 is a trip through those fences.
So, that is my opinon why it is impossible to create a purely software compatibility layer. Maybe I'm wrong and not looking into depth enough.
Emulation (not compatibility layer)...there were many dynamic recompilers and JIT available at the age of transition, even in late 90s there was a PC-on-PC emulator Bosch already in decent state, and people were expected to run 32 bit OS for some time to come. So yeah there was actually always an option to somehow run 16 bit code on 64 bit OS through emulation, I guess even Microsoft's "Virtual PC" could run DOS, so users still had a way.