Certainly when I was a kid I remember it being extremely bad. In games like Golden Axe it took *minutes* to scroll the screen to the next area in some cases. Hard Drivin' was sub 1 FPS. Things like Paratrooper and Striker seemed okay though.
Golden Axe and Hard Driving are 1991 and 1989 vintage games, respectively, and while I haven't tried either of them on my Tandy 1000 to check my guess is that both of them are normally too slow to really be playable on XT-class computers. So I'm not sure that counts as evidence of the graphics in those machines being *unusually* slow.
That said, according to this manual you are right that the CGA graphics are on the "slow side" of an ASIC that separates the 16 bit @ 8mhz "native" 8086 bus from an 8 bit @4mhz "PC-XT compatible" bus that most of the peripherals are on. In theory at least that certainly would make the PC1512's graphics slower than they could be if they were connected at the higher bus speed/slash/had a 16 bit interface. The second part of that, IE, a 16 bit interface, is probably profoundly rare in "XT class" computers whether or not they have an 8086 instead of an 8088; practically all of them only have 8-bit XT compatible slots, and I suspect the vast majority of them with onboard video
and an 8086 also connect the chip to the 8-bit side of any bus transceiver; a possible exception I can think of *might* be the AT&T 6300/Olivetti M24? (* see below)
The popular benchmark "Topbench" has a "VidMem" test, and looking at the results database does seem to back up that there is something "slow" about how the bus transceiver or video memory was implemented in those machines, but maybe not "unexpectedly bad". The timing results for the PC1512 in the database are:
VidMem: 2961 μsec
That's actually a little slower than an IBM 5160's score of:
VidMem: 2678 μsec
(Higher is slower.) And, maybe more importantly/directly comparable, it's considerably slower than the score of an original Compaq Deskpro:
VidMem: 1754 μsec
Which is another 8086-based machine that has its video card on the other side of a 16-to-8-bit bridge. The Compaq's score is right in line with 8088 machines that run at the same CPU clock, which implies that its bridge doesn't insert any unnecessary wait states. (And it also must run its expansion bus at its native 7.16mhz speed instead of halving it.) If we multiply the IBM XT's score by the difference between its 4.77mhz clock and the 4mhz clock the Amstrad manual says its bus runs at we get a number actually a little worse that than what the Amstrad actually scores but close enough to say that, yeah, its video runs about as fast as you'd expect it to.
Realistically I'd think in most games the
significantly better CPU speed of the Amstrad would more than make up for the little ding in VRAM access time compared to an original IBM XT but, sure, it does look like it might be slower than you'd expect for a "Turbo" XT... but only because most Turbo XTs don't divide their bus clock the way it does. (That said, I think *really fast* Turbo XTs, IE, the ones that run at north of 10mhz might. If that's the case then you'd also probably see a video card plugged into them running a little slower than you might expect for the CPU clock rate.) Games that are fine on an original XT should be fine, but for that small category of things that want something a *little* faster without needing to jump up to a full AT-class machine might be a problem.
FWIW, the reason I think the Olivetti M24 may have a 16 bit interface to VRAM is its score in the same tests is:
VidMem: 1049 μsec
For a machine that otherwise runs at about the same speed as the Deskpro or Amstrad with its 8086 CPU.