An admirable goal, but be prepared for some deep research -- most available information/algorithms regarding color reduction and dithering assume uniform/linear colorspaces for both the source and target (or worse, the exact same colorspace for both source and target).
Don't worry, I studied computer graphics and signal processing in university
but since dithering changes the distributed error, how to pick the colors for dithering?
Sometimes the only way is bruteforce (like the CGA tool).
I know there are some tools for Atari 8-bit that bruteforce a ton of colours to try and get the best match for a given image.
You don't always need the 'perfect' image of course. I generally prefer algorithms that are 'good enough' by using some smart heuristics to make them perform in (near-)realtime.
But HAM6, with its unique artifacting, couldn't be done this way.
Yes, that's why I want to make a tool for this sort of thing... Certain modes have very unique characteristics, and there's no way you can properly prepare an image with PhotoShop or other modern tools.
In the case of HAM, there are basically 4 ways to pick each colour: Pick a colour from the 16-entry palette, or take the previous colour and replace the R, G, or B component.
So you have two problems here:
1) Which 16 colours do you pick for the palette?
2) For each pixel, which of the 4 possibilities is the best pick?
The interesting part with 2) is that it leads to a sort of error propagation... It may well be possible that choosing a less optimal colour for one pixel leads to less errors in subsequent pixels. So perhaps some kind of 'greedy' algorithm, trying to match a group of pixels at a time is best.
A common trick with HAM6/HAM8 is to just divide the horizontal resolution by 3, and then use 3 commands to set R, G and B separately. That means every third pixel is the correct RGB value.
Simple, but effective.