NES PPU Tile/PNG Utilities

[segaloco]

Wanted to share a little utility I put together that'll be part of a growing toolchain. The tool is "chrtopng" and can be found here: https://gitlab.com/segaloco/misc/-/blob/master/2C02_tools/chrtopng.c

There is a manual page along with it in the repo. It's the next version of the BMP tools found in the same directory, with the latter being less generalized. I wanted to revamp the interface of that tooling anyway and figured a format with native 2bpp was the way to go.

Anywho, a plethora of examples are available in another thread here: https://forums.nesdev.org/viewtopic.php?t=25701

The short of it is though this utility will take any number of NES PPU (Ricoh 2C02) character tiles from an input file and create a 2bpp indexed PNG on the standard output. A default palette is provided but one can also be given as a file.

Additionally, by default a row is defined as 16 tiles wide, the image will wrap if more than 16 tiles are provided, although this can be adjusted via an option. Finally, the maximum number of tiles processed is also capped at 256 by default, this is used to size a fixed-size buffer. This can be raised or lowered, although the image is generally sized only as large as it needs to be to display the passed data, if it is shy of the maximum, a smaller image is produced.

The linked thread contains numerous examples but one case is inspecting a portion of a larger bank of tiles. Knowing the coordinates of a tile, one can use dd(1) to first pare down a BLOB and then chrtopng to convert the results. Then further tooling can act on that PNG:

This also demonstrates the palette and row width options.

Anywho, it's BSD licensed, I was lazy and haven't provided build instrumentation, my command-line is:

cc -O2 -I/usr/include/libpng16 -o chrtopng chrtopng.c -lpng

My thought is folks who may find this useful can probably figure out how to build a libpng consumer. The larger toolset I'll be putting together around this and some other low-level bits will have a more conventional build provision.

Anywho, enjoy!

P.S. Wasn't sure if this was the best place for it. Out of the various descriptions this feels most like a tool.

 

[segaloco]

Just an update since this lead to a bunch of tooling, including a reference implementation of a pipelined utility using various bits to analyze Nintendo titles specifically.

The culminating tool is dlntopng. This is a utility that takes a terminated Nintendo-format PPU display list and generates a PNG image of its contents. A typical usage is seen here:

dd if=zldata3.bin bs=1 skip=`echo "ibase=16;B59B-B400" | bc` | dlntopng -c zlbg.bin -p pal.bin -t 255 -b 36

So first I use dd(1) to skip to where the display list is in a file. The tool simply expects a stdin (or file argument) that it can read a display list out of, so basically any sort of "find a display list" from automated searching to an already snipped file can be supplied.

At the very least, a tile bank must also be supplied from which to sample the tiles from. This is passed to pipeline member, nttochr, which takes a nametable definiton on stdin and emits the tiles it defines, in order, on stdout.
In this case, this is fed by dlntont, a tool that maps a series of Nintendo-formatted display lists to a series of nametables. Since this process requires a seed tile ID for the non-mapped portions, this is provided as an option "-b" to set this "blank" tile. If omitted, a default of 36 (0x24) is provided, as this is a very common blank tile in Nintendo work.

This tool is then being fed by dlncp, a routine that when passed a display list simply copies it to stdout. However, this tool is sensitive to terminators used in the format and will bail, by default, when a terminator of 0xFF is encountered. This can also be adjusted as different titles use different terminators. This will also close the pipe on EOF, supporting snipped display lists with no terminator. The next step *only* treats EOF as a terminator so this sends everything *except* the last terminating byte down the pipe, the next step must simply consume whatever it is fed.

Anywho, all of that ultimately winds up in the chrtopng tool described in my previous posting, which optionally applies the palette passed to the utility and then generates a 32 tile by 4 nametable image. Note most of this winds up being empty space, but there are display lists in the wild that populate multiple nametables, so I figured handling them all was the more sensible default. I have plans to add another filter that will do some shuffling to present the nametables 2x2 rather than 1x4 since that is how they're arranged in scrolling configurations.

One downside to the pipelined, mostly shell nature of the utility, it is kinda slow. Generating the above took about 10 seconds on my Raspberry Pi. The code purposefully trades off efficiency for simplicity and granularity. Each pipeline member does just one thing, all but the PNG generator in script, and several of them step a stream only a few bytes at a time for processing lists and detecting boundaries. Since this is just an analytical tool for games, I find this sort of time acceptable vs the amount of time it takes to manually map a display list by hand and draw out or assemble in an art program all the various tiles.

One key optimization I'm thinking of is reducing the amount of dd(1) traffic involved by using od(1) to dump whole files or chunks at once, then iterate over the bytes as strings rather than going and getting each byte or two separately with multiple dd(1) invocations. Yeah I could just write it all in C and not use pipes...but then it'd be a lot harder to rapidly clobber together new pipelines or convert individual pieces to support more features. For instance, I want to try and add Sega Mega Drive support to some of the tools. The PNG generator obviously needs the particular definition and pixel depth of VDP tiles, but assuming nametables are still linear lists of tile IDs, nttochr just needs to expose things like the tile size as a configurable option and then bam, you can create Mega Drive mapped nametables through the same filter, just point it at a different PNG generator. This has all sorts of potential applications too, say someone wants to get inventive with writing pixel art, they could write a custom PNG generator for their tile format but then just use linear tile arrays in memory to "draw" their images, using nttochr to map them.

Anywho, all the bits can be found here:

Files · master · Matthew Gilmore / misc · GitLab

Anything without a more specific home

gitlab.com

Any of the BMP stuff is pretty abandoned, I need to put it in an attic, but the rest are all this and the bits driving it. The PNG generator is the only C to compile, as described previously. The rest should be reasonably POSIX-y shell. Manpages abound for all the bits and the overarching tool. Much more detail again can be found in the NesDev thread linked.

Happy to answer any questions or address bugs folks may run up against. This is BSD licensed so has all the usual liability disclaimers but I'll make it clear: this software is pretty minimal on error detection. It'll look for things like invalid files or malformed arguments to a degree, but I wouldn't consider them iron clad. All that to say, as with any random software you get from the net, don't just trust it with your most critical stuff right away. Part of the reason for the simplicity over efficiency in code is to make it easy to audit and fix.

 

[segaloco]

Crosspost from TUHS's COFF mailing list because I'm lazy and don't want to do a separate writeup, I've repackaged these and other tools as a general NES toolset for UNIX-like platforms, not only in implementation but in spirit:

For those whom this sort of thing may interest, I wanted to take
an opportunity to share some tools I've been tinkering on lately
as well as a little background about them. The two main sets are
at:

fc_tools · master · Matthew Gilmore / misc · GitLab

Anything without a more specific home

gitlab.com

and

tools · master · Matthew Gilmore / Super Mario Bros. 3 Disassembly · GitLab

A disassembly of Super Mario Bros. 3

gitlab.com

With the former being tools general to the Famicom/NES and the
latter being tools more specific to Super Mario Bros. 3, my
disassembly of which has served as the testbed for developing
these and other tools.

I share these not only due to their general concern of fitting a
number of different needs relevant to both development and
reverse engineering of NES games, but also due to the influence
that the UNIX philosophy has had on the design patterns and
decisions I've made.

The bulk of these tools act as filters, specifically so that they
can be strung together into pipelines as is tradition.
Furthermore, where a concern matched close enough with an
existing UNIX utility, I used that utility as the interface model
for my own. For instance, my ddnes(1) utility derives its
argument syntax directly from dd(1) and, similar to how dd(1)
abstracts disk blocks and has some basic conversions like ASCII
to EBCDIC, allows for specifying the abstract mapping scheme of
iNES images being dumped from. This sort of replication of
familiar UNIX interfaces as significantly lowered the cognitive
load not only of remembering flags and options, but also
contemplating the logical structure of pipelined operations. I
simply need to do the same thing I would do for a more generic
data operation, except I swap in my tools where necessary.

In some ways I owe it to TUHS and the larger community
surrounding these UNIX history efforts that these tools exist at
all. The field of video game reverse engineering is what I cut
my teeth on as a tech person, and that field has for such a long
time been dominated by the Windows world, graphical applications,
complexity and closed-source solutions, and so on. In other
words, being a ROM hacker on weird UNIX platforms is a lonely,
relatively DIY situation compared with the same in the Microsoft
Windows ecosystem. Learning more about UNIX and more importantly
the UNIX philosophy through discussions here, historical
preservation, studying old manuals and source code, etc. has had
an outsize influence then on my desire to produce what, in my
obviously biased opinion, is a quite comfortable development and
reverse engineering environment for the NES on UNIX. In many
ways I was inspired by the same motivation UNIX was developed
under, to create a simpler, more intuitive technical environment
that avoids the needless complexity of many of the more
established toolkits and workflows.

Only time will tell if my tools see uptake in the niche
communities they concern, but I felt some appreciation was in
order for the fact that UNIX impacted my development of this
toolkit on so many levels.

- Matt G.

P.S. If you're someone who tinkers with this sort of stuff and
have questions or suggestions, I'm always happy to discuss the
finer points. Licenses are provided with the usual disclaimers,
know that there isn't much error checking, so don't feed these
bad data and redirect output into a precious file without a
backup plan. You have been warned these are not hardened for
production workloads.