A few technical questions for CA developers (on the off-chance they can comment)

As the developer of a tool to edit CUF fonts I was wondering if a CA developer might be able to answer a few questions I have.

On the CUF file format
Quite a few of the CUF “properties” in the first table are poorly understood by me and the one other person I've encountered on the Internet who I know has been looking into this format as well. He/she goes by the handle of just on the TWC. That discussion is over at the TWC, starting with this post to be precise. Below is a summary of what just and I so far found out (it's a few forum posts collated into one, so “I” or “me” might refer to either myself or to just). Could you please clarify what these properties refer to, or correct our misunderstanding of the format where appropriate?

Summary of what we do (not) know so far:

• 4 bytes magic 'CUF0'
• short 0: unknown
• short 1: unknown, seems to always match the height of a capital, but is not used for rendering
• short 2: line height, seems to be two pixels lower than the actual line height.
  I think this is used for calculating/allocating a proper height of the entire rendered image (block of text) so that item which extend (far) below the baseline are not cut off.
• short 3: unknown
• short 4: unknown
• short 5: seems to have something to do with baseline, but there is a maximum and a minimum, in one font I tested changing default 7 to 0xF resulting in the text moving down two pixels, while 7 to 1 results in moving up by five pixels.
  
  I think it is the baseline, or at least an equivalent value. Whether signed or unsigned I do not know.
• short 6: also seems related to baseline, changing this matches the changed baseline exactly.
  
  I call it LayoutYOffset. It is a signed short (rather than the unsigned ones that most of the others are). The absolute value is an offset as described earlier, where 0<= offset <= 0x7ff. If the bit corresponding to 0x8000 is set then it is a negative offset, otherwise positive.
• short 7: how wide a space is for justification and text wrapping calculations, the width for rendering a space is as it is specified in the glyph properties and/or kerning.
• short 8: how much less wide a line (or maybe the dynamically sized container with the line) is than the sum of the glyph sizes. If you use a value that is larger than the width of the first word and following space then there is an extra line wrap after the first word and space. In this specific case the space is rendered for justification reasons, normally the space that turns into a line wrap is not rendered/calculated.
  
  I thought it was the horizontal equivalent of #5. At any rate it appears to be a signed short, too; but the above interpretation explains some of the behaviour I saw better.
• short 9: the largest width of any glyph in the font. Anything to the right of this amount of pixels is not rendered.
• short 10: the largest height of any glyph in the font. Anything below this amount of pixels (calculated from the top of that glyph) is not rendered.
• short 11: the number of glyphs
• int 12: total size of the glyph bitmaps
• Array of 65536 shorts/[w]chars. Position in the array indicates UTF-16LE/UCS2 code point, 0xFFFF indicates “not supported”. I call it “chartable” since that's basically what it is.
• for each glyph
  
  • byte 0: starting height of the glyph in pixels above the baseline, 0x80 indicates the glyph doesn't have a height.
  • byte 1: Width that should be allocated to the glyph: this is used to advance to the position at which the next glyph should be drawn.
  • byte 2: width of the bitmap data of the glyph
  • byte 3: height of the bitmap data of the glyph
• for each glyph
  
  • int: offset that indicates where in the bitmap data the data for that glyph is, relative to the start of the bitmap data.
• bitmap data

ETW expanded on the CUF file format seen in M2TW by introducing kerning tables at the end of the file format. Does/will Shogun expand/change the format any further?

Empire extension (kerning), as discovered by just on the TWC:

• short: number of glyphs for which kerning data is available
• short: the amount of glyphs to skip, the first nn glyphs don't have kerning data, some of the last might not either if the number of glyphs with kerning and the skipped glyphs are less than the total amount of glyphs
• for each glyph with kerning
  
  • for each glyph with kerning
    
    • byte: width of the glyph in the outer loop when followed by the glyph in the inner loop, the width is calculated form the left of the glyph.

Tooltip text width
Most description/message text in Medieval II/Kingdoms are rendered in relatively straightforward “block” of text with a static width. This allows us to fix text layout in advance which is very useful for supporting proper indentation based around tabs (not in terms of number of spaces but in terms of pixels, creating an effect similar to that of word processors), as well as prevent/circumvent issues with how Medieval II/Kingdoms text layout algorithms seem to dislike some of our characters.

However, the big exception is tooltips which appear to vary in size depending on context without a clear maximum width limit, so descriptions cannot be made to force a single width for rendering purposes (e.g. to support indentation for a quote in trait description).

Could you suggest another way to predict in advance the dimensions of tooltips? (I can know width of individual glyphs or strings in advance, but right now I do not know how to calculate tooltip dimensions from that.)

Support for more complex descriptions/formatting
From my observation how the engine handles the combination of CUF and strings.bin files, I gathered that it must be possible to support more than plain text in a description.
Currently this is done by using otherwise unused slots in the CUF files to provide multiple glyphs for the same character (in different styles) and compiling the strings.bin files from XML ourselves.

This got me wondering: will CA be moving towards more complex descriptions like these?
Will CA keep the strings.bin format?

Regarding the strings.bin format
(Not questions just some thoughts.)

In testing the CUF tool on some data generated by the game from early Europa Barbarorum II builds, I've seen some “corrupted” strings.bin files: the length field would be bogus (far too long instead of something sensible).

I understand the purpose of the strings.bin format: a cached parse result with fast lookup using length prefixes, instead of having to reparse the original data file on each run. But the format allows no good mechanism for error detection, other than reaching the end of the file unexpectedly or finding more data than expected. For instance a broken length prefix essentially means that the rest of the entire file is corrupted, unless by some lucky chance the corrupted section happens to “end” at a valid length prefix.

Also I once changed one of the shorts (character or length prefix, I can't recall which) into a NULL and the particular item transformed itself into a super fast marquee during runtime. The game (Medieval II) didn't crash; it rendered the UI just fine except for that particular item it simply appeared to cycle through a lot of glyphs/text/strings, too fast to read.