View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0006558Dwarf FortressTechnical -- Input/Keybinding/Macrospublic2014-07-07 21:222014-08-01 12:16
Reporterlue Assigned ToToady One  
PrioritynormalSeverityminorReproducibilityalways
Status resolvedResolutionfixed 
PlatformLinuxOSGentoo Linux 
Product Version0.40.01 
Fixed in Version0.40.06 
Summary0006558: "Broken Unicode" warnings for some keybindings
DescriptionHere is the output I receive upon startup of dwarf fortress, up until the title screen appears (with introductory movie playing):

(the LD_PRELOAD is to workaround another, already-reported bug involving libz and libpng)

$ LD_PRELOAD=/lib32/libz.so.1 ./df
ERROR: ld.so: object '/lib32/libz.so.1' from LD_PRELOAD cannot be preloaded: ignored.
ERROR: ld.so: object '/lib32/libz.so.1' from LD_PRELOAD cannot be preloaded: ignored.
Sound devices available:
OpenAL Soft
Picking OpenAL Soft. If your desired device was missing, make sure you have the appropriate 32-bit libraries installed. If you wanted a different device, configure ~/.openalrc appropriately.
Perfect OpenAL context attributes GET
Loading bindings from data/init/interface.txt
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
Broken unicode: [KEY:�]
New window size: 640x300
Font size: 8x12
Resizing grid to 80x25
Resizing font to 8x12

Resetting textures
Resizing font to 8x12

This appears to be related to data/init/interface.txt
Steps To ReproduceStart up dwarf fortress. This shows up instantaneously.
Additional InformationA comparison for reference. This is a relevant snippet of the 0.34.11 interface.txt, as seen by the program `less`:

[BIND:STRING_A127:REPEAT_SLOW]
[KEY:^?]
[BIND:STRING_A128:REPEAT_SLOW]
[KEY:<U+0080>]
[BIND:STRING_A129:REPEAT_SLOW]
[KEY:<U+0081>]
[BIND:STRING_A130:REPEAT_SLOW]
[KEY:<U+0082>]
[BIND:STRING_A131:REPEAT_SLOW]
[KEY:<U+0083>]
[BIND:STRING_A132:REPEAT_SLOW]
[KEY:<U+0084>]
[BIND:STRING_A133:REPEAT_SLOW]
[KEY:<U+0085>]
[BIND:STRING_A134:REPEAT_SLOW]
[KEY:<U+0086>]
[BIND:STRING_A135:REPEAT_SLOW]
[KEY:<U+0087>]
[BIND:STRING_A136:REPEAT_SLOW]

Here is the same snippet in 0.40.01:

[BIND:STRING_A127:REPEAT_SLOW]
[KEY:^?]
[BIND:STRING_A128:REPEAT_SLOW]
[KEY:?]
[BIND:STRING_A129:REPEAT_SLOW]
[KEY:<81>]
[BIND:STRING_A130:REPEAT_SLOW]
[KEY:?]
[BIND:STRING_A131:REPEAT_SLOW]
[KEY:?]
[BIND:STRING_A132:REPEAT_SLOW]
[KEY:?]
[BIND:STRING_A133:REPEAT_SLOW]
[KEY:?]
[BIND:STRING_A134:REPEAT_SLOW]
[KEY:?]
[BIND:STRING_A135:REPEAT_SLOW]
[KEY:?]
[BIND:STRING_A136:REPEAT_SLOW]
[KEY:?]

As you can see, some of these special characters have transformed into plain question marks (which won't be reported by the console), while others (only one in this snippet, though there are more) are straight byte values, instead of properly encoded codepoints.
TagsNo tags attached.

Relationships

related to 0003497 resolvedToady One incorrect string entry keybindings 

Activities

Khym Chanur

2014-07-08 21:10

reporter   ~0025238

A list of characters which actually shows something:

Broken unicode: [KEY:<81>]
Broken unicode: [KEY:<8d>]
Broken unicode: [KEY:<8f>]
Broken unicode: [KEY:<90>]
Broken unicode: [KEY:<9d>]
Broken unicode: [KEY: ]
Broken unicode: [KEY:¡]
Broken unicode: [KEY:¢]
Broken unicode: [KEY:£]
Broken unicode: [KEY:¤]
Broken unicode: [KEY:¥]
Broken unicode: [KEY:¦]
Broken unicode: [KEY:§]
Broken unicode: [KEY:¨]
Broken unicode: [KEY:©]
Broken unicode: [KEY:ª]
Broken unicode: [KEY:«]
Broken unicode: [KEY:¬]
Broken unicode: [KEY:­]
Broken unicode: [KEY:®]
Broken unicode: [KEY:¯]
Broken unicode: [KEY:°]
Broken unicode: [KEY:±]
Broken unicode: [KEY:²]
Broken unicode: [KEY:³]
Broken unicode: [KEY:´]
Broken unicode: [KEY:µ]
Broken unicode: [KEY:¶]
Broken unicode: [KEY:·]
Broken unicode: [KEY:¸]
Broken unicode: [KEY:¹]
Broken unicode: [KEY:º]
Broken unicode: [KEY:»]
Broken unicode: [KEY:¼]
Broken unicode: [KEY:½]
Broken unicode: [KEY:¾]
Broken unicode: [KEY:¿]
Broken unicode: [KEY:À]
Broken unicode: [KEY:Á]
Broken unicode: [KEY:Â]
Broken unicode: [KEY:Ã]
Broken unicode: [KEY:Ä]
Broken unicode: [KEY:Å]
Broken unicode: [KEY:Æ]
Broken unicode: [KEY:Ç]
Broken unicode: [KEY:È]
Broken unicode: [KEY:É]
Broken unicode: [KEY:Ê]
Broken unicode: [KEY:Ë]
Broken unicode: [KEY:Ì]
Broken unicode: [KEY:Í]
Broken unicode: [KEY:Î]
Broken unicode: [KEY:Ï]
Broken unicode: [KEY:Ð]
Broken unicode: [KEY:Ñ]
Broken unicode: [KEY:Ò]
Broken unicode: [KEY:Ó]
Broken unicode: [KEY:Ô]
Broken unicode: [KEY:Õ]
Broken unicode: [KEY:Ö]
Broken unicode: [KEY:×]
Broken unicode: [KEY:Ø]
Broken unicode: [KEY:Ù]
Broken unicode: [KEY:Ú]
Broken unicode: [KEY:Û]
Broken unicode: [KEY:Ü]
Broken unicode: [KEY:Ý]
Broken unicode: [KEY:Þ]
Broken unicode: [KEY:ß]
Broken unicode: [KEY:à]
Broken unicode: [KEY:á]
Broken unicode: [KEY:â]
Broken unicode: [KEY:ã]
Broken unicode: [KEY:ä]
Broken unicode: [KEY:å]
Broken unicode: [KEY:æ]
Broken unicode: [KEY:ç]
Broken unicode: [KEY:è]
Broken unicode: [KEY:é]
Broken unicode: [KEY:ê]
Broken unicode: [KEY:ë]
Broken unicode: [KEY:ì]
Broken unicode: [KEY:í]
Broken unicode: [KEY:î]
Broken unicode: [KEY:ï]
Broken unicode: [KEY:ð]
Broken unicode: [KEY:ñ]
Broken unicode: [KEY:ò]
Broken unicode: [KEY:ó]
Broken unicode: [KEY:ô]
Broken unicode: [KEY:õ]
Broken unicode: [KEY:ö]
Broken unicode: [KEY:÷]
Broken unicode: [KEY:ø]
Broken unicode: [KEY:ù]
Broken unicode: [KEY:ú]
Broken unicode: [KEY:û]
Broken unicode: [KEY:ü]
Broken unicode: [KEY:ý]
Broken unicode: [KEY:þ]
Broken unicode: [KEY:ÿ]

Quietust

2014-07-22 14:48

reporter   ~0027248

These errors are technically also happening on Windows, though the messages don't seem to show up unless DFHack is installed (in which case they get dumped to stdout.log).

lue

2014-07-22 16:32

reporter   ~0027255

I finally got around to a small script that fixed the file, the results of which are pasted here: https://gist.github.com/lue/8fa0425f6bb806b18baf . This is assuming, of course, that the keybindings are supposed to correlate with the first 256 codepoints of Unicode (instead of, say, CP 437), which appears to be the case.

Interestingly, in addition to the broken unicode errors and the erroneous question marks, codepoint 123 was a } when it should've been a { . I wonder if anyone ran into confusing problems with that in-game :) .

user6

2014-07-22 19:36

  ~0027264

I'm confused, does this problem only happen when using DFHack, or does that just make the error messages visible?

lethosor

2014-07-22 20:11

manager   ~0027267

Last edited: 2014-07-22 20:12

 Happens with a clean installation on OS X and Linux (and Windows, according to comments above, although the messages only show up if DFHack is installed).

lue

2014-07-22 20:41

reporter   ~0027273

For me, the "Broken Unicode" messages are printed to stdout (_not_ stderr). I don't use any utilities.

You can check out the interface.txt file in your installation of DF, and see that the STRING_A keybindings starting with STRING_A128 are either question marks (likely put there by some text editor handling unrecognized input) or raw byte values, as opposed to utf8-encoded codepoints. A123 and A124 were also set incorrectly, for whatever reason.

tesoul

2014-07-30 09:24

reporter   ~0027956

In linux, open the file interface.txt with gedit as ISO-8859-1 and save as UTF-8

lue

2014-07-30 12:24

reporter   ~0027974

tesoul: please note that resaving like that won't fix all the keybindings, just the ones where plain byte values managed to survive to the final file. The ones that were substituted with question marks, as well as the { character keybinding, would still be broken.

Also, the file I posted is still valid for 0.40.05 (that is, running `diff` on the result of fixing 0.40.04's file and the result of fixing 0.40.05's led to no output)

Issue History

Date Modified Username Field Change
2014-07-07 21:22 lue New Issue
2014-07-08 21:10 Khym Chanur Note Added: 0025238
2014-07-22 14:48 Quietust Note Added: 0027248
2014-07-22 16:32 lue Note Added: 0027255
2014-07-22 19:36 user6 Note Added: 0027264
2014-07-22 20:11 lethosor Note Added: 0027267
2014-07-22 20:11 lethosor Assigned To => lethosor
2014-07-22 20:11 lethosor Status new => acknowledged
2014-07-22 20:11 lethosor Note Edited: 0027267
2014-07-22 20:12 lethosor Note Edited: 0027267
2014-07-22 20:41 lue Note Added: 0027273
2014-07-28 16:32 lethosor Status acknowledged => confirmed
2014-07-30 09:24 tesoul Note Added: 0027956
2014-07-30 12:24 lue Note Added: 0027974
2014-08-01 12:16 Toady One Status confirmed => resolved
2014-08-01 12:16 Toady One Fixed in Version => Next Version
2014-08-01 12:16 Toady One Resolution open => fixed
2014-08-01 12:16 Toady One Assigned To lethosor => Toady One
2015-07-28 19:52 lethosor Relationship added related to 0003497