tex.hashtokens incomplete?

Question

This came up while trying to answer another question, and concerns tex.hashtokens() in LuaTeX. Consider the following document:
documentclass{beamer}
begin{document}

message{The meaning of (stringbeamertemplatenavigationsymbolsempty) is (meaningbeamertemplatenavigationsymbolsempty)}

directlua{
for i,v in pairs(tex.hashtokens()) do
   if v:find('beamertemplatenavigation') then
      print('found:', v)
   end
end
}

message{The meaning of (stringbeamertemplatenavigationsymbolsempty) is still (meaningbeamertemplatenavigationsymbolsempty)}

end{document}

When run with lualatex on my system (“This is LuaHBTeX, Version 1.12.0 (TeX Live 2020)”), this prints (with a newline I added for clarity):
The meaning of (beamertemplatenavigationsymbolsempty) is (macro:->setbeamertemplate {navigation symbols}{})
found:  beamertemplatenavigationsymbolsvertical
found:  beamertemplatenavigationsymbolshorizontal
 The meaning of (beamertemplatenavigationsymbolsempty) is still (macro:->setbeamertemplate {navigation symbols}{})

The macro beamertemplatenavigationsymbolsempty clearly exists, so why doesn't the Lua code find it? Have I misunderstood something?
I get similar results with MiKTeX 20.11.

Marcel Krüger · Accepted Answer

LuaTeX (as well as all other TeX versions derived from web2c after 1995) use a hash table separated into two parts: A normal hash table with up to hash_size (65536) entries and an additional part with hash_extra many entries (hash_extra can be set at runtime, TeXLive sets it to 600000 by default). (The hash_extra part is not really a hash table but more of a linear overflow list, but that's irrelevant right now)
The function tex.hashtokens returns a table which contains only the entries of the "normal" hash table and not the hash_extra part. So in some sense it lists the control sequence names which can be accessed particularly fast, not all of them. I don't know if this behavior is intentional, but I personally think that returning all names would be more useful. In any case, as ShreevatsaR mentioned you can force tex.hashtokens to return all names in a particular run by running LuaTeX as hash_extra=0 lualatex foo.tex and therefore forcing the hash_extra part to be empty. In this case, you can't use more than 65536 different control sequences though.

tex.hashtokens incomplete?

One Answer

Add your own answers!

Ask a Question