How to compile the source code of TeX

David Carlisle explains how to compile the sources for the modern versions of Tex that are the basis for Texlive (Pdftex, Xetex, and Luatex, among others). These derive from Karl Berry's Web2c fork of Knuth's source code, which is a mechanical way of translating code Web sources to C code that can be compiled just about anywhere.

If you want to compile sources that are closer to what Knuth wrote (and documents in The TeXbook), take a look at:

http://www.ctan.org/tex-archive/systems/unix/tex-gpc/

This project allows you to compile Pascal WEB sources directly, using GNU Pascal. This apparently wasn't trivial; as the author, Wolfgang Helbig, writes:

I was somewhat intrigued while building TeX from its sources, since some of these depend on others to be built and installed. Knuth wrote these programs in the WEB language (WEB is only remotely related to the last W from CERN's WWW). WEB programs are converted to Pascal sources by tangle and to a TeX input file by weave. Of course, tangle and weave are WEB programs as well. So one needs tangle to build tangle---and weave and TeX to read a beautifully typeset WEB program. But don't despair, I cut this indefinite recursion and provided tangle.p, the Pascal source of tangle, and tex.pdf. It shows what, why and how I changed Knuth's program.

His tex.pdf documents in minute detail these changes.

All the programs in the TeX series (TeX by itself, but also METAFONT and all the auxiliaries) are written in WEB, but to ease the translation, D. E. Knuth has not really written in a flavour of Pascal, but in some very abstract Pascal; much an Algol: there is not a lot of Pascal idiosyncrazies left specifically to allow to convert the programs to whatever. The simplest, nowadays, is to convert the Pascal like code in raw C.

In kerTeX this is what is made by kertex_M: the matrix, the tools needed to obtain C code. The matrix tools are compiled... on the matrix, to obtain C code that can be compiled for whatever target. (Allowing cross-compilation).

If you want to see the thing, just use kerTeX and with the get_mk_install.sh (or rc) program, when asked if you want the program to remove the intermediary products (including the *.c files in this case), just answer : NO to keep them.

For the bootstrapping problem (tangling tangle), D. E. Knuth has written that the first version was done by hand: he mimicked by hand what tangle would do (this is why such a program has to be basic, the same way a compiler has to be able to bootstrap a more basic version of itself to use itself to compile more complex one). This can also be done with text tools (I have sketched in kertex_M/bin1/tangle/tangleboot.sh such a program; it is not used, and lacks change file support).

The conversion from Pascal to C was needed because Pascal was not standardized, Pascal compilers not ubiquitous etc. And D.E. Knuth has also tried several years ago to encourage the possibility to compile TeX and al. with a Pascal compiler, just to be sure his programs will not be unavailable because of some compiling nightmare. And it was a pain.

It is more simple to convert to C, since it is, on purpose, a very basic Pascal (bibtex is far more hard to convert because it uses Pascal idiosyncrazies).

For information, the conversion from Web to C was web-to-c, initially from Tomas Rokicki. This is still what is used with TeXLive, or what has been used as a basis for kerTeX.

The original sources for TeX (and friends) can be found on Knuth's CTAN. All the sources have very comprehensive documentation, but trying to compile them is still an epic task.

TeX is written in WEB, a programming language invented by Knuth. So first we're going to need WEB.

WEB is written in WEB itself, it consists of two programs: weave which produces the TeX documentation from a WEB program, and tangle which produces Pascal code from a WEB program. To compile the WEB system we need an implementation of tangle; you can either get one from an existing TeX package, or you can compile it from Pascal source.

Note that if you want to read about the implementation of any WEB program you can use weave and TeX to produce copious documentation; this is a good starting point to Knuth's code. (For TeX and Metafont you can buy a printed version of the WEB output as Computers and Typesetting volumes B and D respectively.)

Now we need to talk a little about the dialect of Pascal the Knuth uses, which he calls Pascal H. TeX was written before Pascal was standardised; to my knowledge no native Pascal H compiler exists and it is not compatible with modern Pascal compilers. However Knuth wrote the programs in a relatively portable way so it's only moderately Herculean to port them. At this point you have some choices:

1. Write a compiler for Knuth's Pascal H
2. Port the WEB source to an existing Pascal dialect using change files
3. Translate Knuth's Pascal H to another programming language

tex-gpc takes approach 2, TeX live (and Miktex) take approach 3 via web2c.

Now if you can do this the actual process of initialising and running TeX (initex, fonts, etc.) will be relatively easy; make sure you validate your build against TRIP (see the TeX sources). If you're feeling adventurous do the same for Metafont, the TeX tools, Metafont tools, and WEB.

About the design of TeX and Metafont: these programs were designed by Knuth to be highly robust, efficient and portable in the late 1970s. Today programmers take for granted the speed of modern processors and programming standards that allow them to write adequately functioning programs much more quickly. Much of what happens in these programs (e.g. carefully enumerating character codes, statically allocating memory at compile time, on-line error recovery) rarely happens in today's programming; and many of the modern annoyances (having to compile LaTeX twice for back references, difficulty with fonts, the intricacy of the macro language) are a result of these design goals and decisions. I wouldn't advocate Knuth's methods for most projects today involving multiple people, efficient computers, and tight deadlines. Still TeX is among the oldest programs to still be running today (and into the future unless LuaTeX supplants it), delicately designed, intricately implemented, pretty portable and copiously documented.

Good luck!

Talk is cheap. Show me the code. ~Linus Torvolds

1. Install Free Pascal Compiler (fpc) on the system.

   pacman -S --noconfirm fpc # Arch
   apt-get install fpc       # Debian
   

2. Download, then bootstrap tangle and font-related programs.

    export TEX_HOME="$HOME/dev/tex"
    export PATH="${PATH}:${TEX_HOME}/work/bin"
    mkdir -p $TEX_HOME
    cd $TEX_HOME
   
    wget http://mirrors.ctan.org/systems/knuth/dist.zip
    wget http://mirrors.ctan.org/systems/knuth/local.zip
    wget http://mirrors.ctan.org/systems/unix/tex-fpc.zip
    for i in *.zip; do unzip $i; done
    rm *.zip
    mkdir work && cd work
    mkdir -p TeXinputs TeXformats TeXfonts PKfonts DVIPSconf
    mkdir -p MFbases MFinputs
    mkdir -p bin
    cd ..
    cp dist/lib/*.tex work/TeXinputs
    cp dist/cm/*mf dist/lib/*mf work/MFinputs
    cp local/cm/*mf local/lib/*mf work/MFinputs
    cp tex-fpc/shell/* work/bin
    fpc tex-fpc/tangle.p
    mv tex-fpc/tangle work/bin
    itgl dist/mf/mf.web tex-fpc/mf.ch
    itgl dist/tex/tex.web tex-fpc/tex.ch
    mv inimf initex work/bin
    mv tex.pool work/TeXformats
    mv mf.pool work/MFbases
   

3. Update tex-fpc/local.mf to define printer settings. See comments in modes.mf for details. For example:

    % Brother HL-L2320D (name must be one to eight letters, lowercase)
    mode_def brhlld = 
       proofing := 0;
       fontmaking := 1;
       tracingtitles := 0;
       pixels_per_inch := 600;
       o_correction := 1;
    enddef;
    localfont := brhlld;
   

4. Build the tex and weave programs.

    cd work
    inimf ../dist/lib/plain input ../tex-fpc/local dump
    mv plain.base MFbases
    cd ..
    sed -i '99,106s/t//' tex-fpc/mf.ch
    tgl dist/mf/mf.web tex-fpc/mf.ch
    mv mf work/bin
    cd dist/mfware
    tgl gftopk.web ../../tex-fpc/gftopk.ch
    mv gftopk ../../work/bin
    cd ../../work
    ../tex-fpc/MFT/plainfonts
    ../tex-fpc/MFT/webfonts
    ../tex-fpc/MFT/mfwebfonts
    ../tex-fpc/MFT/manfonts
    ../tex-fpc/MFT/logmacfonts
    ../tex-fpc/MFT/tripmanfonts
    cp ../dist/lib/hyphen.tex TeXinputs
    cp ../tex-fpc/webmac-fpc.tex TeXinputs
    initex ../dist/lib/plain dump
    mv plain.fmt TeXformats
    cd ..
    sed -i '198,204s/t//' tex-fpc/tex.ch
    tgl dist/tex/tex.web tex-fpc/tex.ch
    mv tex work/bin
    tgl dist/web/weave.web tex-fpc/weave.ch 
    mv weave work/bin
   

5. Build the xdvi program.

    cd tex-fpc/xdvi
    wget https://math.berkeley.edu/~vojta/xdvi/xdvi-22.86.tar.gz
    tar xf *gz
    rm *gz
    mkdir -p $TEX_HOME/xdvi/bin
    mkdir -p $TEX_HOME/xdvi/man/man1
    mkdir -p $TEX_HOME/xdvi/share/dvips/config
    cd xdvi*
    CPPFLAGS='-DBDPI=600 -DMAKEPK=4' ./configure --prefix=$TEX_HOME/xdvi 
      --with-default-texmf-path=$TEX_HOME/work/DVIPSconf 
      --disable-freetype 
      --enable-old-make-pk 
      --enable-extra-app-defaults=$TEX_HOME/work/DVIPSconf 
      --with-default-font-path=$TEX_HOME/work/PKfonts 
      --with-default-header-path=$TEX_HOME/work/DVIPSconf 
      --with-default-fig-path=.:$TEX_HOME/work/TeXinputs 
      --without-mfmode
   
    make && make install
    cd ../../..
    mv xdvi/bin/xdvi* work/bin
    cd work
   

6. Weave the TeX-FPC document.

    wve ../dist/tex/tex.web ../tex-fpc/tex.ch && xdvi tex
   

This results in:

See Wolfgang Helbig's TeX-FPC README file for details.