linenomath printing extra numbers on last line of multline, align, flalign environments

Problem 1:

The lineno package prints a line number at the end of every display equation, as well between any two lines within the equation (on the first of these two lines). For the amsmath environments, it however also prints an "interdisplay" line number after the last line, so you should suppress the "postdisplay" line number.

The package works by decreasing penalties by a large amount to force (and then abort) page breaks. The linenomath* environment calls linenomathWithnumbers, which contains the line advancepostdisplaypenaltylinenopenalty. Deleting this line from the definition would remove the spurious line number, but it would also remove the only line number for the equation environment.

You could add postdisplaypenalty=0 (which is the default value) after begin{linenomath*} and before begin{<multline/align/flalign>} in your MWE and this would solve problem 1.

Problem 2:

Patching the math environments to include the effect of linenomath* is not too difficult. I have defined commands linenomathpatch and linenomathpatchAMS that do this below. The first is intended for the base math environments (equation and eqnarray) and the latter for the amsmath environments. Both the starred and unstarred versions of these environments are patched. I used patchcmd from the etoolbox package to remove the unwanted change of linenopenalty from linenomath.

Code:

The following code does both of the things described above:

documentclass[12pt,a4paper]{report}
usepackage[mathlines]{lineno} %% <- mathlines turns on line numbering in equations
usepackage{amsmath}           %% <- N.B., mathtools also loads this
linenumbers

usepackage{etoolbox} %% <- for cspreto, csappto, patchcmd, pretocmd, apptocmd

%% Patch 'normal' math environments:
newcommand*linenomathpatch[1]{%
  cspreto{#1}{linenomath}%
  cspreto{#1*}{linenomath}%
  csappto{end#1}{endlinenomath}%
  csappto{end#1*}{endlinenomath}%
}
%% Patch AMS math environments:
newcommand*linenomathpatchAMS[1]{%
  cspreto{#1}{linenomathAMS}%
  cspreto{#1*}{linenomathAMS}%
  csappto{end#1}{endlinenomath}%
  csappto{end#1*}{endlinenomath}%
}

%% Definition of linenomathAMS depends on whether the mathlines option is provided
expandafterifxlinenomathlinenomathWithnumbers
  letlinenomathAMSlinenomathWithnumbers
  %% The following line gets rid of an extra line numbers at the bottom:
  patchcmdlinenomathAMS{advancepostdisplaypenaltylinenopenalty}{}{}{}
else
  letlinenomathAMSlinenomathNonumbers
fi

linenomathpatch{equation}
linenomathpatchAMS{gather}
linenomathpatchAMS{multline}
linenomathpatchAMS{align}
linenomathpatchAMS{alignat}
linenomathpatchAMS{flalign}

begin{document}

begin{equation}
    F = m a
end{equation}

begin{multline}
p(x) = 3x^6 + 14x^5y + 590x^4y^2 + 19x^3y^3
- 12x^2y^4 - 12xy^5 + 2y^6 - a^3b^3
end{multline}

begin{align}
2x - 5y &=  8 
3x + 9y &=  -12
end{align}

begin{flalign}
a &= b + c &
  &= 1 + 1 &
  &= 2  &
end{flalign}

end{document}

Note: I've loaded amsmath after lineno, but the other order also works. If the other order is used, equation and [ will end up being patched twice (because lineno also patches them), which not very elegant but entirely harmless.

Addendum: multline

As you can see in the above screenshot, multline is the only ASM environment that is not playing nice, as it has an additional line number right above the equation. This extra number is caused by an additional interdisplay penalty (not a predisplay penalty) and it occurs during the “measuring” phase, when amsmath typesets the cells that comprise the equation just to determine how wide each of the environment's columns should be. I don't really know how/why multline is different.

This can be fixed by patching amsmath's the internal measure@ command as follows. The following needs to added to the preamble anywhere after usepackage{amsmath}.

% Disable line numbering in measurement phase for multline
makeatletter
patchcmd{mmeasure@}{measuring@true}{
  measuring@true
  ifnum-linenopenaltypar>interdisplaylinepenalty
    advanceinterdisplaylinepenalty-linenopenalty
  fi
  }{}{}
makeatother

I didn't include this in the demonstration above because I believe multline is not used by too many people.

---

Explanation: how lineno works under the hood

The lineno package is rather complicated and I should note that I do not fully understand how it works. This is what I believe to understand:

1. linenumbers decreases the penalty for a page break between two lines in a paragraph by 100000. Since the penalty normally lies somewhere between –10000 and +10000 (one 0 fewer), this is normally pretty much guaranteed to force a page break between any two lines (any value below –10000 forces a page break).

2. The package also hooks into the output routine. Whenever a page is about to be shipped, it tests whether the penalty is below –32000 (which would normally never happen). If it is, it prints a number, increases the penalty by 100000 (undoing the earlier decrease) and essentially tells TeX to reconsider the page break. If it isn't below –32000, the page is shipped as normal.

3. The linenomath* environment decreases the penalties for page breaks above/below an equation and between lines within an equation by 100000. For normal equations, only the "postdisplay" and "predisplay" penalties are relevant, but the "interdisplay" penalty also occurs between any two lines of the amsmath environments (including after the last line).

It is the fake page break caused by the postdisplay penalty that is generating the extra the unwanted numbers, so setting this value back to zero (or never changing it to begin with) solves your problem.