Vertical spacing after a tabu environment

When creating a tabu with two rows, I get more vertical spacing after the environment than when I create a tabu with only one row.

In the document below, the first tabu has two lines, and there is a small space between the end of the table and the rule below it. In the second tabu, I only have one line, and there is a larger space below it.

documentclass{article}
usepackage{geometry}
usepackage{tabu}

geometry{a4paper} 

begin{document}
begin{tabu} to textwidth {X X}%
    These are words & in the first line %
    These are more words & in the second line%
end{tabu}par%
rule{textwidth}{0.4pt}par%
%
%
vspace{2em}par%
%
%
begin{tabu} to textwidth {X X}%
    These are words & in the first line%
end{tabu}par%
rule{textwidth}{0.4pt}par%
%
end{document}

I’ve been quite careful (I think) with whitespace around the tabu, and I see no difference in the two tables aside from the different number of lines.

I know that I could fix this by adding a vspace{0.5em} or similar right after the tabu with the small spacing, but I am using the tables in a custom command and I would like to avoid a hard-coded solution like this if possible.

This problem also comes up if I substitute a tabular for the tabu. Adding tabulinesep=Xem fixes (or appears to fix) the problem for large X (>1em), but I need small spacings (ideally, ~0em).

Any ideas?