How to create search indices for custom documentation pages?

Below I show an example of a minimal package with custom documentation pages that are searchable both in 10.0–11.1 and 11.2.

This is not a definitive solution because it was arrived at by trial and error, and it may cease working in a future version again.

---

What needs to be changed in the above example?

• Pages should not be in the root documentation directory (i.e. Documentation/English). They should reside in subdirectories, such as Guides, Tutorials, ReferencePages, etc. See the documentation of built-in packages for reference.

• The metadata in the tagging rules must also have the "context" entry. Just to be safe, below I included all metadata entries that the built-in documentation pages have.

A working example

Create the TestApp directory within $UserBaseDirectory. Place PacletInfo.m there with the following contents:

Paclet[
    Name -> "TestApp",
    Version -> "0.0.1",
    MathematicaVersion -> "10.0+",
    Extensions -> {{"Documentation", Language -> "English", MainPage -> "Tutorials/TestApp"}}
]

Place TestApp.nb in Documentation/English/Tutorials within the TestApp directory. Other locations are possible too instead of Tutorials, though I have only tried standard ones, such as Guides. If you use a different location, be sure to adjust the metadata below, as well as MainPage within the "Documentation" extension of PacletInfo.m.

The contents of TestApp.nb:

Notebook[{

Cell[CellGroupData[{
Cell["Lorem Ipsum", 
"Section",ExpressionUUID->"cbc045dc-447e-4711-8aac-cb341b739ef9"],

Cell["<
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur non 
volutpat lectus. Orci varius natoque penatibus et magnis dis parturient 
montes, nascetur ridiculus mus. Class aptent taciti sociosqu ad litora 
torquent per conubia nostra, per inceptos himenaeos. Lorem ipsum dolor sit 
amet, consectetur adipiscing elit. Aliquam eget efficitur quam. Sed a aliquam 
libero, non porta erat. Mauris facilisis velit felis, hendrerit consequat 
massa posuere ac. Sed luctus suscipit dictum. Aliquam egestas orci vitae 
turpis ultricies sollicitudin. Etiam vitae nisi lobortis, scelerisque nulla 
ut, euismod lorem. Sed non nulla ex. In vel eros venenatis, mollis tortor ac, 
convallis nibh. Aenean vestibulum dui dolor, sed ultricies lacus dignissim 
nec. Aenean iaculis sapien et tincidunt porttitor. Nulla pulvinar est dapibus 
posuere convallis.
>", "Text",ExpressionUUID->"a609adb0-6225-4cdd-9894-e399706c82ac"]
}, Open  ]]
},
TaggingRules -> {
    "Metadata" -> {
        "built" -> {2017, 9, 18, 10, 0, 0},
        "history" -> {"XX", "", "", ""},
        "context" -> "TestApp`",
        "keywords" -> {"lipsum", "testapp", "foobarbaz"},
        "specialkeywords" -> {}, 
        "tutorialcollectionlinks" -> {},
        "index" -> True,
        "label" -> "TestApp Documentation",
        "language" -> "en",
        "paclet" -> "TestApp",
        "summary" -> "This is a doc page test.",
        "synonyms" -> {},
        "tabletags" -> {},
        "title" -> "TestApp Title",
        "titlemodifier" -> "",
        "windowtitle" -> "TestApp",
        "type" -> "Tutorial",
        "uri" -> "TestApp/tutorial/TestApp"
    }
},
Saveable->False,
WindowTitle -> "TestApp",
DockedCells -> FEPrivate`FrontEndResource["FEExpressions", "HelpViewerToolbar"]
TrackCellChangeTimes->False,
StyleDefinitions->"Default.nb"
]

Now in 11.1 or earlier, evaluate

SetDirectory@FileNameJoin[{$UserBaseDirectory, "Applications", "TestApp", "Documentation", "English"}]
Needs["DocumentationSearch`"]
indexDir = CreateDirectory["Index"]
ind = NewDocumentationNotebookIndexer[indexDir]
AddDocumentationDirectory[ind, "Tutorials"]
CloseDocumentationNotebookIndexer[ind]

spellIndexDir = CreateDirectory["SpellIndex"]
CreateSpellIndex[indexDir, spellIndexDir]

Now start up 11.2 and try searching for "lipsum" or "vestibulum". Both return the TestApp page.

---

Final notes

The following may work in 11.2 for creating new-style indexes directly (and thus avoiding the need to re-index):

Needs["DocumentationSearch`"]
CreateDocumentationIndex[FileNameJoin[{$UserBaseDirectory, "Applications", "TestApp", "Documentation", "English"}]]

Update and warning: If both 11.2 and 11.3 are installed, then creating new-style indices in 11.3 will cause documentation search not to work anymore in 11.2. Indices created with 11.2 will not cause problems in 11.3 though.

---

This QA discusses only how to get search working. There is more to creating a usable documentation page, such as using an appropriate stylesheet, controlling the editability of cells, etc. These are not discussed here.

I tried out Szabolcs' suggestion in his/her "final notes".

I tested this code

Needs["DocumentationSearch`"]
CreateDocumentationIndex[FileNameJoin[{$UserBaseDirectory, "Applications", "TestApp", "Documentation", "English"}]]

in both 11.2 and 12.1. In both cases a search index was built and saved in the folder

...ApplicationsTestAppDocumentationEnglishTextSearchIndex

However, this index at this location is working for 11.2 but not for 12.1.

I found out that coping the content of the folder (built with 12.1) of TextSearchIndex to the location

...ApplicationsTestAppDocumentationEnglishSearchIndex2

makes the index accessable also from the the documentation center of 12.1

Does this makes sense to someone of you?