Husk, 9 bytes

¬ṁ§=Oṙ2Ṗ3

Try it online!

Uses 'disqualifying condition' of any subsequence mid...high...low from tsh's and Lynn's answers.

¬ṁ§=Oṙ2Ṗ3
       Ṗ3   # consider all subsequences of length 3:
     ṙ2     # check whether rotating 2 to the left
  §=O       # is the same as sorting;
¬ṁ          # total of subsequences that satisfy this should be zero

Husk, 10 bytes

Λo≠4Ṡ€oPOQ

Try it online!

Same idea as Kevin Cruijssen's answer.

C, 823 bytes (not counting whitespace characters); 923 bytes (including whitespace)

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
typedef struct tree
{struct tree * left;struct tree * right;int val;}tree;static int * test_p = 0;
void insert_root(tree ** root, int in)
{if (*root == NULL){*root = (tree *)calloc(1,sizeof(tree));(*root)->val = in;return;}else if (in < (*root)->val){insert_root(&((*root)->left),in);}else{insert_root(&((*root)->right),in);}}
void preorder(tree * root)
{if ( root == 0x0 ) { return; }*test_p++ = root->val;preorder(root->left);preorder(root->right);}
int main(int argc, char ** argv)
{int test_list[argc-1];memset(test_list,0,argc*sizeof(int));test_p = test_list;tree * root = (tree *)calloc(1,sizeof(tree));root->val = strtol(argv[1],0x0,10);int i = 1;while ( argv[++i] != 0x0 ){insert_root(&root,strtol(argv[i],0x0,10));}preorder(root);test_p = test_list;i = 1;while ( ( i < argc ) ){if ( *test_p != strtol(argv[i],0x0,10) ){return 0;}test_p++;i++;}return 1;}

The readable version of the program is below:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

typedef struct tree
{
    struct tree * left;

    struct tree * right;

    int val;

} tree;


static int * test_p = 0;

void insert_root(tree ** root, int in)
{
  if (*root == NULL)
  {
    *root = (tree *)calloc(1,sizeof(tree));

    (*root)->val = in;

    return;
  }

  else if (in < (*root)->val)
  {
    insert_root(&((*root)->left),in);
  }

  else
  {
    insert_root(&((*root)->right),in);
  }
}

void preorder(tree * root)
{
    if ( root == 0x0 ) {  return; }
    
        *test_p++ = root->val;

        preorder(root->left);

        preorder(root->right);

}

int main(int argc, char ** argv)
{
    int test_list[argc-1];

    memset(test_list,0,argc*sizeof(int));
    
    test_p = test_list;
    
    tree * root = (tree *)calloc(1,sizeof(tree));

    root->val = strtol(argv[1],0x0,10);

    int i = 1;

    while ( argv[++i] != 0x0 )
    {
        insert_root(&root,strtol(argv[i],0x0,10));
    }
    
    preorder(root);
    
    test_p = test_list;

    i = 1;

    while ( ( i < argc ) )
    {
        if ( *test_p != strtol(argv[i],0x0,10) )
        {
            return 0;
        }

        test_p++;

        i++;
    }

    return 1;   
}

The main method in this program reads the list of numbers that are (allegedly) a legitimate preorder traversal.

The function insert_root that is called inserts the integers into a binary search tree where previous nodes contain lesser values and next nodes contain larger int values.

The function preorder(root) traverses the tree in a preorder traversal and simultaneously concatenates each integer the algorithm comes across to the int array test_list.

A final while loop tests if each of the int values in the stdin list and those in test_list are equivalent at each index. If there is a list element from stdin that fails to match the corresponding element in test_list at the respective index, the main function returns 0. Else the main method returns 1.

To determine what main returned, type echo $status into the bash terminal. BASH will either print out a 1 or a 0.

Scala

All approaches are implementations of the rule shown by tsh.

109

l.zipWithIndex.foldLeft(1>0){case(r,(v,i))=>r&l.zip(l.tail).slice(i+1,l.size).forall(x=>l(i)>x._1|l(i)<x._2)}

101

(for(i<-l.indices)yield l.zip(l.tail).slice(i+1,l.size).forall(x =>l(i)>x._1|l(i)<x._2)).forall(x=>x)

98

l.indices.foldLeft(1>0)((r,i)=>r&(l.zip(l.tail).slice(i+1,l.size).forall(x=>l(i)>x._1|l(i)<x._2)))

78

(for(i<-l.indices;j<-i+1 to l.size-2)yield l(i)>l(j)|l(i)<l(j+1)).forall(x=>x)

If it must be a function and not just an expression then each line have to start with (17 bytes)

def%(l:Seq[Int])=

Oracle SQL, 177 bytes

with r(i,v)as(select rownum,value(t)from table(a)t)
select nvl(min(case when r.v<p.l and r.v>p.v then 0end),1)from r,(select i,lag(v)over(order by i)l,v from r)p where r.i+1<p.i

Since there is no boolean type in Oracle SQL, query returns either 1 or 0.

Oracle SQL 12c, 210 bytes

with function f(n ku$_objnumset,i int)return int as begin return n(i);end;
select min(case when f(c,1)>f(c,2)or f(c,1)<f(c,3)then 1else 0end)from(select value(t)c from table(powermultiset_by_cardinality(a,3))t)

It's not possible access element of the array in SQL the same way as in PL/SQL - i.e. a(i), thus function f was declared in with clause for that purpose. Otherwise solution would have been much shorter.

Other limitations

• throws an exception for arrays shorter than 3 elements (instead of returning 1)
• there is an assumption that powermultiset_by_cardinality preserves order even though it's not explicitly stated in the documentation

sqlplus listing

SQL> set heading off
SQL> with r(i,v)as(select rownum,value(t)from table(ku$_objnumset(6,3,2,4,5,1,8,7,9))t)
  2  select nvl(min(case when r.v<p.l and r.v>p.v then 0end),1)from r,
  3  (select i,lag(v)over(order by i)l,v from r)p where r.i+1<p.i
  4  /

                                            0

SQL> with function f(n ku$_objnumset,i int)return int as begin return n(i);end;
  2  select min(case when f(c,1)>f(c,2)or f(c,1)<f(c,3)then 1else 0end)
  3  from(select value(t)c from table(powermultiset_by_cardinality(ku$_objnumset(6,3,2,4,5,1,8,7,9),3))t)
  4  /

                                                     0

SQL> with r(i,v)as(select rownum,value(t)from table(ku$_objnumset(8,3,1,6,4,7,10,14,13))t)
  2  select nvl(min(case when r.v<p.l and r.v>p.v then 0end),1)from r,
  3  (select i,lag(v)over(order by i)l,v from r)p where r.i+1<p.i
  4  /

                                            1

SQL> with function f(n ku$_objnumset,i int)return int as begin return n(i);end;
  2  select min(case when f(c,1)>f(c,2)or f(c,1)<f(c,3)then 1else 0end)
  3  from(select value(t)c from table(powermultiset_by_cardinality(ku$_objnumset(8,3,1,6,4,7,10,14,13),3))t)
  4  /

                                                     1

Online verification apex.oracle.com

Update

Oracle SQL, 155 bytes

with r(i,v)as(select rownum,value(t)from table(a)t)select nvl(min(case when a.v<b.v and a.v>c.v then 0end),1)r from r a,r b,r c where a.i<b.i and b.i+1=c.i

Haskell, 41 bytes

f(h:t)=t==[]||all(>h)(snd$span(<h)t)&&f t

Try it online!

Uses Lynn's observation that it suffices to check that there's no subsequence of the for mid..high..low. This means that for each element h, the list of elements t that comes after is a block of elements <h followed by a block of elements >h (both blocks may be empty). So, the code checks that after we drop the prefix of elements <h in t, the remaining elements are all >h. Recursing checks this for each initial element h until the list is length 1.

A potential simplification is that it suffices to check for subpatterns mid..high,low where the last two are consecutive. Unfortunately, Haskell's doesn't have an short way to extract the last two elements, as can be done from the front with a pattern match a:b:c. I found a shorter solution that checks for mid,high..low, but this fails to reject inputs like [3,1,4,2].

Formatted test cases taken from Laikoni.

Scala (68 67 Bytes)

def%(i:Seq[Int])= !i.combinations(3).exists(c=>c(0)<c(1)&c(0)>c(2))

Try it online

Port of @nwellnhof's answer.

Scala (122 103 bytes)

def f(i:Seq[Int]):Boolean=if(i.size<1)1>0 else{val(s,t)=i.tail.span(_<i(0));t.forall(_>i(0))&f(s)&f(t)}

Thanks to @Laikoni for the suggestions to make both solutions shorter.

Try it online

Explanation:

1. slice (using Scala's span) the array using array's head as the slicing criteria.
2. Confirm that the first slice of the array is less than the head and the second slice is greater than the head.
3. recursively check that each slice also satisfies (2)

Haskell, 50 bytes

f[]=1<3
f(x:r)|(a,b)<-span(<x)r=all(>x)b&&f a&&f b

Try it online!

Japt, 14 bytes

d@sY ð_§XÃxÈ-Y

Japt Interpreter

Outputs false for BST, true for no BST.

Explanation:

d@                Run on each item X, return true if any aren't 0: 
  sY                  Ignore the numbers before this index
     ð_§XÃ            Get the indexes of numbers less than or equal to X
                          If it is a BST, this list will be e.g. [0,1,2...]
            -Y        Subtract the position within the index list from each index
                          eg. [0,1,2] -> [0,0,0] , [0,1,4] -> [0,0,2]
          xÈ          Sum the resulting array

Perl 6, 38 bytes

!*.combinations(3).grep:{[>] .[1,0,2]}

Try it online!

Explanation

 *.combinations(3)  # All combinations of 3 elements a,b,c
!                 .grep:{            }  # Return false if check succeeds for any combination
                         [>] .[1,0,2]   # Check whether b>a>c, that is b>a and c<a

05AB1E, 15 10 bytes

ŒεD{3.IÊ}P

Port of @Lynn's Jelly answer.
-5 bytes thanks to @Emigna.

Try it online or verify all test cases.

Explanation:"

Π            # Take all sublists of the (implicit) input-list
              #  i.e. [2,3,1] → [[2],[2,3],[2,3,1],[3],[3,1],[1]]
              #  i.e. [1,2,3,4]
              #   → [[1],[1,2],[1,2,3],[1,2,3,4],[2],[2,3],[2,3,4],[3],[3,4],[4]]
 ε      }     # Map each to:
  D           #  Duplicate the current sublist on the stack
   {          #  Sort the copy
              #   i.e. [2,3,1] → [1,2,3]
              #   i.e. [2,3,4] → [2,3,4]
    3.I       #  Get the 4th (3rd 0-indexed) permutation of this list
              #   i.e. [1,2,3] → [2,3,1]
              #   i.e. [2,3,4] → [3,4,2]
       Ê      #  Check that the lists are NOT equal
              #   i.e. [2,3,1] and [2,3,1] → 0 (falsey)
              #   i.e. [2,3,4] and [3,4,2] → 1 (truthy)
         P    # Check whether all are truthy (and output implicitly)
              #  i.e. [1,1,0,1,1,1] → 0 (falsey)
              #  i.e. [1,1,1,1,1,1,1,1,1,1] → 1 (truthy)

Jelly, 7 bytes

ŒPŒ¿€4ḟ

Try it online!

Returns [4] for traversals, otherwise [].

Essentially uses tsh's algorithm: the "disqualifying" condition for a pre-order traversal is a subsequence of 3 elements that looks like [mid, high, low]. (For example, [20, 30, 10].)

We equivalently check for any subsequences of any length that have index 4 in their permutation lists, which are all lists sorted like [a₁ … a_k c d b] where the a_i are sorted and a_i < b < c < d. (Each such list is disqualifying if we look at the last three elements, and each disqualifying list is obviously of this form.)

ŒP          All subsequences.
  Œ¿€       Permutation index of each.
     4ḟ     Set difference of {4} and this list.

Proof

A pre-order traversal contains no disqualifying subsequences

Base case: traversal(•) is the empty list. ✓

Induction: traversal(t) is: t.root ++ traversal(t.left) ++ traversal(t.right).

Let [a, b, c] be a subsequence hereof. We'll show c < a < b is impossible.

• If t.root = a, then c < a < b requires c ∈ t.left and b ∈ t.right, so [a, b, c] is the wrong order.
• If a, b, c ∈ t.left or a, b, c ∈ t.right, use the induction hypothesis.
• If a ∈ t.left and c ∈ t.right then c > a.

A list of distinct integers without disqualifying subsequences is the pre-order traversal of a BST

If the list is empty, it's the traversal of the trivial BST •.

If the list is head followed by tail:

• Let less be the longest prefix of tail of elements less than head, and let more be the rest of the list.
• Then more[1] > head, and all other more[i] are greater than head too (otherwise [head, more[1], more[i]] would be a disqualifying subsequence).
• Recurse: turn less and more into BSTs.

• Now our list is the traversal of

                   head
                  /    
           BST(less)   BST(more),
  

  and this tree is a valid BST.

Ruby, 46 40 38 bytes

f=->r{a,*b=r;!a||b==b&[*0..a]|b&&f[b]}

Try it online!

This works by recursively taking the first element a as a pivot, and checking if the rest of the array can be split in two (using intersection and union: first remove all elements >a, then add them again on the right and check if something has changed).

Java 10, 94 bytes

a->{var r=0>1;for(int j=a.length-1,i;j-->0;)for(i=0;i<j;)r|=a[j]>a[i]&a[j+1]<a[i++];return!r;}

Port of @tsh' JavaScript answer.

Try it online.

Explanation:

a->{                      // Method with integer-array parameter and boolean return-type
  var r=0>1;              //  Result-boolean, starting at false
  for(int j=a.length-1,i;j-->0;)
                          //  Loop `j` in the range (length-1, 0]:
    for(i=0;i<j;)         //   Inner loop `i` in the range [0, j):
      r|=                 //    If any are true, change the result to true as well:
         a[j]>a[i]        //     The `j`'th item is larger than the `i`'th item
         &a[j+1]<a[i++];  //     And the `j+1`'th item is smaller than the `i`'th item
  return!r;}              //  After the nested loop, check if the boolean is still false

Retina 0.8.2, 31 bytes

d+
$*
M`b((1+)1+,).*12b
^0

Try it online! Link includes test cases. Uses @tsh's algorithm. Explanation:

d+
$*

Convert to unary.

M`b((1+)1+,).*12b

Find numbers that lie between two subsequent consecutive descending numbers.

^0

Check that the number of matches is zero.

JavaScript (Node.js), 49 bytes

a=>!a.some((p,i)=>a.some((q,j)=>q>p&a[j+=j>i]<p))

Try it online!

Using the fact that for array $a_0 ... a_{n-1}$, $a$ is the pre-order traversal of some BST iff $ forall 0le i<j<n; a_i<a_{j-1} implies a_i<a_j $ holds.

Thanks to Arnauld, save 1 byte.