Are there any good out-of-the-box language models for python?

The spaCy package has many language models, including ones trained on Common Crawl.

Language model has a specific meaning in Natural Language Processing (NlP). A language model is a probability distribution over sequences of tokens. Given a specific sequence of tokens, the model can assign a probability of that sequence appearing. SpaCy's language models include more than just a probability distribution.

The spaCy package needs to be installed and the language models need to be download:

$ pip install spacy 
$ python -m spacy download en

Then the language models can used with a couple lines of Python:

>>> import spacy
>>> nlp = spacy.load('en')

For a given model and token, there is a smoothed log probability estimate of a token's word type can be found with: token.prob attribute.

I think the accepted answer is incorrect.

token.prob is the log-prob of the token being a particular type . I am guessing 'type' refers to something like POS-tag or type of named entity (it's not clear from spacy's documentation) and the score is a confidence measure over space of all types.

This is not the same as the probabilities assigned by a language model. A language model gives you the probability distribution over all possible tokens (not the type) saying which of them is most likely to occur next.

This repo has pretty nice documentation on using BERT (a state-of-the art model) with pre-trained weights for the neural network,

• BERT

I think the API's don't give you perplexity directly but you should be able to get probability scores for each token quite easily.

You can use the lm_scorer package to calculate the language model probabilities using GPT-2 models.

First install the package as:

pip install lm-scorer

Then, you can create a scorer by specifying the model size.

from lm_scorer.models.auto import AutoLMScorer
scorer = AutoLMScorer.from_pretrained("gpt2-large")

def score(sentence):
    return scorer.sentence_score(sentence)

Apply it to your text and you get back the probabilities.

>>> score('good luck')
8.658163769270644e-11

You can also refer to a blogpost I had written a while back if you're looking for more details.

Here I would show how we can use transformers and the gpt model to compute the perplexity of a given sentence.

import torch
from transformers import GPT2LMHeadModel, GPT2TokenizerFast

# You can change to gpt-large or other pretrained models that you can find in Huggingface.
tokenizer = GPT2TokenizerFast.from_pretrained('distilgpt2')
model = GPT2LMHeadModel.from_pretrained('distilgpt2')

def perplexity(sentence:str, stride:int=512) -> float:
    encodings = tokenizer(sentence, return_tensors='pt').input_ids
    max_length = model.config.n_positions  # 1024
    lls = []
    for i in range(0, encodings.size(1), stride):
        begin_loc = max(i + stride - max_length, 0)
        end_loc = min(i + stride, encodings.size(1))
        trg_len = end_loc - i    # may be different from stride on last loop
        input_ids = encodings[:,begin_loc:end_loc]
        target_ids = input_ids.clone()

        target_ids[:,:-trg_len] = -100
        
        with torch.no_grad():
            outputs = model(input_ids, labels=target_ids)
            log_likelihood = outputs[0] * trg_len
        
        lls.append(log_likelihood)
    ppl = torch.exp(torch.stack(lls).sum() / end_loc)
    return ppl.item()

Then, you can do like this:

perplexity("I love you.")