Chinese word segmentation using neural networks

So, the main question is about how to represent the Chinese characters in your Chinese word segmentation task.

Since effectively these characters are non-ordinal categorical variables, we would represent these as one hot encodings (https://medium.com/@michaeldelsole/what-is-one-hot-encoding-and-how-to-do-it-f0ae272f1179) of dimensionality n = number of unique Chinese characters in your dataset.

Sometimes n can be very large. So in this instance, we typically have an Embedding layer, which can reduce data sparsity and collapse the number of dimensions. (https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding#:~:text=Arguments%20%20%20%20input_dim%20%20%20,the%20embed%20...%20%202%20more%20rows%20)

I think there are three questions hidden here:

1. How to encode Chinese (and potentially other) characters themselves?
2. What is an appropriate neural architecture for this?
3. How to feed these characters to a neural network?

Let's address them one by one:

How to encode characters

The typical way to encode discrete elements like in this case is to use a closed dictionary. In this case the elements to be encoded are characters, so we should:

1. Take our training data and extract all possible characters present there (potentially Chinese characters, roman numerals, and other foreign alphabet letters that were present in the data, like Latin or Cyrillic letters)

2. Create a list with the $N$ most frequent ones. In order to dimension $N$ appropriately, we should take into account that current neural network architectures won't be able to handle more than 50K elements.

Neural Architecture

Given that this problem can be formulated similarly to a language modeling task, I would say that the most appropriate architectures would be LSTM/GRU, 1D Convolutional networks, and the Transformer (or one of its variants). As we may benefit if our architecture can handle infinite context, then I would say that the choice is between LSTMs and TransformerXL. I think that word segmentation should not require very heavy processing, so I would go for an LSTM, which is very light at inference time.

As the characters are discrete, the first layer of the network would be an Embedding layer to encode the discrete characters as continuous vectors. The size of these vectors is a hyperparameter whose value we should decide.

The output of the network could be a 1 at the character positions that start a word and 0 at the other positions.

How to feed the characters to the neural network

Given that we proposed to use a neural architecture that can handle infinite context, then we should support it also in the input data preparation, i.e. by using Truncated Back-Propagation Through Time (TBPTT) if we chose LSTMs, in the style of normal language models, where we prepare the minibatches so that we can take the last hidden state of a batch and use it for the initialization of the next one.