Short Text Classification with Deep Neural Networks: An Experimental Analysis

Abstract

In the era of information explosion, short-text classification is an important task. Short texts are widely presented in social media such as Twitter, news articles, text messages and product reviews. Inspired by the recent success of deep neural networks, we aim to leverage the neural network approaches in feature and structure learning, instead of classification alone. In our experimental analysis, we widely explore and examine numerous neural network approaches and their derivatives. In the context of short texts, we apply LSTMs to perform text classification. We demonstrate that using temporal LSTMs can achieve remarkable performance in capturing syntactic and semantic meaning of short-texts without expensive time and labour cost. Unlike traditional feature engineering, which requires pre-determined dictionary of interest words and hand-crafted features, we showcase that temporal LSTMs can be used as feature extractors of "zero training cost". Our novel approach is done by extracting the intermediate output of the last hidden layer of the LSTMs. The extracted output is considered as meaningful features which a temporal LSTMs has captured. We, thereafter, feed them into machine learning algorithms such as SVMs, KNNs for classification. Our results have proved that such approach can leverage the performance of conventional machine learning algorithms to a much higher accuracy, with an increase by up to 3.5 times. More importantly, they are shown to have reached classification performance that is competitive to many deep neural networks. Our model achieves state-of-art results on three large-scale datasets for topic classification. We run our extensive experiments with two GPU-equipped machines, spending over 580 recorded hours.