Topics in the Haystack: Enhancing Topic Quality through Corpus Expansion

Identifying latent topics in large text corpora is a central task in Natural Language Processing (NLP). With the ever-growing availability of textual data in virtually all languages and about every possible topic, automated topic extraction is gaining increasing importance. Hence, the approaches are manifold. A comprehensive overview over current approaches is, for example, given in Vayansky and Kumar (2020) and Barde and Bainwad (2017). For almost all models, a topic is intuitively defined by a set of words with each word having a probability of occurrence for the given topic. Different topics can share words, and a document can be linked to more than one topic. Generative probabilistic models, such as Probabilistic Latent Semantic Analysis (Hofmann 2001) and Latent Dirichlet Allocation (LDA) (Blei, Ng, and Jordan 2003), are still widely used and inspired multiple adaptations as several studies (e.g., Agarwal and Chen 2010; Blei, Griffiths, and Jordan 2010; Chien, Lee, and Tan 2018; Ramage et al. 2009; Rosen-Zvi et al. 2012) all draw heavily from word co-occurrences. Due to its popularity and general good performance on benchmark datasets, the interpretation of a topic from LDA is seldom challenged. Neural topic models (e.g., Dieng, Ruiz, and Blei 2020; Wang, Zhou, and He 2019; Bianchi, Terragni, and Hovy 2020) further improve upon the existing methods by integrating word-embeddings or variational autoencoders (Srivastava and Sutton 2017) into the modeling approach, but still heavily rely on the ideas from Blei, Ng, and Jordan (2003).

New methods that challenge the typical idea of topic modeling also integrate word- and document-embeddings (Miles et al. 2022; Angelov 2020; Grootendorst 2022; Sia, Dalmia, and Mielke 2020; Thielmann, Weisser, and Säfken 2022). However, improvement over the current state of the art is usually measured in terms of performance as determined by evaluation metrics on standard benchmark datasets. While older models were still evaluated using likelihood-based perplexity metrics (Lafferty and Blei 2005; Larochelle and Lauly 2012; Rosen-Zvi et al. 2012), empirical results showed a negative correlation between perplexity-based metrics and human evaluation of a topic model (Chang et al. 2009). Additionally, Chang et al. (2009) first introduced the idea of intruder words. According to this idea, a topic is considered coherent or simply put, good, if a randomly chosen word, not belonging to that topic, can clearly be identified by humans. As human evaluation of models is cost and time intensive, researchers used new evaluation methods that correlated with human evaluation (Lau, Newman, and Baldwin 2014; Newman et al. 2010). Hoyle et al. (2021) even found no contemporary model at all that used human feedback as a form of model evaluation. Newer models were hence evaluated using coherence scores (Angelov 2020; Dieng, Ruiz, and Blei 2020; Grootendorst 2022; Sia, Dalmia, and Mielke 2020; Srivastava and Sutton 2017). However, Hoyle et al. (2021) found severe flaws in coherence scores. First, they find that coherence scores exaggerate differences between models and, second, they validate the findings from Bhatia, Lau, and Baldwin (2017) and find much lower Pearson correlations between automated coherence scores and human evaluation as compared with Lau, Newman, and Baldwin (2014). New evaluation methods as proposed by Weisser et al. (2023), for example, often lack any form of human verification.

We identify two shortcomings in the current state-of-the-art in topic modeling and document clustering. The first is the significant gap in validated automatic evaluation methods for topic models. The second stems from the continued reliance on evaluation methods based on word co-occurrences and outdated definitions of topics from older models. Current methods rely on limited corpora from which the topic representations are created. However, integrating larger corpora into the modeling process can enhance topic quality by including contextually relevant words that were missing from the original corpus.

The contributions of this paper are hence twofold and can be summarized as follows:

• We propose Context Enhanced Document Clustering (CEDC) which, with only a few adaptations, integrates linguistic ideas into its modeling. Soft-clustering on the document level is integrated, such that P(document|topic) is modeled.

• We introduce new topic modeling performance metrics. The effectiveness of the proposed metrics is validated by demonstrating impressive correlations with human judgment.

• We conduct a benchmark study comparing the presented approach to state-of-the-art topic modeling and document clustering methods. The presented approach outperforms common benchmark models on both coherence scores and the presented new metrics for topic evaluation.

• Our findings illustrate that even without any hyperparameter tuning, CEDC can achieve superior performance compared with existing state-of-the-art topic models that have undergone extensive hyperparameter optimization.

The remainder of the paper is structured as follows: First, we give an overview over related methods. Second, a short introduction into the used linguistic ideas and the definition of topics is presented. Third, we investigate the design of questionnaires employed for the evaluation of topic models, providing insights into the essential factors and considerations inherent in constructing robust questionnaire designs. Fourth, the method of extracting latent topics from documents, incorporating the aforementioned definitions, is presented. Fifth, new evaluation metrics are introduced and validated by presenting correlations with human annotators. Sixth, the proposed model is applied to two common data sets and compared with state-of-the-art topic models. Finally, a discussion of the limitations as well as a conclusion is given in Sections 7 and 8.

Topic modeling has been dominated by Bayesian graphical models. Probabilistic Latent Semantic Analysis (PLSA) (Hofmann 2001) and especially LDA (Blei, Ng, and Jordan 2003) being some of the most notable examples. In the context of these generative models and related structured probabilistic models (Blei and Lafferty 2007; Mazarura, De Waal, and de Villiers 2020), a word w is interpreted as a discrete token in the vocabulary V. The corpus comprises all available documents, where a document d is a sequence of words mostly represented as a bag-of-words vector. The K topics, denoted as t₁, ⋯, t_K, are modeled as categorical probability distributions over the vocabulary. Additionally, the document-specific topic distributions Cat(θ^(d)), are parameterized by θ^(d) ∈ℝ^K for each document d. [[FX-ALGO1]]

Most Bayesian generative models follow the algorithmic structure of LDA with slight adjustments as given in Algorithm 15, where Cat(·) refers to the categorical distribution and Dir(·) refers to the Dirichlet distribution. α and β are thus the distributional parameters for the Dirichlet and the categorical distribution, respectively, and must be specified in advance or optimized according to an optimization criterion. Adaptations of this generative model comprise several techniques: the inclusion of (1) variational autoencoders (Srivastava and Sutton 2017); (2) contextualized word embeddings (Das, Zaheer, and Dyer 2015); and (3) contextualized document embeddings (Bianchi, Terragni, and Hovy 2020). Further adaptations including word, document, or topic embeddings have also achieved remarkable results, slightly adjusting the overall generative process (e.g., Dieng, Ruiz, and Blei 2020).

While the early models like LDA and PLSA suffered from the restrictions imposed by bag-of-words representations, contextualized embeddings offered further possibilities beyond the inclusion into generative modeling (Dieng, Ruiz, and Blei 2020; Bianchi, Terragni, and Hovy 2020). The general idea behind these approaches is to leverage pre-trained word or document embeddings. These, often-times dimensionality-reduced representations are subsequently clustered. Each document or word cluster then represents an individual topic. For example, Sia, Dalmia, and Mielke (2020) demonstrate the efficacy of this conceptually simple approach by applying various centroid-based clustering algorithms to word embeddings. Coherent topics are retrieved from the word clusters.

Angelov (2020) utilizes joint embeddings of words and documents and leverages HDBSCAN (McInnes, Healy, and Astels 2017) to create clusters of documents from which topics are extracted. Closely following Angelov (2020), Grootendorst (2022) uses sentence transformers (Reimers and Gurevych 2019) to obtain document embeddings. Term-frequency inverse-document frequency (tf-idf) scores (Salton 1989) are used to obtain scores for the probabilities of words under a given topic. Miles et al. (2022) also leverage sentence transformers (Reimers and Gurevych 2019) but uses Particle Swarm Optimization to identify latent clusters in the corpus.

While being algorithmically much simpler than classical topic models, these document clustering schemes have proven extremely effective, often outperforming the computationally more demanding generative models (Sia, Dalmia, and Mielke 2020; Grootendorst 2022; Angelov 2020; Thielmann, Weisser, and Säfken 2022).

While there have been numerous approaches to extracting latent topics from large text corpora, little effort has been made in adapting those models to more refined definitions of a topic. We propose a topic model that follows ideas from linguistic definitions of topics (Davison 1982, 1984). We present two simple ideas from linguistic theory in order to construct more humanly interpretable topics:

• A word that most accurately expresses the topic of a document may not necessarily occur in that document.

• (ii) 

  Only using nouns and noun phrases is more appropriate for representing understandable topics.

Idea (i) closely follows (Guijarro 2000): “a topic is, above all, a textual category that is determined by the context and not by purely formal or structural aspects.” Therefore, the topic of a document or even a sentence may go beyond the mere occurrence of all the words in that document. That is, a word that most accurately expresses the topic of a document may not necessarily occur in that document. We leverage a simple example from a New York Times headline to demonstrate that:

That sentence pertains to the financial crisis and the collapse of the Lehman Brothers bank, but neither phrase is explicitly mentioned. A bag-of-words model that only considers words present in the document corpus would not be able to accurately capture the document’s topic. Contextually relevant words, even if not present in the document, can provide better representations. Figure 1 shows the described example. Comparing the cosine distance in a reduced embedding space between the complete embedded sentence (TEXT) and each embedded word or phrase demonstrates how words and phrases not occurring in that text can be a meaningful summary of that text. “Banking crisis” is a more meaningful representation of the sentence than, for example, “global” and lies closer to the text in the semantic space.

Common topic models (e.g., Blei, Ng, and Jordan 2003; Dieng, Ruiz, and Blei 2020; Bianchi, Terragni, and Hovy 2020; Das, Zaheer, and Dyer 2015), as well as document clustering methods (e.g., Grootendorst 2022; Angelov 2020; Sia, Dalmia, and Mielke 2020), face a limitation in that they only consider words that appear in the reference corpus when generating topic representations. This limitation can lead to incorrect topic interpretations, as shown in the example above. Through expanding the reference corpus and leveraging pre-trained embedding models, we make sure that “the indispensability of frame knowledge for understanding texts” (Beghtol 1986) is accounted for.

Point (ii) closely follows Beghtol (1986), after whom one of the features of generalized titles is the absence of verbal forms. Following the idea that a title is the highest macroproposition of a textual unit (Beghtol 1986), we apply this idea to the construction of topics and hence propose to only consider nouns and noun phrases for the proposed method of topic extraction. Martin and Johnson (2015) already demonstrated improvements in topic quality when focusing on nouns, but reduced the complete corpus to nouns only, losing any semantic or syntactic information hidden in the documents.

The findings from Hoyle et al. (2021) show that current state-of-the-art automated topic model evaluation does not represent human judgment of topic quality. However, human topic evaluation is inherently complicated as human evaluation is highly subjective. Moreover, as demonstrated in Section 3 above, topics can be highly complex and theoretical constructs. Simply relying on human judgment to determine whether a topic is good or not does not encompass all of a topic’s properties. The documents that should be represented by the topic or topic diversity are seldom accounted for (Newman et al. 2010; Lund et al. 2019; Clark et al. 2021). Clark et al. (2021) even question human judgment alltogether; however, the used questionnaire design not only does not provide a midpoint but additionally can strongly induce a bias in preference due to a highly biasing follow-up question (Clark et al. 2021) (see, e.g., Lehman et al. 1992). Newman et al. (2010), for example, use the straightforward approach of letting humans rate the created topics quality. Choosing a 3-point scale for model evaluation, however, can induce unreliability of responses (Krosnick 2018). Simple assessment of topic quality therefore does not suffice and creating great questionnaires and adequately operationalizing what researchers are interested in is notably important.

Adapting questions and tasks to the complicated nature of topics can result in promising questionnaire designs. Lund et al. (2019), for example, introduced a topic-word matching task, weighting and selecting answers from participants who have a high confidence and performed well on test questions. Choosing that approach reduces ambiguity in answers, but also induces a bias towards highly confident participants and neglects the subtle differences in perceived quality from humans. Promising results are also achieved with further refined questionnaire designs. Bhatia, Lau, and Baldwin (2017, 2018) introduce document-level topic model evaluation leveraging the intruder-topic task, also introduced in Chang et al. (2009). In this task, participants are presented with a list of words that are related in some way. The task typically involves presenting a series of word lists, with each list containing a set of related words except for one “intruder” word that does not belong to the category or topic (e.g., Apple, Orange, Pineapple, Bicycle, Banana, Mango). The participants’ objective is to identify and quickly recognize the intruder word within each list. Thus, the intruder-task, if cleverly designed and using “intruder” words from different topics, can account for topic diversity and even could account for a topic adequately representing a set of documents, by sampling “intruder” words directly from the set of documents.

Given the seminal works of Grootendorst (2022) and the results shown by Thielmann, Weisser, and Säfken (2022), we propose a simple yet highly effective document clustering and topic extraction method. The pseudo algorithm for the complete model can be seen in Algorithm 15. The proposed method expands a given base-corpus by enriching extracted document clusters with nouns from an external expansion-corpus and can be summarized in a simplified manner as follows: First, the given corpus of documents is embedded using contextualized transformer embeddings. For example Bianchi, Terragni, and Hovy (2020) showed that contextualized embeddings can improve topic quality significantly. Second, the dimensionality of the embedded documents is reduced to alleviate the curse of dimensionality caused by the typically large number of dimensions in text embeddings. Third, the embedded documents are clustered using a Gaussian mixture model (GMM). The central results from this clustering step are the document clusters in general, which can be interpreted as topics among the documents, and the centroids of the found clusters more specifically. Please note that we do not enhance the documents during clustering, thereby ensuring that no erroneous expansion can compromise the quality of the topics. See Section 4.2 for different approaches. Subsequently, all nouns existing in the base-corpus as well as all nouns in the extension-corpus are embedded into the same embedding space as previously the documents using the same text-embedding model as for the texts in the base corpus. After that, we select the prototypical words representing each topic as the words with the most similar embedding to the centroid representing the topic. This allows us to not only obtain soft document-topic-assignment scores via the similarity of documents to centroids of topics, but to also obtain scores for the most likely words given a specific topic. Note that the words representing a topic among documents in the base corpus can also come from the extension corpus provided they enhance the word-level representation of a found topic. Last, a cleaning step can be performed to remove overly similar terms from the topics. Through the simple step of reference corpus expansion and leveraging soft clustering, we can significantly improve upon previous document clustering and topic extraction methods (Grootendorst 2022; Angelov 2020).

More formally and extensively, the proposed approach can be described as follows: Let $V=w1,…,wn$ be the vocabulary of words and $D=d1,…,dM$ be a corpus (i.e., a collection of documents). Each document is a sequence of words $di=wi1,…,wini$ where w_ij ∈ V and n_i denotes the length of document d_i. Further, let $D=δ1,…,δM$ be the set of documents represented in the embedding space, such that δ_i is the vector representation of d_i and let $W=ω1,…,ωn$ be the vocabulary’s representation in the same embedding space. Hence, each word w_i in the embedding space represented as ω_i ∈ℝ^L has the same dimensionality L as a document vector δ_i ∈ℝ^L. There are different representations of topics, but mostly a topic t_k from a set of topics $T=t1,…,tK$ is represented as a discrete probability distribution over the vocabulary (Blei, Ng, and Jordan 2003), such that t_k is often expressed as (ϕ_k,1,…, ϕ_{k, n})^T and $∑i=1nϕk,i=1$ for every k, where ϕ_{k, n} ∈ [0,1]. Thus, ϕ_k = (ϕ_k,1,…, ϕ_{k, n}) simply describes the probability vector over the vocabulary for topic k.¹

Based upon the idea expressed in Section 3, we form clusters from the documents embeddings, $D$⁠, and subsequently extract topics, t_k, that represent these clusters best. Hence, after transforming the raw documents into document vectors, they are clustered. Due to the curse of dimensionality (Aggarwal, Hinneburg, and Keim 2001) we reduce the dimensionality of the document embeddings before clustering using UMAP (McInnes, Healy, and Melville 2018), closely following Angelov (2020) and Grootendorst (2022). However, we allow each document to belong to more than one cluster resulting in document topic matrices θ and word topic matrices β, similar to LDA (Blei, Ng, and Jordan 2003). The documents are clustered with a GMM (Reynolds 2009), as it not only allows for soft-clustering, but also has the advantage of optimizing hyperparameters via, for instance, the Akaike information criterion or the Bayesian information criterion. As a result, CEDC, in contrast to others (Angelov 2020; Grootendorst 2022; Sia, Dalmia, and Mielke 2020), offers not only word-topic distributions but also document-topic distributions.

To avoid having words in a topic that are semantically overly similar as, for example, economics and economy, each topic can be cleaned. The cosine similarity between the top Z words contained in a topic can be computed and all words that exceed a certain threshold, for example, 0.85,² are removed in descending order of the similarity with the clusters centroid. An additional advantage of the corpus expansion is the possibility to model documents in one language, but create topics in a different language, when using a multi-language embedding model.

Cleaning the topics based on a similarity threshold offers maximum flexibility. Depending on the task and the preferences, one could thus create topics with maximally divergent words. In our experiments we find a threshold of 0.85 to be reasonable and delete all words that have a cosine similarity score greater than or equal to 0.85 compared against other words present in the topic in descending order.

In the realm of topic modeling, leveraging additional corpora has long been used to address issues associated with short documents, such as tweets. A straightforward yet effective solution involves merging similar short documents into longer ones, better suited for classical algorithms like LDA (Mehrotra et al. 2013). For example, Kant, Weisser, and Säfken (2020) offer an approach for aggregating tweets based on common hashtags, a strategy also utilized by Luber et al. (2021). On the other hand, Thielmann et al. (2021) and Thielmann, Weisser, and Krenz (2021) use expansion corpora before actual topic modeling to combat severe imbalances in their corpora. Bicalho et al. (2017) present a framework for extending short documents in topic modeling, although it only uses words already present in the main vocabulary. A similar conceptual approach is adopted by Zheng, Liu, and San Wong (2018). These methods, all involving the enrichment of documents, are susceptible to errors, as it is crucial to ensure the correctness of this enrichment. In cases where document enrichment is flawed, the topic model may generate topics that bear no relation to the original corpus, failing to uncover its latent themes and instead presenting erroneous extended topics.

In contrast, CEDC diverges significantly from the aforementioned methods in its conceptual approach. Notably, it refrains from expanding the documents themselves at any stage, guaranteeing the integrity of the topic modeling process. Document expansion occurs only after modeling, when candidate words that best represent a topic are selected. For a concise overview of the method, please refer to the pseudo-algorithm provided in Algorithm 15.

Given the described approach, we are effectively losing any idea of co-occurence based coherence for model evaluation. The words best describing a cluster of documents or topic do not necessarily have to occur together often in documents. In fact, a word capturing the topic of a single document optimally does not necessarily have to be contained in that same document. Additionally, by enhancing the corpus, it might be possible that neologisms are the words best representing a topic. Imagine, for example, a set of documents being equally about software and hardware issues. The neologism software-hardware would be an understandable and reasonable word describing that topic, but would perform poorly in any word co-occurrence-based evaluation measure. [[FX-ALGO1]]

For evaluation, we hence propose new, non-word co-occurrence-based measures and use existing measures leveraging word embeddings (Terragni, Fersini, and Messina 2021). We validate the intruder-based metrics by computing correlations with human annotations. See the Appendix for a comprehensive overview over all introduced and used metrics.

First, we propose a novel measure inherently representing the meaningfulness of a topic. For that, we leverage stopwords, which, as widely recognized, fulfill a grammatical purpose, but transport nothing about the meaning of a document (Salton 1989; Wilbur and Sirotkin 1992). Hence, we compute the vector embeddings of all stopwords and calculate a centroid embedding. Subsequently, we compute the cosine similarity between a topic centroid and the stopword centroid (see Figure 2).

Additionally, we propose three different new metrics, leveraging the idea of intruder words (Chang et al. 2009) and similarly integrating an idea of topic diversity. First, a metric that is based upon unweighted topic centroids.

To validate the intruder word based evaluation metrics we take the publicly available data from Chang et al. (2009). Similar to Lau, Newman, and Baldwin (2014) we compute the metrics over all topics and all models provided in Chang et al. (2009) for the 20 Newsgroups dataset. However, for clear interpretability, we reduce all words that include hyphens, due to the representations from Chang et al. (2009). Hence, we compute the metrics for 7,004 topics in total. We compute the accuracy of the metrics in terms of the true intruder and the humanly detected intruder for all metrics as well as the Pearson-r. While the important measures are here the correlation with the human annotations, reporting the correlations with the true intruder word ensures that the metrics are not inherently biased towards machine selection. For the accuracy, we consider a pre-selected or human-selected intruder to be correctly identified, if the score for this word is the lowest or highest, respectively, among all displayed top words, and for the correlation we consider the average coherence between a human-selected or an intruder-word and the other displayed words in a topic compared with those scores for the rest of the displayed words. The results are shown in Table 1. For all results it must be noted that the human answers have some ambiguity in them. As reported by Lau et al. (2014), the Pearson-r between the human answers was 0.77.

Hence, the results for INT with a maximum correlation of 0.728 is highly credible and outperforms the reported correlations (Lau, Newman, and Baldwin 2014) for coherence evaluation metrics. Interestingly, ISIM performs best when considering the accuracy for the true intruder word, but significantly worse when considering the human selected word. We find that, independent of the chosen model, the newly introduced metrics strongly outperform the results reported by Lau, Newman, and Baldwin (2014) at the topic-level with reported Pearson correlations of around r = 0.6.

Additionally, we compare the accuracy and correlation with four other existing metrics. More specifically, we compute results for normalized pointwise mutual information (NPMI) Coherence (Lau, Newman, and Baldwin 2014), as well as embedding coherence (COH^pw) and word embedding-based centroid coherence (WECC) (Terragni et al. 2021). The results are shown in Table 2. See the Appendix for a comprehensive overview over all of the mentioned evaluation metrics. As a base corpus for metrics leveraging additional documents, we use a Wikipedia dump with a size of around 1.6 million documents. We use the provided implementation from the OCTIS package with default parameters (Terragni et al. 2021). Note that for COH^pw and WECC the used Word2Vec (Le and Mikolov 2014) embeddings are normalized by dividing by the sum of the vector entries before computing the cosine similarity, which we find significantly decreases the quality of those metrics as compared to not including this sum-based normalization. We also include the recently proposed contextualized point-wise mutual information (CPMI) metric proposed by Rahimi et al. (2023), where the classical word probabilities from NPMI are exchanged with an estimate for the probability of words in context based on a pre-trained masked language model. We set the segment length and segment step parameter to 40. Because this metric is highly computationally demanding, we are only able to use a fraction of Wikipedia comprising around 20,000 documents. One can also note that even with this reduced corpus size, computing the CPMI scores for the benchmark study takes around 23 hours using an A100 graphics card when evaluating the word-pair-likelihoods with a BERT model (Devlin et al. 2018) as proposed by Rahimi et al. (2023). Note that all other metrics are computed in less than 3 minutes on inferior CPUs.

Interestingly, we find very bad correlations between CPMI and the human evaluation. However, Rahimi et al. (2023) report very low correlations between classical NPMI and CPMI as well as surprisingly low scores for CPMI for otherwise well performing topics.⁴ Additionally, Rahimi et al. (2023) also find lower correlations between intruder words detected by a chatbot and CPMI compared with NPMI, which is validated in our results. This is also confirmed when we use the same reference corpus of 20,000 documents for NPMI that we used for CPMI. While the performance of NPMI with a smaller reference corpus is drastically reduced, it is still superior to CPMI in terms of intruder metrics which resonates with Rahimi et al. (2023).

Overall, we find that the introduced metrics INT and ISIM achieve the highest correlations with human evaluation. We find a comparably lower correlation for NPMI compared to Lau, Newman, and Baldwin (2014), since they use a different Wikipedia source corpus for computing the metric but are very similar to the results reported by Stammbach et al. (2023) and Hoyle et al. (2021). However, even the reported correlation scores of around 0.6 by Lau, Newman, and Baldwin (2014) are lower than the correlations for INT of 0.73.

While we compute correlation scores between the presented metrics and human evaluation, the metric Topic Expressivity is more heuristically motivated. The general idea is to punish models that create garbage topics more severely as this tends to be an issue when using word- or document clustering techniques (e.g., for BERTopic [Grootendorst 2022]). In addition to Figure 2, a simple example can motivate the validity. Consider two topics:

• Topic 1:

• Spacecraft, Neil Armstrong, Orbit, Spaceship, Satellite, Nasa, Solarwind, Apollo 11, Rocket, Moon

• Topic 2:

• is, in, and, with, have, we, are, about, from, has

Computing the NPMI coherence, without adjusting for stopwords or word length and using the 20 Newgroups corpus as the reference corpus, would lead to the following NPMI coherence scores: Topic 1 =−0.34908 and Topic 2 = 0.2041. However, Topic 2 does not capture any semantic meaning whereas Topic 1 clearly is about outer space. The bad NPMI coherence scores for Topic 1 stem from the fact that neither Neil Armstrong nor Apollo 11 occur in the reference corpus. The topic expressivity, however, would severely punish Topic 2, with scores of 0.31 and 0.9, respectively (smaller scores are better). Note, that we adjusted the NPMI coherence measure in our benchmark study to correctly account for stopwords, by filtering the reference corpus, respectively. However, more or less meaningless words such as without are not accounted for in, for example, the nltk stopword list (Bird, Klein, and Loper 2009). Thus, the introduced measure offers a heuristic solution to penalize topics that uncover little semantic meaning.

To evaluate the proposed model, we compare the model results with different benchmark models. We also demonstrate the validity of our two hypotheses on corpus expansion and noun phrases stated in Section 2.

As a corpus expanding the reference corpus in CEDC for topic extraction, we use the Brown corpus taken from nltk (Bird, Klein, and Loper 2009), which we also use for filtering the vocabulary for noun-phrases. Note that for our applications we do not account for n-grams, which could further improve the results of CEDC. We compute the proposed metrics from Section 5 except for the ISH metric due to its inferior performance on the intruder word detection task (Table 1). Additionally, we compute NPMI scores (Lau, Newman, and Baldwin 2014) with the input corpus as the reference corpus and topic diversity (WESS) and word embedding pairwise coherence scores (COH^pw) using the OCTIS framework (Terragni et al. 2021). All word embedding-based metrics are computed with the paraphrase-MiniLM-L6-v2 model (Reimers and Gurevych 2019) due to the results from Table 1, except for WESS and COH^pw where we use OCTIS’ default pre-trained word2vec (Le and Mikolov 2014) model. The word2vec model is trained on the GoogleNews corpus. The number of top words, Z, taken into account for the metrics EXPRS, COH, WESS, INT, and ISIM is 10. For INT and ISIM, we randomly select an intruder word from a randomly selected topic 50 times and report the averages.

To confirm our two hypotheses from Section 2 that expanding the reference corpus and only considering nouns for topic extraction can increase the topic quality, we perform several analyses. We compare the presented method with and without reference corpus expansion and with and without noun phrase filtering. The averaged results over 3 datasets can be seen in Table 3.

Our results confirm our hypothesis that expanding the reference corpus leads to creating better topics depicted by nearly all metrics. Unsurprisingly, we find that NPMI coherence scores, only using the reference corpus for computing the coherence, are decreased when expanding the reference corpus during topic extraction. Additionally, we find that using a smaller pre-trained model for computing the metrics, as the leveraged word2vec (Le and Mikolov 2014) model for COH^pwand WESS, also shows a decrease in performance when expanding the reference corpus. That is presumably due to the smaller vocabulary size used in these models.

To demonstrate that CEDC actually takes words from the expansion corpus to create the topics, we analyze how many of the top words are actually taken from the expansion corpus; see Table 4. Over all 4 datasets, always using the brown corpus as the expansion corpus, CEDC creates topics with around 50% of words taken from the expansion corpus.

We find that the noun-based models perform worse than the models that consider all types of words and for the different embedding models used to construct the evaluation metrics. However, we find that when cleaning the topics the topic quality increases when using only nouns as compared to using all word types. Additionally we find that expanding the reference corpus and only considering nouns achieves better performance than no expansion and using all word types.

As comparison models, we use BERTopic (Grootendorst 2022) and Top2Vec (Angelov 2020) as closely related models and representatives of clustering based topic models; LDA (Blei, Ng, and Jordan 2003) as a model not leveraging pre-trained embeddings; CTM (Bianchi, Terragni, and Hovy 2020) as a generative probabilistic model leveraging pre-trained embeddings; CTMNeg (Adhya et al. 2023) using CTM as the base model; a simple K-means model, closely following the architecture from Grootendorst (2022), but replacing HDBSCAN with a K-means clustering approach; ETM (Dieng, Ruiz, and Blei 2020) leveraging word2vec (Le and Mikolov 2014); NeuralLDA; and ProdLDA (Srivastava and Sutton 2017). All models are fit using the OCTIS framework (Terragni et al. 2021). Where applicable the same pre-trained embedding model as for CEDC, all-MiniLM-L6-v2 (Reimers and Gurevych 2019), is used. Note that we perform extensive hyperparameter tuning for all models except for CEDC. For comparing CEDC with other models we use 4 standard benchmark datasets, 20 Newsgroups, Reuters (Lewis 1997), BBC News, and M10, as shown in Tables 6 and B.3–B.4. We fix the number of topics to the true number of topics of 20, 90, 10, and 5, respectively. We compute average rank scores for all models over all datasets. Table 5 shows the average rank over all models, all datasets, and all metrics. CEDC performs best and topic expansions clearly improve topic quality. A complete average rank table over all models and datasets can be found in the Appendix, Table B.1. For all tested models, we use the same pre-trained embedding model all-MiniLM-L6-v2 (Reimers and Gurevych 2019), where applicable. NPMI coherence scores are calculated as presented by Lau, Newman, and Baldwin (2014). For the best possible comparison, we use the same dimensionality reduction for CEDC as is used in Toc2Vec (Angelov 2020) and BERTopic (Grootendorst 2022). Hence, we use UMAP (McInnes, Healy, and Melville 2018) and reduce the dimensions to 5, explicitly using the same hyperparameters as done in the mentioned models. The same is done for the simple K-means model.

For LDA, ProdLDA, NeuralLDA, ETM, CTMNeg, and CTM, we optimize over various hyperparameters with Bayesian optimization as provided by the OCTIS package (Terragni et al. 2021). We use model perplexity, measured based on the evidence lower bound of a validation sample of documents, as the objective function in order to not rely on metrics, such as NPMI coherence or WESS, that measure either cohesion or separation of topics. LDA is optimized over the parameters of the two symmetric Dirichlet priors on the topic-specific word distribution and the document-specific topic distribution. For ProdLDA, NeuralLDA, CTMNeg, and CTM, the learning rate parameter, as well as the number of layers and the number of neurons per layer in the inference network, are considered. Finally, for ETM, we tune the learning rate, the number of hidden units in the encoder, and the embedding size.

Since BERTopic and Top2Vec are highly insensitive to different hyperparameter settings of the underlying HDBSCAN algorithm and also do not provide a way to measure the (marginal) likelihood of data, we choose the default hyperparameters for those models. While finding the optimal hyperparameters for these models might improve their performances compared to the models where we implemented hyperparameter tuning, the same is true for CEDC.

For CEDC we do not implement any form of hyperparameter tuning. Hence, the GMM is fit using scikit-learns default parameters. The convergence threshold for the Expectation Maximization (EM) algorithm is 0.0001; each component has its own general covariance matrix and 1e-6 is added to the covariance diagonals for regularization purposes. The maximum number of iterations in the EM algorithm is set to 100 and K-means is used to initialize the weights. Hence, the results achieved by CEDC could be further optimized, for example, by optimizing GMM with respect to the Bayesian- or Akaike Information Criterion. Additionally, the pre-trained embedding could be fine-tuned, which is true for all models leveraging pre-trained embeddings and could additionally improve the models’ performance (Thielmann, Weisser, and Säfken 2022; Bianchi, Terragni, and Hovy 2020).

As expected, the models closely related to CEDC also perform well. However, while Top2Vec, BERTopic, and the used K-means model are closely related to the proposed CEDC, CEDC achieves much better results concerning all metrics. Notably, CTM demonstrates exceptional performance on smaller datasets (please refer to the Appendix material). Since CTM leverages pre-trained document embeddings, the performance improvement compared to, for example, ETM when regarding small corpora is to be expected. CTMNeg exhibits remarkable proficiency in terms of topic diversity due to its utilization of negative sampling, inherently yielding dissimilar topics. Given that we exclusively utilize the source corpus for the computation of NPMI coherence, it is unsurprising that CEDC exhibits considerably diminished performance when considering the expansion corpus. This occurs because the topics generated include words that are absent in the evaluation corpus. Thus, CEDC* without corpus expansion outperforms CEDC ⁺ in terms of NPMI coherence. Notably, CEDC* performs quite similarly across almost all metrics in comparison to Top2Vec and K-means, as they share conceptual similarities. Additionally, CEDC ⁺ consistently demonstrates strong performance for all metrics for all datasets, consistently achieving an average rank within the top four for all metrics except NPMI. CEDC ⁺ performs best in 5 of 8 metrics and achieves an average rank of 3.0 when not considering NPMI coherence. Furthermore, our findings do not support the notion that models employing a hard clustering approach significantly underperform on a multi-label dataset like Reuters, when compared to models incorporating soft clustering techniques (as observed in CTM/ETM vs. Top2Vec/BERTopic results).

This is also confirmed when taking a closer look at the extracted topic words (see Tables C.5–C.6). While CTMNeg also creates coherent topics, it also inflates topics with words that are ambiguous or rather unrelated to the overal theme, such as demonstrated in topic 15: [make, church, people, time, work, president, day, thing, give, job]. The same is true for ProdLDA or K-means. This becomes especially evident when looking at the 20 Newsgroups dataset and the encryption topic.

From the top-5 words it is relatively hard to conclude that the CTMNeg topic is about encryption. Whereas the other words, such as key and chip, certainly are related to encryption they are so only in the context knowledge of what this topic is about. Table 7 shows two selected topics from several models fit on the 20 Newsgroups dataset. The coloring scheme is performed based on cosine similarity in the embedding space. We find that ProdLDA, the worst performing model in terms of automated evaluation metrics, also quite obviously mixes up topics—for example, sports and mideastern politics. On the other hand, especially CEDC and K-means create topics where each word clearly belongs to the overall detected themes.

We develop a novel model for topic extraction beyond the mere occurrence of words in the reference corpus. We are able to show that expanding the reference corpus improves model performance. Additionally, we can confirm that restricting the word types for topic extraction by only considering nouns can also lead to improved topic quality, under certain conditions. CEDC outperforms commonly used state-of-the-art topic models on multiple benchmark datasets, even in cases where the comparison models underwent extensive hyperparameter tuning while no hyperparameter tuning was performed for CEDC.

Given that almost all newly introduced topic models are evaluated automatically (Hoyle et al. 2021), automatic evaluation metrics are of utmost importance. Hoyle et al. (2021) even postulated that automatic topic model evaluation is broken, as the current used metrics have overall low correlations with human judgment of topic quality. Hoyle et al. (2022) even go a step further and argue that neural topic modeling altogether is broken. However, in their comprehensive analysis they fail to address human evaluation or intruder word-based topic evaluation. Looking at Tables C.1–C.3 we cannot conclude that neural topic models perform worse than non-neural models and find that neural topic models can achieve great results also for automated evaluation metrics that strongly align with human judgment. Instead, we find that, for instance, LDA while also creating concise topics creates topics that lack any specific meaning, such as: [time, thing, lot, good, bad, make, feel, pretty, real, experience]. We present multiple novel evaluation metrics closely following state-of-the-art human evaluation of topic model quality and achieve great correlations with human evaluation. We greatly improve upon the correlation with human evaluation compared to the currently most often used metric, NPMI, achieving correlations of around r = 0.73 compared to NPMI correlations of r = 0.63. The proposed approach of using word embeddings and cosine similarity achieves impressive results given the overall lower agreement between human responses (Pearson-r = 0.77).

Additionally, we introduce a novel evaluation metric, based upon the centroid cluster of stopwords in the embedding space. Given the approach of enhancing the reference corpus, the described model might be especially useful when evaluating short texts or identifying sparsely represented topics in a corpus (Thielmann, Weisser, and Krenz 2021; Thielmann et al. 2021). Through the inherent sparsity of the data, the words best describing a topic might not be included in the reference corpus and an enhancement could thus greatly improve the creation of topics.

Automated evaluation of topic model quality is inherently difficult. That difficulty is considerably increased by the fact there is no gold standard or even a ground truth for the quality of a topic. Chang et al. (2009) introduced the reasonable approach of evaluating the coherence of a set of words with intruder-words. However, one cannot expect 100% agreement between people when it comes to judging whether a word is an intruder word in a topic. The proposed evaluation metrics achieve impressive results with human annotations; they cannot, however, reflect human ambiguity or extreme subtlety in perceived topic quality. Additionally, as all evaluation metrics based upon human evaluation and hence experimental results achieved with human participants, the metrics might reflect a selection bias (WEIRD) (Henrich, Heine, and Norenzayan 2010). Further embedding models could be evaluated and tested and larger human evaluation studies could be conducted.

Recent findings about the dominance of certain dimensions in transformer embeddings (Timkey and van Schijndel 2021) suggest an inherent bias in transformer embeddings that could negatively affect similarity measures in the semantic space. Our results do not suggest that such a bias negatively influences the modeling results; however, this study does not look into the dimensionality effects, which could be the topic of further research.

Moreover, the creation of transformer models solely for the purpose of topic extraction that emphasize, for example, the beginnings of phrases due to their increased importance to the underlying topics of a subsubsection (Kieras 1980, 1981) could greatly improve upon the existing methods.

To make reading easier, we provide a full notation list. All used variables and their notation can be found here.

All modeling steps from the proposed method are presented here in extensive form. First, the target corpus should be embedded. This can be done, either using contextualized transformer embeddings, as, for example, Bianchi, Terragni, and Hovy (2020) showed that contextualized embeddings can improve topic quality. However, approaches as used by Sia et al. (2020) where every word is embedded singularly and the documents are represented as centroid vectors of all occurring words are also possible. Second, the dimensions of the embedded documents, δ_i, are reduced due to the curse of dimensionality. Afterwards, the reduced embeddings, $δi^$⁠, are clustered, for example, using GMM such that soft clustering is possible. The centroids for each document cluster, μ_k, are computed. Next, the corpus is filtered for nouns and all nouns present in the corpus supplemented by all nouns present in an expansion corpus are embedded. Note that here the same embedding procedure must be chosen as for the documents (see, e.g., Angelov 2020; Grootendorst 2022). Then, the similarity between all candidate words and all document cluster centroids is computed. Based on the candidate embeddings and the similarity to the document clusters μ_k, the topic centroids γ_k are computed and similar to LDA, we get a document topic matrix, θ, and a word topic matrix, β. Last, a cleaning step can be performed to remove overly similar words from the topics.

In addition to the 20 Newsgroups and Reuters dataset, we fit all models on the M10 and BBC News datasets. Both datasets are taken from OCTIS (Terragni et al. 2021). CEDC again outperforms most other models on nearly all metrics. Interestingly, CTM achieves good results for the BBC News dataset, which is comparably small with <2,000 documents. For the M10 dataset, which is composed of scientific papers and hence a more difficult dataset, we find that topic expansion strongly improves the model performance.

Since we only consider the training corpora for the NPMI metric, CEDC has an inherent disadvantage in this metric. When we compare overall average rankings over all metrics except for NPMI, the advantage of CEDC over the benchmark models becomes even more apparent, which is demonstrated by the average performance rankings omitting NPMI scores shown in Table B.2.

For all tested models, we use the same pre-trained embedding model all-MiniLM-L6-v2 (Reimers and Gurevych 2019), where applicable. NPMI coherence scores are calculated as presented by Lau, Newman, and Baldwin (2014). For the best possible comparison, we use the same dimensionality reduction for CEDC as is used in Doc2Vec (Angelov 2020) and BERTopic (Grootendorst 2022). Hence, we use UMAP (McInnes, Healy, and Melville 2018) and reduce the dimensions to 5, explicitly using the same hyperparameters as done in the mentioned models. The same is done for the simple K-means model.

• NPMI Coherence (NPMI):

  • –

    NPMI Coherence measures the average normalized pointwise mutual information between words in a topic. The word (co)occurrence probabilities are estimated based on document chunks of a reference corpus.

  • –

    Established metric for traditional topic models.

  • –

    No embeddings are required.

  • –

    Needs a potentially large reference corpus.

  • –

    Results depend on hyperparameters for estimating the word (co)occurrence probabilities.

  • –

    Can yield inconsistent or biased results, especially for Neural Topic Models.

• Embedding Coherence (COH):

  • –

    Embedding Coherence measures the average cosine similarity of word pairs within a topic.

  • –

    Simple, easily explainable metric.

  • –

    Requires word embeddings.

    • *

      – Choice of word embeddings is a hyperparameter.

• Word Embedding-based Centroid Coherence (WECC):

  • –

    WECC computes the average cosine similarity of words to the centroid of the words of their topic.

  • –

    Can take advantage of word embeddings.

  • –

    Lower computational complexity than COH.

  • –

    Requires word embeddings.

    • *

      – Choice of word embeddings is a hyperparameter.

• Contextualized Pointwise Mutual Information (CPMI):

  • –

    The CPMI metric works similarly to the NPMI metric. Here the similarity of two words is assessed based on likelihood ratios by a BERT model.

  • –

    Utilizes the powerful BERT model.

  • –

    Potentially very high computational cost.

  • –

    High computational cost can lead to poor performance of this metric since the reference corpus has to be restricted.

• Topic Expressivity (EXPRS):

  • –

    Topic Expressivity is computed as the cosine similarity between the centroid of the embeddings of the stopwords and the centroid of the topic.

  • –

    Novel metric for measuring the expressivity of a topic.

  • –

    Requires word embeddings.

    • *

      – Choice of word embeddings is a hyperparameter.

  • –

    Results depend on the choice of stopwords.

• Word Embedding-based Weighted Sum Similarity (WESS):

  • –

    The diversity metric WESS measures the average cosine similarity between the centroids of a given collection of topics.

  • –

    Simple metric for assessing the diversity of a clustering.

  • –

    Requires word embeddings.

    • – Choice of word embeddings is a hyperparameter.

• Intruder Shift (ISH):

  • –

    Intruder shift assesses how much a topic’s centroid changes when the embedding of an unrelated intruder word is added.

  • –

    Intruder-based metric.

  • –

    Intuitive explanation.

  • –

    Requires word embeddings.

    • – Choice of word embeddings is a hyperparameter.

  • –

    Depends on the choice of intruder (averaging over many choices alleviates this issue).

• Intruder Accuracy (INT):

  • –

    For a given topic and a given intruder word, Intruder Accuracy is the fraction of topwords to which the intruder has the least similar embedding among all topwords.

  • –

    Intruder-based metric.

  • –

    High correlation with human assessments.

  • –

    Requires word embeddings.

    • – Choice of word embeddings is a hyperparameter.

  • –

    Depends on the choice of intruder (averaging over many choices alleviates this issue).

• Average Intruder Similarity (ISIM):

  • –

    ISIM measures the average cosine similarity of topwords of a topic to an intruder word.

  • –

    Intruder-based metric.

  • –

    Intuitive explanation.

  • –

    High correlation with human assessments.

  • –

    Requires word embeddings.

    • – Choice of word embeddings is a hyperparameter.

  • –

    Depends on the choice of intruder (averaging over many choices alleviates this issue).

Funding by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) within project 450330162 is gratefully acknowledged.