Evaluating the BERTopic workflow

The intended application was grouping insurance customer enquiries into understandable topics. Recording and transcription quality constrained access to suitable call data, so the project used scientific abstracts as a proxy. It did not evaluate topic modelling on real insurance calls.

The dataset contained 16,842 abstracts: 5,971 in computer science, 5,513 in mathematics and 5,358 in physics. Their subject labels supported evaluation of overlapping and more distinct topic groups. The reported split was 70% training and 30% test.

SourceOriginal dissertation, pp. 1–3; Table 1. Supervisor affiliations and date: p. 1.

BERTopic supplied the existing modular framework. The study compared MiniLM, BERT and MPNet embeddings; UMAP, PCA and no dimensionality reduction; and k-means, Gaussian mixture and HDBSCAN clustering. Its contribution was configuring and evaluating these components for the chosen dataset.

The selected workflow used pretrained all-MPNet-base-v2 embeddings, retained all 768 dimensions, and applied k-means with 20 clusters. Class-based TF-IDF extracted representative keywords. Stop words were removed after clustering for keyword extraction, preserving fuller text for the embedding stage.

SourceOriginal dissertation, pp. 3, 6–8 and 9–12; Tables 2–5.

The report defines pseudo-accuracy by assigning each cluster its majority subject label, then measuring agreement with the known labels. This substitutes for manually interpreting the extracted keywords. It is a label-based clustering measure; it does not measure an operator’s topic decisions or call-routing accuracy.

UCI coherence separately assessed associations between topic words. Component comparisons used the training set. The cluster-count study examined 9–30 clusters and selected 20 after performance began to level off, rather than choosing the largest observed training score.

SourceOriginal dissertation, pp. 8–9 and 11–12; Figures 2–4.

The final configuration reported 86.25% training pseudo-accuracy and 85.72% test pseudo-accuracy. Mean UCI coherence was 1.28 and 1.27 respectively. Confusion remained concentrated between computer science and mathematics; physics was easier to separate.

Higher scores required context. HDBSCAN’s reported 0.89 pseudo-accuracy excluded approximately 15% of samples as outliers. MiniLM achieved higher keyword coherence than MPNet in Table 2, while MPNet had slightly higher pseudo-accuracy at 20 clusters. These comparisons expose coverage and interpretability trade-offs.

SourceOriginal dissertation, pp. 9–10 and 13–14; Tables 2–4, 6–7; Figures 4–6.

Scientific abstracts are edited, information-dense documents. Their results do not establish performance on noisy conversations. The embedding model was not fine-tuned for insurance, and the study did not measure operational savings. The report also leaves unclear whether test-set cluster labels were fixed from training; the reported score should be read as clustering agreement, not an independently established classification benchmark.

The transferable work is a documented sequence of engineering choices: preserve information before embedding, compare modules under stated metrics, inspect mixed clusters, and account for samples omitted by an algorithm. Applying the workflow to authentic call data remained further work.

SourceOriginal dissertation, pp. 3, 8, 10, 13, 15 and 18.