Hierarchical Indexing for Retrieval-Augmented Opinion Summarization

Online review websites are a useful resource when choosing which hotel to visit or which product to buy, but it is impractical for a user to read hundreds of reviews. Automatic opinion summarization aims to aggregate a large and diverse set of customer reviews about a particular entity into a single, easy to read summary. A good summary should accurately reflect the balance of opinions in the input reviews, highlighting the most common or popular opinions, while omitting unnecessary details. A useful summary should also help compare between competing options, and include points that differentiate the current entity from others.

Early work on opinion summarization extracted reviewers’ sentiment about specific features (Hu and Liu, 2004) or identified salient sentences based on centrality (Erkan and Radev, 2004), while more recent methods have proposed extractive models that use learned feature spaces (Angelidis et al., 2021; Basu Roy Chowdhury et al., 2022). Prior work on abstractive opinion summarization has almost exclusively either required costly supervision (Bražinskas et al., 2021; Cattan et al., 2023) or has assumed that the number of input reviews is limited (Coavoux et al., 2019; Bražinskas et al., 2020; Amplayo et al., 2021a, b; Iso et al., 2021). This defeats the point of a summary: a user could feasibly read 8 reviews in a reasonable period of time. A good summarization system should be scalable, since popular products online may receive thousands of reviews. It should also be attributable, offering some evidence to justify its output. Paraphrasing Rashkin et al. (2023), we say that a statement s is attributable to some evidence E, if a generic reader would agree that ‘According to E, s is true’. Finally, it should generate summaries that are coherent and faithful to the input reviews.

Large Language Models (LLMs) have been shown to generate highly fluent summaries in the news domain (Bhaskar et al., 2023) but are a flawed solution because current instruction-tuned models are not attributable, and because their context windows limit the number of reviews they are able to base their summaries on. Models with long context windows have been proposed (Beltagy et al., 2020; Gu et al., 2022) but these are not currently instruction-tuned, and it has been shown that LLMs are biased toward information at the start and end of the input (Liu et al., 2023).

Our approach, Hierarchical Indexing for Retrieval-Augmented Opinion Summarization (HIRO), identifies informative sentences using hierarchical indexing and then passes the selected sentences as input to a LLM, similar to retrieval-augmented generation (RAG, Lewis et al., 2020). By separating the steps of content selection and generation, we can combine the attributability and scalability of the discrete representation with the strong generative abilities of LLMs, leading both to a higher quality index and to more informative and coherent output summaries.

HIRO consists of three modules, allowing for increased control, flexibility and interpretability. The Hierarchical Indexer is an encoder that maps sentences from reviews to paths through a hierarchical discrete latent space. The Retriever uses the index to identify clusters of sentences for each entity that contain popular and informative opinions. These sentence clusters are passed to a Generator, a pretrained LLM, that generates coherent summaries that are grounded in the retrieved sentences.

Our contributions are as follows:

• We propose a method for learning an encoder that maps sentences to a path through a semantically structured discrete hierarchy.

• We show how to exploit this discrete hierarchy at inference time to identify clusters of related and prevalent sentences from input reviews.

• We introduce an automatic metric that measures whether generated summaries reflect the input reviews, while penalizing overly generic statements.

• Through extensive experiments on two English datasets from different product domains, we demonstrate that passing these retrieved sentences in a zero-shot manner to a pretrained LLM generates summaries that better reflect the distribution of opinions within the input reviews. Human evaluation shows that summaries generated by HIRO are significantly more coherent and accurate than prior work, and are preferred by annotators.

Our code and dataset splits are available at https://github.com/tomhosking/hiro.

Prior approaches to generating summaries of reviews broadly fall into two categories. Extractive methods generate summaries by selecting representative sentences from input reviews to use as the summary (Erkan and Radev, 2004; Angelidis et al., 2021; Basu Roy Chowdhury et al., 2022). These types of approach are scalable and inherently attributable, but result in summaries that are overly detailed and not coherent. Abstractive methods ‘read’ input reviews and generate novel language to use as the summary (Bražinskas et al., 2020; Iso et al., 2021). The resulting summaries are more fluent and coherent, but most prior abstractive methods are only able to consider a limited number of input reviews, whether because of combinatoric complexity or a maximum input length.

Hosking et al. (2023b) propose a hybrid method that represents sentences from reviews as paths through a learned discrete hierarchy, then generates output sentences based on frequent paths through this hierarchy. While their method is abstractive as well as attributable and scalable, the highly compressed bottleneck leads to frequent hallucination and generic output with poor coherence.

Louis and Maynez (2023) use a Natural Language Inference (NLI) model to construct ‘silver standard’ summaries to use as training data for fine-tuning a pretrained language model (T5, Raffel et al., 2020). However, their approach is computationally very expensive, calculating over 1B pairwise entailment relations between sentences in the training data, before fine-tuning a LLM. By contrast, HIRO uses a lightweight indexing encoder, combined with an off-the-shelf LLM that is prompted in a zero-shot manner.

Other work has used specialized training data to train supervised models (e.g., Zhao et al., 2022; Cattan et al., 2023); however, such data is expensive to collect and is not generally available in every language, domain, or setting.

Prior work has investigated how to learn representations of data with richer structure and interpretability. Vendrov et al. (2016) propose learning an embedding space where the ordering of two pairs of samples could be inferred from their relative positions, but their method requires supervision of the correct ordering. Opper et al. (2023) describe a method for learning embeddings that explicitly include structural information, but they focus on representing structures, rather than on learning an ordering within the embedding space itself. Li et al. (2023) learn a tree-based index for passage retrieval concurrently with the dense embedding space, showing that this leads to improved retrieval performance. However, they focus on retrieving passages relevant to a query, rather than aggregating opinions. Sarthi et al. (2024) use a pretrained embedding model to cluster related sentences then generate a summary using an LLM, and repeat this process iteratively to construct a tree-structured index over inputs. However, this process is expensive and must be repeated for each new set of inputs.

The idea of first selecting relevant parts of an input before generating output has been well studied (Kedzie et al., 2018; Puduppully et al., 2019; Amplayo et al., 2021b; Narayan et al., 2023, inter alia), and has been shown to be very effective when used in conjunction with LLMs in the form of retrieval-augmented generation (RAG, Lewis et al., 2020). Xu et al. (2023) found that retrieval augmentation is beneficial even when using models that can accept long inputs. Wang et al. (2023) show that including an additional filtering or selection step to RAG is better than naively passing all retrieved documents as input.

Automatic evaluation of generated summaries is extremely challenging. Prior work has shown that ROUGE (Lin, 2004) scores correlate poorly with human assessments of summary quality (Callison-Burch et al., 2006; Tay et al., 2019; Fabbri et al., 2021; Shen and Wan, 2023; Clark et al., 2023; Aharoni et al., 2023). Some datasets are created automatically, with references that are not directly based on the input reviews (Bražinskas et al., 2021). Modern summarization system outputs are now often preferred to human-written references (Goyal et al., 2022; Bhaskar et al., 2023).

SummaC (Laban et al., 2022) is a reference-free metric that uses an NLI model to evaluate the degree of support between a summary and the input documents, but it overly rewards trivial statements; using the obvious statement “The hotel was a building” as a summary for every entity achieves a near-perfect SummaC score of 99.9% on Space, a dataset of hotel reviews (Angelidis et al., 2021).

Malon (2023) proposes a metric that uses a NLI model to evaluate prevalence, i.e., how many input reviews contain supporting evidence for each sentence in a summary, and explicitly penalizes trivial or redundant output. However, we found it has a similar failure mode to SummaC, with the statement “The rooms are clean and comfortable” achieving a prevalence score of 72% on Space. We propose a modification to prevalence in Section 6.2 that penalizes overly generic summaries.

Let $Re$ be a set of reviews about an entity $e∈E$⁠, where each review $R∈Re$ is composed of a number of sentences x. The goal is to generate a textual summary that includes the most informative opinions from $Re$⁠, while abstracting away the details specific to any one review.

HIRO generates summaries following a modular approach, depicted in Figure 1. We learn an index structure that maps each sentence x from the input reviews to a path q_1:D through a discrete hierarchy. We choose a hierarchical representation so that sentences are grouped at a useful level of abstraction; the upper levels of the hierarchy should partition sentences by topic, while the lowest levels should group together sentences with equivalent meaning.

At inference time, we encode all sentences from the reviews, then identify the paths or subpathsq_1:D within the index that are particularly popular, and retrieve the corresponding sentences. This selection process is query-free, instead relying on properties of the hierarchical index to determine the frequency of different opinions. By indexing sentences hierarchically according to their semantics, we can easily identify opinions or topics that occur frequently by simply counting their occurrence in the index.

Finally, we generate a summary by passing the retrieved clusters of sentences as input to a LLM. This retrieval-augmented usage of LLMs allows us to benefit from the fluency and coherence of LLMs, while retaining the attributability and scalability of extractive opinion summarization methods.

We now detail the three modules that constitute HIRO, evaluating each of them in turn to confirm their efficacy compared to previous methods.

Our goal is to learn an encoding space where sentences with similar meanings are grouped together. The space should be discretized so that frequent opinions can be easily identified by counting the membership of each part of the index, and it should be hierarchical so that opinions may be aggregated at an appropriate level of granularity, rather than by details or phrasings specific to a particular review. Finally, the encoding space should be structured semantically, to enable accurate aggregation of opinions; sentences with equivalent meaning should clearly be indexed to the same point in the hierarchy, while sentences that are topically related but not equivalent should be grouped together at a higher level.

We base our encoder on Hercules (Hosking et al., 2023b). Hercules uses an encoder-decoder architecture with a discrete hierarchical bottleneck to generate summaries. It is trained as a denoising autoencoder, and therefore needs to learn a representation that is both compressed enough to enable aggregation, but also expressive enough for the decoder to be able to generate meaningful output. These factors are in direct competition, with the compressed bottleneck leading to output that is generic and contains hallucinations. By contrast, HIRO uses an external LLM as the ‘decoder’, allowing us to focus on learning a representation that is useful for identifying informative opinions.

The HIRO encoder module maps a single sentence x to a path q_1:D through a discrete hierarchy, using a technique based on residual vector quantization (Chen et al., 2010; Zeghidour et al., 2022; Hosking et al., 2023b).

Since our goal is to learn a representation where semantically similar sentences are grouped together, we use a training objective that explicitly induces this arrangement in encoding space. We train the encoder with a contrastive learning objective, bringing representations of semantically similar sentences (i.e., positive pairs) together, and pushing dissimilar ones apart.

For each sentence in the training data, we construct positive pairs of semantically related sentences x, x₊ as follows: given a random ‘query’ sentence x from the training data, weidentify possible candidate ‘targets’ based on tf-idf similarity; then, we check whether each candidate is entailed by the query using an NLI model (DeBERTa v3, trained on Debiased NLI; He et al., 2021; Wu et al., 2022), and use the sentences which are labelled as entailed as positive targets x₊. An example is shown in Figure 2. We do not use any ‘hard’ negatives; instead, during training we calculate the pairwise tf-idf similarity between all samples in a batch, and include only those samples with similarity below a threshold as negatives x₋. We set this threshold to 0.3 based on validation set performance. This prevents ‘false negatives’ being wrongly forced apart, and allows sentences that are topically related but not strictly equivalent to remain in the same high-level grouping within the index.

We found that it was crucial to include sufficient exploration in the process of constructing positive pairs. The candidate sentences retrieved using tf-idf should be sufficiently similar that we are likely to find ones that are entailed, but not so similar that they only have minor lexical differences.

The similarity is floored at zero, to prevent the model from being able to ‘game’ the loss by pushing negative examples further and further apart. Although the positive pairs are selected based on a directional entailment label, we do not exploit this directionality in our training objective.

We employ the techniques to induce a hierarchical encoding space proposed by Hosking et al. (2023b), including depth dropout, initialization decay, and norm loss. We additionally include the entropy of the distribution over codes, $−∑d,qdlogp(qd)$⁠, as an additional term in the objective, to encourage exploration of the latent space during training.

We now evaluate whether the combination of discrete hierarchical encoding and contrastive objective leads to a more semantically distributed representation than previous methods.

We experiment using Space (Angelidis et al., 2021), which consists of hotel reviews from TripAdvisor with 100 reviews per entity, as well as reference summaries created by annotators. We encode all the review sentences from the Space test set, then calculate both the purity (the mean intra-cluster similarity) and colocation (the mean inter-cluster similarity) for the clusters of sentences assigned to each subpath q_1:D for d≤4. Finally, we take the difference between the purity and colocation as an overall measure of the quality of the representation space.

We compare to Hercules (Hosking et al., 2023b) and a baseline using recursive k-means over embeddings from a pretrained embeddings model (MiniLM, Reimers and Gurevych, 2019). We apply k-means to the embeddings, then calculate the residual errors between cluster centroids and embeddings, apply k-means to those errors, and repeat. All methods use the same number of clusters at each level (K = 12).

We use a 6 layer Transformer, with token embeddings initialized from BERT base (Devlin et al., 2019).¹ We set the codebook size K = 12, with the number of levels D = 12, based on validation set performance. Other hyperparameters are given in Appendix B.

Figure 3 shows the overall quality (using both NLI and tf-idf measures of similarity), indicating that HIRO learns a more semantically distributed space at all levels of depth than comparison approaches, according to both similarity measures. The separation between clusters increases with depth, confirming that the encoder learns a semantic hierarchy. We believe these results indicate that our method could potentially be used for more general purpose document retrieval (similar to Li et al., 2023), which is beyond the scope of this paper.

Recall that a good summary should include opinions that occur repeatedly in the input reviews, as well as opinions that differentiate the current entity from others. In Section 4 we showed how the HIRO encoder maps a single sentence x to a path q_1:D. We now exploit the discrete, hierarchical nature of the representation space to index a large number of review sentences, then identify informative sentences to use to generate a summary. We hypothesize that our content selection method leads to clusters that better represent the balance of opinions in the input reviews.

For each review $R∈Re$ about an entity $e∈E$⁠, we separately encode each sentence within the review to its path q_1:D, giving an indexed review Q(R).

Our scoring scheme conveniently accounts for the fact that short paths are more common;² short paths will also have a higher baseline popularity, leading to an overall score that is effectively normalized for the depth of the subpath.

After retrieving the clusters of sentences corresponding to the top-k highest scoring subpaths, we filter out sentences with very low lexical similarity to other cluster members, and combine any clusters of sentences with high lexical overlap.

We evaluate whether our method successfully selects clusters of sentences containing informative opinions, using reviews from Space, as in Section 4.2. First, we measure the overlap between retrieved clusters and oracle clusters constructed by selecting sentences with high overlap to the reference summaries,³ using the Adjusted Rand Index (ARI, Hubert and Arabie, 1985). Second, we evaluate the average prevalence (Malon, 2023) of sentences in retrieved clusters, which uses a NLI model (ALBERT, trained on VitC; Lan et al., 2020; Schuster et al., 2021) to evaluate how many input reviews support each retrieved sentence.

We compare HIRO to the clusters extracted by Hercules (Hosking et al., 2023b), and the oracle upper bound. As a baseline, we apply k-means to MiniLM embeddings (Reimers and Gurevych, 2019), then extract the 25 sentences whose embeddings are closest to each centroid. For HIRO, we select the top k = 8 subpaths for each entity, and set the smoothing constant α = 6.

Table 1 confirms that HIRO retrieves clusters that more closely match the oracle clusters, and contain opinions that are more prevalent in the input reviews compared to prior work. The oracle ARI score is less than 1 because some sentences appear multiple times in different clusters.

Given the retrieved clusters of sentences for each entity, we want to generate a coherent and fluent textual summary. LLMs are well suited to this constrained rewriting task, and we leverage the zero-shot abilities of instruction-tuned models to map clusters of sentences to a readable summary.

We generate a summary from the retrieved clusters in three ways, with varying trade-offs between coherence and attributability.

We generate extractive summaries by selecting the centroid of each cluster; we compute all pairwise ROUGE-2 scores between sentences in each cluster, and choose the sentence with highest average similarity to other cluster members. This approach is inherently attributable, since each summary sentence is extracted verbatim from a review.

We generate summaries one sentence at a time by passing the contents of a single cluster as input to an instruction-tuned LLM with a simple prompt that requests a single sentence that summarizes the main points. This leads to more fluent output that is likely to be attributable, since each output sentence has an associated cluster of evidential sentences used to generate it.

We generate summaries as a single document, by passing the sentences from all retrieved clusters for an entity to the LLM in one go. This gives summaries that are more coherent and less redundant, since the LLM has control over the whole summary. However, it is not possible to identify which cluster was used to generate each part of the summary, and therefore more difficult to determine the attributability of the output.

The ideal balance between coherence and the granularity of associated evidence is likely to vary by application.

For the variants that require an LLM, we use Mistral 7B Instruct v0.2 to generate summaries from retrieved clusters. Mistral 7B was chosen based on its qualitative performance during model development, but we compare using alternative models in Section 6.4. The LLM is queried in a zero-shot manner, and the prompts used are given in Appendix C. We sample with a temperature of 0.7, and report the mean and standard deviation scores based on 3 samples.

Human evaluation is the gold standard (Section 6.3), but automatic metrics remain useful for model development. ROUGE scores are no longer reliable (Section 2), but we nonetheless report them for consistency with prior work. Malon (2023) propose a prevalence metric, that uses an NLI model to determine how many input reviews contain supporting evidence for each sentence in the summary, but this suffers from a failure mode that overly rewards generic statements. A good summary should include common opinions, but should also help a user to differentiate between multiple entities.

We evaluate summary generation using Space (Section 4.2), which includes multiple reference summaries created by human annotators for each entity. We also include AmaSum (Bražinskas et al., 2021), to evaluate summary generation on a wide range of categories of Amazon products, with an average of 560 reviews per entity. The reference summaries were collected from professional review websites, and therefore are not necessarily grounded in the input reviews. We use the same splits, based on four product categories, as released by Hosking et al. (2023b). Further dataset statistics are given in Appendix D.

We select a random review from the inputs as a lower bound. We include an extractive oracle as an upper bound, by selecting the input sentence with highest ROUGE-2 similarity to each reference sentence.⁴ For a k-means baseline, we run k-means on MiniLM sentence embeddings (Reimers and Gurevych, 2019), then extract the nearest sentence to the cluster centroids. We set k = 8 to match the average sentence length of the reference summaries. Lexrank (Erkan and Radev, 2004) is an unsupervised extractive method using graph-based centrality scoring of sentences. QT (Angelidis et al., 2021) uses vector quantization to map sentences to a discrete encoding space, then generates extractive summaries by selecting representative sentences from clusters. SemAE (Basu Roy Chowdhury et al., 2022) is an extractive method that extends QT, relaxing the discretization and encoding sentences as mixtures of learned embeddings. CopyCat (Bražinskas et al., 2020) is an abstractive approach that models sentences as observations of latent variables representing entity opinions, trained in a ‘leave one out’ manner. BiMeanVAE and COOP (Iso et al., 2021) are abstractive methods that encode full reviews as continuous latent vectors using an autoencoder, and take the average (BiMeanVAE) or an optimized combination (COOP) of review encodings. We compare to a recent open weight instruction-tuned LLM, specifically Mistral Instruct v0.2 7B (Jiang et al., 2023). Since no training data is available, the LLM was prompted zero-shot as per Appendix C, and sampled with temperature 0.7. We report the mean and standard deviation scores based on 3 samples.

Most of the abstractive methods are not scalable and have upper limits on the number of input reviews. CopyCat and Mistral 7B have a maximum input sequence length, while COOP exhaustively searches over combinations of input reviews. We use 8 randomly selected reviews as input to CopyCat, COOP, and Mistral 7B.

The results in Table 2 show that HIRO achieves the highest SAP scores across both datasets, indicating that it generates summaries with the best balance between absolute prevalence and specificity. While CopyCat and COOP achieve the highest prevalence scores on Space and AmaSum respectively, they also display some of the highest genericness; qualitatively, the outputs are very similar to each other, with few specific details. We give example output in Tables 10 and 11.

While the oracle summaries score highly in terms of specificity-adjusted prevalence, some systems (including HIRO) outperform them. This indicates the difficulty with constructing reference summaries for entities with large numbers of reviews; it is not feasible for human annotators to reliably summarize such a large amount of information.

To evaluate how faithful the generated summaries are to the retrieved sentence clusters or evidence sets, we use an NLI model to determine how many sentences in each cluster either entail or are entailed by the corresponding sentence in the output summary, and take the mean. Considering both forward and backward entailment in this manner accounts for the different levels of granularity between the inputs and summary (Zhang et al., 2024); input reviews are likely to be more specific than summary sentences, but concise summary sentences are likely to contain multiple assertions, e.g., “The food was good and the rooms were clean”. HIRO_doc does not align the evidence sets with each generated sentence, so we calculate the maximum support score over all sets for each summary sentence. Most abstractive systems are not attributable, and so we only compare with Hercules. Table 3 shows the percentage of summary sentences that have support from at least one sentence in the evidence (partial support) and from at least half the sentences in the evidence (majority support). HIRO_doc generates summaries that have strong partial support from the associated evidence sets, with improved majority support compared to Hercules despite also being significantly more detailed.

We confirm that HIRO exploits the hierarchical nature of the representation space. Figure 4 shows the distribution of selected subpath depths for both datasets, indicating that HIRO takes advantage of the hierarchy and selects clusters deeper than the top level. This is particularly true for AmaSum, where there is a wider range of product types, causing the selection process to search deeper in the tree.

We conduct a human evaluation to verify that HIRO generates summaries that are coherent and accurately reflect the opinions in the input reviews. We recruit crowdworkers through Prolific, selected to be L1 English speakers from the US or UK with a minimum of 100 previously approved studies, compensated above the UK living wage at 12GBP/hr. Participants were allowed to rate at most 5 samples. We show annotators a set of 50 reviews (chosen based on pilot studies to balance annotator load with reliability), followed by two generated summaries. We solicit pairwise preferences (Louviere and Woodworth, 1990) along three dimensions, as well as an overall preference:

• Accuracy — Which summary accurately reflects the balance of opinion in the reviews?

• Detail — Which summary includes more specific details?

• Coherence — Which summary is easy to read and avoids contradictions?

• Overall — Which summary do you think is better, overall?

The full instructions are reproduced in Appendix A. Ties (i.e., ‘no difference’) were allowed. We gather annotations for 10 entities each from the Space and AmaSum test sets, with 3 annotations for each pairwise combination of system outputs, leading to a total of 900 pairwise ratings. The study was approved by the ethics committee, ref. #491139.

We compare HIRO_doc to the top performing extractive and abstractive systems from Table 2, SemAE and Hercules_abs. HIRO_doc uses Mistral 7B as the generator, so we also compare to Mistral 7B without HIRO (i.e., prompted directly with reviews). Finally, we include a random review as a lower bound, and the references as an upper bound.

The results in Figure 5 show that HIRO_doc produces summaries that outperform comparison systems across all dimensions, producing summaries that coherently and accurately represent the opinions in the input reviews. Differences between HIRO_doc and other systems are significant in all cases (using a one-way ANOVA with post-hoc Tukey HSD test, p <0.05), except for coherence versus Mistral 7B. Both Mistral 7B and HIRO_doc outperform the reference summaries, supporting findings from prior work (Bhaskar et al., 2023; Hosking et al., 2023a).

We run an additional human evaluation to evaluate the tradeoff between quality and attribution between the three HIRO variants, collecting annotations for 5 entities from each dataset from 3 annotators each. Figure 6 shows that annotators prefer the output of HIRO_doc, although the sentence-wise variant generates more detailed summaries. The preferred tradeoff between attribution and coherence will likely vary depending on the application.

Table 4 compares HIRO to zero-shot prompting, for a range of different LLM sizes from the Llama 2 Chat family of LLMs (Touvron et al., 2023). The same prompt template and hyperparameters were used as for Mistral 7B. Increased parameter count does lead to improved SAP scores for the zero-shot approach on Space, but the largest 70B model scores worse on AmaSum than the 7B and 13B models. For all choices of LLM, HIRO leads to summaries with a better balance between prevalence and specificity than zero-shot prompting; HIRO is likely to lead to improvements for any choice of instruction-tuned LLM.

We also compare using HIRO to identify clusters against a k-means baseline, and Hercules (Hosking et al., 2023b). For both datasets, using clusters selected with HIRO leads to summaries that are much less generic, while remaining comparatively prevalent. This confirms the results in Section 5.2, indicating that HIRO selects more informative clusters of sentences than the comparison methods.

Table 5 shows output from Mistral 7B and HIRO as well as a reference summary, for The Grand Hotel Maryland from Space. While all generated summaries are fluent and convincing, Mistral 7B makes reference to the opinion of a , which should not appear in a summary. It also describes positive praise about , but a manual analysis shows that only 3 out of 5 reviews mentioning that individual are positive; the true sentiment is much more mixed than the summary indicates. These extrapolations highlight the problem with models that can only accept a limited number of reviews as input.

The examples of selected subpaths, sentence clusters and corresponding HIRO outputs in Table 12 demonstrate the difficulty of evaluating attribution. The final cluster contains sentences that are all topically related, indicating that HIRO has learned a successful clustering. While the sentences and corresponding HIRO_sent output are all broadly negative, it is not straightforward to determine whether the sentence “Only one mirror in the room” counts as direct evidence towards the output “Rooms were small, loud, and in need of renovation [...]”. This partly explains the relatively low majority support scores in Table 3; some of the selected evidence may be consistent in topic and sentiment, but not directly entail the resulting output.

As well as collecting pairwise preferences in our human evaluation, we allowed annotators to leave qualitative comments. Table 6 shows all non-trivial comments from annotators for pairs including HIRO, with anonymized labels replaced by the true model names. The majority of comments are positive towards HIRO, highlighting improved levels of detail and a better balance of the input reviews. However, some annotators note that HIRO summaries may be less natural or too conservative.

We propose HIRO, a modular method for unsupervised opinion summarization that uses a hierarchical index over sentences to select clusters of prevalent opinions. Our approach leverages pretrained Large Language Models to generate coherent and fluent summaries that are attributable and accurately reflect the popularity of opinions in the input reviews. Extensive experiments show that, as well as generating higher quality summaries, HIRO learns a more semantically distributed representation than competitive baselines. While we limit our experiments to opinion summarization, we believe that HIRO could be usefully applied to a wide range of other retrieval-augmented generation tasks.

We thank the action editor and anonymous reviewers for their constructive feedback. This work was supported in part by the UKRI Centre for Doctoral Training in Natural Language Processing, funded by the UKRI (grant EP/S022481/1) and the University of Edinburgh. Lapata acknowledges the support of the UK Engineering and Physical Sciences Research Council (grant EP/W002876/1).

• Review:

• [...] (x8)

• Write a summary in 70 words or less:

Prompt 1: Baseline LLMs (Mistral 7B and all Llama 2 models)

• Here is a list of sentences taken from reviews of the {entity name}:

• [...]

• In no more than 10 words, write a single concise sentence that includes the main point:

Prompt 2: HIRO_sent

• Here is a list of sentences taken from reviews of the entity name:

• [...]

• In no more than 60 words, write a concise summary that includes the main points:

Prompt 3: HIRO_doc

See Table 10, Table 11, and Table 12.