Let me introduce an interesting paper I have recently read regarding dialogue modeling and evaluation.
This paper suggests a new dialogue model called DialogFlow[1], which I personally consider as a compromise between two multi-turn context encoding methods, concatenation and hierarchical encoding.
Moreover, the pre-trained DialogFlow model works not only for the response generation but also for open-domain dialogue evaluation, in other words, an evaluation model called FlowScore.
Since I have been especially interested in both dialogue generation and open-domain dialogue evaluation, I thought I have several useful insights from this work and decided to review it.
Ok, let’s start.
Introduction
The DialoFlow is suggested as a more representative dialogue modeling approach to detect the change between each turn better since the current dialogue encoding methods cannot appropriately acknowledge the dynamic flows in real-world conversations.
Let’s take a look at the picture.
This shows that although the dialogues in our everyday life seem to be inconsistent and diverse, they are actually conducted focusing on specific topics(or goals) or the semantic influence which each utterance induces.
In other words, the paper contends that there are “flows” to make a certain context move to the next context and it is important to model these patterns properly.
Here, the author calls the current state of dialogue so far as the “context” and the semantic transition between each context as the “semantic influence” which can be interpreted as the result each speaker’s utterance triggers at each turn.
Then do dialogue models catch these flows well enough now?
At least the author says no.
It is argued that the current “flat” concatenation approach cannot separate each turn clearly and fails to detect the change between each context. (I will discuss this more specifically in the next section.)
Therefore, the DialoFlow can be a supplement of this limitation to generate better responses, and moreover, after being trained to identify ideal and universal dialogue flows, it can even evaluate other models’ results by comparing predicted utterances and expected semantic influences.
Previous works
Usually, there are two ways to encode the multi-turn context.
The first one is the “flat pattern” method to concatenate previous utterances into a single sequence and feed it as an input, and the second is the “hierarchical modeling” which first encodes each utterance in word-level and continuously encodes each turn in utterance-level.
The flat pattern approach is the most widely used method, especially has been frequently implemented since the various pre-trained language models based on the transformer[4] came out.
By maximizing the strength of the multi-head attention, the model encodes the input based on the attention score of each token and it is known for the more fluent and decent responses compared to those of the hierarchical encoding.
However, as mentioned before, there is also a weakness that it is hard to catch the changes between turns since it conducts the attention mechanism in token-level, not in turn-level.
To supplement this, special tokens such as a speaker token or separation token are included, but I have seen that they are not as effective as expected sometimes, especially in real-world situations.
Due to this shortcoming, some irregular cases might emerge, for example, the model generates outputs that are not consistent with its previous utterances or directly responds to the question which it made itself.
The flat pattern method was used various models like DialoGPT[5], BST[6], and PLATO[7].
The hierarchical modeling has been commonly used in the RNN[8]-based dialogue models and has recently been implemented in transformer-based models like ReCoSa[9] which I previously reviewed.
This method might cause information loss during the procedure of several compressions since it first encodes each turn separately and combines all vectors after that.
Therefore, it is known that it performs more poorly than the concatenation approach does.
The examples of hierarchical modeling can be seen in HRED[10], VHRED, and DialogBERT[11].
DialoFlow
Eventually, I think the main insight of the DialoFlow is a compromise between the flat pattern and hierarchical modeling to take advantage of the strengths of two ways.
The overall architecture of DialoFlow is as follows.
The model consists of three main parts, the transformer block, flow module, and response generator.
The transformer block is based on the uni-directional transformer structure, which is identical to the GPT[12], and the flow module models context flows hierarchically by extracting the encoded vector of each turn from the predescent stage.
Then the response generator generates actual utterances calculating semantic influences based on these contexts.
Before looking into each part specifically, let’s check the basic definitions from the paper.
Fundamentally, the principle of DialoFlow can be defined as follows.
To sum up, to make a transition from the context $C_k$, which contains the utterance from $1$ to $(k-1)$, to the next context $C_{k+1}$, the semantic influence $I_k$ should occur, which is produced by the $k$th utterance.
That is, if we have a well-extracted $I_k$, it is also available to generate a response that appropriately changes $C_k$ into $C_{k+1}$.
Thus, the ultimate purpose of DialoFlow is to predict the future context $C_{k+1}^{\prime}$ from the context so far $C_k$, get $I_k^{\prime}$ which is the difference between two contexts, and make a final response utilizing this predicted semantic influnce.
If $C_{k+1}^{\prime}$ is obtained similar to the ideal context we want, $C_{k+1}$, we can extract the semantic influence ideally as well, and this eventually leads to a response with high-quality.
Ok, now we are going to look at each part specifically.
First, the transformer block is a uni-directional transformer, which takes a concatenated dialogue context as an input.
As you can see, this is not different from the GPT which we already know.
Here, the author added speaker tokens in front of utterances to separate each turn and another special token $[C]$ at the end of each utterance.
This $[C]$ token is used later in the flow module to be extracted as a context vector for each turn, which has a similar function to the $[CLS]$ token in BERT[13] prepended to an input.
The reason why this token is added to the back of an utterance is that unlike BERT, the GPT conducts the attention from left to right, the whole information from the attention mechanism is accumulated at the very end.
In addition, the author comments that the reason why the pre-normalization used in GPT-2 instead of the post-normalization from BERT is that it is known that the latter is less stable than the former, which can lead to a decrease in performance when the model size increases.
Lastly, since this part is nothing different from the original GPT structure, we can replace this with the pre-trained GPT-2[14] or DialoGPT checkpoint, not training the model from the begging. (We will see this later.)
Next is the flow module.
This is actually an additional uni-directional attention layer, so there is nothing special.
However, unlike the previous part, this flow module conducts the attention only with the vectors extracted from the spots of $[C]$ tokens at the end of all utterances, not with all token representations.
This can be seen as a hierarchical modeling phase, which catches the flow between each turn.
The difference from the original hierarchical encoding is that it uses both encoding approaches since before modeling context flows, each utterance vector is generated from the token-level attention which takes into consideration all tokens from the start, not generated from the independently conducted token-level encoding.
Anyway, the future context $C_{k+1}^{\prime}=\text{Flow}(C_1, C_2, … , C_k)$ is predicted through this module and the semantic influence $I_k^{\prime} = C_{k+1}^{\prime} - C_k$ is calculated accordingly.
Finally, the response generator makes a response based on the predicted semantic influence $I_k^{\prime}$.
This module operates in the next word prediction fashion using a feed-forward layer and a softmax function.
To predict the next token, the vector representations of currently generated tokens, which are outputs from the transformer block and this token vector is concatenated with $I_k^{\prime}$.
Therefore, the predicted semantic influence helps hidden states from previous tokens become more representative not only of the tokens themselves but also of the change of current context flow.
Training objectives
Now, let’s talk about the specific training procedure for the DialoFlow.
There are three objectives to train this model, Context Flow Modeling (CFM), Semantic Influence Modeling (SIM), and Response Generation Modeling (RGM).
First, the Context Flow Modeling is to train the model to predict the future context properly.
This is conducted by making the L2 norm of the difference between the actual context and the predicted context vector.
Second, the Semantic Influence Modeling uses a Bag-of-Words (BoW) loss to approximate the calculated semantic influence to the actual content of the response.
It seems that the reason why the BoW loss is used here is that the purpose of this objective is not auto-regressively fitting the influence into the whole utterance, which might cause a severe over-fitting, but making it just represent the overall meaning of the response.
One thing interesting here is that there is another parameter to train in order to train the model.
So the order seems to be a little bit reversed, considering that originally the loss function itself exists to train parameters.
In my opinion, this objective might not be necessary since by appropriately modeling $L_{CFM}$, the semantic influence should also be calculated properly.
I think the author assumed that there is another layer to convert the semantic influence into the contents in the corresponding response, and tuning this layer can lead to model context flows better, even if this layer is not used in the actual inference.
Lastly, the Response Generation Modeling is the same as the original CrossEntropyLoss, which is broadly used in conditional language modeling.
The only difference is that the semantic influence is included.
And the final loss is the sum of all loss values, which is $L = L_{CFM} + L_{SIM} + L_{RGM}$.
FlowScore
As mentioned above, a well-trained DialoFlow can be also a useful dialogue evaluation model.
This is because the DialoFlow not just generates responses according to given contexts, but also similarly develops the next appropriate dialogue flows just like people usually do.
Therefore, by checking if comparing the semantic influence from a predicted response with the semantic influence which the DialoFlow expects for the next turn, it is available to assess the quality of dialogues generated from other models.
To calculate the turn-level score, the similarity between two semantic influences, one from an utterance generated by another model ($I_k$) and the other which the DialoFlow predicted ($I_k^{\prime}$).
Here, the similarity of the semantic aspect is obtained via the cosine similarity ($\cos(I_k^{\prime}, I_k)$), and the length penalty ($length(I_k^{\prime}, I_k)$) literally penalizes if the difference between two vectors’ size is high.
I think since the cosine similarity is based on the angle between two vectors, the length penalty is adopted to reflect the distance between two points.
Additionally, by combining each turn score, it is also possible to calculate the dialogue-level score.
Here, the dialogue-level perplexity is used, and this final dialogue-level score is called FlowScore, which is mentioned earlier.
This calculation is literally similar to that of the token-level perplexity we already know.
The difference is that instead of the conditional probability of each token, the normalized turn-level score is included.
Experiments
In this paper, the DialoFlow was pre-trained in the same way that the DialoGPT was trained.
As mentioned before, the transformer block is not different from the transformer decoder models, such as GPT-2 or DialoGPT, so the pre-training was conducted starting from the GPT-2 checkpoints (GPT2-base, GPT2-medium, GPT2-large), instead of training it from the initialized state.
The pre-train data is the Reddit comments which are publicly available on pushshift.io[15] and cleaned following the procedure used in DialoGPT too.
After the pre-training, the model was tested using two dialogue datasets, the Reddit test set with the multiple references.
Additionally, DialoFlow was also fine-tuned on DailyDialog[16], which is also a popular multi-turn open-domain dialogue data, for response generation and evaluated with the multi-reference version test set[17].
And then the pre-trained & fine-tuned DialoFlow were compared with the baseline model DialoGPT, the version pre-trained from the GPT-2 checkpoint.
Also, two decoding methods were implemented, the greedy decoding (Reddit) and the beam search (Reddit: beam size 10 and DailyDialog: beam size 5).
I will omit the rest of the training details since they are easily found in the paper.
Also, to evaluate the capacity of the FlowScore as the open-domain evaluation standard, the author used the collected data from the Interactive Evaluation of Dialog Track @ The Ninth Dialog System Technology Challenge (DSTC9)[18].
This contains $2,200$ human-bot conversations from $11$ bots and tagged per each dialogue on the overall quality rating from $0$ to $5$.
To test the model on the human-human dialogues, additional $200$ dialogues were sampled from the BST[19] dataset.
The baseline evaluation scores were the FED[20] and perplexity.
The FED is another dialogue evaluation model based on the DialoGPT using the likelihood.
For response generation, the BLEU[21], METEOR[22], and NIST[23] were used.
The NIST is a variant of BLEU, which penalizes uninformative n-grams to concentrate on the information gain.
The Entropy[24] was also calculated to check the responses in a perspective of lexical diversity.
For the dialogue evaluation task, the Pearson and Spearman correlation between the model results and the human evaluations was computed.
Results & Analysis
First, we are going to look into the response generation results.
On the multi-reference Reddit dataset, the DialoFlow resulted in a higher NIST and METEOR score, and in some cases, the DialoGPT outperformed in BLEU score.
Given that the NIST is more improved evaluation standards than the BLEU, with the uninformative penalty, we can see that DialoFlow generated the responses with more information and less generic.
On the multi-reference DailyDialog dataset, the DialoFlow outperformed DialoGPT more clearly.
It is stated by the author that the DailyDialog has more dialogue turns and is focused more on human conversations in real life, not in an online forum which deals with specific topics, which leads to boosting up the effectiveness of context flow modeling.
Judging by the Entropy, the lexical diversity does not seem to be incredibly different.
Also, the larger the model size, the better the performances are, which is already well-known, but it is a little bit surprising that the DialoGPT with medium size works better than the large one.
Additionally, from figure 3, the DialoFlow outperformed DialoGPT in all history lengths, even if the NIST scores from both drops when the length increases.
It is interesting that even with $1$ turn, the DialoFlow works better, which is also added in the paper that it is thanks to the semantic influence.
There is an additional interesting ablation study here that the model performances were evaluated without SIM and both SIM & CFM to check the contribution of each training objective.
As we can see from table $1$, the difference caused by the absence of SIM is marginal, but when both objectives are excluded the performances dropped slightly much more.
I think as I mentioned in the Training objectives section, the SIM loss does not have a large effect, but is just a supportive means for modeling the context flows.
Of course, the main objective would be the RGM, since it models the ultimate purpose of the model, which is the response generation.
We can see the superiority of the DialoFlow to DialoGPT in human evaluations, as follows.
Ok, next let’s check the dialogue evaluation results.
By checking the two correlation results, we can see that the FlowScore outperformed other baselines, even beating the FED in a huge gap which is based on the widely used pre-trained dialogue model, DialoGPT.
Especially, the perplexity cannot correlate well with human evaluations, since it simply sees the language modeling quality, not other factors of generated responses, such as informativeness, diversity, or human-likeness.
The details on the evaluation results are as follows.
The B1~11 represents each chat-bot which conducted dialogues in the DSTC9 Interactive Dialogue Evaluation Track, and “Human” is sampled human-human dialogues from the BST dataset.
The human and FED scores go up when the quality of dialogue is better, and the FlowScore is the opposite as it is based on the perplexity, which is lower when the quality is better.
The FED didn’t perform well on the human-human dialogues, since it rated the lowest score than other dialogues in which bots participated.
On the other hand, the FlowScore gave the highest credit to it, which means that it detected the most human-like dialogues, the human dialogues themselves.
Additionally, the gap between human-human dialogues and the best performing bot is quite similar to that from the human evaluations.
Lastly, it is also stated in the paper that the FlowScore can be seen as a kind of utterance-level perplexity.
As shown in the FlowScore section, it is calculated by getting perplexity combining normalized turn-level scores.
The original (token-level) perplexity is unstable since if a single token is a serious outlier, it might cause the explosion of value.
We can see that from Table 3, the difference between perplexity of each chat-bot is significant, which means that some words triggered unusual effects on the overall score.
On the other hand, the utterance-level perplexity is much more stable since the effect of perturbations is diminished while getting turn-level scores based on semantic influences.
Ok, this is it for the review on this paper.
First, I was searching for an effective method to evaluate an open-domain generation model, but I also got a useful insight into dialogue modeling itself.
I am going to look into the code which is publicly released by the author and will analyze it to utilize the model diversely.
Thank you for reading this post, and you’re always welcome to leave a comment.