Loading...
Please wait, while we are loading the content...
Similar Documents
Multi-Modal Alignment of Visual Question Answering Based on Multi-Hop Attention Mechanism
| Content Provider | MDPI |
|---|---|
| Author | Xia, Qihao Yu, Chao Hou, Yinong Peng, Pingping Zheng, Zhengqi Chen, Wen |
| Copyright Year | 2022 |
| Description | The alignment of information between the image and the question is of great significance in the visual question answering (VQA) task. Self-attention is commonly used to generate attention weights between image and question. These attention weights can align two modalities. Through the attention weight, the model can select the relevant area of the image to align with the question. However, when using the self-attention mechanism, the attention weight between two objects is only determined by the representation of these two objects. It ignores the influence of other objects around these two objects. This contribution proposes a novel multi-hop attention alignment method that enriches surrounding information when using self-attention to align two modalities. Simultaneously, in order to utilize position information in alignment, we also propose a position embedding mechanism. The position embedding mechanism extracts the position information of each object and implements the position embedding mechanism to align the question word with the correct position in the image. According to the experiment on the VQA2.0 dataset, our model achieves validation accuracy of 65.77%, outperforming several state-of-the-art methods. The experimental result shows that our proposed methods have better performance and effectiveness. |
| Starting Page | 1778 |
| e-ISSN | 20799292 |
| DOI | 10.3390/electronics11111778 |
| Journal | Electronics |
| Issue Number | 11 |
| Volume Number | 11 |
| Language | English |
| Publisher | MDPI |
| Publisher Date | 2022-06-03 |
| Access Restriction | Open |
| Subject Keyword | Electronics Artificial Intelligence Multi-modal Alignment Multi-hop Attention Visual Question Answering Feature Fusion |
| Content Type | Text |
| Resource Type | Article |
