Video Captioning Demo

GT Caption: a black man and a white man on american idol talk to the judges.
Generated Caption (Ours): a man and a woman are talking on a talk show.
Generated Caption (Baseline[1]): a man is talking about a man.

GT Caption: a man wearing a red apron eats food out of a bowl in the kitchen.
Generated Caption (Ours): a man in a white shirt is cooking a food in a kitchen.
Generated Caption (Baseline[1]): a man in a kitchen is cooking food in a kitchen.

GT Caption: a woman is applying eye makeup with a black brush.
Generated Caption (Ours): a woman is applying makeup to her eyebrows with a brush.
Generated Caption (Baseline[1]): a woman is applying makeup on her face.

GT Caption: a boy and a girl are singing for a competition.
Generated Caption (Ours): a group of people are singing on stage.
Generated Caption (Baseline[1]): a boy is singing on stage.

Method

• Baseline[1]

  The method guides the video caption generation with Part-of-Speech (POS) information, based on a gated fusion of multiple representations of input videos .

• Ours

  This method captures video temporal information by using graph convolution network , and puts forward the multi-granularity part-of-speech attention mechanism to compute multi-granularity part-of-speech information; further, this is the first time to introduce meta learning method training video caption model, making the model to maximize reward while guaranteeing the quality of generated statements.

Dataset

Dataset	Video Number	train	validation	test
MSR-VTT	10000	6513	497	2990
MSVD	1970	1200	100	670
Charades Captions	9223	6963	500	1760

Experimental Results

• MSR-VTT

Method	BLUE4	METEOR	ROUGE-L	CIDEr
Baseline[1]	41.3	28.7	62.1	53.4
Ours	45.1	28.9	63.8	54.9

Related Paper

1. B. Wang, L. Ma, W. Zhang and et al. Controllable Video Captioning with POS Sequence Guidance Based on Gated Fusion Network[C]. In Proceedings of the IEEE International Conference on Computer Vision, 2019:2641-2650.