Multimodal Semantics Extraction from User-Generated Videos

<table class="table-group" id="tab3"><tr><td><table class="table"><tr><td class="thead-hr" colspan="6"><hr/></td></tr><tr><td align="left"></td><td align="center"></td><td align="center" colspan="4">Automatic event genre classification</td></tr><tr><td align="left">Event</td><td align="center">Ground truth event genre</td><td align="center">Feature-set <svg height="15.5375" id="M61" style="vertical-align:-3.276pt" version="1.1" viewbox="0 0 14.6125 15.5375" width="14.6125" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g transform="matrix(.017,0,0,-.017,.062,11.4)"><path d="M457 488l-30 -3q-17 148 -131 148q-53 0 -84.5 -34.5t-31.5 -82.5q0 -42 25.5 -72t74.5 -62l33 -22q63 -42 95 -85t32 -102q0 -84 -67 -137t-163 -53q-58 0 -113 22t-70 43l-4 152l27 4q4 -32 15 -62.5t31 -59.5t53.5 -47t76.5 -18q56 0 92 35t36 96q0 39 -25 70t-78 68
l-31 22q-32 23 -53.5 41.5t-45 57t-23.5 77.5q0 82 58 132.5t156 50.5q46 0 101 -17l18.5 -6t17 -6t8.5 -3q-4 -55 0 -147z" id="x1D446"></path></g>
<g transform="matrix(.012,0,0,-.012,8.225,15.475)"><path d="M384 0h-275v27q67 5 81.5 18.5t14.5 68.5v385q0 38 -7.5 47.5t-40.5 10.5l-48 2v24q85 15 178 52v-521q0 -55 14.5 -68.5t82.5 -18.5v-27z" id="x31"></path></g>
</svg> (audio)</td><td align="center">Feature-set <svg height="15.5375" id="M62" style="vertical-align:-3.276pt" version="1.1" viewbox="0 0 14.6125 15.5375" width="14.6125" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g transform="matrix(.017,0,0,-.017,.062,11.4)"><path d="M457 488l-30 -3q-17 148 -131 148q-53 0 -84.5 -34.5t-31.5 -82.5q0 -42 25.5 -72t74.5 -62l33 -22q63 -42 95 -85t32 -102q0 -84 -67 -137t-163 -53q-58 0 -113 22t-70 43l-4 152l27 4q4 -32 15 -62.5t31 -59.5t53.5 -47t76.5 -18q56 0 92 35t36 96q0 39 -25 70t-78 68
l-31 22q-32 23 -53.5 41.5t-45 57t-23.5 77.5q0 82 58 132.5t156 50.5q46 0 101 -17l18.5 -6t17 -6t8.5 -3q-4 -55 0 -147z" id="x1D446"></path></g>
<g transform="matrix(.012,0,0,-.012,8.225,15.475)"><path d="M412 140l28 -9q0 -2 -35 -131h-373v23q112 112 161 170q59 70 92 127t33 115q0 63 -31 98t-86 35q-75 0 -137 -93l-22 20l57 81q55 59 135 59q69 0 118.5 -46.5t49.5 -122.5q0 -62 -29.5 -114t-102.5 -130l-141 -149h186q42 0 58.5 10.5t38.5 56.5z" id="x32"></path></g>
</svg> (sensors)</td><td align="center">Feature-set <svg height="15.6875" id="M63" style="vertical-align:-3.39066pt" version="1.1" viewbox="0 0 14.6125 15.6875" width="14.6125" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g transform="matrix(.017,0,0,-.017,.062,11.4)"><path d="M457 488l-30 -3q-17 148 -131 148q-53 0 -84.5 -34.5t-31.5 -82.5q0 -42 25.5 -72t74.5 -62l33 -22q63 -42 95 -85t32 -102q0 -84 -67 -137t-163 -53q-58 0 -113 22t-70 43l-4 152l27 4q4 -32 15 -62.5t31 -59.5t53.5 -47t76.5 -18q56 0 92 35t36 96q0 39 -25 70t-78 68
l-31 22q-32 23 -53.5 41.5t-45 57t-23.5 77.5q0 82 58 132.5t156 50.5q46 0 101 -17l18.5 -6t17 -6t8.5 -3q-4 -55 0 -147z" id="x1D446"></path></g>
<g transform="matrix(.012,0,0,-.012,8.225,15.475)"><path d="M285 378v-2q65 -13 102 -54.5t37 -97.5q0 -57 -30.5 -104.5t-74 -75t-85.5 -42t-72 -14.5q-31 0 -59.5 11t-40.5 23q-19 18 -16 36q1 16 23 33q13 10 24 0q58 -51 124 -51q55 0 88 40t33 112q0 64 -39 96.5t-88 32.5q-29 0 -64 -11l-6 29q77 25 118 57.5t41 84.5
q0 45 -26.5 69.5t-68.5 24.5q-67 0 -120 -79l-20 20l43 63q51 56 127 56h1q66 0 107 -37t41 -95q0 -42 -31 -71q-22 -23 -68 -54z" id="x33"></path></g>
</svg> (DSIFT)</td><td align="center">Feature-set<svg height="15.5375" id="M64" style="vertical-align:-3.276pt" version="1.1" viewbox="0 0 20.2875 15.5375" width="20.2875" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g transform="matrix(.017,0,0,-.017,5.725,11.4)"><path d="M457 488l-30 -3q-17 148 -131 148q-53 0 -84.5 -34.5t-31.5 -82.5q0 -42 25.5 -72t74.5 -62l33 -22q63 -42 95 -85t32 -102q0 -84 -67 -137t-163 -53q-58 0 -113 22t-70 43l-4 152l27 4q4 -32 15 -62.5t31 -59.5t53.5 -47t76.5 -18q56 0 92 35t36 96q0 39 -25 70t-78 68
l-31 22q-32 23 -53.5 41.5t-45 57t-23.5 77.5q0 82 58 132.5t156 50.5q46 0 101 -17l18.5 -6t17 -6t8.5 -3q-4 -55 0 -147z" id="x1D446"></path></g>
<g transform="matrix(.012,0,0,-.012,13.888,15.475)"><path d="M456 178h-96v-72q0 -51 12.5 -62.5t72.5 -16.5v-27h-256v27q65 5 78 17t13 62v72h-260v28q182 271 300 426h40v-407h96v-47zM280 225v295h-2q-107 -148 -196 -295h198z" id="x34"></path></g>
</svg> (global visual features)</td></tr><tr><td align="left" colspan="6"><hr/></td></tr><tr><td align="left">Football match 1</td><td align="center">Sport</td><td align="center">Live music</td><td align="center">Sport</td><td align="center">Sport</td><td align="center">Sport</td></tr><tr><td align="left">Football match 2</td><td align="center">Sport</td><td align="center">Sport</td><td align="center">Sport</td><td align="center">Sport</td><td align="center">Sport</td></tr><tr><td align="left">Football match 3</td><td align="center">Sport</td><td align="center">Live music</td><td align="center">Sport</td><td align="center">Sport</td><td align="center">Sport</td></tr><tr><td align="left">Ice-hockey match 1</td><td align="center">Sport</td><td align="center">Live music</td><td align="center">Sport</td><td align="center">Live music</td><td align="center">Sport</td></tr><tr><td align="left">Ice-hockey match 2</td><td align="center">Sport</td><td align="center">Live music</td><td align="center">Sport</td><td align="center">Live music</td><td align="center">Live music</td></tr><tr><td align="left">Concert 1</td><td align="center">Live music</td><td align="center">Live music</td><td align="center">Live music</td><td align="center">Live music</td><td align="center">Live music</td></tr><tr><td align="left">Concert 2</td><td align="center">Live music</td><td align="center">Live music</td><td align="center">Live music</td><td align="center">Live music</td><td align="center">Sport</td></tr><tr><td align="left">Concert 3</td><td align="center">Live music</td><td align="center">Sport</td><td align="center">Live music</td><td align="center">Live music</td><td align="center">Live music</td></tr><tr><td align="left">Concert 4</td><td align="center">Live music</td><td align="center">Live music</td><td align="center">Sport</td><td align="center">Live music</td><td align="center">Live music</td></tr><tr><td align="left" colspan="6"><hr/></td></tr><tr><td align="left">Total accuracy (%)</td><td align="center">—</td><td align="center">44.4</td><td align="center">88.9</td><td align="center">77.8</td><td align="center">77.8</td></tr><tr class="table-tr"><td colspan="6"><hr class="tbody-hr"/></td></tr></table></td></tr></table>

Performance comparison for the event genre classification task using different feature-sets.

Advances in Multimedia

Multimodal Semantics Extraction from User-Generated Videos

Table 3