(P26-022-23) Automatic Bite Detection From Meal Videos

Objectives: Eating behavior is a key factor for weight management and eating disorders. The current method to study eating behavior is manual annotation of meal videos, which is time-consuming, laborious, lacks objectivity, and precludes real-time feedback. To replace manual annotation, we employ computer vision methods that can automatically analyze meal videos and predict the number of bites per meal. We aim to use 468 3D facial key points and traditional computer programming to automatically count the number of bites per meal.

Methods: Media pipe was applied on 170 meal videos recorded in 15 participants while they ate 14 different meals for breakfast (fruit or smoothie), lunch (pasta/tortellini with tomato sauce, vegetable, and chicken/pork), dinner (potatoes with vegetables and pork/eggs/chicken schnitzel/meatballs), and dessert (figs, almonds, curd, fig bread, and yogurt). The manual annotation provided the ground truth. The algorithms counted a bite if the mouth ratio was higher than a set threshold. 12 different thresholds were tested for accuracy, one per each algorithm, divided into three different groups: common threshold, gaze direction threshold, and personalized threshold.

Results: The common threshold algorithm achieved the highest accuracy (73%). The algorithms based on the gaze directions threshold and personalized threshold showed lower accuracy (70% ± 0.08 and 50% ± 0.05, respectively). Over/underprediction was high (observed bites ± 75%). While the breakfast, dinner, and lunch groups showed comparable accuracy (66-68%), the dessert group was less accurate (55%) due to the soft food texture. The algorithms were more accurate for some participants (78%) than others (50%).

Conclusions: This study indicates that traditional programming can predict total bites with acceptable accuracy. However, the algorithms failed to perform for some people and especially with soft foods, thus the methods presented here are not accurate enough to replace the manual annotation of meal videos. Machine learning models that use 468 3D facial key points should be explored to increase bite prediction accuracy.

Funding Sources: PROJ-01041/Province of Gelderland, Op Oost, EFRO InToEat