Skip to main content
NUTRITION 2023 Main banner
Icon Legend
This session is not in your schedule.
This session is in your schedule. Click again to remove it.
Back

Session: Methodological Approaches in Nutritional Epidemiology (Oral 9)

(OR09-01-23) Surveying Nutrient Assessment With Photographs of Meals (SNAPMe): A Benchmark Dataset of Food Records Paired With Meal Photos

Sunday, July 23, 2023
2:00 PM – 2:12 PM ET
Location:  Constitution A
NTAC ELPC

    Presenting Author(s)

  • Jules A. Larke, PhD

    Postdoctoral Researcher
    USDA-ARS Western Human Nutrition Research Center
    Davis, California, United States

    Disclosure(s): No relevant financial relationship(s) with ineligible companies to disclose.

    • Co-Author(s)

  • EC

    Elizabeth L. Chin

    USDA-ARS Western Human Nutrition Research Center

  • Yasmine Y. Bouzid, BS (she/her/hers)

    Doctoral Candidate
    University of California, Davis
    Davis, California, United States

  • TN

    Tu Nguyen

    USDA-ARS Western Human Nutrition Research Center

  • YV

    Yael Vainberg

    University of California, Davis

  • JS

    Jennifer T. Smilowitz, PhD

    Faculty Affiliate
    University of California, Davis
    Moraga, California, United States

  • DL

    Danielle G. Lemay, PhD

    Research Molecular Biologist
    USDA-ARS Western Human Nutrition Research Center
    Davis, California, United States

Disclosure(s):
Jules A. Larke, PhD: No relevant financial relationship(s) with ineligible companies to disclose.
Yasmine Y. Bouzid, BS: No relevant financial relationship(s) with ineligible companies to disclose.
Objectives: The objective of the current study is to develop a benchmark data set for the evaluation of computer vision algorithms for the use of food photos in dietary intake assessment.

Methods: We conducted the Surveying Nutrient Assessment with Photographs of Meals (SNAPMe) Study (ClinicalTrials ID: NCT05008653) to pair meal photographs with traditional food records. Participants were recruited nationally and completed study visits via web-based video conferencing. Participants uploaded and annotated their meal photos using a mobile phone app called BiteSnap and completed food records using the Automated Self-Administered 24-hour Dietary Assessment Tool (ASA24®) on the same day. Participants included photos before and after eating non-packaged and multi-serving packaged meals, as well as photos of the front package label and ingredient label for single-serving packaged foods.

Results: The SNAPMe Database (DB) contains 3,311 unique food photos linked with 275 ASA24 food records from 95 participants who photographed all foods consumed and recorded matching food records for up to 3 study days. Each line in the ASA24 food record was linked to a food photo. The publicly available Facebook Inverse Cooking algorithm (F1 score = 0.23) was used to predict ingredients from the SNAPMe DB, suggesting that other algorithms developed to predict not only recipes, but also single ingredient foods can improve accuracy. To evaluate prediction of nutrient estimates, a 44-nutrient vector was constructed based on estimates common to BiteSnap and ASA24, revealing a modest relationship after adjusting for BMI, ethnicity, and education level (R2 = 0.577, p < 1.6e-15).

Conclusions: The SNAPMe benchmark data set is publicly available for photo-based dietary assessment in nutrition research. This study demonstrates its utility and areas of improvement needed in publicly available AI algorithms.

Funding Sources: This work was supported by the United States Department of Agriculture (USDA)/NSF AI Institute for Next Generation Food Systems (AIFS).