How are advances in technology improving dietary research?
To uncover how diet influences health, dietitians and nutritionists need easy but accurate ways to assess what people eat. However, current methods of dietary assessment are either time-consuming to record or rely on participant memory for recalling what they eat. To overcome these challenges, dietitians Professor Deborah Kerr and Associate Professor Carol Boushey have teamed up with computer engineer Associate Professor Fengqing (Maggie) Zhu. By combining their skills and expertise, they are developing new technologies to improve dietary assessment.
GLOSSARY
Dietary intake — all the food and drink a person consumes
Dietary patterns — the quantities, proportions, variety or combination of different foods, drinks and nutrients in diets, and the frequency with which they are consumed
Fiducial marker — an object used as a point of reference for image processing
Machine learning — a type of artificial intelligence in which computers learn to complete tasks
Nutritional value — the amount of nutrients (e.g., carbohydrates, proteins, vitamins) in food or drink
How would you describe what you eat and drink, and how much? Without being a trained dietitian, evaluating dietary intake is not easy! However, analysing what people eat and drink can be important, as accurate assessment of dietary intake is essential to understanding how people’s diet may be associated with health and well-being. As people eat food, not nutrients, dietitians describe what people eat in terms of dietary patterns. Dietitians and nutritionists study what people eat and drink, both to help individuals monitor their health and to investigate the dietary patterns of populations and how these compare with patterns recommended in dietary guidelines.
How is dietary intake assessed?
Common methods to assess food and drink intake include 24-hour dietary recalls (24HR) and keeping dietary records. The 24HR method typically involves an interviewer prompting participants to recall the food and drink they consumed the previous day. A significant limitation of this method is that people often forget what they ate and drank and struggle to explain exactly how much they consumed. Dietary records require the participant to keep detailed notes about everything they consume, each time they eat or drink. A dietitian or nutritionist then enters the food and beverages into a special food composition program so that dietary intake can be assessed for food groups and nutritional value. However, this is very time-consuming for both participants and researchers.
How can technology improve dietary assessment?
Addressing the challenges of accurately assessing dietary intake while minimising the burden on participants requires an innovative, interdisciplinary approach. As researchers who specialise in dietary assessment, Professor Deborah Kerr and Associate Professor Carol Boushey realised they needed to combine their knowledge of diet and food composition with the skills and expertise of computer engineers Associate Professor Fengqing (Maggie) Zhu and Professor Edward Delp, if they were to overcome the challenges of assessing diet.
Deborah, Carol, Maggie and their team are harnessing the power of technology to improve dietary assessment methods. By combining expertise from different disciplines, the team has created a Technology Assisted Dietary Assessment (TADA) system. Participants simply take a photo of their meal on their phone using the app designed by the TADA team, and the software behind the system recognises what food or drink is in the image.
Determining the volume of food is still a challenge and requires a trained analyst to estimate the portion size and calculate the nutritional content, but the aim is for this process to also be automated. TADA dramatically reduces the burden on the participant and eliminates the need for them to remember what they have eaten and drunk. However, while this makes dietary assessment easy for the participant, developing TADA was far from simple.
How does TADA identify food?
The TADA system consists of two main components: an image-based dietary recording app called Mobile Food Record™ (mFR), and a server that processes and stores the food images. While the use of facial recognition technology is well-developed, Maggie was one of the first people to apply the principles of image recognition to food. “As no one had done this before, it was quite challenging to get started,” she says. “It was hard to know what tools we needed to build.”
Maggie used machine learning to train the TADA system to recognise different types of food. For example, by showing it hundreds of pictures of an apple, it learnt to recognise the features of an apple. However, teaching TADA to differentiate between foods with similar appearances, such as a bowl of mashed potato and a bowl of porridge, and to recognise foods that come in many forms (e.g., a potato may be mashed, boiled, roasted, baked, etc.) is challenging. “Factors such as food preparation and personal preferences can also affect the appearance of food ingredients,” explains Maggie.
The advantage of machine learning is that TADA keeps learning as it is exposed to new images, so its capabilities are still improving. When participants use the mFR, the app labels what food and drink it thinks are present in the image and the user can confirm or correct this information. Over time, the system learns the dietary patterns of each individual, enabling it to become more accurate in its food identification.
How will TADA calculate the nutritional value of a meal?
Once the system has correctly identified the food in the image, the next challenge to overcome is how to convert a 2D picture into a numerical value of a food’s nutritional content. Maggie is using image processing and machine learning techniques to estimate the 3D volume of the portion sizes in the 2D image. When using the mFR, participants include a fiducial marker (an object used as a point of reference for image processing) in the photo of their food. The TADA fiducial marker is a small piece of white card covered in coloured squares. TADA knows the exact size of each square and will use this information, along with image processing techniques, to calculate the 3D volume of the food. Automatically estimating the volume of foods from images is still a challenge, so the portion size estimation is currently still done by trained analysts.
With the portion size estimated, the nutritional value of the food can be calculated by linking to a food composition database that contains details of the food and nutrient content of different foods. TADA uses these data to calculate the amount of food groups consumed or the nutritional value of all the food and drink in the photos.
Testing TADA
The team has been developing TADA for several years and the system has undergone many tests to evaluate its effectiveness. In the recent ACE-TADA study, the team conducted a controlled feeding study. “Participants visited our food laboratory, where they selected their food choices from a menu, just like at a restaurant, and took photos of their food with the mFR,” says Deborah. As the researchers knew exactly what types and amounts of food and drink the participants consumed during the study, they could compare the real nutritional values with the values calculated by TADA.
“We also asked participants what they thought about the system, as usability and consumer acceptance of the mFR has been a critical part of the development process,” says Carol. “We have found that most people like using the mFR for dietary assessment. When we first started this research, it wasn’t common to take photos of your food, but now everyone seems to have their phone handy when they’re eating!”
“Dietary assessment provides rich data on what people are eating and can inform key decisions related to nutrition policy,” says Deborah. By developing new techniques for dietary assessment, Deborah, Carol and Maggie are contributing to improved health and nutrition outcomes for individuals and whole populations.
Professor Deborah Kerr
Curtin School of Population Health, Curtin University, Australia
Fields of research: Dietetics, Nutrition Science
Associate Professor Carol Boushey
Director of the Nutrition Support Shared Resource, University of Hawai’i Cancer Center, University of Hawai’i at Mānoa, USA
Fields of research: Dietetics, Nutritional Epidemiology
Associate Professor Fengqing (Maggie) Zhu
School of Electrical and Computer Engineering, Purdue University, USA
Field of research: Electrical and Computer Engineering
Research project: Accuracy and cost-effectiveness of technology-assisted dietary assessment (ACE-TADA)
Funders: This work is supported in part by the Australian Research Council (ARC) under the discovery grant (DP190101723). The Image-Assisted Mobile Food Record 24-Hour Recall app is funded by National Institutes of Health – National Cancer Institute (1U01CA130784-01) and National Institutes of Health – National Institute of Diabetes and Digestive and Kidney Diseases (1R01-DK073711-01A1 and 2R56DK073711-04). The term Mobile Food Record™ is a registered trademark.
Meet Deborah
Food and cooking were a big part of my childhood. My family are all excellent cooks, and I began experimenting in the kitchen from a young age. My other interests revolved around sports and being outdoors.
In high school, my friends and I entered a state science competition. I grew up in Kalgoorlie, a country town in Western Australia, and my parents had to drive us 350 km to reach the competition. We didn’t win, but it was a great experience which sparked my interest in designing science experiments.
It was my combined love of food and science that led me to qualify as a dietitian. After graduating, I worked as a clinical dietitian for several years before specialising as a sports dietitian due to my love of sport. At the time, to assess what people were eating involved them keeping detailed records of everything they ate and drank for up to four days. I soon realised how burdensome it was for elite athletes, who were very busy training for long hours, to weigh and record everything they ate – especially as they ate a lot!
These experiences shaped my interest in improving dietary assessment. I wanted to make it easier to record what people were eating. In 2005, Carol and I asked teenagers to test different dietary assessment methods, and discovered they weren’t keen on many of our suggestions. However, when we gave them cameras to take pictures of their food, they loved recording what they were eating.
I enjoy designing studies to test research questions. I especially love working with the volunteers who participate in our studies. Without them, we wouldn’t be able to conduct research, so I believe it is important for them to enjoy the experience. For example, our current project involves testing different dietary assessment methods using a feeding study. It was fun to develop a menu for the feeding study that the participants would enjoy eating.
Reference
https://doi.org/10.33424/FUTURUM403
The ACE-TADA project has been both exciting and challenging. Most research projects face unplanned challenges, so it is important to be creative and good at problem solving. Completing the feeding study successfully and safely, while overcoming the problems posed by COVID-19, has been a highlight, as has knowing that the participants enjoyed taking part in the study and helping us discover and test new ways to advance the field of dietary assessment.
Deborah’s top tips
1. Follow your passions. The combination of what you love to do plus what you are good at may lead to a career you had never considered.
2. Be open to opportunities that lead you in different directions. When I began my career as a clinical dietitian, I never expected to end up as a research dietitian.
3. If you have a curious nature and enjoy puzzling over things, consider a career in science. Although ‘scientists’ are commonly portrayed wearing lab coats and examining test tubes, there are so many different areas of research to explore. You will never be bored as there is still so much to discover.
Meet Maggie
When I was younger, I was always curious about the world outside of where I grew up, in a large city in China. I enjoyed reading – for a time I was interested in Greek mythology, then I became interested in the real-life stories of people and started reading lots of autobiographies. I was fascinated by how people think and what inspires them to do the things they do and make the decisions they make. This curiosity into how different people choose their paths in life motivates me to do the work I do.
At school, I was good at math and physics, but I wanted to study something that had a practical aspect at university. Engineering was a natural choice – I didn’t know much about engineering at the time, but I thought I might enjoy it, so I signed up.
I studied electrical and computer engineering, which is a very rapidly growing field. It is also a very interesting field as it involves both the hardware and software side of computing. Electrical and computer engineers design and build the physical components of computers, such as semiconductor devices, and develop the software and algorithms that support computer functioning.
My research focuses on signal and image processing, which require both abstract mathematical thinking and analytical skills to apply theories to engineering challenges. I enjoy the fact that I can do fundamental science that has impacts for real-world problems. For example, one project involved developing image processing software that could analyse videos of poultry farms to assess animal welfare. In another project, I worked with infant speech development researchers to create software to analyse the interactions between caregivers and babies in research study videos.
As a professor, I enjoy teaching and mentoring students. I like motivating them, but, even more importantly, I love that they motivate me. Interacting with my students exposes me to their brilliant minds and their curiosity.
Maggie’s top tips
1. Whatever discipline you chose to study, it is important to have both depth and breadth of knowledge in your subject.
2. Try new things and don’t be afraid to explore outside your comfort zone.
3. Cherish and enjoy the good things you are blessed with.
Meet Carol
My mother was a registered dietitian. She and her colleagues were passionate about their work and very effective at communicating the opportunities it presented. They sold me on the idea of being a dietitian, too!
I went to university to study dietetics and nutrition, and never looked back. After graduating, I was fortunate to start my career with the Washington State Health Department, as part of the team initiating the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC). This important national scheme still operates today, providing low-income pregnant and breastfeeding women and their children with nutritious foods and nutrition education.
I became interested in developing better methods to estimate dietary intake, so I moved to Hawai’i to work with Dr Jean Hankin, a pioneer in the field of recording dietary intake and the founder of the Nutrition Support Shared Resource (NSSR). The NSSR assists researchers as they establish what people, from birth to old age, eat and drink, and I became the director of the NSSR in 2011.
The idea for TADA was born when Deborah and I asked teenagers to take photos of their daily meals. There were no mobile devices at the time, so we gave them cameras, and the teenagers really liked the idea. I realised that image capture methods were evolving, so I looked for computer engineers working in this field who could assist us. I discovered Edward and Maggie, and the rest is history! Since then, we have worked together to successfully advance the methods and technology for dietary assessment, and I look forward to our anticipated future developments.
Getting official patents for the technology we have developed has been an exciting highlight of this project. It has also been inspiring to establish connections between different disciplines and institutions.
Carol’s top tips
1. Stay positive about your ideas.
2. Welcome feedback and embrace it, both the good and not so good.
About dietetics and technology
Advances in technology are improving many aspects of our everyday lives, and diet and health are no exception. If you are interested in how diet influences health or would like to contribute to new inventions that improve our understanding of this topic, then there are many different options for becoming involved in the interdisciplinary field of dietetics and technology, as shown by Deborah, Carol and Maggie’s team.
What disciplines contribute to advances in technology assisted dietary assessment methods?
The TADA project is a unique collaboration between research dietitians, nutrition scientists and computer engineers. “As dietitians, we have skills in dietary assessment and food composition,” says Deborah. “Through our shared experiences and expertise, we try to balance our pursuit of accurate dietary assessment with finding a method that people will want to use. We have the ideas but it’s the brilliant engineers that make it happen!” It is the engineers’ job to translate the dietary problem into an engineering problem. “As engineers, we learn from the subject experts about the challenges in the field,” says Maggie. “We have to take this information and develop engineering-based solutions.”
Creating technology to solve dietary challenges also relies on several other disciplines. “Biostatisticians help us undertake statistical analyses to test the accuracy of our methods,” says Carol. Using statistical modelling, the team can assess whether people with different characteristics (e.g., age, gender or other lifestyle factors) report different outcomes when using TADA. Health psychology plays a significant role in dietary assessment. “It is important to have a deep understanding of human behaviour, as individual factors might influence how people report their dietary intake,” says Deborah. For example, 24-hour dietary recall methods rely on participants’ memory and participants might not be able to recall as accurately if they are under time pressure or distracted with the stresses of everyday life.
The team has developed TADA to help research dietitians and nutrition scientists study dietary patterns and associated health outcomes. This means that as well as being accurate and easy to use, the system must also be cost effective. “Health economists helps us determine both the financial costs and time costs of different dietary assessment methods, to both researchers and participants,” says Carol. “By comparing the costs and outcomes of different methods, health economists can determine which methods are the best use of resources.” This is particularly important information for organisations such as the Australia Bureau of Statistics (ABS), responsible for collecting Australian population health statistics, including dietary intake data. Paul Atyeo, who works at ABS, is in the TADA team to ensure the project is policy-relevant for planning future population surveillance to assess dietary intake.
Why is consumer input also essential?
Engaging with participants is a key part of research. “As researchers, we need to put ourselves in our participants shoes and understand what we are asking them to do,” explains Deborah. “Would we be prepared to do what we are asking our participants to do?” From the start, the TADA team has actively engaged with the end-users of the technology. “Over the years, the many people who have taken part in our research studies have helped shape the TADA system to what it is today.”
Pathway from school to dietetics or technology
• Undergraduate degrees in dietetics, nutrition science or food science could all lead to a career as a research-based or clinical dietitian
• Undergraduate degrees in electrical and computer engineering or computer science could lead to a career developing the technology to address dietary challenges.
• To get excited by research, science and technology, investigate outreach programmes run by universities near you. For example, Curtin University, where Deborah works, offers outreach programmes so high school students can engage with research activities (www.curtin.edu.au/engage/schools-teachers-advisors/education-outreach) and Purdue University, where Maggie works, hosts a ‘Girls Excelling in Math and Science’ (GEMS) club (www.gems.education.purdue.edu)
Explore careers in dietetics or technology
• A career as a clinical dietitian or nutritionist will involve working with individuals to help them manage their diet for their health, while a career as a research dietitian or nutrition scientist will involve conducting diet-based research to understand how diet impacts health.
• Dietitians Australia (www.dietitiansaustralia.org.au/working-nutrition-and-dietetics-australia), the Association of UK Dietitians (www.bda.uk.com/about-dietetics/how-become-a-dietitian/career-faqs.html) and the Academy of Nutrition and Dietetics (www.eatright.org/become-an-rdn-or-ndtr) all provide information and advice about careers in dietetics and nutrition.
• A huge range of careers are available for computer engineers, as computing skills can be applied to many topics.
• Computer engineers working in signal processing can join the Signal Processing Society (www.signalprocessingsociety.org), a branch of the Institute of Electrical and Electronics Engineers (www.ieee.org).
Meet the team
Do you have a question for Deborah, Maggie or Carol?
Write it in the comments box below and Deborah, Maggie or Carol will get back to you. (Remember, researchers are very busy people, so you may have to wait a few days.)