Automatic Sensing and Characterization of Social Interactions

Computer Sciences; Psychology; Engineering; data science

The objective of this project is to design technology to measure the presence of and quality of social interactions.

The environment we live in can be thought of both in terms of physicaland social dimensions, with the social environment including our virtual/realfriend networks, organization and community memberships, etc. 

The quality of our social environment is known to be associated withdisease mortality and risks independent of other risk factors. In fact, havingsupportive social relationships through the lifespan has demonstrated value forboth physical and psychological well-being. Social support has been shown to influence everything from the commoncold, incidence of and progression of cancer, and mental health.  However, there is still much that remains tobe understood about the relationship between our social environment, interactionswith the physical environment, and near and long-term health outcomes.  The ability to automatically recognize socialinteractions and the quality of those interactions in real-world settings wouldenable researchers to more rigorously study social interaction patterns, and todesign and optimize interventions targeted at improving social and emotionalfunctioning, with direct effects on physical and mental health and well-being.

Currently, social interactions are predominantly measured usingethnographic studies, surveys, and via proscribed behavioral tasks incontrolled laboratory settings (such as videotaping pre-planned conversations, whichtend to lack ecological validity). However, recent advances have made itpossible to monitor how behavioral systems unfold in people's natural settings,such as by leveraging sensors embedded in ubiquitous devices such as personalsmartphones and watches.

In this project, wewill leverage ubiquitous computing technology and machine learning to bothdetect the presence of social interactions as well as the quality ofinteractions in natural settings by leveraging multi-modal sensor data, such asaudio, Bluetooth, location, accelerometer, and gyroscope.  The technology will support multiple researchthreads that explore the relationship between health and an individual’s socialand physical context, including design ofjust-in-time interventions for social anxiety and facilitating socialinteractions.

Desired outcomes

The objective of this project is to design new technology and associatedmethods to automatically recognize social interactions in context, and evaluate the nature and quality ofthose interactions.  Desired outcomes ofthe project are peer-reviewed publications in both engineering and psychology,and multiple follow-on grants to NIH and NSF leveraging the new technology tounderstand human behavior in real-time in people’s daily lives.