Task Models

During an activity, the elder is likely to perform one or more tasks. Indeed, executing a series of tasks constitutes an activity. Lois maintains a family of task models (elements of L_D) allowing it to both monitor the elder's activity but also assist with the activity when needed. Sample task models are:

Lois has the ability to learn new task models by observing and being taught by the elder. Via the Internet, Lois may also download new task models when necessary. We expect Lois to maintain hundreds if not thousands of task models. Since models are dynamic data structures and are associatively linked to other models, an activity model is linked to a number of task models comprising the activity as appropriate. This association can change over time allowing an activity and the associated tasks to evolve as necessary. For example, an "eating breakfast" activity model may be linked to the "grits," "coffee," "eggs," and "toast" task models. Using the evaluate skill, activity and task models gives Lois the ability to compare actual execution with expectations allowing Lois to detect departures from normal behavior.

Episodic Memory and Diagnostic Conversation

An important ability of an elderly caregiver is to remember and learn from past experiences. Lois maintains a collection of episodic memories (K_E) allowing it to remember all past interactions with the elder. The recall and extract skills allows Lois to utilize these past experiences as needed.

Lois is able to solve newly encountered problems, like any expert, by matching and extracting general, domain-specific, and problem-solving knowledge about previous situations from episodic memory. Lois can adapt a previously successful solution to the new situation and apply the new solution. As Lois lives with the elder, it not only gets to know the elder better but also becomes a better problem solver.

We envision Lois to be a conversational chatbot meaning Lois is able to carry on a verbal (or textual) dialog in natural language for an extended period of time (essentially continuously). In fact, we see spoken language to be the most common way the elder interacts with Lois on a daily basis. Over time, with episodic memory, Lois accumulates an extensive and valuable collection of diagnostic information. As a result, Lois can detect emotional and cognitive problems early. Departures from normal behavior and the onset of new and different idiosyncrasies can alert Lois to problems such as: depression, loneliness, dementia, stroke, brain seizures, etc. To recognize signs of cognitive, emotional, and mental abnormalities, Lois maintains a collection of models (M_maladv) including:

For example, one sign of cognitive decline in the elderly is loss of memory (M_mem,_oss). By comparing current experiences and knowledge to past experiences, Lois is able to detect lapses in memory. Remembering the fact a friend named Kate visited on Monday, Lois could engage in the following conversation on Wednesday:

Lois: "Wasn't it good to see Kate the other day?"

Elder: "Who? I haven't seen Kate in a long time."

Lois: "Kate was here on Monday talking about her new grandbaby."

Elder: "Oh yes, I remember now."

Using the ability to remember and recall past experiences, Lois has detected a possible indication of short-term memory degradation. If this is the only occurrence over a period of time, it will not be anything to worry about. Therefore, Lois updates the constantly evolving M_mentai model of the elder and creates a new goal (G) to query the elder at a future date with a similar challenge. If multiple occurrences happen, Lois will alert a loved one or the elder's medical personnel.

Once an elder's memory has started to degrade, Lois can use episodic memory to remind the elder of recent events. Constantly refreshing the elder's memory can improve memory in addition to helping the elder remain independent. Lois can also cue the elder. For example, if Kate visits at some point in the future, Lois can recall Kate's last visit and remind the elder before Kate arrives when she last visited and what they discussed.

Cognitive and neuropsychological tests are often used to diagnose dementia and other maladies. These tests measure attention span, concentration, ability to learn and remember, perception, problem-solving, decision-making, verbal abilities, etc. Presently there is no definitive cure or prevention for dementia but there are measures able to help. Keeping the mind active with activities such as reading, puzzles, word games and memory training might delay the onset of dementia or reduce its effects (Mayo Clinic, 2019). Lois can weave tests and exercises like these into daily conversations in the form of conversation or games. For example, image- discrimination games such as the Frankfurt Adaptive Concentration Test (FACT) could be used to test the elder's concentration and ability to stay focused (Goldhammer et al., 2009; Mentalup, 2019).

Using episodic memory and family models, Lois can interlace questions and recollections about the elder's family members, past experiences, and current events into daily conversational dialog. Reminiscing will help the subject recall family-related facts and memorable experiences. These sort of conversations and word games will improve the elderly person's memory and maintain other cognitive skills.

Distributed Sensors

Lois is continually receiving information from a variety of sensors throughout the home. For example, monitoring sleep quantity and quality is important for the elderly. There are already many sleep monitoring products on the market today and the future promises to bring many more. For example, smart pillows sense how well you slept, breathing, heart rate, and overall quality of sleep (Lacoma, 2019). Cameras, including

360-degree cameras (Fisher, 2019), positioned throughout the house allows Lois to monitor sleep position throughout the night. All information is aggregated into the elder's M_slwpmodel.

Lois uses various types of sensors, some of them wearable. For example, one possibility is a ring worn by the elder in many ways similar to the Echo Loop currently available as an Amazon Echo accessory (Smith, 2019). The Lois smart ring monitors the elder's movements, updates the location, and takes vitals periodically (e.g., heart rate, blood pressure, blood sugar, and body temperature). All information gathered by the ring is transmitted to Lois wirelessly via Bluetooth or local WiFi shared by Lois.

The smart ring is envisioned to vibrate and display different colors as needed to communicate caution or alarm and an embedded speaker/ microphone combination allows the elder to communicate via voice interaction with Lois. Red indicates a critical alert. When vitals or activities are critical, the ring will vibrate and turn red. The red alert immediately triggers a verbal alert to the elder, a call to 911 if necessary, optionally transmit relevant information to the hospital and emergency response personnel, and sends a message to the elder's loved ones via email, text, or phone call. Yellow alert is a cautionary alert. During a yellow alert, the ring vibrates and turns yellow and the yellow alert will be enunciated verbally. The elder can choose to have Lois contact medical personnel or loved ones. Instead of a smart ring, one can envision similar functionality in other commonly worn devices such as: a necklace, bracelet, watch, or eyeglasses.

Interactive Smart Mirror Displays

We envision combination mirror/computer display devices to be primary interface points to Lois for the elder. Smart mirror technology is already available, however this technology is at the beginning stages presently. Mirrors able to display basic information such as time, date, weather, and news are available. Recently, however, this class of technology took a leap forward when the MIRROR device was released to the public (Mirror, 2019). The MIRROR is a two-way smart mirror with a rear- mounted computer display allowing computer-generated content to be shown through the mirror and combined with the reflection of whoever or whatever is in front of the device. When not activated, MIRROR functions as a standard mirror so fits anywhere in the home a mirror would (e.g., on the back of a door, a wall, bedroom, bathroom). When activated, a personal trainer appears on the display screen and leads the user through facilitated workouts as a virtual personal trainer.

As Brynn Putnam, MIRROR'S founder and CEO states, "We are building a best-in-class fitness product today, but that's not where we will be in the near future. We're building the third screen in your life that you're going to turn to for all immersive interactive experiences going forward" (Raphael, 2019). Today, people routinely use the smartphone screen and the computer/tablet screen as their interactive medium. The MIRROR device is being positioned as the next screen people use on a daily basis.

For this reason, we envision one of Lois's primary interface devices to be the MIRROR or other versions of smart mirror technology. A camera built into the MIRROR allows Lois to recognize the elder (or family member) when he or she stands in front of the MIRROR. Lois can then use the embedded computer display, along with speakers and microphones, to interact with the elder. Lois displays information on the screen so it is superimposed on the reflected image of the elder—a form of mixed and augmented reality (Milgram and Kishino, 1994). For example, Lois might display vitals, such as heart rate, blood pressure, or respiration rate near the elder's heart and lungs. In another example, if the elder complains of localized pain (e.g., shoulder), Lois might display a transparent color swatch over the elder's shoulder in the reflected image with color of the swatch indicating severity of the pain being felt. Lois drives this display with information from the M_health model it continually maintains. New information, such as the shoulder pain, would be added to Lois's models for later reference and use such as creating a goal to check on the shoulder pain daily expecting the severity to decline over the near future.

Lois can tailor the information and interaction according to who it recognizes as standing in front of the MIRROR. For example, when a family member on the FIPPA-approved list stands in front of the MIRROR, Lois might display the elder's health overview (icons shown above) and invite the family member to tap on an icon to call up detail information about the category. In the event of emergencies, say when a paramedic is called to the elder's home, Lois, having recognized the paramedic, could display medical status information needed for the situation as defined in the M_emergencv model.

When the elder is in the bathroom, the mirror is a natural interface medium, so we envision replacing existing bathroom mirrors in the elder's home with MIRROR-type devices integrated with Lois. Sensors and cameras allow Lois to observe the elder's hygiene regimen and also obtain valuable medical data. Sensors in "smart toilets" can provide valuable data about the elder's gastrointestinal health (and general health overall). Toilet manufacturers Toto and Matsushita have already released WiFi- connected toilets able to measure body mass index, biochemical makeup (sugar, protein), flow rate, and temperature of urine (Zolfagharifard, 2015). Inui Health has announced approval for a smartphone-connected home-based urine analysis able to detect bladder infections, pre-diabetes, gestational diabetes, and kidney disease (Comstock, 2018). Integration of these and other types of sensors is a natural evolution of the technology and should be expected to continue to be integrated into the smart toilet. We envision Lois being the recipient of information generated by smart toilets. Understanding the day-to-day health of the elder outside of a clinic or doctor's office is key to a holistic assessment of well-being and better care (Berry, 2018).

Augmented and Mixed Reality

Lois uses forms of augmented/mixed reality in addition to the interactive smart mirrors described above. Augmented reality and mixed reality refers to the presence of digital, computer-generated information added to the real world (Caudell and Mitzell, 1992; Milgram and Kishino, 1994). Smartphone apps already exist using the phone's camera to superimpose digital information onto the real scene being imaged by the camera (Kahney, 2018; Rasool, 2019). Lois could use smartphone or tablets in similar fashion. One can imagine an elder holding a tablet in front of an appliance and receiving guidance by Lois in augmented reality fashion on how to operate or troubleshoot the appliance.

Recently, the introduction of "smart glasses" such as Google Glass and Microsoft Hololens represents a new generation in wearable augmented reality technology. These devices are part of the fifth generation of media, also known as wearable augmented reality devices (Brem et al., 2015). These devices are also naturally worn items, such as reading glasses, rather than clunky head mounted displays of past generations. Many elderly people wear eyeglasses, so this is a natural interactive display medium for Lois not requiring the elder to have to use a smartphone or tablet intermediary.

In Spatially Augmented Reality (SAR), the user's physical environment is augmented with digital images and information integrated directly in the user's environment, not requiring a special device for viewing (Raskar et al., 1998). This involves projecting information into the real world. We imagine an elder getting out of bed in the middle of the night and Lois, having sensed the elder's motion and having determined the intended activity (M_reslroom), could slightly raise the light level in the room and project lines on the floor showing the obstacle-free path.

Gait/Body Language Analysis

When a human caregiver interacts with an elderly person for a period of time, they begin to notice subtle changes in the elder's body language, range of motion, posture, etc. Through its cameras, daily monitoring, and episodic memory, Lois is in the best position to monitor the elder's body language for clues to declining health and the elder's general wellbeing.

Gait analysis is the study of how someone moves and current technology goes further than just studying the simple motion of walking. The elder's gait signature, as unique as a fingerprint, can be learned from a frame by frame examination of the elder's body in motion. Gait analysis takes into account parameters such as: posture, length of stride, movement of hands, head tilt, distribution of weight, feet angle, pelvic rotation, head roll, shoulder position, torso flex, arm swing, knee position, etc. (Ahaskar, 2018; Birch et al., 2016).

Recently, a group of researchers from the University of North Carolina at Chapel Hill associated a person's emotions to the way they walk (Randhavane et al., 2019). Emotions such as anger, fear, stress, sadness, boredom, calm, contentment, happy, and elation can be detected by analysis of 16 joint and use positions. By watching and studying how the elder moves during activities, task execution, exercise, and physical therapy, Lois can not only detect deviations from normal movement (M_kineljcs) but also detect mood and emotional changes (M_mood).

Barbara and Allan Pease have studied body language for several decades examining each component of body language and giving the basic vocabulary to read attitudes and emotions through behavior (Pease and Pease, 2004). In 1978, Paul Ekman and Wallace V. Freisen developed the Facial Action Coding System (FACS) used today in emotion expression recognition (Ekman et al., 1980; Noroozi et al., 2018; Ekman, 2019). FACS breaks down a human's facial expressions into separate components of muscle movement, these components are called Action Units. Researchers at Carnegie Mellon University have developed computers to understand the movements of multiple individuals and their body poses, even the pose of each person's finger, in real time. Using this technology can open up new ways for humans to interact with computers (Spice, 2017).

Through body language, facial expressions, and gestures humans communicate as much, or more, information non-verbally than we do with our voice. Through cameras mounted in the elder's home and in the interactive MIRROR devices, Lois will have ample opportunity to observe and analyze the elder's body language. Being able to detect the nuances of nonverbal communication of individuals means Lois can recognize intent, behavior, and infer meaning.

Elder Care Research

Each Lois/elder pairing represents an engagement lasting a significant length of time, perhaps many years. With hundreds of millions of elders living with their own version of Lois, an enormous knowledge base relating to elder care will be amassed. De-personalized to protect privacy, this knowledge base can be mined and explored for new insights on the needs of the elderly and how to care for them. Not only will Lois dramatically improve the lives of the elderly, the aggregation of Lois knowledge promises to revolutionize elder care.