Let Computer Vision Be The Eye: "Sense+" For the Visually Impaired

Let Computer Vision Be The Eye: "Sense+" For the Visually Impaired

Derek Mingyu MA, Winona Shi WANG, Constan Shuhao CHANG

Mentored by Dr Henry C.B. Chan

Demo Video

What is Sense+

There are more than 285 million visually impaired people in the world, which means among every 20 of us, there is one person who can not see the colorful world as others do. They may encounter many difficulties every day, such as reading, looking for personal belonging, recognizing friends, let alone obtaining more information about target objects.

However, according to the survey we did with visually impaired people, they did use the smartphones every day. Then, we came up with the solution: use state-of-the-art Computer Vision and Artificial Intelligence to read the visual scene, and link with more comprehensive online information.

Users only need to raise the phone and take a picture using the phone camera, objects in the scene can be recognized intelligently and be described in natural language. It can help the visually impaired to recognize friends’ identities, distinguish cash value, search dropped belongings, read the text and so on. Especially, Sense+ can read books for the amblyopia children which will significantly decrease their difficulties in reading. Besides recognizing objects, Sense+ can also provide information beyond the image such as objects’ price, online searching result, event information and restaurant ratings based on content in the captured image. If the user takes a picture of a poster of an event, the schedule and introduction of that event can also be read out.

Sense+ also support offline real-time machine learning. The neural network model is stored on the phone so that real-time image processing results can be obtained even without internet connection. The model can also learn users’ behaviour and their inputs and iterates automatically.

For image processing and object detection, Microsoft Cognitive Services and TensorFlow deep learning framework are utilized. Microsoft Language Understanding Intelligent Service is used to understand natural language commands input by the users, online information APIs such as Yelp, Eventbrite and Bing are deployed to obtain online information. Apple iOS CoreML and TensorFlow Lite are used for offline real-time machine learning.

Sense+ is designed in two version: the free version contains basic computer vision functions. The subscription fee will be charged in the premium version, which links comprehensive online information. Our target users are visually impaired people, amblyopia children, elders with eye diseases and NGOs such as Hong Kong Blind Union. We can also get sponsorships form manufactures of assistive tools.

Sense+ is designed for the visually impaired, which tries to solve the social problem and improve their life quality. Sense+ applies newly emerging Artificial Intelligence technologies to address the urgent needs of the visually impaired and make their daily life more convenient. We are collaborating with the Hong Kong Blind Union for product validation and their members gave very positive feedback. By far, Sense+ has won several awards: Champion and Most Innovative Award of Microsoft Imagine Cup 2017 Hong Kong National Final, Champion of PolyU Smart Computing Competition and Winner of Hong Kong Techathon. Sense+ is also invited to exhibit at InnoCarnival 2017 organized by Innovation and Technology Commission at Hong Kong Science Park.

Avatar
Mingyu Derek MA
Research Intern at UC Santa Cruz, UG student at Hong Kong PolyU working on Language Technologies

Related