SBIR-STTR Award

The purpose of the proposed research is to take advantage of recent advances in object recognition algorithms and availability of nationwide wireless broadband networks to develop "See Scan"; a camera enabled mobile PDA phone service providing access to object recognition for people with vision loss. The intent is to increase functional independence by providing a mobile system with which the person can: (1) easily and quickly detect and locate a specific object, and (2) be apprised of the presence of an object within a specific set of objects. In the first case, the object to be located may be a specific box of cereal; in the second case, the object being recognized may be a denomination of money (e.g., $5 bill) within a set of monetary denominations. Such service will be accomplished through the development of a system that employs an accessible mobile camera phone to capture and transmit still images and live video to a remote computer (or server) for object recognition processing. Once an object is recognized, text to speech output will name the object and assist the person in moving toward it. The intent is to increase functional independence by providing a mobile system with which the person with vision loss can: (1) easily and quickly detect and locate a specific object, and (2) be apprised of the presence of an object within a specific set of objects. In the first case, the object to be located may be a specific box of cereal; in the second case, the object being recognized may be a denomination of money (e.g., $5 bill) within a set of monetary denominations.

Thesaurus Terms:
There Are No Thesaurus Terms On File For This Project.

The purpose of the proposed research is to continue development of the "SeeStar" system based on Phase 1 proof of concept results. Participant experiences with SeeStar under a variety of use conditions demonstrated the value of the system, and indicated where further development efforts will be required. SeeStar offers persons who are blind a "pair of eyes" via the use of a cell phone camera and data link to a remote assistant who can look at live images through that cell phone camera and then describe the person's surroundings, indicating the location of salient landmarks and pathways to specific destinations. Also, using GPS, the assistant can employ Google Maps to orient the person to their farther reaching surroundings. The experiences of Phase 1 participants, as well as remote operators, made it clear that continued SeeStar development should focus on static rather than dynamic use scenarios -- that is, not attempt to provide "blow by blow" directions as the person is walking. Further, participants made it clear that in the absence of an available remote assistant, an automated means of detecting salient visual landmarks and specific destination points would be most useful. The participants were most enthusiastic when landmark and text recognition algorithms were demonstrated in this regard. The experience of the remote operators of SeeStar also highlighted specific issues to be addressed in Phase 2: (1) difficulties in communicating caused by video and two-way voice reception latencies greater than 1 second caused by fluctuations in data channel capacity, (2) difficulty in orienting to a setting given the very restricted field of view of the camera lens, (3) low video image resolutions made it difficult to read signs and recognize objects easily, and (4) GPS coverage and errors. (5)Further, participants were split over whether they felt it was easier to hold the camera in their hands, or wear it around their neck, or perhaps even on their head. (6)Participants also wanted a means of verifying their intended routes (indoor and outdoor) and destinations, locating and recognizing pedestrian crossing buttons, bus stop signs, bus or train numbers, elevator buttons, exits, bathrooms, office numbers, and street signs. The development the automated capabilities will include a significant amount of existing of state of the art but off the shelf, image based landmark recognition, text sign detection and recognition, and automatic panorama creation software. Such landmarks and text signs when referenced to a GIS database can provide a redundant method of positioning to GPS which may suffer from coverage and availability issues. The combination of object and text recognition will also enable location and identification of critical salient features in the absence of an assistant such locating and recognizing bus numbers, street signs, elevator buttons, or pedestrian crossing buttons. Finally, more extensive and statistically relevant focus group and end user tests will be conducted to insure we have adequately addressed issues of concern for this very heterogeneous population.

Public Health Relevance:
Using a from the consumer's perspective, SeeStar must level the playing field in a manner not yet in evidence for: "In order to effectively function within society, all people must have the ability to plan comprehensible routes, follow routes, and maintain orientation along the way. People must also be able to traverse any given location without being struck by moving traffic, confounded by new construction sites, or stumbling upon a decaying sidewalk or roadway. People who have visual impairments face unique challenges in achieving independent mobility, as there is no easy way for them to identify potential hazards, or to immediately access information about their surroundings without an assistive technology device or assistance from a sighted person" [T2RERC's Stakeholder Forum, 2003].

Public Health Relevance:
This Public Health Relevance is not available.

Thesaurus Terms:
There Are No Thesaurus Terms On File For This Project.