Demo Video

Download Full Video

We introduce the preliminary design of a novel vision-augmented touch system called HandSight intended to support activities of daily living (ADLs) by sensing and feeding back non-tactile information about the physical world as it is touched. Though we are interested in supporting a range of ADL applications, here we focus specifically on reading printed text. We discuss our vision for HandSight, describe its current implementation and results from an initial performance analysis of finger-based text scanning. We then present a user study with four visually impaired participants (three blind) exploring how to continuously guide a user’s finger across text using three feedback conditions (haptic, audio, and both). Though preliminary, our results show that participants valued the ability to access printed material, and that, in contrast to previous findings, audio finger guidance may result in the best reading performance.

Selected Reviews from ECCV 2014

R1: The project is very ambitious because of the system complexity from both hardware and software points of view. With respect to already existing reading machines for the blind, the proposed system intends to be easier to use and more effective. The results of a system performance evaluation involving four visually impaired people are reported.
R2: The framework is very interesting and the novelties introduced for supporting ADLs sound very promising.
R3: In the proposed paper the author provide a preliminary design for a system called HandSign that allows reading print text for VI people. The analysis of the proposed system seem promising containing adequate preliminary results to evaluate the performance.
ECCV 2014 Workshop Paper

Paper

In European Conference on Computer Vision (ECCV), Workshop on Assistive Computer Vision and Robotics. 2014 | See Video

ASSET 2015 Poster

Poster

In The 17th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2015).

BibTeX

@inproceedings{
    lee2014handsight,
    title={The Design and Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind},
    author={Stearns, Lee and Du, Ruofei and Oh, Uran and Wang, Yumeng and Findlater, Leah and Chellappa, Rama and Froehlich, Jon E.},
    booktitle={European Conference on Computer Vision (ECCV) 2014 Workshops},
    pages={615--631},
    year={2014},
    url={http://dx.doi.org/10.1007/978-3-319-16199-0_43},
    doi={10.1007/978-3-319-16199-0_43},
    publisher={Springer}
}
@inproceedings{
    findlater2015assets,
    title={Supporting Everyday Activities For Persons With Visual Impairments Through Computer Vision},
    author={Findlater, Leah and Stearns, Lee and Du, Ruofei and Oh, Uran and Ross, David and Chellappa, Rama and Froehlich, Jon E.},
    booktitle={The 17th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2015)},
    pages={1--2},
    year={2015},
    url={http://dx.doi.org/10.1007/978-3-319-16199-0_43},
    doi={10.1007/978-3-319-16199-0_43},
    publisher={ACM}
}

Citation

Stearns, L., Du, R., Oh, U., Wang, Y., Chellappa, R., Findlater, L., & Froehlich, J. (2014) The Design and Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind. European Conference on Computer Vision (ECCV) 2014 Workshops. pp. 615--631. Springer. Findlater, L., Stearns, L., Du, R., Oh, U., Ross, D., Chellappa, R., & Froehlich, J. (2015) Supporting Everyday Activities For Persons With Visual Impairments Through Computer Vision. The 17th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2015)