Non-Visual Drawing with the HIPP Application

So what is it?

There’s a force feedback device, a pen, that gives haptic feedback. There’s several components actually, for example, a sonic feedback component as well, and also a speech feedback which is different than sonic, which is non-speech audio. Changes, deletions, those sorts of things cause sound effects to occur. There’s also screen reader output. Currently using jaws.

A Demo

So we hear speech as she draws and reviews the drawing. We also hear sound effects, sonfiication, about the fact that something got drawn.

A question comes up about whether it can normalize lines, for example, can it take a drawn line and lengthen or shorten it for the user automatically? The answer is no for now.

We hear her go over more items on the screen, and we can hear things like the fact that if you draw a wave, you can then associate wave sounds like splashing water with that part of the image.

The application is written in c++ as well as python. Various haptic and audio APIs are used.

Why did we do this?

Lots of inaccessible materials, especially in schools for children. Tactile material can be costly, and so that’s another reason that it doesn’t get used a lot in schools. Haptics, on the other hand, via technology like the pen she’s using could be used to solve this problem.

Question: what do you see as typical projects that this would be used for? Answer, we’ve seen that it’s used in different ways, but there’s more later in the talk.

Free and Open Source?

It’s an open source project, which is fantastic. I’ll include URLs later on.

We took haptic devices, laptops, etc. and put them in schools with kids that we worked with. We talked with teachers and found out how/on what they want to use it with/on, and then we saw what happened. Context is very important e.g. using it for drawing or exploring graphics, but also for other reasons. Teachers were actually thinking that they could give the students materials and have them consume it. We tried that, and it took a long time for teachers to provide the exact material and so forth, so there’s something missing there. So we then consulted pedogical experts.


Some things we learned

Students need to learn how to draw. Even doodling, making dots, etc. That’s a useful skill, and one that might not be as prevalent in blind children. So, kids draw. They make shapes and draw things, and they get positive feedback form parents and teachers, and so this encourages them. Also, though, those individuals give the children positive reenforcement and they even provide context such as “oh, that’s a wave” or “oh, that’s a dolphin”, and so this process continues and there’s feedback to move the kids from doodling to drawing actual things.

Question: did they ever ask the kids what they were trying to draw? Somewhat, but often the children start drawing something, with an intention, such as “I want to draw a dolphin” and then get some help drawing that … maybe with the body or the hsape or what not.

Interesting Query/future work: How can you draw 3D? For example, thehre’s a guy who wants to draw a mountain on the moon, and then a vally etc.

One Takeaway: One child started with “I can’t draw at all” and ended with “when it comes to drawing, I’m the best”.

Current Status:  The project has ended, but the code and all is available. Also the kids who were in the project have kept it, some of them anyways, and are continuing to use it. We are looking for folks to help us with the project, to move it forward, do new things, etc. The application is in English, but the website and all is in Swedish.

Question: how much spoken output is there? When you draw, it speaks, or erase something, it speaks.

Question thought about SVG? Yes, it’s an SVG variant, but we didn’t have a need just yet for it.

Have you looked at vibrotactile feedback Vs. haptic? That way, no pen force feedback required. Yes, we looked into this some, but nothing official.

Question: what is the major contribution that this project represents? General access to graphics, but also this is a bottom up approach to think about drawing and exploring graphics. Drawing might not be the only goal here, for example, something interactive like a graphing calculator, or other possibilities.

Haptics in pedogical practice is what HIPP stands for.

2 thoughts on “Non-Visual Drawing with the HIPP Application

  1. Pingback: The Great Big List from the 2013 CSUN International Technology & Persons with Disabilities Conference | Curb Cut

Comments are closed.