LiDAR Haptic Feedback Wearable

Joe Lavelle, Computer Science & Data Analytics

Joe Lavelle - Computer Science and Data Analytics Double Major Working with Dr. Ryan Mattfeld, Associate Professor of Computer Science

The Problem

Most navigational assistive devices for people with visual impairments require use of other important sense. Some provide auditory warnings based on nearby objects - the issue with this is it can divert attention away from actively listening nearby objects and can have a slight delay between when an object is detected and when you receive the full message about it. Another example of an assistive device is a walking stick. Although it is effective, a walking stick does not give a full picture of surroundings and can miss things like low hanging objects. They also don't offer much range for object detection.

Early Prototyping

Part 1 - Reading in Data

This image shows the data read in from the LiDAR camera along with the normals calculated for each point. These normals tell us the angle of the object relative to the camera. This can help remove a lot of the noise from things like floors, ceilings, and other flat objects that can be detected by the camera but aren't really important

These are similar normal vectors as shown above, just a bit smaller so the objects in the image are a bit clearer. It may still be hard to see, but on the left wall there is a window, with two chairs along the right wall

Here is an image of the processed data read in by the LiDAR camera. At this point the data was also already divided into quadrants - eventually each quadrant was mapped to a motor on the belt - vibrating at variable intensity depending on how close the nearest point is. In this image the window on the left wall and the two chairs along the wall are much clearer (although the seats on the chair are missing due to the filtering to remove noise)

Part 2 - Integrating the Motors

Once the script was running properly and values were being output, it was time to add in the new motors. We ended up choosing Titan Haptics TacHammer Motors connected to an Arduino MKR Wifi 1010 through haptic motor drivers

Initial Testing

Initial tests with the motor involved wiring one motor to the Arduino and running it independent of the LiDAR script. This was necessary to determine what sort of motor settings to use/gauge how powerful the motors actually were

Running the motors with the LiDAR script

After testing out motor settings we were able to wire up all four motors. As you can see above, the Arduino connects to a mux which splits output to each of the the four haptic drivers, each connected to one of the motors. This setup allows the motors to trigger independent of one another based on the nearest object in their given quadrant shown at the end of Part 1.

Final Prototype

This image is a bit of a mess, but (believe it or not) this is actually a fully functioning design. The four pieces covered in tape (one on the middle left is covered by the electronics box lid) are the motors, with pieces of foam taped to either side to secure them. The electronics box houses all of the hardware in the image above (except the motors) and required soldering some longer wires so that the haptic drivers could remain safely inside the box and still reach the motors.

Materials I Used

As explained above, this setup involved wiring an Arduino MKR Wifi 1010 to a Sparkfun qwiic I2C Mux. The Mux then connected to four separate I2C Haptic drivers (DRV2605L), each connected to one of the Titan Haptics TacHammer Carlton motors. The specific camera used is a Robosense E1R LiDAR camera.

Resources that helped me

In terms of software, the most helpful resource for this project was the ROS2 Jazzy Jalisco documentation. The activities/lessons built into the documentation were extraordinarily helpful in familiarizing me with the library. This library handled most of the heavy-lifting on the software end, including things like transferring the data from the camera to the computer, and visualizing and processing the data. Another key resource was the demo-code provided with the Carlton haptic motors. This code demonstrated how to activate the motors at different intensities - exactly what we were aiming to do. The most important resource of all throughout the entire project has been collaborating with Dr. Mattfeld. Not only did he point me in the direction of the resources mentioned above, but also through our meetings we have been able to work through all sorts of obstacles that have come up throughout the project (of which there have been many) and kept me on track to make meaningful progress each week

The next step for this project is to get clearance and begin testing it on volunteers. This will help refine the design as well as the software - making sure the motors are running at the correct strength when needed and that the feedback is intuitive. The main goal of this project is to develop a functional proof-of-concept prototype, laying out the foundations in terms of software and hardware for future products to be developed and made available on a larger scale for people to actually use.