Nicholas Colonnese

Explorations of Wrist Haptic Feedback for AR/VR Interactions with Tasbi

E. Pezent, M. K. O’Malley, A. Israr, M. Samad, S. Robinson, P. Agarwal, H. Benko, and N. Colonnese. Conference on Human Factors in Computing Systems (CHI). 2020.
[Video] [DOI] [PDF]

Leveraging Tasbi’s haptic rendering, and using standard visual and audio rendering of a head mounted display, we present several interactions that tightly integrate sensory substitutive haptics with visual and audio cues. Interactions include push/pull buttons, rotary knobs, textures, rigid body weight and inertia, and several custom bimanual manipulations such as shooting an arrow from a bow.

Hasti: Haptic and Audio Synthesis for Texture Interactions

S. Chan, C. Tymms, N. Colonnese. In IEEE World Haptics Conference (WHC). 2021.
[Video] [Presentation] [DOI] [PDF]

Best Video Presentation
Best Technical Paper Candidate

This work presents a method for real-time synthesis of vibrotactile haptic and audio stimuli for interactions with textured surfaces in 3-D virtual environments.

PneuSleeve: In-fabric Multimodal Actuation and Sensing in a Soft, Compact, and Expressive Haptic Sleeve

M. Zhu, A. H. Memar, A. Gupta, M. Samad, P. Agarwal, Y. Visell, S. J. Keller, and N. Colonnese. Conference on Human Factors in Computing Systems (CHI). 2020.
[Short Video] [Full Video] [Presentation] [DOI] [PDF]

Best Technical Paper Candidate

In this paper, we introduce PneuSleeve, a fabric-based, compact, and highly expressive forearm sleeve which can render a broad range of haptic stimuli including compression, skin stretch, and vibration.

SleeveIO: Modular and Reconfigurable Platform for Multimodal Wearable Haptic Feedback Interactions

A. Shtarbanov, M. Zhu, N. Colonnese, and A. H. Memar. In ACM Symposium on User Interface Software and Technology (UIST). 2023.
[DOI] [PDF]

SleeveIO is a modular and reconfgurable hardware platform for rapid prototyping of multimodal wearable haptic feedback interactions. It features an engineered machine-knitted sleeve and band substrates, and fve categories of haptic feedback actuator modules including vibrotactors, bellows, muscles, suction/puffing cups, and quad-chamber actuators.

Multisensory Pseudo-Haptics for Rendering Manual Interactions with Virtual Objects

E. Pezent, A. Macklin, J. M. Yau, N. Colonnese, and M. K. O’Malley. In Advanced Intelligent Systems. 2023.
[DOI] [PDF]

We present and analyze a multisensory approach to communicate proprioception and effort through illusory visual feedback and referred wristband haptic feedback. Results demonstrate that users reliably discriminate the stiffness of virtual buttons when provided with multisensory pseudohaptic feedback, comprising tactile pseudohaptic feedback (discrete vibrotactile feedback and continuous squeeze cues in a bracelet interface) and visual pseudohaptic illusions of touch interactions.

Soft Modular Shape-Changing Displays with Hyperelastic Bubble Arrays

M. R. Devlin, T. Liu, M. Zhu, N. S. Usevitch, N. Colonnese, and A. H. Memar. In International Conference on Intelligent Robots and Systems (IROS). 2023.
[DOI] [PDF]

We demonstrate a soft, shape-changing, bubble aray in a range of applications including a wearable shape changing wristband and a robotic skin on mouse for interaction.

HAPSEA: Hydraulically Amplified Soft Electromagnetic Actuator for Haptics

N. D. Kohls , N. Colonnese, Y. C. Mazumdar, and P. Agarwal. In IEEE/ASME Transactions on Mechatronics. 2023.
[DOI] [PDF]

In this work, we present a new type of soft electromagnetic actuator architecture for haptics. These low-cost, easy-to-manufacture, and conformal actuators are composed of a coil, magnet, thin-film material, and water. Adding a thin ferromagnetic sheet further enables the creation of a latching actuator variant, which can improve force output while reducing power consumption.

Fingertip Wearable High-resolution Electrohydraulic Interface for Multimodal Haptics

Purnendu, J. Hartcher-O’Brien, V. Mehta, N. Colonnese, A. Gupta, C. J. Bruns, P. Agarwal. In IEEE World Haptics Conference (WHC). 2023.
[DOI] [PDF]

We present a wearable electrohydraulic haptic interface that can produce high-fidelity multimodal haptic feedback at the fingertips. This novel hardware can generate high intensity fine tactile pressure (up to 34 kPa) as well as a wide range of vibrations (up to 700 Hz) through 16 individually controlled electrohydraulic bubble actuators.

A Hybrid Active-Passive Actuation and Control Approach for Kinesthetic Handheld Haptics

P. Dills, N. Colonnese, P. Agarwal, M. Zinn. In IEEE Haptics Symposium (HS). 2020.
[Video] [DOI] [PDF]

Best Video Presentation
Best Technical Paper Runner Up

This work presents a handheld kinesthetic (force feedback) controller employing a hybrid actuation approach and control topology to achieve crisp and accurate haptic rendering.

PneuSleeve: In-fabric Multimodal Actuation and Sensing in a Soft, Compact, and Expressive Haptic Sleeve

M. Zhu, A. H. Memar, A. Gupta, M. Samad, P. Agarwal, Y. Visell, S. J. Keller, and N. Colonnese. Conference on Human Factors in Computing Systems (CHI). 2020.
[Short Video] [Full Video] [Presentation] [DOI] [PDF]

Best Technical Paper Candidate

In this paper, we introduce PneuSleeve, a fabric-based, compact, and highly expressive forearm sleeve which can render a broad range of haptic stimuli including compression, skin stretch, and vibration.

Design, Control, and Psychophysics of Tasbi: A Force-Controlled Multimodal Haptic Bracelet

E. Pezent, P. Agarwal, J. H. O'Brien, N. Colonnese, M. K. O’Malley. Transactions on Robotics 2022.
[DOI] [PDF]

In this paper, we expand upon our previous Tasbi research. We present a detailed description of the design and experimental results demonstrating closed-loop control. Additionally, we present the results of psychophysical experiments that quantify user perception of the vibration and squeeze cues, including vibrotactile identification accuracy in the presence of varying squeeze forces, discrimination thresholds for the squeeze force, and an analysis of user preferences for squeeze actuation magnitudes.

Explorations of Wrist Haptic Feedback for AR/VR Interactions with Tasbi

E. Pezent, M. K. O’Malley, A. Israr, M. Samad, S. Robinson, P. Agarwal, H. Benko, and N. Colonnese. Conference on Human Factors in Computing Systems (CHI). 2020.
[Video] [DOI] [PDF]

Leveraging Tasbi’s haptic rendering, and using standard visual and audio rendering of a head mounted display, we present several interactions that tightly integrate sensory substitutive haptics with visual and audio cues. Interactions include push/pull buttons, rotary knobs, textures, rigid body weight and inertia, and several custom bimanual manipulations such as shooting an arrow from a bow.

Tasbi: Multisensory Squeeze and Vibrotactile Wrist Haptics for Augmented and Virtual Reality

E. Pezent, A. Israr, M. Samad, S. Robinson, P. Agarwal, H. Benko, N. Colonnese. In IEEE World Haptics Conference (WHC). 2019.
[Video] [DOI] [PDF] [Poster]

Best Technical Paper Candidate

In this work, we present Tasbi, a multisensory haptic wristband capable of delivering squeeze and vibrotactile feedback. The device features a novel mechanism for generating evenly distributed and purely normal squeeze forces around the wrist. Our approach ensures that Tasbi’s six radially spaced vibrotactors maintain position and exhibit consistent skin coupling. In addition to experimental device characterization, we present early explorations into Tasbi’s utility as a sensory substitution device for hand interactions, employing squeeze, vibration, and pseudo-haptic effects to render a highly believable virtual button.

Bellowband: A Pneumatic Wristband for Delivering Local Pressure and Vibration

E. M. Young, A. H. Memar, P. Agarwal, and N. Colonnese. In IEEE World Haptics Conference (WHC). 2019.
[Video] [DOI] [PDF]

We present Bellowband, a pneumatic wristband for localized pressure and vibration haptic feedback. The wristband has eight equally spaced pneumatic bellows that extend into the wrist, constructed from layers of polyester thermoplastic polyurethane (TPU), resulting in a flexible, lightweight (11 g) band capable of rendering complex pressure and vibration cues to the user.

Hasti: Haptic and Audio Synthesis for Texture Interactions

S. Chan, C. Tymms, N. Colonnese. In IEEE World Haptics Conference (WHC). 2021.
[Video] [Presentation] [DOI] [PDF]

Best Video Presentation
Best Technical Paper Candidate

This work presents a method for real-time synthesis of vibrotactile haptic and audio stimuli for interactions with textured surfaces in 3-D virtual environments.

HYFAR: A Textile Soft Actuator for Haptic Clothing Interfaces

J. Barreiros, T. Liu, M. Chiaramonte, K. Jost, Y. Menguc, N. Colonnese, P. Agarwal. In SIGGRAPH. 2022.
[Full Video] [Short Video] [PDF]

We introduce a HYperelastic FAbric-Reinforced (HYFAR) soft actuator that is pneumatically powered and suitable for haptic clothing.

Photo‑Responsive Liquid Crystal Network‑Based Material with Adaptive Modulus for Haptic Application

I. Kurylo, J. V. D. Tol, N. Colonnese, D. J. Broer, D. Liu In Nature Scientific Reports. 2022.
[DOI] [PDF]

This work presents the application of dynamic coatings based on photo-responsive liquid crystal network (LCN) material. This material adapts upon UV light with a power intensity of 50–90 mW/cm2 that changes the coating's elastic properties (87% decrease of the modulus for 90 mW/cm2 power intensity of 365 nm UV light).

Z-Qualities and Renderable Mass-Damping-Stiffness Spaces: Describing the Set of Renderable Dynamics of Kinesthetic Haptic Displays

N. Colonnese and S. Chan. IEEE World Haptics Conference (WHC). 2019.
[DOI] [PDF] [Poster]

In this paper we define language and definitions to define the renderable set of dynamics that a general kinesthetic haptic display can render to a human operator.

A Balanced Hybrid Active-Passive Actuation Approach for High-Performance Haptics

C. Parthiban, P. Dills, I. Fufuengsin, N. Colonnese, P. Agarwal, M. Zinn. IEEE World Haptics Conference (WHC). 2019.
[Video] [DOI] [PDF]

This paper describes the design of a high performance balanced hybrid haptic device, which addresses the asymmetry by combining a high-power, low-impedance active compliant actuation (series-elastic actuator) with energy absorbing high-force passive actuation in parallel with a fast, low-power secondary active actuation. We describe the actuation, design and control approaches, and experimentally validate the approach with a one degree-of-freedom testbed.

Stability and Quantization-Error Analysis of Haptic Rendering of Virtual Stiffness and Damping

N. Colonnese and A.M. Okamura. International Journal of Robotics Research (IJRR). 2015.
[DOI] [PDF] [Matlab Function] [Matlab Example]

Stable, quantization error noise free, rendering of high-stiffness dynamics can be challenging using impedance type haptic displays. In this paper, we construct various stability and quantization error regions as a function of system parameters, showing the necessary trade-offs that occur between them.

Rendered and Characterized Closed-Loop Accuracy of Impedance-Type Haptic Displays

N. Colonnese, A. F. Siu, C. M. Abbott, A. M. Okamura. IEEE Transactions on Haptics (ToH). 2015.
[DOI] [PDF]

In this work, using our previously introduced "effective impedance" concept, we establish bandwidth limits for rendering effective stiffness and damping. We also show that a general system impedance can be characterized by a mass, damper, and spring optimally by the solution to a convex optimization problem, and we present a quantitative metric, the Average Distortion Error (ADE), to describe the fidelity of this model.

Closed-Loop Stiffness and Damping Accuracy of Impedance-Type Haptic Displays

N. Colonnese, S. M. Sketch, A. M. Okamura. IEEE Haptics Symposium (HS) 2014.
[DOI] [PDF]

Best Technical Paper Candidate

Impedance-type kinesthetic haptic displays aim to render accurate desired dynamics to a human operator using force feedback. In this paper, we analyze accuracy using “effective impedances,” a conceptual tool that decomposes the display’s closed-loop impedance to components with physical analogs.

M-Width: Stability, Noise Characterization, and Accuracy of Rendering Virtual Mass

N. Colonnese, A.M. Okamura. International Journal of Robotics Research (IJRR). 2015.
[DOI] [PDF] [Matlab Function] [Matlab Example]

In this paper, we extend our analysis of M-width: we identify important parameters for system passivity and stability, present passivity and stability boundaries, predict noise limit cycles and establish conditions for their existence, and describe the expected accuracy of rendered virtual mass.

M-width: Stability and Accuracy of Haptic Rendering of Virtual Mass

N. Colonnese and A.M. Okamura. In Robotics: Science and Systems (RSS). 2012.
[DOI] [PDF]

In many physical human-robot interaction scenarios, such as haptic virtual environments for training and rehabilitation, it is desirable to carefully control the apparent inertia of a robot. In this work, we introduce “M-width”, which we define as the dynamic range of virtual masses renderable in a stable manner.

Analysis of Effective Impedance Transmitted to the Operator in Position-Exchange Bilateral Teleoperation

N. Colonnese, A.M. Okamura. IEEE World Haptics Conference (WHC). 2017.
[DOI] [PDF]

In this paper, we analyze the impedance transmitted to the operator in position-exchange bilateral teleoperation including the effects of master and slave dynamics, local and communication time delay, low-pass filtering of the velocity estimate, and controller stiffness and damping, for three different environment dynamics: free, clamped, and a mass-damper-spring. We show the impedance transmitted the operator numerically using effective impedances, a conceptual tool that decomposes the impedance into components with physical analogs, and also present symbolic expressions for the effective stiffness and damping transmitted to the operator at low frequencies.

Noise, but not Uncoupled Stability, Reduces Realism and Likeability of Bilateral Teleoperation

J. M. Walker, N. Colonnese, A. M. Okamura. IEEE Robotics and Automation Letters (RA-L). 2016.
[DOI] [PDF]

In bilateral teleoperation, user performance, user acceptance, and transparency are functions of the control laws that govern slave tracking and master force feedback. This study investigates the effects of teleoperator stability margin and quantization error noise on performance, likeability, and realism for a palpation task.

Propagation of Joint Space Quantization Error to Operational Space Coordinates and Their Derivatives.

N. Colonnese and A.M. Okamura. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). 2017.
[DOI] [PDF] [Longer Unpublished PDF]

In this paper, we characterize encoder error in a robotic system. Given encoder specifications, robot kinematics, and discrete transfer functions mapping coordinates to their derivatives, we describe worst case and average error.

Rigid Body Inertia Estimation in Torque Free Motion

N. Colonnese. 2011.
[PDF]

This paper concerns estimating the inertia matrix of an object undergoing torque free motion using various optimization techniques.

Mode Evolution of Cyclic Symmetric Rotors Assembled to Flexible Bearings and Housing

H. Kim, N. Colonnese, I. Y. Shen. ASME Journal of Vibration and Acoustics. 2009.
[DOI] [PDF]

This paper analyzes how the vibration modes of a cyclic symmetric rotor evolve when it is assembled to a flexible housing via multiple bearing supports

Stability and Transparency of Bilateral Teleoperators and Haptic Displays

N Colonnese. Stanford University. 2015.
[PDF]

This thesis focuses on the design and control of impedance-type bilateral teleoperators and kinesthetic haptic displays. The results aim to serve as design and control guidelines for haptic devices, teleoperators, and exoskeletons, and are particularly relevant for applications such as surgery and rehabilitation.