I am a Research Science Director at Meta Reality Labs Research, working on novel interfaces for augmented and virtual reality. I lead a multidisciplinary haptic displays program, which includes researchers with expertise in materials, actuation, rendering algorithms, perception, and interaction design. My mission is to bring the sense of touch to the metaverse.
I am an active member of several academic and industrial communities, including Haptics Symposium, World Haptics, UIST, CHI, ISWC, SIGGRAPH, Smart Haptics, and others.
I hold a Ph.D. in mechanical engineering from Stanford University.
E. Pezent, M. K. O’Malley, A. Israr, M. Samad, S. Robinson,
P. Agarwal, H. Benko, and N. Colonnese. Conference on Human
Factors in Computing Systems (CHI). 2020.
[Video]
[DOI]
[PDF]
Leveraging Tasbi’s haptic rendering, and using standard visual and audio rendering of a head mounted display, we present several interactions that tightly integrate sensory substitutive haptics with visual and audio cues. Interactions include push/pull buttons, rotary knobs, textures, rigid body weight and inertia, and several custom bimanual manipulations such as shooting an arrow from a bow.
S. Chan, C. Tymms, N. Colonnese. In IEEE World Haptics Conference
(WHC). 2021.
[Video]
[Presentation]
[DOI]
[PDF]
Best Video Presentation
Best Technical Paper Candidate
This work presents a method for real-time synthesis of vibrotactile haptic and audio stimuli for interactions with textured surfaces in 3-D virtual environments.
M. Zhu, A. H. Memar, A. Gupta, M. Samad, P. Agarwal, Y. Visell,
S. J. Keller, and N. Colonnese. Conference on Human Factors
in Computing Systems (CHI). 2020.
[Short Video]
[Full Video]
[Presentation]
[DOI]
[PDF]
Best Technical Paper Candidate
In this paper, we introduce PneuSleeve, a fabric-based, compact, and highly expressive forearm sleeve which can render a broad range of haptic stimuli including compression, skin stretch, and vibration.
A. Shtarbanov, M. Zhu, N. Colonnese, and A. H. Memar.
In ACM Symposium on User Interface Software and Technology (UIST). 2023.
[DOI]
[PDF]
SleeveIO is a modular and reconfgurable hardware platform for rapid prototyping of multimodal wearable haptic feedback interactions. It features an engineered machine-knitted sleeve and band substrates, and fve categories of haptic feedback actuator modules including vibrotactors, bellows, muscles, suction/puffing cups, and quad-chamber actuators.
E. Pezent, A. Macklin, J. M. Yau, N. Colonnese, and M. K. O’Malley.
In Advanced Intelligent Systems. 2023.
[DOI]
[PDF]
We present and analyze a multisensory approach to communicate proprioception and effort through illusory visual feedback and referred wristband haptic feedback. Results demonstrate that users reliably discriminate the stiffness of virtual buttons when provided with multisensory pseudohaptic feedback, comprising tactile pseudohaptic feedback (discrete vibrotactile feedback and continuous squeeze cues in a bracelet interface) and visual pseudohaptic illusions of touch interactions.
M. R. Devlin, T. Liu, M. Zhu, N. S. Usevitch, N. Colonnese, and A. H. Memar.
In International Conference on Intelligent Robots and Systems (IROS). 2023.
[DOI]
[PDF]
We demonstrate a soft, shape-changing, bubble aray in a range of applications including a wearable shape changing wristband and a robotic skin on mouse for interaction.
N. D. Kohls , N. Colonnese, Y. C. Mazumdar, and P. Agarwal.
In IEEE/ASME Transactions on Mechatronics. 2023.
[DOI]
[PDF]
In this work, we present a new type of soft electromagnetic actuator architecture for haptics. These low-cost, easy-to-manufacture, and conformal actuators are composed of a coil, magnet, thin-film material, and water. Adding a thin ferromagnetic sheet further enables the creation of a latching actuator variant, which can improve force output while reducing power consumption.
Purnendu, J. Hartcher-O’Brien, V. Mehta, N. Colonnese, A. Gupta, C. J. Bruns, P. Agarwal.
In IEEE World Haptics Conference (WHC). 2023.
[DOI]
[PDF]
We present a wearable electrohydraulic haptic interface that can produce high-fidelity multimodal haptic feedback at the fingertips. This novel hardware can generate high intensity fine tactile pressure (up to 34 kPa) as well as a wide range of vibrations (up to 700 Hz) through 16 individually controlled electrohydraulic bubble actuators.
P. Dills, N. Colonnese, P. Agarwal, M. Zinn. In IEEE Haptics
Symposium (HS). 2020.
[Video]
[DOI]
[PDF]
Best Video Presentation
Best Technical Paper Runner Up
This work presents a handheld kinesthetic (force feedback) controller employing a hybrid actuation approach and control topology to achieve crisp and accurate haptic rendering.
M. Zhu, A. H. Memar, A. Gupta, M. Samad, P. Agarwal, Y. Visell,
S. J. Keller, and N. Colonnese. Conference on Human Factors
in Computing Systems (CHI). 2020.
[Short Video]
[Full Video]
[Presentation]
[DOI]
[PDF]
Best Technical Paper Candidate
In this paper, we introduce PneuSleeve, a fabric-based, compact, and highly expressive forearm sleeve which can render a broad range of haptic stimuli including compression, skin stretch, and vibration.
E. Pezent, P. Agarwal, J. H. O'Brien,
N. Colonnese, M. K. O’Malley.
Transactions on Robotics 2022.
[DOI]
[PDF]
In this paper, we expand upon our previous Tasbi research. We present a detailed description of the design and experimental results demonstrating closed-loop control. Additionally, we present the results of psychophysical experiments that quantify user perception of the vibration and squeeze cues, including vibrotactile identification accuracy in the presence of varying squeeze forces, discrimination thresholds for the squeeze force, and an analysis of user preferences for squeeze actuation magnitudes.
E. Pezent, M. K. O’Malley, A. Israr, M. Samad, S. Robinson,
P. Agarwal, H. Benko, and N. Colonnese. Conference on Human
Factors in Computing Systems (CHI). 2020.
[Video]
[DOI]
[PDF]
Leveraging Tasbi’s haptic rendering, and using standard visual and audio rendering of a head mounted display, we present several interactions that tightly integrate sensory substitutive haptics with visual and audio cues. Interactions include push/pull buttons, rotary knobs, textures, rigid body weight and inertia, and several custom bimanual manipulations such as shooting an arrow from a bow.
E. Pezent, A. Israr, M. Samad, S. Robinson, P. Agarwal, H. Benko,
N. Colonnese. In IEEE World Haptics Conference (WHC). 2019.
[Video]
[DOI]
[PDF]
[Poster]
Best Technical Paper Candidate
In this work, we present Tasbi, a multisensory haptic wristband capable of delivering squeeze and vibrotactile feedback. The device features a novel mechanism for generating evenly distributed and purely normal squeeze forces around the wrist. Our approach ensures that Tasbi’s six radially spaced vibrotactors maintain position and exhibit consistent skin coupling. In addition to experimental device characterization, we present early explorations into Tasbi’s utility as a sensory substitution device for hand interactions, employing squeeze, vibration, and pseudo-haptic effects to render a highly believable virtual button.
E. M. Young, A. H. Memar, P. Agarwal, and N. Colonnese. In IEEE
World Haptics Conference (WHC). 2019.
[Video]
[DOI]
[PDF]
We present Bellowband, a pneumatic wristband for localized pressure and vibration haptic feedback. The wristband has eight equally spaced pneumatic bellows that extend into the wrist, constructed from layers of polyester thermoplastic polyurethane (TPU), resulting in a flexible, lightweight (11 g) band capable of rendering complex pressure and vibration cues to the user.
S. Chan, C. Tymms, N. Colonnese. In IEEE World Haptics Conference
(WHC). 2021.
[Video]
[Presentation]
[DOI]
[PDF]
Best Video Presentation
Best Technical Paper Candidate
This work presents a method for real-time synthesis of vibrotactile haptic and audio stimuli for interactions with textured surfaces in 3-D virtual environments.
J. Barreiros, T. Liu, M. Chiaramonte, K. Jost,
Y. Menguc, N. Colonnese, P. Agarwal.
In SIGGRAPH. 2022.
[Full Video]
[Short Video]
[PDF]
We introduce a HYperelastic FAbric-Reinforced (HYFAR) soft actuator that is pneumatically powered and suitable for haptic clothing.
I. Kurylo, J. V. D. Tol, N. Colonnese, D. J. Broer, D. Liu
In Nature Scientific Reports. 2022.
[DOI]
[PDF]
This work presents the application of dynamic coatings based on photo-responsive liquid crystal network (LCN) material. This material adapts upon UV light with a power intensity of 50–90 mW/cm2 that changes the coating's elastic properties (87% decrease of the modulus for 90 mW/cm2 power intensity of 365 nm UV light).
N. Colonnese and S. Chan. IEEE World Haptics Conference (WHC).
2019.
[DOI]
[PDF]
[Poster]
In this paper we define language and definitions to define the renderable set of dynamics that a general kinesthetic haptic display can render to a human operator.
C. Parthiban, P. Dills, I. Fufuengsin, N. Colonnese, P. Agarwal,
M. Zinn. IEEE World Haptics Conference (WHC). 2019.
[Video]
[DOI]
[PDF]
This paper describes the design of a high performance balanced hybrid haptic device, which addresses the asymmetry by combining a high-power, low-impedance active compliant actuation (series-elastic actuator) with energy absorbing high-force passive actuation in parallel with a fast, low-power secondary active actuation. We describe the actuation, design and control approaches, and experimentally validate the approach with a one degree-of-freedom testbed.
N. Colonnese and A.M. Okamura. International Journal of Robotics
Research (IJRR). 2015.
[DOI]
[PDF]
[Matlab Function]
[Matlab Example]
Stable, quantization error noise free, rendering of high-stiffness dynamics can be challenging using impedance type haptic displays. In this paper, we construct various stability and quantization error regions as a function of system parameters, showing the necessary trade-offs that occur between them.
N. Colonnese, A. F. Siu, C. M. Abbott, A. M. Okamura. IEEE
Transactions on Haptics (ToH). 2015.
[DOI]
[PDF]
In this work, using our previously introduced "effective impedance" concept, we establish bandwidth limits for rendering effective stiffness and damping. We also show that a general system impedance can be characterized by a mass, damper, and spring optimally by the solution to a convex optimization problem, and we present a quantitative metric, the Average Distortion Error (ADE), to describe the fidelity of this model.
N. Colonnese, S. M. Sketch, A. M. Okamura. IEEE Haptics Symposium
(HS) 2014.
[DOI]
[PDF]
Best Technical Paper Candidate
Impedance-type kinesthetic haptic displays aim to render accurate desired dynamics to a human operator using force feedback. In this paper, we analyze accuracy using “effective impedances,” a conceptual tool that decomposes the display’s closed-loop impedance to components with physical analogs.
N. Colonnese, A.M. Okamura. International Journal of Robotics
Research (IJRR). 2015.
[DOI]
[PDF]
[Matlab Function]
[Matlab Example]
In this paper, we extend our analysis of M-width: we identify important parameters for system passivity and stability, present passivity and stability boundaries, predict noise limit cycles and establish conditions for their existence, and describe the expected accuracy of rendered virtual mass.
N. Colonnese and A.M. Okamura. In Robotics: Science and Systems
(RSS). 2012.
[DOI]
[PDF]
In many physical human-robot interaction scenarios, such as haptic virtual environments for training and rehabilitation, it is desirable to carefully control the apparent inertia of a robot. In this work, we introduce “M-width”, which we define as the dynamic range of virtual masses renderable in a stable manner.
N. Colonnese, A.M. Okamura. IEEE World Haptics Conference (WHC).
2017.
[DOI]
[PDF]
In this paper, we analyze the impedance transmitted to the operator in position-exchange bilateral teleoperation including the effects of master and slave dynamics, local and communication time delay, low-pass filtering of the velocity estimate, and controller stiffness and damping, for three different environment dynamics: free, clamped, and a mass-damper-spring. We show the impedance transmitted the operator numerically using effective impedances, a conceptual tool that decomposes the impedance into components with physical analogs, and also present symbolic expressions for the effective stiffness and damping transmitted to the operator at low frequencies.
J. M. Walker, N. Colonnese, A. M. Okamura. IEEE Robotics and
Automation Letters (RA-L). 2016.
[DOI]
[PDF]
In bilateral teleoperation, user performance, user acceptance, and transparency are functions of the control laws that govern slave tracking and master force feedback. This study investigates the effects of teleoperator stability margin and quantization error noise on performance, likeability, and realism for a palpation task.
N. Colonnese and A.M. Okamura. IEEE/RSJ International Conference
on Intelligent Robots and Systems (IROS). 2017.
[DOI]
[PDF]
[Longer Unpublished PDF]
In this paper, we characterize encoder error in a robotic system. Given encoder specifications, robot kinematics, and discrete transfer functions mapping coordinates to their derivatives, we describe worst case and average error.
N. Colonnese. 2011.
[PDF]
This paper concerns estimating the inertia matrix of an object undergoing torque free motion using various optimization techniques.
H. Kim, N. Colonnese, I. Y. Shen. ASME Journal of Vibration and
Acoustics. 2009.
[DOI]
[PDF]
This paper analyzes how the vibration modes of a cyclic symmetric rotor evolve when it is assembled to a flexible housing via multiple bearing supports
N Colonnese. Stanford University. 2015.
[PDF]
This thesis focuses on the design and control of impedance-type bilateral teleoperators and kinesthetic haptic displays. The results aim to serve as design and control guidelines for haptic devices, teleoperators, and exoskeletons, and are particularly relevant for applications such as surgery and rehabilitation.