top of page

MIDI Controlled Realtime 3D Animation

May 2021 - August 2021 (3-month), 2021 NYUSH Deans' Undergraduate Research Fund 

Individual Project

Supervised by Prof Jyun Hyun Moon


      In this project, I built up the connection between MIDI signals (received from virtual/physical MIDI keyboard) and 3D character animations utilizing the three.js and p5.js libraries. The final outcome is an interactive website and a workflow of 3D character music visualization with Blender, Maya, Three.js, and p5.js:

workflow

This project proves, technically, the feasibility of visualizing music through 3D character animations online. 

Character Design

      Originally, I wanted to narrate a story between Eggy and Cube (written by myself) in accord with the speed of my piano performance. So I designed and modeled the two characters as follows:

DURFmodel1.png
DURFmodel3.png
DURFmodel2.png
DURFmodel4.png

During the execution, I found two major problems:

 

1) The connection between the overall story and the classical piano piece is awkward;

2) In some of the segments, the intervals between chords are too short for signal processing during the live performance;

 â€‹â€‹

Thus, I redesigned a character "Brushman", who dances to a simple melody (one-note-based song) without a given narration/story. 

 

In this way, I strengthened the connection between music and its 3D visualization.

Screen Shot 2023-11-10 at 4.03.46 PM.png

©2023 by Xinyu Liu Carrot

bottom of page