The PixelPlayer AI Can Identify and Isolate Instruments in an Orchestra

 - Jul 9, 2018
References: venturebeat & engadget
Developed by MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) the PixelPlayer AI is capable of recognizing instruments in a video, identifying specific ones and isolating the sounds they produce. Identification of sound by the PixelPlayer AI occurs at the smallest pixel level, which allows the AI to accurately identify and isolate sounds no matter how loud or lost they are in a greater orchestra. The isolation of a specific sound can be played back at any volume and will allow users to engage in full manipulation of it.

To train this instrument-identifying AI, CSAIL utilized a self-supervised deep learning technique and fed the AI over 60 hours of videos to learn from. The PixelPlayer AI can currently identify the sounds of 20 instruments and is being continually improved to increase this number. In the future, the team at CSAIL hopes to further develop the AI into an audio editing tool and use it for robots aimed at sound identification.