Overview

Originally, my teammate Donny and I will program motion detection using the difference method, and implement difference matting in the GUI interface, which should assuage the issue of changes in the background. By doing so, we can see if our method, being a lot easier than Nikolov and Kostov’s machine learning method, is just as effective or almost as effective. The result will be should be a motion tracked object or person that has its background cleanly removed by the color difference method and placed onto a new background. In addition, we will aim to cleanly remove all background motion with the keying options provided in the GUI.

After implementing the difference matting (see Work Description section in the report attached), we have decided to replace it with additional functions including sliders which changes gamma of the output image, changes the proxy image by reducing the number of number of pixels needed to be processed, frame stacking where separate frames are stacked together, and delta which controls how many frames of difference. We have also implemented multi-threading to improve the performance of our code. This report will walk through the steps that we have done, including implementations that we have decided to remove once implemented.

Quick Sample

Click here to view the actual prototype

Key Implementations

Multi-Threading to Improve Performance

Multi-threading is considered a bonus functionality for our project because it does not affect the front end of the software, but the back end. By allocating a specific thread for a specific task, the overall performance of our program will improve resulting in faster a computing speed. If this is not implemented, only one thread will be used and multiple tasks cannot be executing simultaneously.

Interactive UI

Our UI is very interactive as it allows the user to choose a file of their choice, use the sliders to make manipulations, and see the result frame by frame as each frame is being displayed as a proxy image inside the UI. This will allow the user to make adjustments while looking at their changes and export only when they are happy with what the file will look like. Considering that exporting takes some time, the proxy image viewer will allow them to only export once rather than making test exports.

Challenges

The first challenge encountered was linking the libraries in a way that it is linked not only on my computer, but also on other computers. Our project uses a .dll extension for the library, which only works for PC Windows computers, not Macs.

One challenge was learning how to extract the frames from a video and save them as images. It took a white to research existing libraries, learning the OpenCV library, and understanding how to implement this library. This library requires specific format such as instantiating the class for VideoCapture and Mat in order for the library to read the data.

Another challenge was learning how to compile the images which were subtracted in the BufferedImage array into a video and exporting that video. Similar to the previous challenge, it took a white to research existing libraries, learning the JavaCV library, and understanding how to implement this library. This library requests the instantiation of the class FFmpegFrameRecorder which can select the type of file export, bitrate and so on for the exported video.

By using the second method, one that stores the frames in the BufferedImage ArrayList rather than exported as files, an issue could be not enough random-access memory (RAM). I was filming the sample video at 4k resolution; hence, if the video is short and is captured at 720p, this should not be an issue.

In the end, we decided to not use ram and save the frames in a folder instead. This way, running out of ram will not be an issue, especially for larger videos.

Download Files

Final Report PDF