Machine Learning component of Devices

The goal of this project is transform Isabel into a statue as she performs her song Devices, which is about the feelings of being trapped as someone who suffers from PTSD.

 

The footage comes from a depthkit shoot I did with Isabel. I have the depth data aligned with RGB data.

 

The dataset:

I first scavenged for a scrapped dataset of the Metropolitan Museum for busts that I knew Golan had (because I scrapped it for his class when I was a sophomore). I then filtered that by material, taking marble, plaster, stone and leaving the rest. This got me about 400 good pictures to train on. I then ran a face detection algorithm and extracted the faces and added that into the dataset as well. I thought it looked cool so I made this video.

The model:

I trained the cycleGAN using the pytorch implementation by the original researches.

https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix

I trained it for 200 epochs.

epoch 3

 

epoch 18

epoch 62

epoch 124

epoch 150

This one I forgot to set the load size to a larger size, so it randomly sampled cropped the picture, hence why it jumps around, but this may be my favorite epoch:

I wasn’t super happy with the results, so I decided to give it a second go, this time experimenting with the parameters of the model more. I had decided to set the weights for the cycles loss function from Isabel-> statues ->Isabel to 100 which would, in comparison make the weight for the loss cycle of statues->Isabel->statues negligible in comparison and thus produce better fake generations of statues given Isabel’s data, but I was wrong.

 

For CMU’s Art and Machine Learning class Assignment 5 (https://sites.google.com/site/artml2018/) Taught by: Dr. Eunsu Kang Dr. Barnabas Poczos TA: Jonathan Dinu