Generative AI


Volunteer Basis, Potential for UROP Funding, Potential for Work-Study Funding, Potential for Academic Credit


There has recently been much progress in generative AI, generating images, video, and 3D models from text. However, in all these cases, the output is directly the image, video, or 3D model, and the methods use either diffusion models or auto-regressive transformers. In this UROP we will synthesize rendering code to render three-dimensional scenes in all their complexity. Given an image, we will generate rendering code for a graphics engine along with the textures, polygons, and illumination required to render the scene. The graphics engine will render an image visually similar to a target image. This capability will unlock many new applications in computer vision, graphics, and virtual reality. This project will require: (i) fine-tuning a large text-to-code transformer such as OpenAI’s codex for graphics rendering code, (ii) using advanced perceptual multi-resolution metrics for the visual similarity between images, and (iii) using deep reinforcement learning as an outer loop for searching the space of rendering programs that, when used in a graphics engine, result in the desired image.

Generative AI

Posted 3 months ago on

Back to On-Campus Opportunities