So we need a way to make sure the camera poses are aligned into a global coordinate system that doesn’t change throughout the video. COLMAP is used to do thisįor the bullet time case we have as many “scenes” as there are timesteps in our exported videos ( i.e. The first step of NeRF is to compute camera poses for all images in the scene. The next step is training the NeRFs on all the data – a few issues came up here worth mentioning.įirst some background on the NeRF training process: Once the videos were all aligned to the same global time, I exported a set of ~200 synced frames from all 15 phones. To make this more robust, we brought along two metal rods that we clanked together at the start of the recording.Ĭross Correlation shows a clear peak where the two signals have maximum alignment: I ended up writing a small python script that does cross correlation on the audio streams to find the best time offset between two videos. Once we captured all the videos, we need to time sync them.Īs a developer I wanted to automate this process even though manually syncing the videos was the faster option for only doing it once. With more time I would probably test some different camera setups to see what works best. We had a pretty short time window of maybe 10 minutes to setup and capture the scene. The app did lock focus / exposure on recording, but did not sync those values across phones, so you can see some color differences in the images. Here are some of the captured images from the different phones: We built some basic tripods out of cheap lumber and asked a few family members to hold the other phones: The app also shows the battery level and video upload status for each device: Pressing record on one phone starts recording on all connected devices. Once videos were captured, they were automatically uploaded to google cloud storage. I then created a simple Firebase Realtime database app that allows me to start recording simultaneously on all phones. ( Though typically a NeRF scene would use more like 40 - 100 images. Newer iPhones can capture video at 240 frames per second at 1920x1080 resolution which is good enough to train the NeRF with.Īfter gathering up all our testing devices, and asking family members, we came up with 15 iPhones. The first requirement for the project is capturing slow motion videos of a scene from different angles. The pace of research is moving so quickly around NeRFs that in a few months there will likely be better / faster methods for creating something like this. It even has support for “DNerf” for dynamic scenes, but I have not had a chance to test it yet. Nerfstudio is another NeRF training toolkit worth checking out. Given that this was a fun / quick side project, I was not able to test out any of the newer approaches and opted for the ease of use and fast training times Instant NGP gives. Instant NGP decreased NeRF training time from days to minutes.īullet time with NeRFs has been done before, and new NeRF extensions allow representing dynamic scenes by adding a time component to existing NeRF models. In 2021, NVIDIA released Instant NGP – an open source toolkit for rapidly creating NeRFs from images or video. , or watch some cool NeRF renders on YouTube. You can read more about NeRFs here or read the original paper here The field is moving so quickly that new approaches and improvements are appearing almost daily. NeRFs can produce high-quality and photo-realistic renders of complex 3D scenes, and have the potential to revolutionize the field of computer graphics. The resulting representation is a continuous function that can be evaluated at any point in 3D space, allowing for highly realistic rendering of the scene from any viewpoint. Given a set of input images of a scene, the neural network learns to predict the 3D structure of the scene, as well as the surface reflectance and illumination at each point. The main idea behind NeRFs is to use a neural network to predict the radiance of a scene at each point in 3D space. Neural Radiance Fields are a method for representing and rendering 3D scenes using deep learning. NeRFs, aka Neural Radiance Fields, have rapidly come onto the scene as an amazing new method for view synthesis and 3d reconstruction.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |