Pin and Align Video (Pano2VR)
Sometimes it's the small details. In this tutorial, we show you how to pin and align video within the panorama using the tools you already have.
A special thanks to the PanoTwins for a great hint that made this tutorial all that much better.
What we're using
- Stitch the panorama.
- Create an image sequence from the video.
- Extract a Patch from the panorama.
- Use PTGui to match (stitch) the patch to the video frames
- Layer the video and stitched video frames in Photoshop.
- Add the new video to the Media Editor in Pano2VR.
Create the image sequence
- Load your video in Photoshop by doing either of the following: dropping it on to the workspace, or choosing Open from the File menu.
For older Photoshop versions, go to File > Import > Video frames to Layers….
- Export the video frames as an Image Sequence:
- Go to File > Export.
- Choose Photoshop Image Sequence.
- Hit Render.
Grab the patch
- Import your panorama into Pano2VR.
- Open the Patch Editor. Here, you'll be grabbing a patch that is similar as possible to the video frame. Center the patch so that the action is centered. You can use the gridlines for help.
◊ NOTE: The field of view should be less than 90º.
- Remember the Pan, Tilt and Field of View values.
The easiest way to remember is to not remember: Go to the Pano2VR Preferences/Settings, open the Advanced Tab and select Add position to file name. This will add the location information right to the outputted file name. You will need the Field of View for PTGui and then all three values for pinning the video into the Media Editor.
- Extract the patch. You do not need to update the patch so you can hit No here.
Stitch the video frame and patch
- Open PTGui (or other stitcher) and load the extracted patch (image 0).
- Enter the Lens data.
If you're using our example, the lens is 10.5mm with a 1.5 crop sensor.
- Load a video frame into the project (image 1).
- Open the Advanced settings.
- In the Lens Settings, select Lens and Shift for the Individual settings for the patch image. Nothing should be selected for the video frame.
- For the Image Parameters, change the Lens type to Rectilinear for the patch image. Also change the FoV to what was set in Pano2VR. Keep all the rest of the parameters at zero.
- Manually add control points. (The more, the better!)
- Go to the Optimizer tab and click Advanced. We will now optimize for the video frame.
- For the frame (image 1), select yaw, pitch and roll and keep the rest of the settings deselected.
- For the patch (image 0), keep everything deselected.
- Globally optimize (left-side panel) the Field of View, lens distortion a, b and c. Also select Horizontal and Vertical shift.
- Run the Optimizer. You should get a good result. If not, go back and check for bad control points and/or add more control points.
- Open the Panorama Editor and use only the FoV sliders to crop the image to show only the main action. Try to keep the FoV the same as the original patch. Do not move the image. You can go to Image Parameters and make sure that the settings are still set to zero.
- Change the projection to Rectilinear if it isn't already selected.
- Go to the Create Panorama tab and set the dimensions of the frame. Start by clicking Set optimum size... and then Maximum size. Make sure both the width and height are divisible by 16. It's okay if the height, for instance, is a few pixels off from the original aspect ratio. It's also OK to have a higher resolution than the original video.
◊ TIP: Deselect Link width and height to change the parameters individually.
- Save the project.
- Export the transformed patch image. In the Include images section of the Create Panorama tab, deselect the video frame and then click Create Panorama. This will be used as a background image when we edit our video later, in Photoshop.
- Add all the video frames to the project (Source Images tab). It's easiest to drag them in.
- In the Source Images tab, remove the patch image.
- In the Image Parameters tab, double-click the Link column to select (link) all images.
- Go to Create Panorama and select a file format.
- Choose Individual layers only for layers.
- Choose a location for the transformed frames.
◊ NOTE: If you didn't remove the reference frame when you added all the frames in step 17, then make sure, Image 0 is not selected. You would then have a extra frame in your movie and in the wrong order (if the image was taken from the middle of the sequence).
- Click Create Panorama.
Layer and color correct
- Open the edited image sequence in Photoshop (or other image editor). Go to File > Open and select a frame of the sequence. Then choose Image Sequence before hitting Open. Make sure to choose the same frame rate as the video.
◊ TIP: To see the timeline, go to Window > Workspace > Motion.
- Add the transformed patch to the image sequence as a layer above the movie (drag it in).
◊ TIP: If you're wondering where your layer went in the Timeline, scroll to the right. It has been placed at the end of the image sequence. Move it inline with the movie and make sure it is the same length.
- Resize the image to match the image sequence.
- Add a mask to the background layer and mask out the areas where the video will shine through. We only want the video (interesting parts) to show in the panorama. This also makes it easier to match the video to the panorama.
- Add audio to the timeline (optional). Make sure the audio is the same length as the video.
- Edit the image a bit more if you wish.
◊ NOTE: Images and video use different color spaces (or sample color differently), and you might notice changes in color.
- If your movie is large (over 2000 pixels), go to Image > Image Size… and change the image size to what you want the video size to be. In our example, we simply cut the size in half.
◊ TIP: You might need to deselect Constrain Proportions.
- Finally, export the video. Go to Export > Render Video. We exported our video with the preset, High Quality.
◊ NOTE: You may want to experiment with different codecs and qualities. A good place to start to get good quality is to change the data rate. The movie's resolution will depend on which data rate to use. 1080 pixels or less can use a range of 3000 - 5000 kbits/s. More than 1080 pixels, you'll want to bump it up to 5000 - 10,000 kbits/sec.
◊ TIP: Make sure that the image size matches the exported resolution. If these do not match, you may have a black border around the video when pinning it into Pano2VR.
Pin the video in Pano2VR
- Open the Pano2VR project with the panorama.
- Open the Media Editor.
- Drag the video to the preview window. Pano2VR will ask you if a preview image should be generated. You can say no here, it's not needed.
- Now, enter the Pan, Tilt and Field of View from the patch. The video should line up with the panorama.
- Generate an alternate version of the movie by clicking Generate…. Choosing WebM is a good idea to make sure the movie plays back in Firefox.
- Change Loop to 0 for continuous playback of the video.
- Click OK to close the Media Editor.
- Add an output format and create the panorama.
In our example, we used HTML5 with multiresolution and Flash fallback.
- The video will have a border in Safari due to a bug in Safari.
- Due to a bug in Firefox, the video will "slip" from the panorama and appear misaligned.