A video consists of a sequence of images, each of which is called a frame of the video. To refer a particular frame, we use a number called video frame number. For example, frame 4 of a video is the image that immediately follows frame 3.
Photo4D does not require multiple videos to be synchronized so that low-end video cameras, such as home video cameras, can be used in 3D motion capture. This introduces an issue: the same frame number in different videos does not correspond to the same moment that the frames were recorded. Therefore, the concept of reference frame number is introduced to provide a time reference.
Photo4D uses two parameters for each video to represent the correspondence between the reference frame number and the video frame number:
videoFrameNumber = speed * referenceFrameNumber + offsetwhere "speed" represents the difference in the sampling frequency and "offset" represents the difference in the starting frame.
If the sampling rate of the video is greater than the reference frame rate, "speed" > 1.0, otherwise, "speed" <1.0. In most cases, the sampling rate difference between videos is quite small, leading to a "speed" closer to 1.0. Photo4D contains algorithms which can be used to compute the speed and offset parameters.
The (x,y,z) coordinates of a point are always computed at a reference frame number. This introduces another complication. For example, say the reference frame 8 corresponds to frame number 5.2 in video B and 7.3 in video A. Since images are only taken at the integer frame number in a video, we do not have any image to mark at frame 5.2 or 7.3 in order to computes the (x,y,z) coordinates of feature points at reference frame 8! Photo4D solves this problems by automatically interpolating mark locations from adjacent video frames. In the example, marks in frame 5 and 6 of video B are used to estimate their locations in frame 5.2 of video B, and so do marks in frame 7 and 8 of video A for frame 7.3. We still mark feature points in the images, but keep in mind that its location may not be directly used in the 3D computation.