Merciful 3/Merciful_Release_3.bin/software/f/flyer/flyerhdtools.lha/FHDT/FHDTeknotes.txt

Text File | 1996-01-11 | 22KB | 438 lines

HOW TO CHOOSE HARD DRIVES FOR THE FLYER:

Nonlinear does not mean offline:
NewTek's Video Flyer is a nonlinear editor (NLE), a computer-based
video editing system which records video and audio to hard drives.
Unlike offline editing systems that work with reduced-quality video,
the Flyer offers broadcast-quality recording and output. You can
produce a finished product directly with the Flyer without going
through the additional step of an on-line tape edit. That means the
Flyer is an online nonlinear editing system.

Most online nonlinear editing systems in today's market limit the
user's freedom to choose hard drives. Such systems, in order to support
broadcast quality, may require specific drives in proprietary RAIDs
(Redundant Arrays of Independent Disks) or customized hard drives that
cost a great deal more.

Not so, the Flyer! Throughout the development process NewTek has tested
the Flyer with a wide variety of hard drives. These drives are the same
off-the-shelf items that our customers can buy locally, however they
must have specific firmware revisions and cache settings to work
properly. We'll discuss these needs below.


Greater freedom of choice:
This approach frees the user to select the drives that best fit his or
her needs and build the most cost-effective Flyer system. Flyer owners
can weigh their needs and resources in selecting a drive. You can take
into account the storage capacity you require, the performance
characteristics of available drives, the type of production work you
plan for the Flyer, and current drive costs. Some users may need
high-performance drives which support demanding program sequences and
high-detail video imagery for broadcast or entertainment. Others can
use less expensive drives for more modest production needs.

There are a number of video-capable drives available and drive
technology is advancing very rapidly. The drive market sees a constant
flow of exciting new products heralding dramatic increases in
performance and capacity.


FlyerHDTools:
To help Flyer owners select drives, NewTek provides a utility program
called FlyerHDTools. FlyerHDTools is included with the Flyer system
software, and is also available via the NewTek Technical Support BBS
(913- 271-9299) or the InterNet at NewTek's anonymous FTP site
(ftp.newtek.com). This utility indicates how well a drive will perform
with the Flyer. Combining FlyerHDTools with "real-world" evaluation
allows our users and dealers to test various drives and assess them for
Flyer use.

The NewTek BBS also is a resource for timely hard drive information.
Users and dealers upload their FlyerHDTools test result files and
evaluations of various drives to the "Drive Data" area of the Flyer
Zone (in our Uploads and Downloads area). The results of FlyerHDTools
tests are available to other users and dealers who may need to find
drives suitable for their particular Flyer applications. Once someone
has realized the proper configuration settings for a particular drive
to perform well with the Flyer (see below), that information can be
shared with others.

NewTek recommends that most Flyer end-users seek the assistance of a
trusted dealer who has experience with Flyer drive selection and
knowledge of the Flyer-related drive performance issues.


What does FlyerHDTools do?
FlyerHDTools evaluates a number of Flyer-related hard drive performance
criteria. This data can be used to make an informed decision about the
drives you want to utilize with the Flyer.

These criteria include the minimum data transfer rate, the drive's
cache parameters, thermal calibration, and error mapping.

1) Minimum data transfer rate
Not all portions of a drive can transfer data at the same rate. The
inner tracks of a drive cannot deliver information at the same speed
that the outer tracks can. Drive manufacturers typically list the "best
case" transfer speed that the drive can achieve, so manufacturer's
specs are not always reliable source of transfer speed data.

FlyerHDTools defines the minimum data transfer rate as the slowest rate
at which the drive can read or write data. For Flyer operation it is
important to know this "worst case scenario" because most users will
probably use a drive to its maximum capability. This means that
real-time video data will likely be recorded on both faster and slower
parts of the drive. The slowest part of the drive is considered its
baseline performance ability.

For example, a typical drive we tested delivered a healthy 5.6
megabytes per second (MB/sec) at its outer tracks. On the inner tracks,
however, the drive slowed to only 3.9 MB/sec. FlyerHDTools reported the
minimum transfer speed for that drive as 3.9 MB/sec.

Knowing this, you can determine whether this drive will suit your
needs. When the Flyer records in Standard Mode, the drive must have a
minimum transfer rate of 3.7 MB/sec. In Flyer HQ5 Mode (High Quality 5
Megabyte), the drive needs a minimum data transfer rate of 4.8 MB/sec.
This example drive would suffice for Standard Mode video, but would not
be acceptable for HQ5 video. If you required the best video quality in
your work, you would want to pass this drive up in favor of one that
met the Flyer's HQ5 requirements.

2) Drive Cache Parameters
Hard drives have onboard RAM used as temporary data storage areas
(caches). Data being written to, or read from, the drive is held here.
How the drive uses these caches can be changed for different
applications. To do so you alter aspects of the drive's "Mode
Sense/Mode Select Pages," or just Mode Pages, which retain this data
even after power-down.

The cache settings for a drive affect the minimum data transfer rate.
Most drive caches are optimized for DOS type programs. The caches
retain frequently-accessed information so that data can be retrieved
quickly if needed again. This avoids having the drive repeatedly look
up the same information again and again, which takes longer than simply
retrieving the data from a memory cache.

For video, however, this is not good. Video data transfers best from
the drive when it passes through the caches as quickly as possible.
Factory cache settings, therefore, may not be best for video uses.

If FlyerHDTools indicates that a drive performs below the performance
level you desire (and believe that it can produce), you can employ
PC-based drive utility programs to experiment with different cache
settings until the drive operates at higher minimum transfer rates.
Once you find these settings, we ask that you post the FlyerHDTools log
containing them to the NewTek BBS for others who have the same make,
model, and revision hard drive.  If you are not thoroughly familiar
with and experienced in this technical aspect of drive configuration,
you would be best to leave such experimentations to knowledgeable
technicians.

3) Thermal Calibration
As the temperature inside a hard drive changes, its platters (which
actually store the data) will slightly shrink or expand.
Temperature-induced changes in a platter's size can shift the position
of the data tracksÑconcentric rings where information is storedÑenough
to cause the drive to exhibit head-position errors. For example, an
attempt to locate track 20 might result in the head being positioned at
track 23 instead.

To compensate for these changes some hard drives perform a periodic
(e.g. every 10 minutes) "thermal calibration," or T-CAL. T-CAL
recalibrates the head positioning system to true track locations given
the current thermal state of the platters. When it occurs, T-CAL
interrupts all other drive activity. Read and write operations stop
while the heads recalibrate. This interval is displayed by FlyerHDTools
as idle time.

Naturally, if you're reading or writing video data, the interruption is
a potential problem! The Flyer has its own memory buffer to partly
overcome short "idle time" interruptions in drive data. This memory is
pre-loaded before you see a clip play so that when a T-CAL occurs, data
can continue to flow to the Flyer's VTASC compression/decompression
chips. If the T-CAL delay lasts longer than 10 frames (about 1/3
second, or 333 milliseconds), the Flyer runs out of video data and
playback may appear to stutter while the buffer redisplays the last
frame it received from the drive.

In practice, other factors can reduce this built-in safety zone to less
than 333 milliseconds (ms). There is a statistical gray area between
200ms and 333ms. When idle time or T-CAL delays are less than 200ms,
chances of stuttering are minimal.

Some hard drives, called "embedded-servo drives," have special servo
tracks on the drive surface. They can always position the heads
properly with no need for T-CAL. These drives can have an idle time of
essentially zero milliseconds and reduce the possiblity of stutter
caused by drive delays to nil.

4) Error map settings
A rare but possible drive consideration is platter imperfection. When a
drive is made there may be microscopic pits or bumps on some of its
surfaces. The manufacturer may opt to replace a badly pitted platter
or, if there are relatively few problems, may instead program the drive
to to "remap" a bad sector to another location on the platter surface.
The record of remapped locations is called an error map.

Although error maps allow a drive to function perfectly well for data
storage, they can introduce cache delays while the heads jump to the
remap location and back in order to supply a steady stream of video
data. FlyerHDTools will show such delays as part of the drive's idle
time. Depending on the severity of the delay, it may or may not cause a
video stutter.

For this reason it's a good idea to run FlyerHDTools on the entire
drive to uncover potential problems caused by imperfect platter
surfaces and sector remap delays.  As with Mode Page settings, only
skilled technical personnel should attempt to change error map
settings.


Other Factors that Affect Drive Performance:
Other factors can affect the performance of the Flyer. These are listed
below.

1) Complex Video
Video may be complex due to high detail levels, high noise levelsÑor
both. A user working with complex video should use a drive which can
support Flyer HQ5 mode. A user making extremely fast-cut Flyer
sequences using complex video will want a drive with excellent seek
time and data throughput performance in addition to HQ5 capability.

Seek time is not considered a primary performance factor because modern
high speed hard drives generally have more than acceptable seek times.
Seek time could be a problem only in exceptional circumstances, such as
a combination of very complex video material being used in very fast
cuts of short duration (four-field cuts).

Note: You may wonder why seek time (the time a drive takes to shift its
heads from one track to another) is not mentioned as a primary
performance factor. Virtually all modern drives with video capabilities
have acceptable seek times. Seek time could be a problem only in
exceptional circumstances, such as very complex video material
sequenced with a large number of fast cuts of short duration (i.e.,
four-field cuts).

2) SCSI Cable Quality
Always use high quality SCSI cables that are as short as possible. The
best quality SCSI cables are double-shielded. No single SCSI cable
should be over 6 feet long, and keep the total length of any SCSI chain
under 15 feet.

Also, it's wise to check the integrity of the connector itself. The
connector (on either the cable and the drive enclosure) can become
intermittent over time, or from constant cable swapping. Wiggling the
connection can make a bad connection begin working again. If you find
this to be the case, you may wish to try another cable (or have the
connectors checked by a dealer or service center).

3) SCSI Termination
Always terminate the last drive in a SCSI chain, and only the last
drive.

4) SCSI ID Numbers
Every device connected to a SCSI chain must have a unique ID number
(between 0 and 6). If two devices share the same number, one of them
will not be seen by the host computer (or errors may be reported, or
the system may not even power on). Make sure your drives are numbered
differently.

5) Termination Power Source
Your drives may have termination power jumpers (these select either the
SCSI host controller or the drive as the source for termination power).
Set the jumpers so that the drive is the source.

6) Grounding and Ground Loops
Ground loops can introduce noise into SCSI cabling. This noise can
prevent commands or data from passing back and forth between the Flyer
and the drives, which can induce stuttering (the Flyer makes several
attempts to retrieve data and is forced to redisplay the last received
data until the new data arrives).

Ground loops can be caused by the arrangement of data and power cables
on external drive cases. Try to keep data and power cables short, and
try to plug all external drives into the same set of AC outlets
powering the computer system.


Which drives support HQ5 mode?
The significant factor is a minimum of 4.8 megabytes per second data
transfer rate over the entire surface of the drive for both reading and
writing. This assumes that idle time, T-CAL delays, etc., are within an
acceptable range for Flyer use. But what can do you for a drive that
can sustain the high data rate that the HQ5 mode requires for only a
portion of its total storage capacity?

The Flyer always records clips from the outermost tracks (progressing
toward the center). The idea that you could use a fully formatted drive
with a fast transfer rate on its outer tracks but a slow transfer rate
on its inner tracks is a good one, but partially flawed. In preparation
for playback, the Flyer must often "cross-copy" portions of clips to
different drives on the system. One or more of these A/V temp files
might be written to a slower portion of a drive, causing problems on
playback (because that portion of the drive does not sustain the
minimum transfer rate). Stutter would likely result.

The way to make such a drive reliable for A/B roll playback is to
short-stroke the drive. This means that you format only the portion of
the drive that can handle the data rate you want.

FlyerHDTools can not only tell you what percentage of the drive can
sustain a given data rate, but also it can format that amount of the
drive for you. Short-stroking a drive is a valid means of assuring that
its entire capacity is available for high quality recording and
playback.

Note: That portion of the drive that cannot be used at high speed will
be unavailable if you choose to short-stroke a drive. Only the
short-stroked portion can be used by the Flyer.


RAID systems:
Although there are Flyer users who report success with RAID devices,
NewTek neither requires nor recommends them. An advantage of the Flyer
as compared to other broadcast-quality NLE systems is that the Flyer
has been designed for use with off-the-shelf drives. Our perspective is
that hard drive technology is rapidly advancing to meet the needs of
users working in desktop video.


What are audio drive requirements for Flyer?
Audio requirements are different from video requirements. SCSI-2 is
preferred (although most recently-made SCSI drives should suffice). We
recommend a drive with at least 1 megabyte per second data transfer
speed,  20 millisecond (or faster) seek time, and 100 milliseconds or
less  of idle time, for T-CAL, etc. This figure represents the slowest
SCSI drive performance that will yield acceptable results when using
multiple audio tracks.  Note that FlyerHDTools reports on data transfer
rates and idle time, but not seek time.  The manufacturer's data label
on the drive usually provides this information.

Removable media drives, if they meet these requirements, should also
work.

The termination power jumper (if available) should be set to provide
termination power from the drive itself.


What tape backup drives will work?
For a tape drive to work with the Flyer, it must meet the following
requirements:

a) Tape units MUST be SCSI-2. Many current tape drives have SCSI-1 or
SCSI-2 selectable via a jumper or switch. Check your tape drive
documentation and set the drive to SCSI-2.

b) The tape drive must use 512-byte data blocks or must be able to be
set for that block size.

c) The tape drive MUST NOT compress the data. Most tape drives which
are capable of compressing data can be configured to do so via a jumper
or switch; others may require software selection of this feature. Set
the drive so it does NOT compress data. Some drives (e.g., Exabyte) use
a custom "command tape" which must be inserted and read by the drive to
turn this feature on or off. Command tapes should be used to turn the
data compression feature OFF. You may need to obtain command tapes from
the drive manufacturer, possibly at nominal cost.



A QUICK COURSE IN FLYER THEORY:

The following information is provided for Flyer owners who wish to gain
a greater understanding of how the Flyer operates and achieve a more
thorough awareness of the factors that affect drive performance.


What data transfer rate is required in order to achieve D2 quality?

NewTek defines D2 quality as a video signal made up of 60
fields-per-second, full overscan (approximately 752 pixels wide by 480
lines high) image data within a digital composite video signal that is
8 bits at 14.3 MHz sampling rate.

This describes the digital video signal of a D2 video tape deck. It is
precisely the digital signal the Flyer uses internally. Since the Flyer
uses a variable compression ratio this set of signal characteristics
does not dictate a specific rate for data transfer because Flyer data
rate varies in proportion to picture complexity.

In previous publications such as the Flyer FAQ we described the various
Flyer operating ranges which preserve D2 at full broadcast bandwidth or
better. The low-end data rate, which could preserve very simple and
clean (noise-free) video is 1.4 MB/sec. The upper requirement for
highly complex video recorded at full-fidelity and
greater-than-broadcast resolution is 8.0 MB/sec. This is well beyond
the transfer rate of hard drives made at the time of this writing.

The Flyer can operate within the limits of current drive technology by
limiting the signal we record to broadcast resolution. By law, the FCC
limits the frequency response of broadcast video to 4.2 MHz. The Flyer,
in HQ5 and Standard Modes, limits frequency response to this figure
through a programmable low-pass filter. This first compression step
saves data and bandwidth that would be otherwise wasted by including
detail which must be discarded during broadcasting anyway.

Standard Mode is based on an assumed drive transfer rate limited to 3.7
MB/sec. HQ5 assumes a drive capable of about 5 MB/sec. Both will
deliver lossless D2 video when they can (when complexity, or noise and
detail, is low) and both modes will appear identical when complexity
permits a data rate of 3.7 MB/sec or lower.

If complexity increases, however, Standard Mode will respond with
higher compression ratios to keep the data rate down to 3.7 MB/sec or
less. The Flyer's VTASC compression system introduces more noise as its
compression ratio increases. The VTASC noise resembles random or
ordinary tape noise. At the highest compression ratios caused by
maximum video noise or detail, VTASC can also reduce the pixel sampling
rate to keep the data rate within the 3.7 MB/sec limit. In the worst
situations, this can introduce chroma noise.

HQ5 mode also employs variable compression, but the threshold (for the
shift to higher compression ratios) is 4.8 MB/sec instead of 3.7
MB/sec. VTASC will introduce less noise, if any, in HQ5 mode even with
very complex video.

In fact, with low noise or smooth rather than detailed video, there may
be no difference between the two modes.

Extended Mode also uses variable ratio compression, but always employs
reduced pixel sampling to reduce data flow. This results in slightly
lower resolution Flyer video and increased chroma noise.


"I've recorded various tape formats with great results, but with VHS
..."

All compression systems have difficulty compressing noisy or snowy
video. Such video may result from low quality tape, bad heads,
multi-generation dubs, or low light shooting conditions. There is no
way for video encoding systems to differentiate between noise and
detail, so a noisy image acts like an image that requires high
compression or a high data rate. Any compression system, including
VTASC, must increase its compression when it encounters noisy video.
This can result in a lack of detail and even more noise. Noisy video
does not compress well. In such cases, HQ5 mode will probably give you
far better results than Standard Mode.

Video shot in low light usually has very high noise levels. Consumer
camcorders often have built-in gain boosters for low-light shots. They
allow recording in low light, but they introduce noise from the
camera's light-sensitive CCD chip(s) and the video amplifier circuits.
Professional cameras usually have an option whether or not to boost
gain in low-light and can provide some control over introducing noise
to a low-light shot.

Tape noise can be eliminated entirely by recording directly through the
Flyer to hard disk.  You'll get best possible video recording and
lowest possible data rates if you bypass tape.


----------------------------------------------------
All material written by Tim Jenison and Chuck Baker
(with contributions from Dan Wolf and James Hebert).
----------------------------------------------------