Don't Panic - Yet.

     I work in a small data processing shop in Indianapolis.  Where we
maintain data for the purpose of direct mailing.  On a daily
basis our data entry staff makes accesses to fifty or sixty files
for the purpose of updates to the mailing lists.  Our system contains
somewhere in the neighborhood of 11,000 client mailing lists, very few
of which are accessed on an every day basis.

     Our backup requirements suggested to us that use of the SAVCHGOBJ
command would serve the purpose of daily backups.  In the event of a
"crash", we would need only to restore last months save lib nonsys and
then "apply" each of the 30 change tapes.  Sounds like a safe plan,
and hundreds of 38/400 shops are using this technique, but what you are
about to read may shock you.

     In early June of 1990, our S38 had a 3370 mod B12 crash in the HDA.
Yes, on a scale of 1 to 10, its a 9; I figure it would only be worse if
there were no backup tapes or, no system.  IBM to the rescue with a new
HDA and the system is at best running.  Next day, restoring all of this
mess begins.  The SAVLIB LIB(*NONSYS) runs smooth and by the end of the
day all access paths are rebuilt, all 52,000.  Now restore the change
tapes and everything is O.K., right?  Sorry folks, its not quite that
easy.

P A N I C!

     I shall attempt to stay on the track of the problem here and not
go off and attack "Big Blue".  Day two, first change tape applies first
level of changes and all is O.K. until tape 2.  At this point we notice
certain members are not being restored, in particular SOURCE members.
Spending some time analyzing the error messages, we determine that the
file member level IDs do not match.  Right to the point, there is no
combination of parameters on the RSTOBJ command to get around this.

     Three days have passed and enter the IBM support center LVL 1.
IBM show 1 hit on this problem, and resolving the problem is undocumented.
We are now on our way up from level 1, next stop "development center".
In less than an hour we are put in contact with one of the maintenance
programmers that takes care of CPF, who in not such a polite manor tells
us: "Save changed object does not update the last save date...". Argument
begins, why is this true when the UPDHST(*YES) is specified, answer
"... it is a design consideration."  IBM can offer no solution to the
problem, other than, "it is a design consideration.", I say: "How #$%#@
inconsiderate of IBM, to not document this in ANY manual!".

     Solution, 24hrs pass, all attempts at restoring the changed files,
with out deleting the old version have failed.  The only visible solution
would be to change the file member level ID, which can only be done by
updating the last save history.  To do this on a global level, this would
require another save of the object with SAVOBJ or SAVLIB.  We decide to
make sure we hit all of them and do the SAVLIB using the nonsys parameter.
There is a God, tape 2 applies all changes, but tape 3 will not apply
correctly with out another 5hr save lib nonsys.  No better solution could
be found, so we spent several days in the 24hr go change a tape mode.

     I guess the real solution to this problem is, design a better back
up process.  Some day it will happen, an HDA will crash on your system.
Make sure you don't end up in the same shape we were in, the support
center will be of little assistance, other than to tell you, a hit
on this problem occurred and that problem #3X278 has no documented fix.
There is no fix if you make it that far.  Make sure your back up routine
is updating the save history of all your objects.

          Jeff Price
          Sr. Systems Analyst
          Faris Mailing
          [71601,3231]
 ...... ... ...-....1200 N81N         ......................... ... ...-....1200 N81N         ......................... ... ...-....1200 N81N         ......................... ... ...-....1200 N81N         ......................... ... ...-....1200 N81N         ................