home *** CD-ROM | disk | FTP | other *** search
- Xref: sparky sci.math.stat:2099 sci.math.num-analysis:2977
- Newsgroups: sci.math.stat,sci.math.num-analysis
- Path: sparky!uunet!gatech!ncar!csn!teal.csn.org!sullivan
- From: sullivan@teal.csn.org (Steve Sullivan)
- Subject: SUMMARY: Principal Components Analysis
- Message-ID: <BvvMxu.Cpy@csn.org>
- Sender: news@csn.org (news)
- Nntp-Posting-Host: teal.csn.org
- Organization: Colorado SuperNet, Inc.
- Date: Fri, 9 Oct 1992 23:08:16 GMT
- Lines: 560
-
-
- Here is a summary of responses to my request
- on info about principal components analysis.
-
- Many thanks to all who replied!
-
- Steve Sullivan
- sullivan@csn.org
-
- =================================================
-
- From: dtm1@Ra.MsState.Edu (David T. Morse)
-
- Amick, D. J., & Walberg, H. J. (Eds.) (1975). Introductory
- multivariate analysis for educational, psychological,
- and social research. Berkeley, CA: McCutchan. Chapters
- 5-7 cover exploratory, rotation schemes, and
- confirmatory factor analysis, respectively.
-
- Cooley, W. W., & Lohnes, P. R. (1971). Multivariate data
- analysis. New York: Wiley. Chapter 4 covers principal
- components; chapter 5 discusses factor analysis.
- FORTRAN programs are supplied for data analysis.
-
- Cooper, J. C. B. (1983). Factor analysis: An overview.
- American Statistician, 37, 141-147.
-
- Forsythe, G. B., McGaghie, W. C., & Friedman, C. P. (1986).
- Construct validity of medical clinical competence
- measures: A multitrait-multimethod matrix study using
- confirmatory factor analysis. American Educational
- Research Journal, 23, 315-336.
-
- Gorsuch, R. L. (1983). Factor analysis (2nd ed.).
- Hillsdale, NJ: Lawrence Erlbaum.
-
- Hair, J. F., Jr., Anderson, R. E., Tatham, R. L., & Black,
- W. C. (1992). Multivariate data analysis with readings
- (3rd ed.). New York: Macmillan.
-
- Harman, H. H. (1967). Modern factor analysis (2nd ed.
- rev.). Chicago: The University of Chicago Press. One
- of the classic references for factor analysis.
-
- Kachigan, S. K. (1986). Statistical analysis: An
- interdisciplinary introduction to univariate &
- multivariate methods. New York: Radius Press. Chapter
- 15.
-
- Kim, J., & Mueller, C. W. (1978). Factor analysis:
- Statistical methods and practical issues. SAGE, 07-014.
- Beverly Hills, CA: Sage Publications.
-
- Kim, J., & Mueller, C. W. (1978). Introduction to factor
- analysis: What it is and how to do it. SAGE, 07-013.
- Beverly Hills, CA: Sage Publications.
-
- Kroonenberg, P. M., & Lewis, C. (1982). Methodological
- issues in the search for a factor model: Exploration
- through confirmation. Journal of Educational
- Statistics, 7, 69-89.
-
- Loehlin, J. C. (1987). Latent variable models: An
- introduction to factor, path, and structural analysis.
- Hillsdale, NJ: Lawrence Erlbaum.
-
- Lohnes, P. R. (1986). Correlated factors modeling via weak
- mathematics. American Educational Research Journal, 23,
- 289-302.
-
- Long, J. S. (1983). Confirmatory factor analysis. SAGE,
- 07-033. Beverly Hills, CA: Sage Publications.
-
- Magidson, J. (Ed.) (1979). Advances in factor analysis and
- structural equation models. Cambridge, MA: Abt Books.
-
- McDonald, R. P. (1985). Factor analysis and related
- methods. Hillsdale, NJ: Lawrence Erlbaum. Includes
- sections on linear structural relations, confirmatory
- factor analysis.
-
- Mislevy, R. J. (1986). Recent developments in the factor
- analysis of categorical variables. Journal of
- Educational Statistics, 11, 3-31.
-
- Overall, J. E., & Klett, C. J. (1972). Applied multivariate
- analysis. New York: McGraw-Hill. Chapters 4-7 include
- discussion of factor analysis topics; the book also
- includes sample FORTRAN programs for factor analysis.
-
- Ramsey, F. L. (1986). A fable of PCA. American
- Statistician, 40, 323-324.
-
- Rummel, R. J. (1970). Applied factor analysis. Evanston:
- Northwestern University Press.
-
- Stevens, J. (1986). Applied multivariate statistics for the
- social sciences. Hillsdale, NJ: Lawrence Erlbaum.
- Chapter 11 covers principal components analysis.
-
- Veldman, D. J. (1967). FORTRAN programming for the
- behavioral sciences. New York: Holt, Rinehart and
- Winston. Chapter 9 provides techniques for principal
- axis, common factor analysis, image analysis, and
- comparison of factor structures. FORTRAN programs
- included for analyzing data.
-
- Williams, F. (1986). Reasoning with statistics: How to read
- quantitative research (3rd ed.). New York: Holt,
- Rinehart and Winston. Chapter 13 provides an easy
- introduction to factor analysis.
-
- ................
- Also, Harris, R.J. A primer of multivariate statistics. New York:
- Academic Press (now in 2nd edition, about 1988 date, I think).
-
- ==========================================================
-
- From: sam@kalessin.Jpl.Nasa.Gov (Sam Sirlin)
-
- This is more usually called singular value decomposition. I have no
- idea what you want it for, so can't help with anything but basic
- references:
-
- Dongarra et al, LINPACK manual
-
- Greville and Ben-Isreal, "Generalized Inverses" had what I thought was
- a good treatment of SVD.
-
- --
- Sam Sirlin
- Jet Propulsion Laboratory sam@kalessin.jpl.nasa.gov
-
- ==========================================================
-
- From: scotbri@rosevax.rosemount.com (Scott Brigham)
-
- The book that I used that I found to be pretty helpful is
-
- Multivariate Calibration by Harald Martens and Tormod Naes
-
- John Wiley and Sons, 1989. (419 pages)
-
- It discusses PCA as well as PLS (partial least squares) and covers
- a lot of the other stuff you'll want to know when doing PCA.
-
- ==========================================================
-
- From: ph@physiol.ox.ac.uk (Patrick Haggard)
-
- I Recommend Krzanowski's Book
- Principles of Multivariate Analysis (OUP),
- which is very strong on concepts of multivariate
- stuff, inc. PCA.
- Patrick Haggard --- ph@uk.ac.ox.physiol
-
- ==========================================================
-
- From: Steve Smith <bugman@sciborg.uwaterloo.ca>
-
- Try the following for principal components. It's written for end users like
- me (a biologist) rather than for mathies, but in my opinion, it's first rate.
- If you get several replies, you can post to the group.
-
- Cheers!
-
- Jackson, J. Edward. 1991.
- A user's guide to principal components.
- John Wiley & Sons, Inc., New York.
-
- ==========================================================
-
- From: stewart@cs.umd.edu (G. W. Stewart)
-
- Here is a dump from my bibliography. Keep in mind that it represents
- a numerical analysts view. A statistician could give you many
- more references.
-
- Pete Stewart
-
- \begin{thebibliography}{10}
-
- \bibitem{cham:72}
- J.~M. Chambers.
- \newblock Stabilizing linear regression against observational error in
- independent variats.
- \newblock Manuscript, Bell Laboratories, Murray Hill, New Jersey., 1972.
-
- \bibitem{ecyo:36}
- C.~Eckart and G.~Young.
- \newblock The approximation of one matrix by another of lower rank.
- \newblock {\em Psychometrika}, 1:211--218, 1936.
-
- \bibitem{gaza:79}
- K.~R. Gabriel and S.~Zamir.
- \newblock Lower rank approximation of matrices by least squares with any choice
- of weights.
- \newblock {\em Technometrics}, 21:489--498, 1979.
-
- \bibitem{golu:69}
- G.~H. Golub.
- \newblock Matrix decompositions and statistical calculations.
- \newblock Technical Report 124, Computer Science Department, Stanford
- University, 1969.
-
- \bibitem{golu:69a}
- G.~H. Golub.
- \newblock Matrix decompositions and statistical computation.
- \newblock In R.C. Milton and J.~A. Nelder, editors, {\em Statistical
- Computation}, pages 365--397, New York, 1969. Academic Press.
- \newblock Cited in {\AA ke Bj\"orck's} bibliography on least squares, which is
- available by anonymous ftp from {\tt math.liu.se} in {\tt pub/references}.
-
- \bibitem{good:69}
- I.~J. Good.
- \newblock Some applications of the singular decomposition of a matrix.
- \newblock {\em Technometrics}, 11:823--831, 1969.
-
- \bibitem{guns:78}
- R.~F. Gunst.
- \newblock Similarities among least squares, principal component, and latent
- root regression estimators.
- \newblock Manuscript. Department of Statistics, Southern Methodist University,
- 1978?
-
- \bibitem{guwm:76}
- R.~F. Gunst, J.~T. Webster, and R.~L. Mason.
- \newblock A comparison of least squares and latent root regression estimators.
- \newblock {\em Technometrics}, 18:75--83, 1976.
- \newblock Cited in \cite{govl:89}.
-
- \bibitem{hamm:85}
- S.~J. Hammarling.
- \newblock The singular value decomposition in multivariate statistics.
- \newblock {\em {ACM} {SIGNUM} Newsletter}, 20:2--25, 1985.
- \newblock Cited in \cite{govl:89}.
-
- \bibitem{hano:81}
- R.~J. Hanson and M.~J. Norris.
- \newblock Analysis of measurements based on the singular value decomposition.
- \newblock {\em {SIAM} Journal on Scientific and Statistical Computing},
- 2:363--374, 1981.
- \newblock Cited in \cite{govl:89}.
-
- \bibitem{hawk:73}
- D.~M. Hawkins.
- \newblock On the investigation of alternative regressions by principal
- component analysis.
- \newblock {\em Appl.\ Statist.}, 22:275--286, 1973.
-
- \bibitem{hote:33}
- H.~Hotelling.
- \newblock Analysis of a complex of statistical variables into principal
- components.
- \newblock {\em Journal of Educational Psychology}, 24:417--441 and 498--520,
- 1933.
-
- \bibitem{hote:57}
- H.~Hotelling.
- \newblock Relation of the newer multivariate statistical methods to factor
- analysis.
- \newblock {\em Br. J. Static. Psychol.}, 10:69--79, 1957.
-
- \bibitem{hucu:62}
- J.~R. Hurley and R.~B. Cattell.
- \newblock The {Procrustes} program: {Direct} rotation to test a hypothesized
- factor structure.
- \newblock {\em Behavioral Science}, 7:258--262, 1962.
-
- \bibitem{paig:85}
- C.~C. Paige.
- \newblock The general linear model and the generalized singular value
- decomposition.
- \newblock {\em Linear Algebra and its Applications}, 70:269--284, 1985.
- \newblock Cited in \cite{govl:89}.
-
- \bibitem{pear:01}
- K.~Pearson.
- \newblock On lines and planes of closest fit to points in space.
- \newblock {\em Philosophical Magazine}, 2:559--572, 1901.
- \newblock Cited in \cite{govl:89}.
-
- \bibitem{rao:80}
- C.~R. Rao.
- \newblock Matrix approximations and reduction of dimensionality in multivariate
- statistical analysis.
- \newblock In P.~R. Krishnaiah, editor, {\em Multivariate Analysis--V}.
- North-Holland, Amsterdam, 1980.
-
- \bibitem{vhva:91}
- S.~Van~Huffel and J.~Vandewalle.
- \newblock {\em The Total Least Squares Problem: Computational Aspects and
- Analysis}.
- \newblock SIAM, Philadelphia, 1991.
-
- \bibitem{wegm:74}
- J.~Webster, R.~Gunst, and R.~Mason.
- \newblock Latent root regression analysis.
- \newblock {\em Technometrics}, 16:513--522, 1974.
-
- \bibitem{wegm:76}
- J.~Webster, R.~Gunst, and R.~Mason.
- \newblock A comparison of least squares and latent root regression estimators.
- \newblock {\em Technometrics}, 18:75--83, 1976.
-
- \end{thebibliography}
-
- ==========================================================
-
- From: rmyers@ics.uci.edu (Richard E. Myers)
-
- I've become interested in using PCA to analyze a sequence of high
- dimensional vectors (speech frames). My SYSTAT manual and a look on
- mlvl turned up the following references:
-
-
- 5. MCDONALD RP.
- COMMON PRINCIPAL COMPONENTS AND RELATED MULTIVARIATE METHODS - FLURY,B.
- Pub type: Book Review.
- JOURNAL OF CLASSIFICATION, 1990, V7 N2:310-312.
-
- 6. Flury, Bernhard, 1951-
- Common principal components and related multivariate models / Bernhard
- Flury. New York : Wiley, c1988.
- Series title: Wiley series in probability and mathematical statistics.
-
- 8. Jackson, J. Edward.
- A user's guide to principal components / J. Edward Jackson. New York :
- Wiley, c1991.
- Series title: Wiley series in probability and mathematical statistics.
- Applied probability and statistics.
-
- 2. Harman, Harry Horace, 1913-1976.
- Modern factor analysis. [Chicago] University of Chicago Press [1960].
-
- 1. Mulaik, Stanley A., 1935-
- The foundations of factor analysis [by] Stanley A. Mulaik. New York,
- McGraw-Hill [1971, c1972].
- Series title: McGraw-Hill series in psychology.
-
- 1. Gnanadesikan, Ram, 1932-
- Methods for statistical data analysis of multivariate observations / R.
- Gnanadesikan. New York : Wiley, c1977.
- Series title: Wiley series in probability and mathematical statistics.
- Series title: A Wiley publication in applied statistics.
-
- 40. Mardia, K. V.
- Multivariate analysis / K. V. Mardia, J. T. Kent, J. M. Bibby. London ;
- New York : Academic Press, 1979.
- Series title: Probability and mathematical statistics.
-
- ==========================================================
-
- From: olevin@random.hwr.arizona.edu
-
- Cattell, r.b. (965) Factor analysis. An introduction to
- Essentials. I: the purpose and underlying models. Biometrics 21,
- pp. 190-210
- II: the role of factor analysis in research, Biometrics, 21,
- pp405-435.
-
- Childs d 1970, The essentials of factor analysis. London, Holt,
- Rinehart, and Wisnton.
-
- Cooley W.W> and Lohnes (1971) Multivariate data analysis, NY
- J-Wiley.
-
- Davis, J./c. 1973) Statistics and data analyiss i geology,
- Academic press
-
- Fruchter b. (1954) /Introduction to factor analysis D. van
- Nostrand Co.
-
- Seyhan E. 1985 Introduction to multivariate statistical analysis
- Free university, Institute of Earth sciences Netherlands
-
-
- ==================================================================
-
- From: whitbeck@equinox.unr.edu (Michael Whitbeck)
-
- PCA is highly flavored by the area of application
- but a very readable book is "Factor Analysis in Chemistry" by
- E. Malinowski, Wiley-Interscience, 1991.
- Heartily recommended! (despite some typos)
-
- Not very readable but more mathematical, less science application
- oriented, is "Factor Analysis as a Statistical Method" by
- Lawley and Maxwell, Butterworths, 1963. But read the HISTORICAL
- INTRO in Malinowski's book if nothing else!
-
- Recently matlab code has been posted that will do pca.
- To 'roll' your own in FORTRAN or C or C++.... start with
- a singular value decomposition routine from netlib or NR.
- Sort the eigenvalues, largest first, determine which
- eigenvalues are significant (see Malinowski)... Quite
- easy.
-
- ==================================================================
-
- From: jsvrc@rc.rit.edu (J A Stephen Viggiano)
-
- Edward (Ted) Jackson, of Eastman Kodak, wrote a three-part series on PCA for
- the _Journal_of_Quality_Technology_ which appeared in 1981. It's a great
- introduction. I think he also wrote a book on the subject.
-
- There are a variety of books on a related topic, Factor Analysis. Be advised
- that this is only a related topic; PCA is only treated as a special case, if
- at all. One of the best is the Third Edition of _Modern_Factor_Analysis_ by
- Harry Horace Harmon. BTW, Part 3 of Ted Jackson's series in JQT covers
- Factor Analysis.
-
- Finally, there are books on multivariate analysis that discuss PCA.
- Gnanadesikan's _Methods_for_Statistical_Data_Analysis_of_Multivariate_
- _Observations_ provided me with a starting point for my thesis work on
- non-linear PCA about 8 or 9 years ago.
-
- ==================================================================
- ==================================================================
-
- From: beaucham@uxh.cso.uiuc.edu (James Beauchamp)
-
- A summary from a previous query, including the following:
-
- I found the Sage University Papers series Quantitative Applications in the
- Social Sciences to provide very accessable introductions to such topics.
- The Principal Components Analysis paper is #69 by Dunteman; I have not
- seen the particular paper. If your library doesn't have it, Sage Publications
- is out of Newbury Park, California (fax: 805-499-0871).
-
- Peter Palij, Columbia University, pbp1@cunixb.cc.columbia.edu
-
- ------------
-
- i'm not an expert on PC, but i have had occassion to use it as a tool in
- descriptive data analysis. i also noticed the multiple methods/explanations
- you mention. if you haven't looked at it already, you might want to look
- at a book by morrisson - something like "multivariate analysis". he talks
- a bit about different methods, for example, when PC would makes sense based
- on the covariance matrix vs based on the correlation matrix.
-
- john watts, University of Chicago, stuw@midway.uchicago.edu
-
- ------------
-
- Duda & Hart is a great reference for statistical
- pattern recognition techniques. While PCA is a
- general tool, they have a decent explanation,
- plus references.
-
- Dave DeMers, UC San Diego, demers@cs.ucsd.edu
-
- ------------
-
- Yes, the stuff on principal components analysis often IS confusing and
- technical. One reason for the confusion is that a lot of analysis techniques
- are quite similar. Principal components, Correspondence Analysis, Canonical
- Analysis (and maybe another two or three) are all techniques based on the
- exact SAME mathematical manipulation (the singular value decomposition or
- SVD). The differences amongst the techniques comes in how the data are
- treated before the SVD is conducted.
-
- I think that the best combination of explanation, technical sophistication
- without obfuscation, and examples is (as is often the case) in the little
- green Sage book from their series on quantitative analysis. The title
- is something like Classification and Scaling Analysis, but I am not
- certain of the title. The authors, however, are Susan Weber (sp?) and
- A Kimball Romney.
-
- Raymond V Liedka, Cornell University, RJOY@cornellc.cit.cornell.edu
-
- ------------
-
- Venables (the guy who followed up with a posting) is good, and
- PC must be one of his specialties, but if you need something more
- elementary, try
-
- Pielou, E.C. 1984
- The interpretation of ecological data: A primer on classification
- and ordination
- Published by Wiley
- LC call number: QH541.15.S72P54 1984
-
- Lots of simple, worked-out examples with diagrams. The different
- varieties of PC are identified.
-
- Charles Packer, PACKER@amarna.gsfc.nasa.gov
-
- ------------
-
- This is one well-defined method. For the theory see for example:
- Lebert, Morineau and Warwick: Multivariate Descriptive Analysis (Wiley, 1984).
- Mardia, Kent and Bibby: Multivariate Analysis (Academic Press, 1979).
- SAS/STAT User's Guide 1&2.
-
- Helgi Thorsson, helgith@rhi.hi.is
-
- ------------
-
- Regarding your query about references to the PC techniques, I talked to
- Doris (who is the real expert here on that topic), and she suggested:
-
- 1) Brillinger, D. R. (1981) Time Series, Data Analysis and Theory.
- Holden Day, San Francisco. (esp. Chapter 9)
-
- 2) Molenaar, P. C. M. (1987) Dynamic factor analysis in the frequency
- domain: causal modeling of multivariate psychophysiological time
- series. Multivariate Behavioral Research, 22, 329-353. (this is a dandy
- since it deals explicitly with changing spectra, but I find it rough
- going.)
-
- Fred Wightman, Univ. of Wisconsin at Madison, WIGHTMAN@waisman.wisc.edu
-
- ------------
-
- A standard reference is Morrison's Multivariate Statistical Methods,
- that shows how PC fits into the scheme of things. Preisendorfer's
- PC analysis in meterology and oceanography is also a good book and
- has a very thorough list of references of PC applications.
-
- You should read (though it is difficult) Richman's (1986) Rotation of PC
- in J. of Climatology, 6, 293-335.
-
- Arthur J. Mariano U. of Miami, mariano@umigw.miami.edu
-
- ------------
-
- One book that may be of use is part of the Sage series:
- Dunteman, George H. (1989). Principal Components Analysis. Sage Publications.
- ISBN number: 0-8039-3014-2
-
- Wynne Chin, The University of Calgary, chin@acs.ucalgary.ca
-
- ------------
-
- Try:
-
- Principal Component Analysis
- by I. T. Jolliffe
- Springer Series in Statistics
- 1986
-
- Stephanie Butler, TI Semiconductor Process and Design Center,
- butler%epcot@ti.com
-
-
- Aside from Bill Venables remarks, there are several basic statistical
- texts which may help. Here's a reference to one with a more rigorous
- text listed second.
-
- Cooley and Lohnes "Multivariate Data Analysis"
-
- Preisendorfer "Principal Component Analysis in Meteorology and
- Oceanography"
-
- Mike Richman, The University of Oklahoma, richman@reepicheep.gcn.uoknor.edu
-
- =====================================================================
-
-
-