OS/2 Shareware BBS: 35 Internet

home *** CD-ROM | disk | FTP | other *** search

/ OS/2 Shareware BBS: 35 Internet / 35-Internet.zip / httpan22.zip / http-analyze.man < prev next >

Wrap

Text File | 1999-03-14 | 116KB | 2,113 lines

hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) NNNNAAAAMMMMEEEE hhttttpp--aannaallyyzzee - a fast log analyzer for web servers SSSSYYYYNNNNOOOOPPPPSSSSIIIISSSS hhttttpp--aannaallyyzzee [--{{hhddmmVV}}] [--33aaeeffggnnqqvvxxyy] [--cc _c_f_g_f_i_l_e] [--ll _l_i_b_d_i_r] [--oo _o_u_t_d_i_r] [--pp _p_r_i_v_d_i_r] [--ss _s_u_b_o_p_t,...] [--tt _n_u_m,...] [--uu _t_i_m_e] [--ww _h_i_t_s] [--FF _f_o_r_m_a_t] [--GG _s_u_f_f_i_x,...] [--HH _i_d_x_f_i_l_e,...] [--II _d_a_t_e] [--EE _d_a_t_e] [--OO _v_i_r_t_n_a_m_e,...] [--PP _p_r_o_l_o_g] [--RR _d_o_c_r_o_o_t] [--SS _s_r_v_n_a_m_e] [--TT _T_L_D_f_i_l_e] [--UU _s_r_v_u_r_l] [--WW _3_D_w_i_n] [_l_o_g_f_i_l_e[...]] DDDDEEEESSSSCCCCRRRRIIIIPPPPTTTTIIIIOOOONNNN hhttttpp--aannaallyyzzee analyzes the logfile of a web server and creates a detailed summary of the servers's access load in graphical, tabular, and three- dimensional form. In auto-sense mode (default), hhttttpp--aannaallyyzzee recognizes the logfile format automatically. Supportet formats for logfiles are the _C_o_m_m_o_n _L_o_g_f_i_l_e _F_o_r_m_a_t (_C_L_F) and two forms of the so-called _E_x_t_e_n_d_e_d _L_o_g_f_i_l_e _F_o_r_m_a_t (_E_L_F), which is basically the CLF plus user-agent and referrer URL information. All web servers support at least the _C_o_m_m_o_n _L_o_g_f_i_l_e _F_o_r_m_a_t and most of them can be configured to produce the _E_x_t_e_n_d_e_d _L_o_g_f_i_l_e _F_o_r_m_a_t. hhttttpp--aannaallyyzzee has been highly optimized to process large logfiles at maximum speed. There are two modes of operation with different levels of detail in the logfile analysis: _S_h_o_r_t _s_t_a_t_i_s_t_i_c_s ("daily" mode, option --dd): hhttttpp--aannaallyyzzee generates a short summary of the server usage per day for the current month. In this mode, it uses a history file to skip entries which have been processed already. By avoiding detailed analysis of the logfile entries, hhttttpp--aannaallyyzzee requires only a fraction of the time which would be required to generate a full statistics report. _F_u_l_l _s_t_a_t_i_s_t_i_c_s ("monthly" mode, option --mm): In full statistics mode, the analyzer generates a complete report for a whole month, which contains much more details than the short statistics report. The history file is used only to produce a summary for the last 12 months without having to analyze the logfiles for those previous periods again. In full statistics mode the actual period to analyze is determined by analyzing the timestamps of the first and last logfile entry read. This is the default if no mode is specified explicitely. Usually you run hhttttpp--aannaallyyzzee in full statistics mode only, since this report also includes all the information available in short statistics. However, if your logfiles are rather large and if the analyzer causes significant load while generating the full statistics report, you could run it more frequently in short statistics mode with update intervals in the range of 30 minutes to some hours to create an up-to-date report, and then run it in full statistics mode less often, for example once per day or week, to generate a detailed report. The operation modes have been named after their periods covered, namely _d_a_i_l_y for the short and _m_o_n_t_h_l_y Page 1 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) for the full statistics mode. Note that in full statistics mode the analyzer needs to process all logfile entries since the beginning of the current month, while in short statistics mode it skips all entries up to the current day if it finds a valid history. Therefore you should rotate the logfile at the first day of a new month and then generate a final statistics report for the previous month using the logfile just rotated. If disk space is a concern, you can set up a scheme where the logfiles are rotated and compressed using some compression program once per week or even once per day. In this case, you have to concatenate all logfiles for this month in order of ascending date before feeding them into the analyzer to have it generate a full statistics report. On the first day of the new month, if a detailed report for the previous month has been generated, you can save the corresponding logfile(s) somewhere and finally remove it or them from your production system. LLLLOOOOGGGGFFFFIIIILLLLEEEE FFFFOOOORRRRMMMMAAAATTTTSSSS hhttttpp--aannaallyyzzee recognizes three logfile formats, which can be configured in most web servers: CCoommmmoonn LLooggffiillee FFoorrmmaatt ((CCLLFF)) The _C_o_m_m_o_n _L_o_g_f_i_l_e _F_o_r_m_a_t is supported by all web servers. The entries contain the following information: dns-name - auth-user [date] "clf-request" clf-status ct-length where the fields have following meaning: _d_n_s-_n_a_m_e The IP number of the system accessing the web server. If there is an entry in the _D_o_m_a_i_n _N_a_m_e _S_y_s_t_e_m (_D_N_S) for this IP number and the web server is configured to do DNS lookups, the corresponding hostname is logged instead. - Unused. _a_u_t_h-_u_s_e_r The username provided by the client to access files which require authentication. [_d_a_t_e] The date of the access as [DD/MMM/YYYY:HH:MM:SS +-ZZZZ]. _c_l_f-_r_e_q_u_e_s_t The request in format "method URI proto", where _m_e_t_h_o_d is one of GGEETT, HHEEAADD, PPOOSSTT, PPUUTT, BBRROOWWSSEE, OOPPTTIIOONNSS, DDEELLEETTEE or TTRRAACCEE; _U_R_I is the _U_n_i_f_o_r_m _R_e_s_o_u_r_c_e _I_d_e_n_t_i_f_i_e_r, and _p_r_o_t_o is the protocol parameter containing the HTTP version. The _c_l_f- _r_e_q_u_e_s_t field is surrounded by double quotes. _c_l_f-_s_t_a_t_u_s This is the (numerical) response code from the server. _c_t-_l_e_n_g_t_h Depending on the server, this number is either the size of Page 2 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) the document or the data actually sent over the wire. Following is an example for an entry in _C_o_m_m_o_n _L_o_g_f_i_l_e _F_o_r_m_a_t: car.4rent.de - - [01/Aug/1998:00:00:02 +0100] "GET /doc.html HTTP/1.1" 200 393 CCoommbbiinneedd LLooggffiillee FFoorrmmaatt ((DDLLFF)) Some server use the so-called _C_o_m_b_i_n_e_d _L_o_g_f_i_l_e _F_o_r_m_a_t to add the referrer URL and user-agent (browser) identification to the logfile entries. It looks like the CLF format followed by the referrer URL and the user- agent, where the latter two fields are surrounded by double quotes: CLF "referrer_URL" "user_agent" This is an example for an entry in _C_o_m_b_i_n_e_d _L_o_g_f_i_l_e _F_o_r_m_a_t (wrapped on two lines here for readability only): car.4rent.de - - [01/Aug/1998:00:00:02 +0100] "GET /doc.html HTTP/1.1" 200 393 "http://inet-tv.net/hot.html" "Mozilla/4.05 (X11; I; IRIX64 6.4 IP30)" Unfortunately, the double quotes sometimes appear in broken referrer URLs, as for example in: "http://www.some.host/wiredlink.html TARGET=newwin"" Sometimes there are even referrer URLs which contain double quotes followed by blanks, which make such entries not parseable in an unambiguous way. Although hhttttpp--aannaallyyzzee recognizes the _C_o_m_b_i_n_e_d _L_o_g_f_i_l_e _F_o_r_m_a_t automatically, and tries to do it's best to parse the referrer URL correctly, the following format, which avoids this ambiguity, should be preferred if possible. EExxtteennddeedd LLooggffiillee FFoorrmmaatt ((EELLFF)) The _E_x_t_e_n_d_e_d _L_o_g_f_i_l_e _F_o_r_m_a_t contains also the user-agent and the referrer URL as in the _C_o_m_b_i_n_e_d _L_o_g_f_i_l_e _F_o_r_m_a_t, but in the opposite order and without the surrounding double quotes: CLF user_agent referrer_URL If this _E_x_t_e_n_d_e_d _L_o_g_f_i_l_e _F_o_r_m_a_t is used, hhttttpp--aannaallyyzzee searches backwards for the protocol specification of the referrer URL (to be precise, it looks for the colon in hhttttpp::) and then for the preceeding blank. This way, even broken referrer URLs which contain blanks are handled correctly. To select this format, just edit the configuration file of your web server and select the _E_L_F order of the user-agent and referrer URL fields. This is an example for an entry in _E_x_t_e_n_d_e_d _L_o_g_f_i_l_e _F_o_r_m_a_t (wrapped on two lines here for readability only): Page 3 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) car.4rent.de - - [01/Aug/1998:00:00:02 +0100] "GET /doc.html HTTP/1.1" 200 393 Mozilla/4.05 (X11; I; IRIX64 6.4 IP30) http://inet-tv.net/index.html SSSSTTTTAAAATTTTIIIISSSSTTTTIIIICCCCSSSS RRRREEEEPPPPOOOORRRRTTTT Depending on the operation mode, there are two reports: a full statistics report and a short statistics report, which might be updated more frequently. While the full statistics report contains much more details, the short statistics report covers only the most important values. FFuullll ssttaattiissttiiccss mmooddee By default, hhttttpp--aannaallyyzzee runs in full statistics mode. Due to technical reasons, a full statistics report will not be created before the second day of a new month, although the totals for the first day of the new month on the summary main page of the report will be updated. A full statistics report contains a detailed summary including (see the section _I_n_t_e_r_p_r_e_t_a_t_i_o_n _o_f _t_h_e _r_e_s_u_l_t_s for an explanation of the terms): o the number of hits, files, pageviews, sessions, and data sent by year, month, and day o the total amount of data requested, transferred, and saved by cache o the total number of unique URLs, sites, sessions, agents, and referrers o the total number of all response codes other than 200 (_O_K) o the total number of reuqests which required authentication o the average load per week, day, hour, minute and second o the top 7 days, 24 hours, 5 minutes and 5 seconds o the top 30 most commonly accessed URLs (hits, pageviews, sessions, data sent) o the 10 least frequently accessed URLs (hits, pageviews, sessions, data sent) o the top 30 client domains, browser types, and referrer hosts o the overview/detailed list of all files, sitenames, browser types, and referrer URLs o the list of all Code 404 (_N_o_t _F_o_u_n_d) responses SShhoorrtt ssttaattiissttiiccss mmooddee In short statistics mode, hhttttpp--aannaallyyzzee creates a short summary including only the number of hits, files, pageviews, sessions, and the amount of data sent per day. Since the short statistics report does not contain as many details as a full statistics report, it requires only a fraction of processing time to create it. A short statistics report is created if requested explicitely and also in full statistics mode for the current month. This way, on the first day of a new month, when no full statistics can be generated due to technical reasons, a short statistics report is available at least. Running hhttttpp--aannaallyyzzee in short statistics mode explicitely may be useful Page 4 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) if the load on the server increases when creating full reports very frequently. For example, a short statistics report can be generated twice per hour, while a full statistics report is created only twice per day. UUUUSSSSEEEERRRR IIIINNNNTTTTEEEERRRRFFFFAAAACCCCEEEESSSS There are two user interfaces to the statistics report: a conventional interface suitable for any browser and a frames-based interface which requires JavaScript. TThhee ccoonnvveennttiioonnaall iinntteerrffaaccee The conventional interface appears as in version 1.9e if JavaScript is disabled in your browser or the option --gg was specified at invocation of hhttttpp--aannaallyyzzee. If JavaScript is enabled, the following separate windows are used for different parts of the report to allow for easy navigation: _T_h_e _M_a_i_n _w_i_n_d_o_w This window is used for most parts of the report such as the yearly, monthly, daily and weekly summaries, the _T_o_p _N lists and the overviews. Hotlinks in the _T_o_p _N most often point to the corresponding page, which is then displayed in the _V_i_e_w_e_r _w_i_n_d_o_w if the link is followed, while hotlinks in the overviews point to the detailed lists, which show up in the _L_i_s_t _w_i_n_d_o_w. _T_h_e _N_a_v_i_g_a_t_i_o_n _w_i_n_d_o_w If JavaScript is enabled in your browser and a summary for a year or a month is loaded in the main window, a small window containing a navigation panel will pop up. If JavaScript is disabled, the navigation links appear at the bottom of the monthly summary pages. In this case, use the _B_a_c_k button of your browser for navigation. _T_h_e _L_i_s_t _w_i_n_d_o_w This window is used for the detailed lists of URLs, sites, browser types and referrer URLs. A separate window for those (often large) lists causes them to be loaded only once if the links in the _M_a_i_n _w_i_n_d_o_w are followed and the _L_i_s_t _w_i_n_d_o_w is still open. _T_h_e _V_i_e_w_e_r _w_i_n_d_o_w This window is used for external pages which are loaded by following hotlinks in the statistics report. This way, you can visit the pages referred to in the report without having to go forth and back between the report and the pages listed there. _T_h_e _3_D _w_i_n_d_o_w This window is used for the 3D (VRML) model of the statistics. If you have JavaScript enabled, the window's size will be set to the smallest possible size so that the 3D model fits onto the screen or to the dimensions given in the 33DDWWiinnSSiizzee directive. Page 5 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) TThhee ffrraammeess--bbaasseedd iinntteerrffaaccee The frames-based interface requires a JavaScript-enabled browser. It contains the following frames and windows: _T_h_e _N_a_v_i_g_a_t_i_o_n _f_r_a_m_e This frame contains navigation buttons and text. You can specify it's width using the NNaavviiggFFrraammee directive in the configuration file. _T_h_e _M_a_i_n _f_r_a_m_e This frame is used for most parts of the report such as the yearly, monthly, daily and weekly summaries, the _T_o_p _N lists and the overviews. Hotlinks in the _T_o_p _N lists point most often to the corresponding page, which is displayed in the _V_i_e_w_e_r _w_i_n_d_o_w if the link is followed, while hotlinks in the overviews point to the detailed lists, which show up in the _L_i_s_t _w_i_n_d_o_w. _T_h_e _L_i_s_t _w_i_n_d_o_w This (separate) window is used for the detailed lists of URLs, sites, browser types and referrer URLs. A separate window for those (often large) lists causes them to be loaded only once if the links in the _M_a_i_n _w_i_n_d_o_w are followed and the _L_i_s_t _w_i_n_d_o_w is still open. _T_h_e _V_i_e_w_e_r _w_i_n_d_o_w This (separate) window is used for external pages which are loaded by following the hotlinks in the statistics report. This way, you can visit the pages referred to in the report without having to go forth and back between the report and the pages listed there. _T_h_e _3_D _w_i_n_d_o_w This window is used for the 3D (VRML) model of the statistics. Depending on the setting of the 33DDWWiinnddooww directive in the configuration file, this is either a separate window (_e_x_t_e_r_n_a_l) or a new frame (_i_n_t_e_r_n_a_l) inside the _M_a_i_n _f_r_a_m_e (actually, two frames are created which replace the former _M_a_i_n _f_r_a_m_e when the 3D model is being displayed). In case of a separate (external) _3_D _w_i_n_d_o_w, you can specify it's dimensions using the 33DDWWiinnSSiizzee directive. TThhee 33DD mmooddeell The 3D model requires a VRML 2.0 plug-in such as CosmoPlayer from Cosmo Software (http://cosmo.sgi.com/). Using this plug-in, which is available for Netscape on Silicon Graphics systems and Netscape/MSIE on Windows NT, you can "walk" or "fly" through the model and view the scene from all sides. And if you look at the models, don't forget to touch the buddha appearing in our 3D logo on top of the statistics report in the yearly summary pages! The 3D model contains two _s_c_e_n_e_s (statistics models): one scene showing the hits, 304's, sites and data sent by day and another scene showing the server's load by weekday and hour. To view the second scene, click on the _s_c_e_n_e _s_w_i_t_c_h on the right top of the model. To navigate through the Page 6 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) 3D space, use the _V_i_e_w_p_o_i_n_t_s and the CosmoPlayer _N_a_v_i_g_a_t_i_o_n _p_a_n_e_l. For customization of CosmoPlayer use the pop-up menu, which appears if you press the right-most mouse button. The 3D representation of hits by weekday and hour in the second scene allow easy identification of the time your server has been most busy serving requests. IIIINNNNTTTTEEEERRRRPPPPRRRREEEETTTTAAAATTTTIIIIOOOONNNN OOOOFFFF TTTTHHHHEEEE RRRREEEESSSSUUUULLLLTTTTSSSS hhttttpp--aannaallyyzzee shows you a summary of the content of your server's logfile. It collects information from the logfile entries, sets them into some relationship and creates a summary as a result of this analysis. The following is an explanation of the terms used in the report: HHiittss (color key: green) A hit is any response from the server on behalf of a request sent from a browser. This includes aannyy response from the server, not only text files or documents. For example, if a HTML page is requested, which has two inline images, the server would generate three hits: one hit for the HTML page itself and two hits for the inline images. On the other side, if an invalid URL is requested, the server would respond with a Code 404 (_N_o_t _F_o_u_n_d) status code, which also generates a hit. FFiilleess (color key: blue) If the user requests a document and the server successfully sends back a file for this request, this is counted as a Code 200 (_O_K) response. Any such response is counted for as a file. Again, "file" here means any kind of a file, no matter whether it contains text (documents, directory listings) or binary data (images, applets, etc.). CCooddee 330044 ((NNoott MMooddiiffiieedd)) (color key: yellow) A Code 304 (_N_o_t _M_o_d_i_f_i_e_d) response is generated by the server if a document hasn't changed since the last time it was transferred to some site. If a browser has access to a local copy of a document requested by the user - either through it's local disk cache or through a caching server on the way between the browser and the web server -, it sends out a conditional request, which contains the modification date of the document as stored in the browser's or the caching server's local cache. If the document has changed since then, the server re-transmits the new document. If it hasn't changed meanwhile, the server sends back a Code 304 response and the browser uses it's local copy. While this technique can significantly reduce network traffic, it causes an inaccuracy in the statistics report regarding the number a document is actually transmitted to some visitor because of two reasons: First, the browser usually sends only one such a conditional request per session if it still holds an up-to-date copy of the file. Second, caching servers often serve many thousands of Page 7 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) users. So if you see some requests from a caching server of an online service for example, this could be caused by thousands of users requesting a certain document or just one person with a browser configured to not cache anything at all. However, the ratio between "files" and "304's" reflects the efficiency of overall caching mechanisms for at least those hits which made their way to the server. PPaaggeevviieewwss (color key: magenta) The analyzer classifies all URLs which match certain patterns as pageviews (text files). Patterns may be defined using an option or a directive in the configuration file. The analyzer automatically pre-defines the suffix ..hhttmmll as a pageview. Classifying requests of certain files as pageviews allows you to estimate the number of "real" documents transmitted by your server. If used correctly, hhttttpp--aannaallyyzzee rates text files (documents) as pageviews, which do not include images, CGI scripts, Java applets or any other HTML objects. KKBByytteess ttrraannssffeerrrreedd (color key: orange) This is the amount of data sent during the whole summary period as reported by the server. Note that some servers do log the size of a document instead of the actual number of bytes transferred. While in most cases this is the same, if a user interrupts the transmission by pressing the browser's stop button before the page has been received completely, some servers (for example all Netscape web servers) do not log the amount of data transferred but the amount of data which _w_o_u_l_d have been transferred if the user would have completely loaded the page. KKBByytteess rreeqquueesstteedd This is the amount of data requested during the whole summary period. hhttttpp--aannaallyyzzee computes this number by summing up the values of _K_B_y_t_e_s _t_r_a_n_s_f_e_r_r_e_d and _K_B_y_t_e_s _s_a_v_e_d _b_y _c_a_c_h_e (see below). KKBByytteess ssaavveedd bbyy ccaacchhee The amount of data saved by various caching mechanisms. This value is computed by multiplying the number of Code 304 (_N_o_t _M_o_d_i_f_i_e_d) requests per file with the size of the corresponding file. Because hhttttpp--aannaallyyzzee can determine the size of a file only if the file has been requested at least once in the same summary period, the values for _K_B_y_t_e_s _s_a_v_e_d _b_y _c_a_c_h_e and _K_B_y_t_e_s _r_e_q_u_e_s_t_e_d are just approximations of the actual values. UUnniiqquuee UURRLLss Unique URLs are the number of all different, valid URLs requested in a given summary period. This shows you the number of all different files on your web server requested at least once in the corresponding summary period. RReeffeerrrreerr UURRLLss If a document on your server is requested because a hypertext link Page 8 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) to it on a page of a foreign web server is followed, the name of this server gets logged as the _r_e_f_e_r_r_e_r _U_R_L (the URL of the page referring to your document). Note, that if the URL is specified manually in the browser, no referrer URL gets logged. Such requests are collected under _U_n_k_n_o_w_n in the referrer URL part of the report. SSeellff--rreeffeerrrreerr UURRLLss If a document loaded by the browser contains any inline objects (images, applets, etc.) on the same server, they are requested for in separate requests. Those requests are so-called self-referrers, because they have the own hostname in the referrer URL. If configured correctly, hhttttpp--aannaallyyzzee separates all self-referrer URLs from the rest of the external referrer URLs in the statistics report. UUnniiqquuee ssiitteess This is the number of all unique hosts which did access the server during the period of the statistics report. Each different host gets counted only once per period, so this number tells you how many sites did request documents from your server per month. SSeessssiioonnss (color key: red) Similar to unique sites, this is the number of unique hosts which did access the server during a given _t_i_m_e-_w_i_n_d_o_w, which defaults to one day for backward compatibility. This number therefore reflects the different sites per day if the time-window hasn't be changed with the option --uu or the SSeessssiioonn directive in the configuration file. You can increase or decrease the time-window used to calculate sessions. For example, if you set a time-window of two hours, all accesses from the same host in less than 2 hours after it's first access are lumped together into one session. Any access more than 2 hours later will be counted as a new session. RReeqquueesstt MMeetthhoodd The browser uses a certain method to request a document from a web server. For example, documents, images, applets, etc. are usually requested using the GGEETT method. Other often used methods are the HHEEAADD method to request information about a document such as it's size without have the server send it's actual content, and the PPOOSSTT method, a special way to transfer user input from forms into CGI scripts. Although all logfile entries with a valid request method are accounted for as hits, only URLs requested using either the GGEETT or the PPOOSSTT method are processed further. The remaining hits are summarized under _R_e_q_u_e_s_t _M_e_t_h_o_d_s _o_t_h_e_r _t_h_a_n _G_E_T/_P_O_S_T. RReessppoonnssee CCooddeess In reply of a request from a browser, the server sends back a status code such as a Code 200 (_O_K) or Code 404 (_N_o_t _F_o_u_n_d) response. Similar to the request methods, the analyzer will account any valid response code as a hit, but it will only process those URLs, which Page 9 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) did cause a Code 200 (_O_K), Code 304 (_N_o_t _M_o_d_i_f_i_e_d), or Code 404 (_N_o_t _F_o_u_n_d) response from the server. All other responses are summarized in the monthly summary page under _O_t_h_e_r _R_e_s_p_o_n_s_e _C_o_d_e_s. See the HTML specification at _h_t_t_p://_w_w_w._w_3._o_r_g/ for information about all valid response codes. hhttttpp--aannaallyyzzee recognizes HTTP/1.1 responses according to RFC2068. WWhhaatt tthhee rreeppoorrtt ddooeess nnoott sshhooww ...... Due to the nature of the HTTP protocol used for communication between the browser and the server and due to the type of information available in the server's logfile, the analyzer can nnoott: o identify a person as a visitor of your server, o count the number of visitors of your server, o track the way a visitor takes through your site, o measure the time a visitor sees a page of your server, o inform you about the sudden death of the visitor while looking at your homepage, o nor show any other information not in the server's logfile. Even if you classify certain URLs as pageviews or use a specific time- window to count sessions, this does in no way tell you anything about the number of visitors of your server. However, if you use an appropriate server structure with files grouped by type or if you use the HHiiddeeUURRLL directive to group unstructered files together, the statistics report can show a trend or a tendency. Following the numbers for some time, you soon get a feeling which documents are most interesting for the visitors of your site. OOOOUUUUTTTTPPPPUUUUTTTT FFFFIIIILLLLEEEESSSS A statistics report is created in the output directory specified at invocation of hhttttpp--aannaallyyzzee or in the current directory if no output directory is given. Starting with version 2.0, all output files are placed into separate subdirectories to reduce the number of directory entries per report. Those subdirectories are named wwwwww_Y_Y_Y_Y, where _Y_Y_Y_Y is the year of the period covered by the report. This ensures century compliance for the latest version of hhttttpp--aannaallyyzzee and also makes older (non-compliant) files from the 1.9e version fully Year 2000 compliant without having to re-generate the old statistics. Of course, all HTML output files created by hhttttpp--aannaallyyzzee are HTML 3.2 compliant and have been validated using _w_e_b_l_i_n_t. The analyzer can be instructed to place files with "private" data such as overviews and detailed lists of files, hosts, browser types, and referrer URLs in a separate ("private") subdirectory. The web server then can be configured to request authentication for access of files in this directory (see the option --pp and the PPrriivvaatteeDDiirr directive in the configuration file). NNoottee:: for protection of the whole report, you would configure your web server to request authentication for any file in the Page 10 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) statistics output directory. A "private" area is needed only if you want to secure certain lists, while granting access to the rest of the statistics report. The following list shows all files created for a full statistics report: iinnddeexx..hhttmmll is the main page for a given year and contains the total number of _h_i_t_s, _f_i_l_e_s, _p_a_g_e_v_i_e_w_s, _s_e_s_s_i_o_n_s and _d_a_t_a _s_e_n_t per month in tabular and graphical form for the last 12 months. At the end of the year, this file reflects the values for the whole year, while the values for the last 12 months will be written into another index file in a new directory wwwwww_Y_Y_Y_Y. This page is displayed in the _M_a_i_n _w_i_n_d_o_w. ssttaattss_M_M_Y_Y..hhttmmll and ttoottaallss_M_M_Y_Y..hhttmmll contain the total summary for the month _M_M of year _Y_Y in tabular form. The file ttoottaallss_M_M_Y_Y..hhttmmll is the frames version of the report in ssttaattss_M_M_Y_Y..hhttmmll. In the conventional interface, this page is displayed in the _M_a_i_n _w_i_n_d_o_w. jjssnnaavv..hhttmmll and nnaavv_M_M_Y_Y..hhttmmll Navigation panels for JavaScript-enabled browsers, shown in the _N_a_v_i_g_a_t_i_o_n _w_i_n_d_o_w. ddaayyss_M_M_Y_Y..hhttmmll contains the number of hits, files, pageviews, sessions and data sent per day for the month _M_M of year _Y_Y. This report is displayed in the _M_a_i_n _w_i_n_d_o_w. aavvllooaadd_M_M_Y_Y..hhttmmll contains a graphical representation of the average hits per weekday/hour and the top seconds, minutes, hours, and days of the current period. Appears in the _M_a_i_n _w_i_n_d_o_w. ccoouunnttrryy_M_M_Y_Y..hhttmmll contains the list of all countries the visitors of your web server came from. This information is determined by analyzing the _t_o_p- _l_e_v_e_l _d_o_m_a_i_n (_T_L_D) of the hostname assigned to a system in the _D_o_m_a_i_n _N_a_m_e _S_y_s_t_e_m (_D_N_S). The country report is displayed in the _M_a_i_n _w_i_n_d_o_w. Note 1: The country list is meaningful only for ISO two-letter domains. All other domains (..ccoomm, ..oorrgg, ..nneett, etc.) are used by organizations world-wide, so they are not assigned a country, but listed literally in the charts. The ISO country code for the U.S. is ..uuss, by the way ... Note 2: If DNS lookups are disabled in your web server or if the system accessing your server has not been assigned a symbolic hostname in the _D_o_m_a_i_n _N_a_m_e _S_y_s_t_e_m for whatever reason, hhttttpp--aannaallyyzzee can not determine the country (domain) a system is located in. All hosts without a hostname registered in the DNS will show up as Page 11 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) _U_n_r_e_s_o_l_v_e_d in the country list. Since some systems are intentionally not registered in the DNS, a percentage of up to 40% for unresolved IP numbers is absolutely normal. 33DDssttaattss_M_M_Y_Y..hhttmmll, 33DDssttaattss_M_M_Y_Y..wwrrll..ggzz, 33DDssttaattss_Y_Y_Y_Y..hhttmmll, 33DDssttaattss_Y_Y_Y_Y..wwrrll..ggzz are pre-requisite files for the 3D statistics models in the _V_i_r_t_u_a_l _R_e_a_l_i_t_y _M_o_d_e_l_i_n_g _L_a_n_g_u_a_g_e (_V_R_M_L). Those models are created if the option --33 is given at invocation of hhttttpp--aannaallyyzzee. To view those models, you need a VRML2.0 compatible plug-in such as the free _C_o_s_m_o_P_l_a_y_e_r from Cosmo Software, which is currently available for Netscape Communicator and MS Internet Explorer. See _h_t_t_p://_c_o_s_m_o._s_g_i._c_o_m/ for more information about Cosmo Software. All 3D models are displayed in the _3_D _w_i_n_d_o_w, so that you can compare them against the graphs in the conventional report. While the monthly models may be displayed separately on any system with a VRML-compliant browser, the yearly model (with all other twelve monthly models embedded in it) is suitable only on a fast graphics workstation due to it's increased complexity. Therefore, if only --33 is given, the yearly model is replaced by a logo which can be displayed again on any system. In case you have a workstation available for display of the model, you can generate a world with all twelve monthly models embedded in it by specifying a prolog file using the option --PP or the VVRRMMLLPPrroolloogg directive in the configuration file (the file 33DDpprroolloogg..wwrrll is provided as an example). The report then will include a button to choose between the workstation ("SGI") and the PC version of the yearly model. ttooppuurrll_M_M_Y_Y..hhttmmll, ttooppddoomm_M_M_Y_Y..hhttmmll, ttooppuuaagg_M_M_Y_Y..hhttmmll, ttoopprreeff_M_M_Y_Y..hhttmmll Those files contain the _T_o_p _T_e_n lists (actually it's _T_o_p _N, where _N is a configurable number) of the files, sites, browser types and referrer URLs. The URLs shown in ttooppuurrll_M_M_Y_Y..hhttmmll are either the real URLs requested by the visitor or an _i_t_e_m (arbitrary text) you choosed to collect certain file names under (see the HHiiddeeUURRLL directive in the configuration file). The domain names shown in ttooppddoomm_M_M_Y_Y..hhttmmll are either the second- level domains of the hosts accessing your server if the DNS name is available or an item you choosed to collect certain hostnames under (see the HHiiddeeSSyyss directive in the configuration file). Unresolved IP numbers show up as _U_n_r_e_s_o_l_v_e_d. The file ttooppuuaagg_M_M_Y_Y..hhttmmll contains a list of all different browser types (_u_s_e_r _a_g_e_n_t_s) which have been used by visitors to access your web site. The browser type is an identification string sent by the browser and logged by the web server. Although the format for this identification string is well-defined, it isn't obeyed by any browser. If possible, hhttttpp--aannaallyyzzee reduces the name of the browser in the Top lists to the browser model including the first digit of it's version number. If it is not possible to determine this information, the full name as sent by the browser is used. The referrer URLs are the URLs of those web pages, which have a link Page 12 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) to a page on your server, and which have been visited by the user just before following the link. Note that if the user did address a document on your server manually in his browser, no referrer URL gets logged. The browser can also choose to not send a referrer URL at all. Entries without a referrer URL appear as _U_n_k_n_o_w_n in the report. The list of referrer URLs is displayed in the _M_a_i_n _w_i_n_d_o_w. ffiilleess_M_M_Y_Y..hhttmmll, ssiitteess_M_M_Y_Y..hhttmmll, aaggeennttss_M_M_Y_Y..hhttmmll, rreeffeerrss_M_M_Y_Y..hhttmmll Those files contain a complete overview of the files, sites, browser types and referrer URLs, similar to the Top N lists. llffiilleess_M_M_Y_Y..hhttmmll, llssiitteess_M_M_Y_Y..hhttmmll, llaaggeennttss_M_M_Y_Y..hhttmmll, llrreeffeerrss_M_M_Y_Y..hhttmmll Those files contain the detailed lists of all files, sites, browser types and referrer URLs, similar to the previous lists, but sorted by item (if any) and hits. On frequently accessed sites those lists can become rather large, so they are shown in the separate _L_i_s_t _w_i_n_d_o_w. rrffiilleess_M_M_Y_Y..hhttmmll contains all invalid URLs which caused the server to respond with a _C_o_d_e _4_0_4 (_N_o_t _f_o_u_n_d) status. If there are large number of hits for certain files the server couldn't find, it's probably due to missing inline images or other HTML objects embedded in other pages. This report is displayed in the _M_a_i_n _w_i_n_d_o_w. rrssiitteess_M_M_Y_Y..hhttmmll contains the list of reverse domains. This report is displayed in the _M_a_i_n _w_i_n_d_o_w. ffrraammeess..hhttmmll, hheeaaddeerr..hhttmmll This two files are required for the frames-based user interface. All other files are shared with the ones for the non-frames UI. In the frames-based UI, the _M_a_i_n window is inside the frame, while the _L_i_s_t window is still an external window. The _3_D _w_i_n_d_o_w may be inside the frame or an external window (see the 33DDWWiinnddooww directive). ggrr--iiccoonn..ggiiff This is a small icon displayed on the main page under the base directory for the statistics report (option --oo or the OOuuttppuuttDDiirr directive in the configuration file). OOOOPPPPTTTTIIIIOOOONNNNSSSS --hh print a short help list explaining the usage of the options. Use --hhhh to print an even more detailed help. --dd (_d_a_i_l_y _m_o_d_e) generate a short statistics report for the current month only. If a history file exists, the values for the previous days will be read from this history file and the corresponding logfile entries are skipped. If the history file does not exist, the whole logfile will be processed and a history file will be created (unless --nn is also given). Page 13 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) --mm (_m_o_n_t_h_l_y _m_o_d_e) generate a full statistics report for a whole month. In this mode, the values from the history file for previous month are used to create a summary page for the last 12 months. However, the logfile entries feed into the analyzer always take preceedence over the records in the history unless the option --ee is given. --VV (_v_e_r_s_i_o_n) print the version of hhttttpp--aannaallyyzzee and exit immediately. --33 create a 3D (VRML) model of the statistics in addition to the regular statistics report. You need a VRML2.0 compliant plug-in such as _C_o_s_m_o_P_l_a_y_e_r from Cosmo Software to view the model. --aa ignore all URLs which required authentication. If your statistics report is available to the public, you probably do not want to have those secret URLs listed in the report. See also the AAuutthhUURRLL directive in the configuration file. --ee use the history file even in full statistics mode. If this option is given and you analyze the logfiles for several months at once (either in different files or in one single logfile), hhttttpp--aannaallyyzzee uses the values recorded in the history file for previous months and skips all logfile entries up to the first day of a month not recorded in the history (usually the current month). This option is useful if, for example, you rotate your logfile once per quarter and want to have the analyzer skip all entries for a previous month which already has been processed before. --ff create also a frames-based user interface for the statistics report (requires JavaScript). --gg (_g_e_n_e_r_i_c _i_n_t_e_r_f_a_c_e) create a conventional (non-frames) user interface for the statistics report without the JavaScript-based navigation window. --nn (_n_o _u_p_d_a_t_e) do not update the history file. Useful to generate statistics for previous months (before the last month) without overwriting the current state of the history. Since the history is used to create the report for the last 12 months, this option must be used to not mess up the actual statistics report when analyzing an older period. --qq do not strip arguments to CGI scripts in URLs. By default, hhttttpp-- aannaallyyzzee strips arguments to CGI scripts from their URLs to be able to lump them together. If your server creates HTML files dynamically through a CGI script, they are reduced to the URL of the script. The option --qq causes the analyzer to leave those argument lists intact. This way, CGI URLs with different arguments are treated as different URLs. Note that this only works for requests passing arguments using the GGEETT method (see the section _I_n_t_e_r_p_r_e_t_a_t_i_o_n _o_f _t_h_e _r_e_s_u_l_t_s for an explanation of the request methods and the SSttrriippCCGGII directive in the configuration file). Page 14 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) --vv (verbose) comment ongoing processing. Warnings are printed only in verbose mode. Use this option to see how hhttttpp--aannaallyyzzee processes the logfile. If --vv is doubled, the analyzer prints a dot for each new day discovered in the logfile. --xx list each image URL literally rather than lumping them together under the item "All images". Without this option, hhttttpp--aannaallyyzzee comprises all images (*._g_i_f, *._j_p_g, *._i_e_f, *._p_c_d, *._r_g_b, *._x_b_m, *._x_p_m, *._x_w_d, *._t_i_f) under the item "All images" to avoid cluttering up the lists with lots of image URLs. If --xx is given, each image URL is listed literally unless matched by an explicit HHiiddeeUURRLL directive in the configuration file. --cc _c_f_g_f_i_l_e use _c_f_g_f_i_l_e as the configuration file. A configuration file allows you to define the behaviour of hhttttpp--aannaallyyzzee and to define the ;look & feel+ of the statistics report. See the section _C_o_n_f_i_g_u_r_a_t_i_o_n _F_i_l_e for a description of possible settings, which are called _d_i_r_e_c_t_i_v_e_s in the following text. --ll _l_i_b_d_i_r use _l_i_b_d_i_r as the central library directory where hhttttpp--aannaallyyzzee looks for the pre-requisite files, buttons, and license information (usually /_u_s_r/_l_o_c_a_l/_l_i_b/_h_t_t_p-_a_n_a_l_y_z_e). --oo _o_u_t_d_i_r use _o_u_t_d_i_r to create the statistics report in. If no output directory is given, the report is created in the current directory. See also the OOuuttppuuttDDiirr directive. --pp _p_r_i_v_d_i_r place the detailed list of files, sites, browsers and referrer URLs into the subdirectory _p_r_i_v_d_i_r. Because _p_r_i_v_d_i_r is created directly under the output directory specified with --oo, it's name may not contain any slashes ('/'). This option is useful to restrict free access to only certain parts of the statistics report. See also the PPrriivvaatteeDDiirr directive. --FF _f_o_r_m_a_t use this logfile format. Valid values for _f_o_r_m_a_t are aauuttoo for auto- sensing the logfile format, ccllff for the _C_o_m_m_o_n _L_o_g_f_i_l_e _F_o_r_m_a_t, or ddllff and eellff for the two supported forms of the _C_o_m_b_i_n_e_d/_E_x_t_e_n_d_e_d _L_o_g_f_i_l_e _F_o_r_m_a_t. See also the section _L_o_g_f_i_l_e _F_o_r_m_a_t_s above. --GG _p_a_t_t_e_r_n,... define additional pageview patterns. All URLs matching one of the _p_a_t_t_e_r_n_s are classified as pageviews (text files). If _p_a_t_t_e_r_n starts (doesn't start) with a slash (/), _i_t _i_s _t_r_e_a_t_e_d _a_s _a _p_r_e_f_i_x (_s_u_f_f_i_x) _e_a_c_h _U_R_L _i_s _c_o_m_p_a_r_e_d _w_i_t_h. _T_h_e _s_u_f_f_i_x ..hhttmmll is pre-defined by default. You can add 9 more patterns here, for example ..sshhttmmll, ..tteexxtt and //ccggii--bbiinn//. To specify more than one suffix with a single --GG option, use commas to separate them. See also the PPaaggeeVViieeww Page 15 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) directive. --HH _i_d_x_f_i_l_e,... define additional directory index filenames. The name _i_n_d_e_x._h_t_m_l is pre-defined by default. hhttttpp--aannaallyyzzee truncates URLs containing an index filename so that they merge with `/' (their "base URL"). For example, /_d_i_r/_i_n_d_e_x._h_t_m_l is truncated to /_d_i_r/. You can add up to 9 more names for directory index files, for example _W_e_l_c_o_m_e._h_t_m_l or _h_o_m_e._h_t_m_l. See also the IInnddeexxFFiilleess directive. --II _d_a_t_e skip all logfile entries until this day (exclusive). The date may be specified as _D_D/_M_M/_Y_Y_Y_Y _o_r _M_M/_Y_Y_Y_Y , where _M_M is the number or the name of a month. Note that in full statistics mode, _D_D defaults to the first day of the month if absent. If you specify any other day in this mode, unpredictable results may occur. For example, -I Feb _r_e_s_t_r_i_c_t_s _t_h_e _a_n_a_l_y_s_i_s _t_o _t_h_e _F_e_b_r_u_a_r_y _o_f _t_h_e _c_u_r_r_e_n_t _y_e_a_r. --EE _d_a_t_e skip all logfile entries starting from this day on (inclusive). The date format is the same as in --II. To restrict analysis to a certain period, specify the starting date using --II and the first date to be ignored using --EE. For example, -I Jan/98 -E Feb/98 restricts the analysis to January 1998. --OO _v_i_r_t_n_a_m_e,... define additional ("virtual") names for this server to be classified as _s_e_l_f-_r_e_f_e_r_r_e_r _U_R_L_s. The server's primary name (from --SS or --UU) is pre-defined already. If _v_i_r_t_n_a_m_e doesn't include a protocol spcifier, two URLs with the http _a_n_d _t_h_e _h_t_t_p_s protocol specifier are added for each name. See also the VViirrttuuaallNNaammeess directive. --PP _p_r_o_l_o_g use _p_r_o_l_o_g as the prolog file for a yearly VRML model (optional). The file 33DDpprroolloogg..wwrrll is included in the distribution as an example. Note that the resulting VRML model for a whole year is suitable only for viewing on a graphic workstation. The monthly VRML models do not need a prolog file and can be viewed on any platform without problems. See also the VVRRMMLLPPrroolloogg directive. --RR _d_o_c_r_o_o_t restrict logfile analysis to the given Document Root. If _d_o_c_r_o_o_t is prefixed by a `!', analysis takes place for all directories except _d_o_c_r_o_o_t. If _d_o_c_r_o_o_t does not start with a slash ('/'), it is interpreted as the name of a virtual server, which is matched against the (normally unused) second field of a logfile entry. Intented for use with (software-) virtual servers with a separate Document Root or for which the hostname is recorded in the second field of a logfile entry. See also the DDooccRRoooott directive in the configuration file. --SS _s_r_v_n_a_m_e Page 16 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) use _s_r_v_n_a_m_e for the server name. If no server name is defined, hhttttpp--aannaallyyzzee uses the hostnamename of the system. The server name must be a full qualified domain name, not an URL. See also the SSeerrvveerrNNaammee directive. --TT _T_L_D_f_i_l_e use _T_L_D_f_i_l_e for the list of valid top-level domains (TLDs). This list currently includes all ISO two-letter country domains, the well-known domains ..nneett, ..iinntt, ..oorrgg, ..ccoomm, ..eedduu, ..ggoovv, ..mmiill, ..aarrppaa, ..nnaattoo, and the new _C_O_R_E top-level domains ..ffiirrmm, ..iinnffoo, ..sshhoopp, ..aarrttss, ..wweebb, ..rreecc, and ..nnoomm. The length of a top-level domain in the TLD file may not exceed 6 characters. If no TLD file is given, hhttttpp--aannaallyyzzee uses it's built-in defaults. See also the TTLLDDFFiillee directive and the sample file TTLLDD included in the distribution. --UU _s_r_v_u_r_l define _s_r_v_u_r_l as the server URL which should be used as a prefix for the hotlinks in the URL list. Useful if the statistics report is created on a different system than the server is running on and for virtual hosts. See also the SSeerrvveerrUURRLL directive. --WW _3_D_w_i_n define the window for the VRML model. The keyword _3_D_w_i_n may be either eexxtteerrnn or iinntteerrnn for display of the VRML model in a new, external window or in the lower half of the main frame respectively (meaningful only in the frames-based interface). --ss _s_u_b_o_p_t,... suppress certain lists in the report. See also the SSuupppprreessss directive. _s_u_b_o_p_t may be: AAVVLLooaadd to suppress the average load report (top seconds/minutes/hours), UURRLLss to suppress the overview and list of URLs/items, UURRLLLLiisstt to suppress the list of URLs/items only, CCooddee440044 to suppress the list of Code 404 (_N_o_t _F_o_u_n_d) responses, SSiitteess to suppress the overview and list of client domains, RRSSiitteess to suppress the overview of reverse client domains, SSiitteeLLiisstt to suppress the list of all client domains/hostnames, AAggeennttss to suppress the overview and list of browser types, RReeffeerrrreerr to suppress the overview and list of referrers URLs, CCoouunnttrryy to suppress the list of countries, PPaaggeevviieewwss to suppress pageview rating (304's are shown instead), AAuutthhRReeqq to suppress requests which required authentication, GGrraapphhiiccss to suppress images such as graphs and pie charts, HHoottlliinnkkss to suppress hotlinks in the list of all URLs, IInntteerrppooll to suppress interpolation of values in graphs. --tt _n_u_m define the size of certain lists. _n_u_m is either a positive number or the value 0 to suppress the corresponding list. You specify the list by appending one of the following characters to the number shown here as '#' (note that the characters are case-sensitive): #UU # is the number of entries in the Top URL list (default: 30), Page 17 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) #LL # is the number of entries in the Least URL list (default: 10). #SS # is the number of entries in the Top domain list (default: 30), #AA # is the number of entries in the Top agent/browser list (default: 30), #RR # is the number of entries in the Top referrer URL list (default: 30), #dd # is the number of entries in the Top days table (default: 7), #hh # is the number of entries in the Top hours table (default: 24), #mm # is the number of entries in the Top minutes table (default: 5), #ss # is the number of entries in the Top seconds table (default: 5), #NN # is the size of the navigation frame (default: 120 pixels) You can specify more than one _n_u_m with a single --tt option by separating them with a `,' as in -t 20U,0L,20S.. SSeeee aallssoo tthhee TToopp** directives in the configuration file. --uu _t_i_m_e define the time-window for counting _s_e_s_s_i_o_n_s. See _S_e_s_s_i_o_n_s in the section _I_n_t_e_r_p_r_e_t_a_t_i_o_n _o_f _t_h_e _r_e_s_u_l_t_s for an explanation of this term. --ww _h_i_t_s set the noise-level to _h_i_t_s. If a noise-level is defined, all URLs, sites, agents and referrer URLs with hits below this level are collected under the item _N_o_i_s_e in the Top N lists and overviews to avoid cluttering up those lists. See also the NNooiisseeLLeevveell directive. _l_o_g_f_i_l_e(_s) This are the name(s) of the logfile(s) to process. If more than one file is given, they are processed in the order in which their names appear on the command line. hhttttpp--aannaallyyzzee checks for the existance of all files before processing them. If a `-' is specified as the filename, standard input is read. If no file is given, the analyzer either processes the default logfile specified in the configuration file or the standard input. CCCCOOOONNNNFFFFIIIIGGGGUUUURRRRAAAATTTTIIIIOOOONNNN FFFFIIIILLLLEEEE The option --cc and the environment variable HHAA__CCOONNFFIIGG allow to define a configuration file which contains server-specific configuration settings for hhttttpp--aannaallyyzzee. However, command line options always take preceedence over the definitions in this configuration file. The configuration file contains a single directive per line. Except for IInnddeexxFFiilleess, PPaaggeeVViieeww, AAddddDDoommaaiinn, VViirrttuuaallNNaammeess, IIggnn**, and HHiiddee**, each directive may appear only once in the configuration file. Following a directive field there are one or two value fields, which must be separated from the directive and each other by one or more tabulators. Blanks are considered a part of the string for the third field only if there is such a field. All directive names are case-insensitive. Comment lines starting with a hash character (#) are ignored. 33DDWWiinnSSiizzee _w_i_d_t_hx_h_e_i_g_h_t Defines the size of the 3D window. Useful for Netscape Page 18 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) Navigator 3.X, which displays scrollbars in the 3D window with standard size (520x420 pixels). Example: 3DWinSize 540x450 33DDWWiinnddooww _k_e_y_w_o_r_d Defines the 3D window the VRML model is displayed in (same as option --WW). The _k_e_y_w_o_r_d may be eexxtteerrnn (default) or iinntteerrnn for display of the VRML model in a new, external window or in the lower half of the main frame respectively. Example: 3DWindow intern AAddddDDoommaaiinn _d_o_m_a_i_n _s_t_r_i_n_g Add entries to the domain table causing certain _d_o_m_a_i_n_s to be allocated to the "mock" domain _s_t_r_i_n_g. Wildcards in _d_o_m_a_i_n are ignored. This directive is useful to collect certain hostnames (for example the hosts of world-wide operating online services), under some _s_t_r_i_n_g (item) instead of the country they seem to originate from. Example: AddDomain .compuserve.com CompuServe AAuutthhUURRLL _b_o_o_l_e_a_n _v_a_l_u_e Defines whether URLs which required authentication are to be skipped. By default, such URLs show up in the report just like all other URLs. Setting AAuutthhUURRLL to _O_f_f, _N_o, _N_o_n_e, _F_a_l_s_e, or _0 causes the analyzer to skip those URLs in the logfile (if your statistics report is available to others, you probably do not want to have secret URLs listed there). Example: AuthURL No CCuussttLLooggooWW _i_m_a_g_e _s_r_v_u_r_l and CCuussttLLooggooBB _i_m_a_g_e _s_r_v_u_r_l Define images for use as customer logos in the statistics report. This feature is available only in the commercial version of the analyzer. _i_m_a_g_e is the name of the image file relative to the output directory OOuuttppuuttDDiirr and _s_r_v_u_r_l is the URL to be followed if the user clicks on the image. To use your own logos create two images - one for use with a white background (CCuussttLLooggooWW) and the other one for use with a black background (CCuussttLLooggooBB). The images should be approximately 72x72 pixels in size and must be placed into the buttons subdirectory of the output directory (_O_u_t_p_u_t_D_i_r/btn). _T_h_e_n _d_e_f_i_n_e _t_h_e _a_p_p_r_o_p_r_i_a_t_e CCuussttLLooggoo _d_i_r_e_c_t_i_v_e_s _a_n_d _g_e_n_e_r_a_t_e _a _n_e_w _r_e_p_o_r_t _w_i_t_h _y_o_u_r _c_o_m_p_a_n_y'_s _l_o_g_o. _E_x_a_m_p_l_e: CustLogoW btn/mycompany_sw.gif http://www.mycompany.com/ CustLogoB btn/mycompany_sb.gif http://www.mycompany.com/ DDeeffaauullttMMooddee _m_o_d_e The default operation mode of hhttttpp--aannaallyyzzee. The value field contains either the keyword ddaaiillyy for short statistics mode or mmoonntthhllyy for full statistics mode (see also options --dd and --mm). If left undefined, the default is full statistics mode (mmoonntthhllyy). Example: DefaultMode daily DDooccRRoooott _d_o_c_r_o_o_t Page 19 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) Restricts logfile analysis to the given Document Root (same as option --RR). If _d_o_c_r_o_o_t is prefixed by a `!', analysis takes place for all directories except _d_o_c_r_o_o_t. If _d_o_c_r_o_o_t does not start with a slash ('/'), it is interpreted as the name of a virtual server, which is matched against the (normally unused) second field of a logfile entry. Intented for use with (software-) virtual servers with a separate Document Root or for which the hostname is recorded in the second field of a logfile entry. Example: DocRoot /customer/ DocRoot www.customer.com HHTTMMLLPPrreeffiixx _p_r_e_f_i_x and HHTTMMLLTTrraaiilleerr _t_r_a_i_l_e_r The HTML _p_r_e_f_i_x and _t_r_a_i_l_e_r to be printed after the header section and at the end of the page. If defined, the HHTTMMLLPPrreeffiixx string must include the <BODY> tag. If a _f_i_l_e_n_a_m_e is given instead of the _p_r_e_f_i_x or _t_r_a_i_l_e_r, the HTML code is taken from this file. Example: HTMLPrefix <BODY BGCOLOR="#FF0000"> HTMLTrailer <A HREF="/intern/">Back</A> to the internal page. HHeeaaddSSiizzee _s_i_z_e, TTeexxttSSiizzee _s_i_z_e, SSmmaallllSSiizzee _s_i_z_e and LLiissttSSiizzee _s_i_z_e The font sizes for headings (navigator default, usually 3), regular text (default: 2), small text (default: 1) and lists (default: 2). TTeexxttSSiizzee replaces the former FFoonnttSSiizzee, which is still recognized. Example: HeadSize 4 TextSize 3 SmallSize 2 HHeeaaddFFoonntt _f_o_n_t_l_i_s_t, TTeexxttFFoonntt _f_o_n_t_l_i_s_t and LLiissttFFoonntt _f_o_n_t_l_i_s_t The fonts to use for headers, for regular text, and for the detailed lists. If unset, the analyzer uses a list of common serif-less fonts for headers and regular text and a monospaced (fixed) font for the detailed lists. To force the navigator's default for fonts, use the keyword ddeeffaauulltt as the fontname. Example: HeadFont Helvetica,Arial,Geneva,sans-serif TextFont Helvetica,Arial,Geneva,sans-serif ListFont Courier,fixed HHiiddeeAAggeenntt _a_g_e_n_t _s_t_r_i_n_g Hide certain browsers under an arbitrary _s_t_r_i_n_g (item). Needed only for a certain browser whose vendor still can't spell it's name correctly. Only the leading part of the browser type is compared against _a_g_e_n_t, so no wildcards are needed in the second field. Example: HideAgent Mozilla/4.0 (compatible; MSIE 4. MSIE 4.* HideAgent Mozilla/3.0 (compatible; MSIE 3. MSIE 3.* HHiiddeeRReeffeerr _r_e_f_e_r_r_e_r _s_t_r_i_n_g Hide certain referrer URLs under an arbitrary _s_t_r_i_n_g (item). Useful to map different referrer URLs for a given host to a common name. Since only the leading string of the referrer URL is compared against _r_e_f_e_r_r_e_r, there is no need to specify wildcards. As in HHiiddeeAAggeenntt, a Page 20 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) wildcard suffix is removed from the string, while a wildcard prefix is taken literal. If the second argument contains a string in square brackets, this defines the CGI parameter which specifies the search key for search engines. In this case, the search key will be extracted from the argument list and prominently displayed after the name of the search engine/web server. See also the file ssaammppllee..ccoonnff included in the distribution for more examples on how to use the HHiiddeeRReeffeerr directive. Example: HideRefer http://altavista.digital.com/ AltaVista [q=] HideRefer http://lycospro.lycos.com/ Lycos [query=] HideRefer http://www.excite.com/ Excite [search=] HideRefer http://www.dino-online.de/ Dino Online [query=] HHiiddeeSSyyss _h_o_s_t_n_a_m_e _s_t_r_i_n_g Hide a _h_o_s_t_n_a_m_e under an arbitrary _s_t_r_i_n_g (item). The string may contain blanks. If the first character of _s_t_r_i_n_g is a `[', _t_h_i_s _i_t_e_m _i_s _s_u_p_p_r_e_s_s_e_d _i_n _t_h_e _T_o_p _N lists. Hidden items are accounted for separately, but in the summary they are collected under the description defined with this directive. You may use the wildcard character `*' as either a prefix or as a suffix of the _h_o_s_t_n_a_m_e (as in **..hhoosstt..ccoomm and 119922..116688..1122..**), bot not as both. Hostnames are case-insensitive. When building the list of countries, hhttttpp--aannaallyyzzee determines the country from the top-level domain given in _h_o_s_t_n_a_m_e. If _h_o_s_t_n_a_m_e is an IP number, you can optionally define the top-level domain it should be accounted for by appending the domain in square brackets to the _s_t_r_i_n_g as shown below. Example: HideSys *.mycompany.com MY COMPANY HideSys 192.168.12.* MY COMPANY [COM] HHiiddeeUURRLL_u_r_l _s_t_r_i_n_g Hide an _U_R_L under an arbitrary _s_t_r_i_n_g (item). The string may contain blanks. If the first character of _s_t_r_i_n_g is a `[', _t_h_i_s _i_t_e_m _i_s _s_u_p_p_r_e_s_s_e_d _i_n _t_h_e _T_o_p _N lists. Hidden items are accounted for separately, but in the summary they are collected under the description defined with this directive. You may use the wildcard character `*' as either a prefix or as a suffix of the _U_R_L (as in **..mmaapp and //ssuubbddiirr//**), bot not as both. URLs are case-sensitive. Note that images are hidden automatically under _A_l_l _i_m_a_g_e_s unless --xx was specified. See the ssaammppllee..ccoonnff file included in the distribution for more examples. Example: HideURL *.map [All image maps] HideURL /robots.txt [Robot control file] HideURL /newsletter/* MyCompany's Monthly Newsletter HideURL /~delta-t/ DELTA-t Homepage IIggnnUURRLL _u_r_l and IIggnnSSyyss _h_o_s_t_n_a_m_e Ignore entries with a specific URL or accesses from a certain system. You may use the wildcard character `*' as either a prefix or as a suffix of the URL or the hostname (as in **..ggiiff, //ssuubbddiirr//ffiillee** and **..hhoosstt..ccoomm), but not as both. Note that all logfile entries are Page 21 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) compared against this list while hhttttpp--aannaallyyzzee reads the logfile opposed to the HHiiddeeUURRLL and HHiiddeeSSyyss directives, which are looked up for when all entries have been reduced to the set of unique URLs and hostnames, respectively. Therefore, many IIggnnUURRLL/IIggnnSSyyss definitions will significantly increase processing time of hhttttpp--aannaallyyzzee. Example: IgnURL *.gif,*.jpg,*.jpeg IInnddeexxFFiilleess _i_d_x_f_i_l_e[,_i_d_x_f_i_l_e...] Define additional directory index filenames (same as option --HH). The name _i_n_d_e_x._h_t_m_l is pre-defined by default. hhttttpp--aannaallyyzzee truncates URLs containing an index filename so that they merge with `/' (their "base URL"). For example, /_d_i_r/_i_n_d_e_x._h_t_m_l is truncated to /_d_i_r/. You can add up to 9 more names for directory index files. Note that each name requires another table lookup, which may significantly increase processing time. Example: IndexFiles Welcome.html,home.html,index.htm LLaanngguuaaggee _l_o_c_a_l_e NNoott aavvaaiillaabbllee yyeett:: Use given message catalogue for the language in the statistics report. By default, the message catalogue selected by the current locale is used. This directive may be used to overwrite the locale used by hhttttpp--aannaallyyzzee to find the correct message catalogue. LLooggFFiillee _f_i_l_e_n_a_m_e The name of the server's logfile. If you define a default name for the logfile, this file is processed if no other filenames are explicitely specified on the command line. Without such a definition, hhttttpp--aannaallyyzzee always reads _s_t_d_i_n if no other filename is given. Example: LogFile /usr/ns-home/www/logs/access LLooggFFoorrmmaatt _f_o_r_m_a_t use this logfile format. Valid values for _f_o_r_m_a_t are aauuttoo for auto- sensing the logfile format, ccllff for the _C_o_m_m_o_n _L_o_g_f_i_l_e _F_o_r_m_a_t, or ddllff and eellff for the two supported forms of the _C_o_m_b_i_n_e_d/_E_x_t_e_n_d_e_d _L_o_g_f_i_l_e _F_o_r_m_a_t. See the section _L_o_g_f_i_l_e _F_o_r_m_a_t_s above for a description of the formats supported by hhttttpp--aannaallyyzzee. Example: LogFormat clf NNaavvWWiinnSSiizzee _w_i_d_t_hx_h_e_i_g_h_t Defines the size of the navigation window which pops up in the conventional interface if JavaScript is enabled. Useful if the browser displays scrollbars when the default size of 420x190 is used. Example: NavWinSize 440x200 NNaavviiggFFrraammee _s_i_z_e Defines the size of the navigation frame in pixels. Useful if the browser displays scrollbars when the default size of 120 pixels is used. Example: Page 22 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) NavigFrame 140 NNooiisseeLLeevveell _h_i_t_s set the noise-level to _h_i_t_s. If a noise-level is defined, all URLs, sites, agents and referrer URLs with hits below this level are collected under the item _N_o_i_s_e in the Top N lists and overviews to avoid cluttering up those lists. Example: NoiseLevel 7 OOuuttppuuttDDiirr _d_i_r_e_c_t_o_r_y The name of the directory where the output files should be created (same as option --oo). If left undefined, output files are created in the current directory. Example: OutputDir /usr/www/htdocs/stats PPaaggeeVViieeww _p_a_t_t_e_r_n[,_p_a_t_t_e_r_n...] define additional pageview patterns (same as option --GG). All URLs matching one of the _p_a_t_t_e_r_n_s are classified as pageviews (text files). If _p_a_t_t_e_r_n starts (doesn't start) with a slash (/), _i_t _i_s _t_r_e_a_t_e_d _a_s _a _p_r_e_f_i_x (_s_u_f_f_i_x) _e_a_c_h _U_R_L _i_s _c_o_m_p_a_r_e_d _w_i_t_h. _T_h_e _s_u_f_f_i_x ..hhttmmll is pre-defined by default. You can add 9 more patterns here, for example ..sshhttmmll, ..tteexxtt and //ccggii--bbiinn//. Note that each pattern requires another table lookup, which may significantly increase processing time. Example: PageView .shtml,.text,/cgi-bin/ PPrriivvaatteeDDiirr _p_r_i_v_d_i_r The name of a private directory where the detailed lists of files, sites, browsers, and referrer URLs should be created (same as option --pp). Because _p_r_i_v_d_i_r is created directly under the output directory specified with --oo, it's name may not contain any slashes ('/'). This option is useful to restrict free access to certain parts of the statistics report only: Instead of securing the whole statistics report, you can have certain lists separated from the rest of the report and then have the server request authentication for access of this lists. Example: PrivateDir lists RReeggIInnffoo _c_u_s_t_o_m_e_r__n_a_m_e _r_e_g_i_s_t_r_a_t_i_o_n__I_D Defines the customer's name and the registration ID, which are both shown on the main page in the summary report. Example: RegInfo MyCompany 3745JMJZ00000311300000682344 RReeppoorrttTTiittllee _t_i_t_l_e The document title to use in the statistics report. Example: ReportTitle Access Statistics for MyCompany SSeerrvveerrNNaammee _s_r_v_n_a_m_e The official name of the server (same as option --SS). If no server name is defined, hhttttpp--aannaallyyzzee uses the hostnamename of the system. The server name must be a full qualified domain name, not an URL. Example: Page 23 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) ServerName www.mycompany.com SSeerrvveerrUURRLL _s_r_v_u_r_l The URL of the server to be used for hotlinks in URL lists (same as option --UU). Useful if the report for your web server is published on another server, for example on an internal developement machine. Also necessary for (software-) virtual servers to have hhttttpp--aannaallyyzzee generate correct hypertext links in the report. Example: ServerURL http://www.mycompany.com SSeessssiioonn _t_i_m_e The time-window for counting _s_e_s_s_i_o_n_s. All unique hosts accessing your server more than once inside this time-window, are accounted for as the same session. If the distance between two adjacend accesses from the same host is greater than the time-window, the accesses from this host are accounted for as different sessions. Example: Session 4 hours SSttrriippCCGGII _b_o_o_l_e_a_n _v_a_l_u_e Defines the handling of arguments to CGI scripts in URLs. By default, hhttttpp--aannaallyyzzee strips arguments to CGI scripts from their URLs to be able to lump them together. If your server creates HTML files dynamically through a CGI script, they are reduced to the URL of this script. Setting SSttrriippCCGGII to _O_f_f, _N_o, _N_o_n_e, _F_a_l_s_e, or _0 causes the analyzer to leave those argument lists intact. This way, CGI URLs with different arguments are treated as different URLs. Note that this only works for requests passing arguments using the GGEETT method (see _I_n_t_e_r_p_r_e_t_a_t_i_o_n _o_f _t_h_e _r_e_s_u_l_t_s for information about request methods). Example: StripCGI No SSuupppprreessss _s_u_b_o_p_t,... Suppress certain lists in the report (same as --ss). _s_u_b_o_p_t may be one of: AAVVLLooaadd to suppress the average load report (top seconds/minutes/hours), UURRLLss to suppress the overview and list of URLs/items, UURRLLLLiisstt to suppress the list of URLs/items only, CCooddee440044 to suppress the list of Code 404 (_N_o_t _F_o_u_n_d) responses, SSiitteess to suppress the overview and list of client domains, RRSSiitteess to suppress the overview of reverse client domains, SSiitteeLLiisstt to suppress the list of all client domains/hostnames, AAggeennttss to suppress the overview and list of browser types, RReeffeerrrreerr to suppress the overview and list of referrers URLs, CCoouunnttrryy to suppress the list of countries, PPaaggeevviieewwss to suppress pageview rating (304's are shown instead), AAuutthhRReeqq to suppress requests which required authentication, GGrraapphhiiccss to suppress images such as graphs and pie charts, HHoottlliinnkkss to suppress hotlinks in the list of all URLs, IInntteerrppooll to suppress interpolation of values in graphs. Example: Suppress Country,Interpol Page 24 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) TTLLDDFFiillee _f_i_l_e_n_a_m_e use _f_i_l_e_n_a_m_e for the list of top-level domains (same as option --TT). This list includes all ISO two-letter country domains, the well-known domains ..nneett, ..iinntt, ..oorrgg, ..ccoomm, ..eedduu, ..ggoovv, ..mmiill, ..aarrppaa, ..nnaattoo, and the new _C_O_R_E top-level domains ..ffiirrmm, ..iinnffoo, ..sshhoopp, ..aarrttss, ..wweebb, ..rreecc, and ..nnoomm. The length of a domain in the TLD file may not exceed 6 characters. hhttttpp--aannaallyyzzee uses it's built-in defaults, if no TLD file is given. Example: TLDFile /usr/local/lib/http-analyze/TLD TToopp{DDaayyss,,HHoouurrss,,MMiinnuutteess,,SSeeccoonnddss,,UURRLLss,,SSiitteess,,AAggeennttss,,RReeffeerrss}, LLeeaassttUURRLLss Defines the size of certain Top N tables and lists. If set to zero, the corresponding list will be suppressed. Example: TopURLs 20 LeastURLs 0 TopDays 14 VViirrttuuaallNNaammeess _v_i_r_t_n_a_m_e,... The list of additional ("virtual") names for this server to be classified as _s_e_l_f-_r_e_f_e_r_r_e_r _U_R_L_s. The server's primary name (from SSeerrvveerrNNaammee or SSeerrvveerrUURRLL) is pre-defined already. If _v_i_r_t_n_a_m_e doesn't include a protocol specifier, two URLs with the http _a_n_d _t_h_e _h_t_t_p_s protocol specifier are added for each name. Since self-referrers are suppressed from the list of referrer URLs, the remaining entries give a good impression about external pages referring to some document on your site. Example: VirtualNames www2.mycompany.com,mycompany.com VirtualNames www.customer.com,customer.com VirtualNames http://www.other.com,https://secure.other.com VVRRMMLLPPrroolloogg _f_i_l_e The name of a prolog file for a yearly VRML model (same as option --PP). Pathnames not beginning with a `/' are relative to OOuuttppuuttDDiirr. If a prolog file is given, an additional yearly model with all 12 monthly models embedded as inlines is created. This model may be displayed only on graphics workstation. See the section _O_u_t_p_u_t _f_i_l_e_s for further information about this yearly model. Example: VRMLProlog 3Dprolog.wrl EEEEXXXXAAAAMMMMPPPPLLLLEEEESSSS After successful compilation of hhttttpp--aannaallyyzzee you can create a statistics report before you choose to install the program permanently. To do so, create a subdirectory for the output files to avoid cluttering up the directory and install the required files using the hhaa--sseettuupp utility: http-analyze setup ------------------ 1) Set up an analyzer configuration for a virtual web server 2) Install the required files in a statistics output directory 3) Brand your copy of http-analyze with the registration ID 4) Exit Page 25 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) Please select a function (1-4) [1]: 2 Install required files for http-analyze --------------------------------------- This script copies the required files (3D*, btn/*) into the statistics ... Name of the HTML output directory: testd Directory testd doesn't exist, create it (y/n) [y]: <RETURN> Now enter the name of the directory containing the required files. ... Directory containing required files (3D*, btn/*) [files]: <RETURN> Required files have been copied into testd Then, run the analyzer on your web server's logfile. For example, if the name of the logfile is /_u_s_r/_n_s-_h_o_m_e/_w_w_w/_l_o_g_s/_a_c_c_e_s_s, use the following command to create a full statistics including a frames-based interface and a 3D (VRML) model in the newly created directory tteessttdd: $ http-analyze -vm3f -o testd /usr/ns-home/www/logs/access http-analyze 2.2 (IP22; IRIX 6.2), Copyright 1998 RENT-A-GURU(TM) Generating full statistics in output directory `testd' Reading data from `/usr/ns-home/www/logs/access' Best blocksize for I/O is 64 KB Hmm, looks like Extended Logfile Format (ELF) Start new period at 01/Sep/1998 Creating VRML model for September 1998 Creating full statistics for September 1998 ... processing URLs ... processing hostnames ... processing user agents ... processing referrer URLs Statistics complete until 30/Sep/1998 $ After the analyzer terminates, start your browser and open the file tteessttdd//iinnddeexx..hhttmmll. To permanently install the program, issue a make install which copies the required files in the appropriate places. To set up an analyzer configuration for a web server, choose an output directory for the statistics report and use the hhaa--sseettuupp utility to install the required files there. Following are some more examples, which assume that the analyzer has been installed permanently. The first command processes an archived logfile _l_o_g_Y_Y_Y_Y/_a_c_c_e_s_s._M_M from the server's log directory to create a report for January 1998 in the directory //uussrr//hhttddooccss//ssttaattss: $ cd /usr/ns-home/www/logs $ http-analyze -vm3f -o /usr/htdocs/stats log1998/access.01 Page 26 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) The next command reads the logfile entries from a pipeline and creates the statistics report for a whole year using a customized configuration file: $ gzcat log1997/access.[01]?.gz | > http-analyze -c /usr/httpd/analyze.conf - RRRREEEEGGGGUUUULLLLAAAARRRR IIIINNNNVVVVOOOOCCCCAAAATTTTIIIIOOOONNNN VVVVIIIIAAAA CCCCRRRROOOONNNN To have statistics generated on a regular base, use the following scheme: 1) Optionally install a cron job which calls hhttttpp--aannaallyyzzee --dd frequently to create a short statistics report. The execution interval may range from once per day up to twice per hour depending on the size of your logfile and the time needed to analyze it. On our server, we run the daily statistics once per hour. 2) Install a cron job which calls hhttttpp--aannaallyyzzee --mm to create a full statistics report once per week or once per day (again depending on the size of your logfile). Note that the full statistics report is created for the first time at the second day of a new month. On our server, we create a monthly summary two times per day. 3) Create a script which rotates the server's logfile, restarts the http server, and then creates the final summary for this period. Have _c_r_o_n execute this script at 00:00 on the ffiirrsstt ddaayy of a new month. See the script rroottaattee--hhttttppdd for an example on how to do this for several virtual web servers running on the same machine. 4) Because of _c_r_o_n's scheduling overhead and delays in execution of the script which rotates the logfile, heavy used servers sometimes writes a few entries for the new month in the old logfile. hhttttpp--aannaallyyzzee usually detects and ignores such "white noise" at the end of a month. However, to get correct figures, in this last step you should run hhttttpp--aannaallyyzzee --mm on the logfile for the current month immediately after generating the statistics for the previous month. Note that the cron jobs must run with the user ID of the owner of the directory where the HTML output files are to be created, except for rroottaattee--hhttttppdd, which must run with the user ID of the server user. You should also take care to avoid running more than one hhttttpp--aannaallyyzzee processes at the same time. Here are some sample _c_r_o_n_t_a_b(1) entries for the scheme described above: # Generate a full report twice per day at 01:17 and 13:17 17 1,13 * * * /usr/local/bin/http-analyze -m -c /usr/httpd/analyze.conf # Generate a short summary each hour except at 01:17 or 13:17 17 2-12 * * * /usr/local/bin/http-analyze -d -c /usr/httpd/analyze.conf 17 14-23 * * * /usr/local/bin/http-analyze -d -c /usr/httpd/analyze.conf Page 27 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) # Rotate the HTTPD logfiles at the first day of a new month at 00:00 0 0 1 * * /usr/local/bin/rotate-httpd TTTTRRRROOOOUUUUBBBBLLLLEEEESSSSHHHHOOOOOOOOTTTTIIIINNNNGGGG If you discover any problems using the analyzer you may find the verbose mode helpful. Each --vv option increases the verbosity level. In verbosity level 1, hhttttpp--aannaallyyzzee comments ongoing processing; in level 2 it indicates progress by printing a dot for each new day discovered in the logfile. In level 3, a debug message for each logfile entry parsed successfully is printed and in level 4 an even more detailed message appears on standard error. Furthermore, compiling hhttttpp--aannaallyyzzee without the macro _N_D_E_B_U_G includes various assertion checks in the executable. $ http-analyze -vvvm3f -o testd log1998/access.08 http-analyze 2.2 (IP22; IRIX 6.2), Copyright 1998 RENT-A-GURU(TM) Generating full statistics in output directory `testd' Reading data from `log1998/access.08' Best blocksize for I/O is 64 KB Hmm, looks like Extended Logfile Format (ELF) 1 01/Aug/1998:00:02:14 [262929738], req=/stats/, sz=2656 <- Code 200 OK Start new period at 01/Aug/1998 2 01/Aug/1998:00:02:17 [262929741], req=/logo.gif, sz=5880 <- Code 200 OK 3 01/Aug/1998:00:02:17 [262929741], req=/btns.gif, sz=4713 <- Code 200 OK ... RRRREEEEGGGGIIIISSSSTTTTRRRRAAAATTTTIIIIOOOONNNN The distribution of hhttttpp--aannaallyyzzee on our web site is made available to you for evaluation purposes only. In this version an "unregistered" button will show up in the statistics report. To replace this button with the Netstore logo of the free version (for personal and educational use), just click on this "unregistered" button to follow the link to our registration form on our web site and register for a free, non-commercial version. NNNNOOOONNNN----CCCCOOOOMMMMMMMMEEEERRRRCCCCIIIIAAAALLLL VVVVEEEERRRRSSSSIIIIOOOONNNN After registration you will receive a registration ID and two registration images as replacements for the "unregistered" buttons by email. In the private version, the Netstore logo, a Copyright note and a link to the homepage of hhttttpp--aannaallyyzzee appears in the statistics report, which must be left intact according to the license, under which this software is made available to you. CCCCOOOOMMMMMMMMEEEERRRRCCCCIIIIAAAALLLL VVVVEEEERRRRSSSSIIIIOOOONNNN If you use hhttttpp--aannaallyyzzee for commercial purposes such as providing statistics services for your customers, you must buy a _C_o_m_m_e_r_c_i_a_l _S_e_r_v_i_c_e _L_i_c_e_n_s_e available from RENT-A-GURU(R) and authorized resellers. You will receive a registration ID and two registration images as replacements for the "unregistered" buttons by email from our office. In the commercial version, the Netstore logo, the Copyright note and the Page 28 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) link to the homepage of hhttttpp--aannaallyyzzee are supressed from the statistics report (except for the logo and Copyright, which appears only once on the main page and inside the navigation frame). You can also add your Company's logo to the report using the CCuussttLLooggooWW and CCuussttLLooggooBB directives in the configuration file, which are enabled by branding the software. Except for this feature and the individual support for users of a commercial license, both versions have identical functionality. BBBBRRRRAAAANNNNDDDDIIIINNNNGGGG TTTTHHHHEEEE SSSSOOOOFFFFTTTTWWWWAAAARRRREEEE For all license types, you have to brand your copy of hhttttpp--aannaallyyzzee with the registration ID and the registration images. The registration ID may be set either in a system-wide file (usually /_u_s_r/_l_o_c_a_l/_l_i_b/_h_t_t_p- _a_n_a_l_y_z_e/_R_E_G_I_D) or via the RReeggIInnffoo directives in an analyzer configuration file. The latter method requires specification of the configuration file each time hhttttpp--aannaallyyzzee is invoked. If you create a system-wide registration file, you need to brand the software only once. To do so, issue the following commands as root (if you can't become root, use another directory you can write to and set the environment variable HHAA__LLIIBBDDIIRR to it's name): # mkdir _l_i_b_d_i_r # http-analyze -r "_C_u_s_t_o_m_e_r _N_a_m_e" _r_e_g_I_D Registration information saved in file `_l_i_b_d_i_r/REGID' # where _l_i_b_d_i_r is the library directory, _C_u_s_t_o_m_e_r _N_a_m_e is the name of the organization this license is registered for and _r_e_g_I_D is the registration ID assigned to the license. Next, install the two registration images we sent you by email into the appropriate buttons subdirectory: o If you use hhttttpp--aannaallyyzzee for only one web server, copy the registration images into the buttons subdirectory (bbttnn) of the corresponding statistics output directory (OOuuttppuuttDDiirr). o If you analyze several virtual servers on the same platform, install the registration images in the buttons subdirectory (usually /_u_s_r/_l_o_c_a_l/_l_i_b/_h_t_t_p-_a_n_a_l_y_z_e/_b_t_n), which should have been created during the installation process. Then, you can easily install the required files and buttons using the hhaa--sseettuupp utility by copying or linking them into the several OOuuttppuuttDDiirr subdirectories for the virtual web servers. After installing the buttons you have completed the registration. Now run the analyzer to create the statistics with the registered version. Page 29 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) YYYYEEEEAAAARRRR 2222000000000000 CCCCOOOOMMMMPPPPLLLLIIIIAAAANNNNCCCCEEEE Versions 2.0 and above of hhttttpp--aannaallyyzzee are fully Year 2000 compliant. There are no problems with date-related functions after the year 1999. Year 2000 compliant means, that the software does not produce errors in date-related data or calculations or experience loss of functionality as a result of the transition to the year 2000. This Year 2000 compliance statement is not a product warranty. The hhttttpp--aannaallyyzzee software is provided under the terms of the license agreement included in each distribution. DDDDAAAATTTTEEEE UUUUSSSSAAAAGGGGEEEE IIIINNNN HHHHTTTTTTTTPPPP----AAAANNNNAAAALLLLYYYYZZZZEEEE The analyzer depends on the timestamp found in the logfile of the web server. A Year 2000 compliant date format was choosen for the _C_o_m_m_o_n and _E_x_t_e_n_d_e_d _L_o_g_f_i_l_e _F_o_r_m_a_t_s from the very beginning on. This unique date format is - and ever was - required by hhttttpp--aannaallyyzzee to be able to generate a statistics report, so there are no problems unless those caused by your OS (see below). Although hhttttpp--aannaallyyzzee 22..XX generates two-digit years in some output filenames to retain compatibility with previous versions of the log analyzer, those files are placed in a subdirectory containing the year in four digits, which make aallll output filenames - even those generated by older versions of the log analyzer - fully Year 2000 compliant. This way, statistics reports generated by the 1.9e version of the analyzer, which originally were not Year 2000 compliant, will become compliant during the upgrade to version 2.X automatically - without re-running the statistics with the original logfiles! The date format in the --II and --EE options allows a year to be specified with two digits only. hhttttpp--aannaallyyzzee interprets values greater than 70 in 1900 and values lower than 70 in 2000. This way, the analyzer covers the whole range of the time representation in modern Operating Systems. However, any other date can be specified unambiguously by using four digits for the year. DDDDAAAATTTTEEEE UUUUSSSSAAAAGGGGEEEE IIIINNNN TTTTHHHHEEEE OOOOPPPPEEEERRRRAAAATTTTIIIINNNNGGGG SSSSYYYYSSSSTTTTEEEEMMMM Actually, there is a date-related function in modern operating systems, which may cause problems after the year 2037. For those interested in the technical details, here's why: In operating systems the date is often represented in seconds since a certain date. For example, in Unix systems the date is represented as seconds since the birth of the OS at January, 1st 1970. This value is stored in a _s_i_g_n_e_d _l_o_n_g (4-byte) data object, so it can represent as much as 2147483648 seconds, which equals 35791394 minutes = 596523 hours = 24855 days = 68 years. Therefore, most clocks in traditional Unix systems will overflow at January, 1st 2038 if the OS is not updated before this date. It has been reported that a certain version of Windows doesn't recognize the Year 2000 as a leap year. Although hhttttpp--aannaallyyzzee computes leap years Page 30 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) for itself, it maps dates into weekdays using the _l_o_c_a_l_t_i_m_e function, which may work correctly only if the OS itself knows about Year 2000 being a leap year. Since hhttttpp--aannaallyyzzee uses several data structures depending on the operating system's idea of the time (for example, the _t_m__y_e_a_r variable contains the years since 1900), the software has to be updated also before the year 2038 in order to take advantage of the time representation in future OS'es. CCCCOOOOPPPPYYYYRRRRIIIIGGGGHHHHTTTT Copyright (C) 1996-1998 by Stefan Stapelberg, RENT-A-GURU(R), <stefan@rent-a-guru.de> Please see the file LLIICCEENNSSEE included in the distribution for the license terms under which this program is made available to you in the free, non-commercial version. RENT-A-GURU(R) is a registered trademark of Martin Weitzel, Stefan Stapelberg, and Walter Mecky. Netstore(R) is a registered trademark of Stefan Stapelberg. CCCCRRRREEEEDDDDIIIITTTTSSSS Thanks to the numeruous users of hhttttpp--aannaallyyzzee for their valuable feedback. Special thanks to Lars-Owe Ivarsson forr his suggestions to optimize the parser algorithm and the code he provided as an example. Special thanks also to Thomas Boutell (_h_t_t_p://_w_w_w._b_o_u_t_e_l_l._c_o_m/) for his great GD library for fast GIF creation, without hhttttpp--aannaallyyzzee couldn't produce such fancy graphics in the statistics report. _g_d _1._2 _i_s _c_o_p_y_r_i_g_h_t _1_9_9_4, _1_9_9_5, _Q_u_e_s_t _P_r_o_t_e_i_n _D_a_t_a_b_a_s_e _C_e_n_t_e_r, _C_o_l_d _S_p_r_i_n_g _H_a_r_b_o_r _L_a_b_s. _P_e_r_m_i_s_s_i_o_n _g_r_a_n_t_e_d _t_o _c_o_p_y _a_n_d _d_i_s_t_r_i_b_u_t_e _t_h_i_s _w_o_r_k _p_r_o_v_i_d_e_d _t_h_a_t _t_h_i_s _n_o_t_i_c_e _r_e_m_a_i_n_s _i_n_t_a_c_t. _C_r_e_d_i_t _f_o_r _t_h_e _l_i_b_r_a_r_y _m_u_s_t _b_e _g_i_v_e_n _t_o _t_h_e _Q_u_e_s_t _P_r_o_t_e_i_n _D_a_t_a_b_a_s_e _C_e_n_t_e_r, _C_o_l_d _S_p_r_i_n_g _H_a_r_b_o_r _L_a_b_s, _i_n _a_l_l _d_e_r_i_v_e_d _w_o_r_k_s. EEEENNNNVVVVIIIIRRRROOOONNNNMMMMEEEENNNNTTTT VVVVAAAARRRRIIIIAAAABBBBLLLLEEEESSSS Environment variables might work only in the Unix version of hhttttpp-- aannaallyyzzee. HHAA__LLIIBBDDIIRR name of the library directory (default: /usr/local/lib/http-analyze)) HHAA__CCOONNFFIIGG nnaammee ooff tthhee ccoonnffiigguurraattiioonn ffiillee ffoorr hhttttpp--aannaallyyzzee ((nnoo ddeeffaauulltt)) FFFFIIIILLLLEEEESSSS This section lists all files required by hhttttpp--aannaallyyzzee to create a statistics report. Those files are usually installed in the library directory as defined by the environment variable HHAA__LLIIBBDDIIRR or the hard- coded default (usually /_u_s_r/_l_o_c_a_l/_l_i_b/_h_t_t_p-_a_n_a_l_y_z_e) defined at compile- time. See also the section _S_t_a_t_i_s_t_i_c_s _R_e_p_o_r_t above for the names of the HTML output files. _b_t_n/*._g_i_f Buttons and icons used in HTML output files _T_L_D List of all top-level-domains _h_a_2._0_*._g_i_f hhttttpp--aannaallyyzzee logos for your web site (black/white bg) _l_o_g_f_m_t.[_c_d_e]_l_f Sample logfiles in CLF, DLF and ELF format Page 31 (printed 9/15/98) hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) UUUUnnnniiiixxxx CCCCoooommmmmmmmaaaannnndddd RRRReeeeffffeeeerrrreeeennnncccceeee hhhhttttttttpppp----aaaannnnaaaallllyyyyzzzzeeee((((1111)))) _3_D_p_r_o_l_o_g._w_r_l Prolog file for the yearly VRML model on SGI workstations _3_D_s_h_e_l_f_M_o_t_i_o_n._w_a_v Sound file for the yearly model on SGI workstations _3_D_l_o_g_o._w_r_l._g_z A stubs file for the yearly VRML model on PCs SSSSEEEEEEEE AAAALLLLSSSSOOOO _r_o_t_a_t_e-_h_t_t_p_d Script to rotate the web server's logfiles _h_a-_s_e_t_u_p Script to set up the analyzer configuration for a web server _c_v_t__f_i_l_e_s Script to convert older files into new 2.0 directory structure _h_t_t_p://_w_w_w._n_e_t_s_t_o_r_e._d_e/_S_u_p_p_l_y/_h_t_t_p-_a_n_a_l_y_z_e/ Homepage of hhttttpp--aannaallyyzzee NNNNOOOOTTTTEEEESSSS Logfile entries must be sorted in order of ascending date and time. If hhttttpp--aannaallyyzzee detects logfile entries from an older month between newer ones, it prints a warning and skips all entries up to the date of the last entry processed. Page 32 (printed 9/15/98)