Hi there,

We try to generate usage statistics for http://www.xxxx.ch/yyyy/ but the Monthly Summary Table stays empty. Here some details…

I removed all old data and executed webalizer like this:
cat log | webalizer -c /etc/webalizer/www.xxxx.ch/yyyy/webalizer.conf -t "Stats for yyyy - "

Stone Steps Webalizer v3.8.0.4 (Linux 2.6.29.6)

Processed configuration file /etc/webalizer/www.xxxx.ch/yyyy/webalizer.conf
Using database /var/webalizer/www.xxxx.ch/yyyy/webalizer.db
History file not found...
Creating output in /var/apache22/var/apache22/export/webalizer/www.xxxx.ch/yyyy
Hostname for reports is 'www.xxxx.ch'
Using logfile STDIN (clf)
Saving current run data... [11/30/2009 23:50:54]
Saving history information...
=> removed some months here
Generating report for November 2009 (HTML)
Saving current run data... [12/21/2009 23:40:27]
Saving history information...
Generating report for December 2009 (HTML)
Generating summary report (HTML)
Processed 89394 records (6899 ignored) in 0.99 seconds
DNS wait time is 0.00 seconds
Generated reports in 1.38 seconds
Maintenance time is 1.04 seconds
Total run time is 3.41 seconds


After this run I have some files containing a month of data each (usage_200912.html) and a summary file (index.html).

All the monthly files are completely filled with data and look good. The summary page also shows a nice graph containing all data but the table below the graph is empty except for header and footer. This is how the html looks:

<div id="monthly_summary_graph" class="graph_holder" style="width: 967px"><img src="usage.png" alt="Usage summary for www.xxxx.ch" width="967" height="256" ></div>
<p class="note_p">Pages - document requests (e.g. html, asp, txt); Files - successful requests of all types (e.g. html, gif, css); Hits - all requests, including errors;</p>

<!-- Monthly Summary Table -->
<table class="report_table monthly_summary_table">
<thead>
<tr class="table_title_tr"><th colspan="11">Summary by Month</th></tr>
<tr><th rowspan="2" style="text-align: left;">Month</th>
<th colspan="4">Daily Avg</th>
<th colspan="6">Monthly Totals</th></tr>
<tr><th class="hits_th">Hits</th>
<th class="files_th">Files</th>
<th class="pages_th">Pages</th>
<th class="visits_th">Visits</th>
<th class="hosts_th">Hosts</th>
<th class="kbytes_th">KBytes</th>
<th class="visits_th">Visits</th>
<th class="pages_th">Pages</th>
<th class="files_th">Files</th>
<th class="hits_th">Hits</th></tr>
</thead>
<tbody class="summary_data_tbody">
</tbody>
<tbody class="summary_footer_tbody">
<tr class="table_footer_tr"><th colspan="5">Totals</th>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td>
<td>0</td></tr>
</tbody>
</table>

<!-- Page Footer -->
<div class="page_footer_div">
<a href="http://www.stonesteps.ca/webalizer">Stone Steps Webalizer</a> (v3.8.0.4)
</div>


Any ideas why this table stays empty?

thanks a lot
Thomas

Can you post the content of your webalizer.hist file?

Andre

Thanks for your response. For me the .history-file looks good. This is the content:

# cat webalizer.hist
1 2009 525 0 240 181 1 31 520 471
2 2009 669 0 301 231 1 28 663 536
3 2009 678 0 292 233 1 31 673 582
4 2009 499 0 237 173 1 30 490 435
5 2009 476 0 241 165 1 31 469 430
6 2009 533 0 233 183 1 30 513 479
7 2009 511 0 252 177 1 31 502 475
8 2009 515 0 260 178 1 31 508 475
9 2009 631 0 309 217 1 30 615 572
10 2009 737 0 341 251 1 31 711 678
11 2009 561 0 246 193 1 30 546 514
12 2009 412 0 167 125 1 21 406 337

Thomas

There is something we noticed:
The field "Total files" in the monthly reports is always 0. But why? We have .html, .php, .gif and .jpg files in this logfile.

here the complete configuration with comments stripped:


LogType clf
OutputDir /var/apache22/var/apache22/export/webalizer/www.xxxx.ch/yyyy
HistoryName /var/webalizer/www.xxxx.ch/yyyy/webalizer.hist
Incremental yes
IncrementalName /var/webalizer/www.xxxx.ch/yyyy/webalizer.current
ReportTitle Usage Statistics for
HostName www.xxxx.ch
PageType htm*
PageType cgi
PageType php
PageType asp
UseHTTPS no
DNSCache /var/webalizer/dns_cache.db
DNSChildren 0
CountryGraph yes
DailyGraph yes
DailyStats yes
HourlyGraph yes
HourlyStats yes
GraphLegend yes
TopSites 30
TopKSites 10
TopURLs 30
TopKURLs 10
TopReferrers 30
TopAgents 15
TopCountries 10
TopEntry 10
TopExit 10
TopSearch 20
TopUsers 20
HideURL *.gif
HideURL *.GIF
HideURL *.jpg
HideURL *.JPG
HideURL *.png
HideURL *.PNG
HideURL *.ra
HideURL /statistics/*
HideURL *.css
HideURL *.js
GroupAgent MSIE Microsoft Internet Explorer
GroupAgent Mozilla Netscape
GroupAgent Lynx* Lynx
IncludeURL *
SearchEngine yahoo.com p=
SearchEngine altavista.com q=
SearchEngine google.com q=
SearchEngine eureka.com q=
SearchEngine lycos.com query=
SearchEngine hotbot.com MT=
SearchEngine msn.com MT=
SearchEngine infoseek.com qt=
SearchEngine webcrawler searchText=
SearchEngine excite search=
SearchEngine netscape.com search=
SearchEngine mamma.com query=
SearchEngine alltheweb.com query=
SearchEngine northernlight.com qr=
HTMLCssPath /statistics/
DbPath /var/webalizer/www.xxxx.ch/yyyy

A file is anything that is served with the HTTP code 200 or 206. It's strange that you have them all as zeros. Can you post a couple of lines from your log files?

Regarding your original question. Can you try processing files directly and not through the standard input?

Also, when you provide explicitly SearchEngine parameters in your configuration file, you suppress more advanced default configuration, which would report whether people searched for full text, all words, etc. See here for more information:

http://www.stonesteps.ca/projects/webalizer/faq.asp?qid=q20050122-01&topic=webalizer

Andre

Here some log-lines:
(Client IP removed) - - [19/Dec/2009:01:05:13 +0100] "GET /yyyy/cgi-bin/mitteilung_detail.asp?PKey_Mitteilung=816&Typ1&Monat=M%C3%83%C2%A4rz&Jahr2000 HTTP/1.1" 301 420 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)" "-" "- - - CN=-" "zip: -/- (-%)" www.xxxx.ch:80 5952
(Client IP removed) - - [19/Dec/2009:06:54:39 +0100] "GET /yyyy/medienmitteilungen.htm HTTP/1.1" 301 345 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" "-" "- - - CN=-" "zip: -/- (-%)" www.xxxx.ch:80 3065
(Client IP removed) - - [19/Dec/2009:08:55:08 +0100] "GET /yyyy/cgi-bin/mitteilung_detail.asp?PKey_Mitteilung=1121 HTTP/1.1" 301 373 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" "-" "- - - CN=-" "zip: -/- (-%)" www.xxxx.ch:80 2573
(Client IP removed) - - [19/Dec/2009:10:11:11 +0100] "GET /yyyy/cgi-bin/mitteilung_detail.asp?PKey_Mitteilung=816&Typ1&Monat=M%E4rz&Jahr2000 HTTP/1.1" 301 411 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-" "- - - CN=-" "zip: -/- (-%)" www.xxxx.ch:80 2459
(Client IP removed) - - [19/Dec/2009:11:33:42 +0100] "GET /yyyy/cgi-bin/mitteilung_detail.asp?PKey_Mitteilung=2080&Tage=30&Typ=0 HTTP/1.1" 301 395 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-" "- - - CN=-" "zip: -/- (-%)" www.xxxx.ch:80 2956
(Client IP removed) - - [19/Dec/2009:16:28:13 +0100] "GET /yyyy/datenbanken/images/bild2_kl.gif HTTP/1.1" 301 354 "http://www.xxxx.ch/yyyy/de/pub/yyyy/mitteilung_suchen.php?controller=Mitteilung&MitteilungsId=177&navId=yyyy" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; de; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6" "-" "- - - CN=-" "zip: -/- (-%)" www.xxxx.ch:80 3148
(Client IP removed) - - [19/Dec/2009:16:28:13 +0100] "GET /yyyy/datenbanken/images/bild1_kl.gif HTTP/1.1" 301 354 "http://www.xxxx.ch/yyyy/de/pub/yyyy/mitteilung_suchen.php?controller=Mitteilung&MitteilungsId=177&navId=yyyy" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; de; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6" "-" "- - - CN=-" "zip: -/- (-%)" www.xxxx.ch:80 2996
(Client IP removed) - - [19/Dec/2009:18:27:40 +0100] "GET /yyyy/cgi-bin/mitteilung_detail.asp?PKey_Mitteilung=685&Tage=30&Typ=0 HTTP/1.1" 301 394 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" "-" "- - - CN=-" "zip: -/- (-%)" www.xxxx.ch:80 2461
(Client IP removed) - - [19/Dec/2009:19:11:09 +0100] "GET /yyyy HTTP/1.1" 301 322 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" "-" "- - - CN=-" "zip: -/- (-%)" www.xxxx.ch:80 2816
(Client IP removed) - - [19/Dec/2009:19:34:05 +0100] "GET /yyyy/cgi-bin/mitteilung_detail.asp?PKey_Mitteilung=1345&Tage=30&Typ=0 HTTP/1.1" 301 395 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "-" "- - - CN=-" "zip: -/- (-%)" www.xxxx.ch:80 2017

It looks like all the pictures get a 301 result, not 200. So they are not counted by webalizer. This explains why we don't have "files".

Maybe there is a bug in webalizer when files==0. I'll try to inject some pictures with code=200 in this logfile and test again.

I replaced one of these 301 codes from a picture with 200 and now we see the month in the summary table!

So the problem looks clear to me. Webalizer does not generate the table if files==0. We'll try to dig into the code and locate the problem or do you know a workaround?

Possible fix (not yet tested)

--- html_output.cpp-orig 2009-12-23 11:09:08.000000000 +0100
+++ html_output.cpp 2009-12-23 11:09:21.000000000 +0100
@@ -2800,7 +2800,7 @@
iter = state.history.rbegin();
while(iter.more()) {
hptr = &iter.next();
- if(hptr->files==0) continue;
+ if(hptr->hits==0) continue;
days_in_month=(hptr->lday-hptr->fday)+1;
fprintf(out_fp,"<tr><th><a href=\"usage_%04d%02d.%s\">%s %d</a></th>\n", hptr->year, hptr->month, config.html_ext.c_str(), lang_t::s_month[hptr->month-1], hptr->year);
fprintf(out_fp,"<td>%lu</td>\n", hptr->hits/days_in_month);

Yes, that's definitely a bug. There's a similar line in the XML report generator, in case if you want to use those. I will release a fix some time next week. Thanks for your help.

Andre