NOTE:Due to the potentially large volume of WEBLOG data needing to be processed, the default implementation of WEBLOG support pre-summarizes the raw data and places it in the DAY reduction level summarized at the hour.
There are several different formats for WEB logs and IT Service Vision can process most of them without modification.
With ITSV V2.3 and higher you can specify a COLLECTR=WEBLOG and TOOLNM=AUTO in your %CPPROCES macro and ITSV will try to detect the WEB log format. Unfortunately, not all WEB logs can be processed this way as they contain insufficient or no header information. See the bottom of this page for some tips in identifying your log format.
Use the QuickStart Wizard to process your WEB log data for the first time. The batch job, PDB and reports that are generated can be tailored to suit your requirements.
You may want to investigate the following areas as they may help reduce resource consumption (both disk space and CPU):-
Due to the typically large volume of data associated with WEB logs, the detail data is not written to the DETAIL level of the PDB, instead, using exit points it is possible to write out to a SAS data set. This data set is not included as part of the reduction process and will be overwritten when the next batch runs.
Two of the Character-Delimited support exit points are used to output this detail data, both these and other exit points are discussed in Generic Collector Appendix 2: Using Character-Delimited Data.
To keep the detail data you will have to do the following :-
The WEB* tables will require will require their default definitions to be updated to ensure that the data is summarized correctly and the reduction levels and also so that the statistics are correct at the reduction levels.
A variable INTID has to be added to the WEBRES table. The INTID (Interval ID) is a value that represents which interval of any hour (based on duration) that this observation belongs. For example, if we are summarizing to the half hour, then the value of this variable will either be '1' for the first half hour or '2' for the second half hour.
Use the following DDUTL control statements to update your WEBRES table.
set table name=WEBRES; create variable name=INTID interpret=STRING type=CHARACTER label='Interval ID' description='This value represents an interval within an hour.' kept=YES format=$2. length=2 ;
For any other WEB* tables that you have in your PDB you will have to update them accordingly.
Add the variable INTID to all the WEB* tables. Some example DDUTL control statements follow.
set table name=WEBxxx; create variable name=INTID interpret=STRING type=CHARACTER label='Interval ID' description='This value represents an interval within an hour.' kept=YES format=$2. length=2 ;
You will also need to add a formula variable DTADJ which will adjust the DATETIME value according to the interval so that it is correct when viewed.
set table name=WEBxxx;
delete formula name=DTADJ noerror;
create formula name=DTADJ
interpret=INT
levels='DAY WEEK MONTH YEAR'
type=NUMERIC
label='DATETIME Adjuster'
description='Adjust DATETIME to show values on the Interval boundary.'
kept=YES
format=2.
length=4
source={dtadj=((input(intid,2.)-1)*(duration/60));
datetime=datetime+dtadj*60;
}
;
Finally, you will need to add the INTID variable to the BY/CLASS list at each reduction level for this PDB. It is the INTID variable which prevents the data being summarized into hourly intervals.
The example below is actually using the WEBBRW table. Each WEB* table will have different BY/CLASS lists which you will have to generate.
Tip: Use the GENERATE SOURCE DDUTL control statement to generate the tables DDUTL and then cut and paste the update table part (similar to below) to a separate file and edit accordingly. This is quicker and probably more accurate than manually writing the DDUTL control statements.
UPDATE TABLE NAME= WEBxxx
DETAIL =(
BYVARS= ' DATETIME INTID HOUR MACHINE SITENAM DOMAIN HOST BROWNAM BROWVER BROWPLT REMHOST'
INDEXVARS= ' BROWNAM BROWPLT DATETIME DOMAIN HOST'
)
DAY =(
CLASSVARS= ' DATETIME INTID HOUR MACHINE SITENAM DOMAIN HOST BROWNAM BROWVER BROWPLT REMHOST'
INDEXVARS= ' BROWNAM BROWPLT DATETIME DOMAIN HOST'
)
WEEK =(
CLASSVARS= ' DATETIME INTID MACHINE SITENAM DOMAIN HOST BROWNAM BROWVER BROWPLT REMHOST'
INDEXVARS= ' BROWNAM BROWPLT DATETIME DOMAIN HOST'
)
MONTH =(
CLASSVARS= ' DATETIME INTID MACHINE SITENAM DOMAIN HOST BROWNAM BROWVER BROWPLT REMHOST'
INDEXVARS= ' BROWNAM BROWPLT DATETIME DOMAIN HOST'
)
YEAR =(
CLASSVARS= ' DATETIME INTID MACHINE SITENAM DOMAIN HOST BROWNAM BROWVER BROWPLT REMHOST'
INDEXVARS= ' BROWNAM BROWPLT DATETIME DOMAIN HOST'
)
;
The CPSUMDUR macro variable has to be set to a valid value prior to running %CPPROCES against the WEBRES table. Valid values are :-
If any other value is used then the default of 3600 (1 hour) will be used.
%let cpsumdur=900; /* 15 minutes */
%cpproces(webres,
collectr=weblog,
toolnm=clf);