Generating Source Dynamically

Introduction

Collectors such as Demand Technology's NTSMF and BMC Patrol have the potential for producing information from a large number of data sources. Although IT Service Vision will typically supply table and variable definitions for the more popular data sources, it is not practical to supply definitions for them all. To address this issue additional functionality has been added to the GENERATE SOURCE ddutl control statement to assist in creating table and variable definitions.

The process for creating the table and variable definitions is the same for both NTSMF and PATROL, it is only the input that is different.

Overview

1. Collect the appropriate input data to pass to the GENERATE SOURCE ddutl statements
   
   For NTSMF the input to GENERATE SOURCE is the NTSMF discovery record for the object that you are creating the table and variable definitions.
   For PATROL the input is the data file that you would normally feed to the %CPPROCES macro.
   
2. Run the %CPDDUTL macro with the appropriate statements and create a 'first cut' of the table and variable definitions in a SAS catalog entry or a text file.
   
3. Review the generated table and variable definitions.
   
4. Run the %CPDDUTL macro with the reviewed table and variable definitions, adding the new table to the PDB.
   
5. Run %CPPROCES as per any other NTSMF table.

Overview

Assume that you are a customer with NTSMF and a new object has appeared in your NTSMF log that you want to include in your PDB.

1. Collect the appropriate input data to pass to the GENERATE SOURCE ddutl statements.
   
   The information required to construct table and variable definitions is the meta data information. The source of this information will vary from collector to collector.
   
   
   • NTSMF
     
     The meta data for NTSMF tables comes from the NTSMF Discovery records which are collected as part of your NTSMF log when the Discovery record option is switched on. (As part of   IT Service Vision 2.2 we require that all logs contain NTSMF Discovery records). The following is an example of an NTSMF Discovery record.
     
     3,0,5,CARYNT,nightingale,1998,4,24,17,25,8,375,0,4,LogicalDisk,0,24,28,2,PhysicalDisk,LogicalDisk,% Free Space,,Free Megabytes,Current Disk Queue Length,% Disk Time,Avg. Disk Queue Length,% Disk Read Time,Avg. Disk Read Queue Length,% Disk Write Time,Avg. Disk Write Queue Length,Avg. Disk sec/Transfer,,Avg. Disk sec/Read,,Avg. Disk sec/Write,,Disk Transfers/sec,Disk Reads/sec,Disk Writes/sec,Disk Bytes/sec,Disk Read Bytes/sec,Disk Write Bytes/sec,Avg. Disk Bytes/Transfer,,Avg. Disk Bytes/Read,,Avg. Disk Bytes/Write,,
     
     Although you can pass a complete NTSMF log file as input to the GENERATE SOURCE ddutl statement, it will create definitions for each NTSMF Discovery record in that file. It is recommend that you cut and paste just the NTSMF Discovery record(s) that you are interested in into a separate file and use this as the input.
     
     
   • PATROL
     
     The meta data for PATROL tables and variables come from the same Patrol History data that is passed to the %CPPROCES macro. The following example is output from the dump_hist command (although data from the Patrol History Loader KM will also work). In the small example shown below a single table  for NT_CPU would be defined with 2 metrics. See your BMC Patrol documentation for more information on extracting data using the dump_hist command or through the Patrol History Loader KM.
     
     nightingale/NT_CPU.CPU_0/CPUprcrUserTimePercent
         Thu Jul 23 10:00:57 1998 26.981
         Thu Jul 23 10:01:58 1998 5.35963
         Thu Jul 23 10:02:58 1998 0.598205
         Thu Jul 23 10:03:58 1998 0.333915
     nightingale/NT_CPU.CPU_0/CPUprcrPrivTimePercent
         Thu Jul 23 10:00:57 1998 61.0279
         Thu Jul 23 10:01:58 1998 1.20528
         Thu Jul 23 10:02:58 1998 1.56053
         Thu Jul 23 10:03:58 1998 1.05312
     
     
     
2. Use GENERATE SOURCE to create DDUTL control statements for this object (table). See the Macro Reference guide for more information on the GENERATE SOURCE control statement and syntax.
   
   The following code will read the file from INFILE= and generate table and variable definitions which it will save in a SAS catalog entry (this could be a text file if the FILENAME= parameter was used).
   
   %cpcat;
   cards4;
   generate source infile='location.of.file.with.metadata'
                            intype=NTSMF | PATROL
                            entryname='work.ddutl.gensrc.source';
   ;;;;
   %cpcat(cat=work.cpddutl.gen.source);
   
   %cpddutl(entrynam=work.cpddutl.gen.source);
   
   
   
3. Review the generated source, particularly the interpretation types and statistics for each variable. For information on the interpetation types, see the 'How To/Macro Reference Guide'.
   
   GENERATE SOURCE scans the meta data it is fed and will attempt to match a valid interpretation type to a variable based on its label or description. If its interpretation type cannot be determined it is assigned the default interpretation type and statistics. If you feel that GENERATE SOURCE has not assigned the correct value then change it appropriately.
   
   Typically, the only other parameters that you are likely to need to update are the DESCRIPTIONS and LABEL parameters. Do NOT change the EXTNM parameter as this is used to map the external variable name to the variable name used by IT Service Vision. We also recommend that you do not alter the CLASSVARS statements in particular the DURGRP variable should not be removed. The DURGRP variable is used at the summary levels to ensure that each variable is weighted using the correct duration value.
   
   
4. Use the %CPDDUTL macro to add the table to the PDB using the reviewed ddutl statements..
   
   %cpddutl(entrynam=work.cpddutl.gensrc.source);
   
   
5. Use %CPPROCES to process the raw data into the PDB.
   
   %cpproces(,collectr=NTSMF | PATROL
                    ,rawdata=location.of .data
                     ,toolnm=sasds);