Softpanorama
(slightly skeptical) Open Source Software Educational Society

May the source be with you, but remember the KISS principle ;-)

Google   


Tivoli TEC Logfile Adapter

News Recommended Links Reference TEC Documentation TEC Rules Programming
Installation Adapter files Operations with Adapter: Starting, Stopping and Reloading Starting the adapter Stopping the adapter

 

      Default rules
Format file Troubleshooting the UNIX logfile postmsg adapter Humor Etc

The TEC logfile adapter is used to relay events from the clients to the TEC event server.  It also can be installed on the TEC server itself.  ACP is much more powerful and using the latest version very scalable and easy to configure.  there are two types of logfile adapter: TME and and not-TME

The UNIX logfile adapter adds modifies the /etc/syslog.conf file to enable the adapter to monitor events that the syslogd daemon writes to various log files. The adapter can also be configured to monitor any ASCII log file for information that is important to the operation of your enterprise.

Installation

A set of configuration steps must be performed to enable logadapter adapter. IBM( Tivoli Enterprise Console Installation Guide is almost useless as for installation. The key idea is that you need to create a profile and distribute it to endpoints. Distribution of profile actually creates the logadapter files.

The first step is to create an Adapter Configuration Profile (ACP). In order to do so, you must insure that ACPs are managed resources. From the pull-down menu on the selected policy region, select Managed Resources and make sure ACP is in the Current Resources list.

After the format files have been distributed, the events generated from logfile entries should show up in the event console. When the adapter configuration is distributed, the Logfile Adapter for the subscribe endpoint will automatically restart the adapter with the new configuration file. Add entries to the log file to generate new events.

Adapter files

The UNIX logfile adapter package consists of the following files:

  1. tecad_logfile.conf The configuration file.  This file along with format file is the most important for the adapter configuration. It is located at /opt/tivoli/lcf/bin/platform/TME/TEC/adapters/etc
  2. tecad_logfile.fmt  The format file.  The second most important file for the adapter. Provide parsing of log entries using an ideosyncratic language. It is also located at /opt/tivoli/lcf/bin/platform/TME/TEC/adapters/etc
  3. log_default.rls The default rule file.  Located in rules directory of TEC server.
  4. tecad_logfile.baroc The BAROC file.  Located in BAROC directory of TEC server.
  5. tecad_logfile.cfg The installation script.
  6. init.tecad_logfile The adapter startup and shutdown script. Never stop the adapter using signals. Use this script to ensure that the syslogd daemon remains running and functional.
  7. tecad_logfile The executable file of the adapter that receives the log information and transforms it into events.
  8. logfile_gencds The executable file that converts a format file to a CDS file.
  9. tecad_logfile.cds The CDS file. This file is created by running logfile_gencds on the format file.
  10. tecad_logfile.err The error file.

Before you start the event server and UNIX logfile adapter, check each adapter file to determine if it defines the behavior you want from the adapter.

See also Adapter files

Configuration file

The configuration file defines the behavior of the adapter. The configuration file can have the common keywords described in Configuration file, as well as the following custom keywords:

LogSources
Specifies the log files to poll. The complete path to each file must be specified, and file names must be separated by commas. Within each file name, you can also use an asterisk (*) to represent any sequence of characters, or a question mark (?) to represent any single character. For example, mylog* would result in polling all log files whose names begin with mylog, while mylog??? would result in polling all log files whose names consist of mylog followed by exactly three characters. These wildcard characters are supported only within the file name; the path must be explicitly specified.

A log source need not exist when the adapter is started; it is polled when it is created.

Each line in the file must end with a newline character. If a file truncates while the adapter is active, the adapter automatically resets its internal pointer to the beginning of the file. If during the polling interval the file is overwritten, removed, or recreated with more lines than the previous poll, only the number of lines greater than the previous line count is read. For example, the file has one line. After the poll interval elapses, the file is overwritten with two lines. Only the second line is read on the next polling.

Note:
The maximum number of lines that can be concatenated to a log file is 16 384.
 
NewLogBasedOn
Specifies whether a log file should be treated as new when the time stamp of the file changes but the size remains the same. When a file is treated as new, the adapter re-sends every event contained in the file. The possible value is:
mtime | MTIME
The file is treated as new if the modification time stamp changes.

This keyword is optional. If NewLogBasedOn is not specified, a preexisting log file is treated as new only if its size decreases.
 

PollInterval
Specifies the frequency, in seconds, to poll each file listed in the LogSources field for new messages. The default value is 120 seconds.
 
ProcessPriorityClass
Specifies the process priority for the adapter. You can adjust this value to improve system performance if the adapter processes large volumes of events and is using too many processor resources. The possible values are:
A
Very low priority (20)
B
Low priority (10)
C
Typical priority (0)
D
Above typical priority (-5)
E
High priority (-10)
F
Very high priority (-20)

The default value is C (typical priority).
 

UnmatchLog
Specifies a file to log discarded events that cannot be parsed into a Tivoli Enterprise Console event class by the adapter. The discarded events can then be analyzed to determine if modifications are needed to the adapter format file.

See also Configuration file

 

Format file

Creating the format file

The format file contains message format specifications and their mappings to BAROC events. The message fields of a record in the log file are matched against the format descriptions in this file and when a match succeeds, the corresponding Tivoli Enterprise Console® event is generated by the adapter.

The default format files (Windows®: tecad_win_C.fmt and Linux®: tecad_logfile_C.fmt) contain predefined mappings for some common events for Windows and System logs for Linux. They can also be customized to add new messages.
You can create a customized format file for a specific log file. The following examples show sample entries from customized format file tecad_wassample_C.fmt used for the WebSphere® Application Server SystemOut.log and sending events of classes to Tivoli Enterprise Console:
FORMAT WAS_Base 
[%s+] %s %s* 
hostname DEFAULT
fqhostname DEFAULT
origin DEFAULT
msg $3
-sysout $3
msg PRINTF("%s", sysout)
END

FORMAT WAS_INFO FOLLOWS WAS_Base
[%s+] %s %s I %s: %s*
-s1 $3
-s2 $4
-s3 $5
msg PRINTF("%s %s %s", s1, s2, s3)
severity "HARMLESS"
END

FORMAT WAS_INFO FOLLOWS WAS_Base
[%s+] %s %s A %s: %s*
-s1 $3
-s2 $4
-s3 $5
msg PRINTF("%s %s %s", s1, s2, s3)
severity "HARMLESS"
END

FORMAT WAS_WARNING FOLLOWS WAS_Base
[%s+] %s %s W %s: %s*
-s1 $3
-s2 $4
-s3 $5
msg PRINTF("%s %s %s", s1, s2, s3)
severity "WARNING"
END

FORMAT WAS_ERROR FOLLOWS WAS_Base
[%s+] %s %s E %s: %s*
-s1 $3
-s2 $4
-s3 $5
msg PRINTF("%s %s %s", s1, s2, s3)
severity "MINOR"
END

To use the sample format files for SystemOut.log (WebSphere Application Server), db2diag.log (DB2®), and AMQERR01.log (WebSphere MQ), perform the following steps:

On your Tivoli Enterprise Console server, copy the necessary adapter format files to the appropriate directories. To do this, perform the following steps:
  1. Upload the following files to the Tivoli Enterprise Console server from the following directories on CD 1:
    • logfilead\samples\tecad_wassample_C.fmt
    • logfilead\samples\tecad_db2v82sample_C.fmt
    • logfilead\samples\tecad_wmqsample_C.fmt
  2. Copy the files to the ACF_REP directory under the TME® installation directory. For example: /opt/Tivoli/bin/generic_unix/TME/ACF_REP
  3. Follow the steps in the Defining event classes in a BAROC file section to create BAROC file that defines the necessary Tivoli Enterprise Console event classes.
     

A convenient way to get events into the Tivoli Enterprise Console from an application or resource is to use the existing Logfile Adapter. This is easier than writing a custom adapter.

Our goal is to be able to generate Tivoli Enterprise Console events from a new application or resource. There is an existing mechanism designed to read information from a logfile and generate messages that correspond to the log entries. This mechanism can be configured to work with other sources.

While this is the easiest way to hook up your application with Tivoli Enterprise Console, you can alternatively write your own event adapter.

Tivoli provides a Logfile Adapter for its UNIX platforms. It reports on the kinds of events monitored by syslogd in UNIX such as su failures, disk space exhaustion, and permissions problems. Syslogd puts this information into the log file specified in its /etc/syslog.conf file.

While the various varieties of UNIX all report similar information, the different varieties do not report it in exactly the same way. For this reason, Tivoli devised format files, which map logfile entries to BAROC event definitions. Using format files, each variety of UNIX can map its own log into the common BAROC definitions.

This mechanism turns out to be more generally useful. Tivoli allows you to point to your own log file and specify your own format. Then the existing Logfile Adapter does the work of creating the appropriate event.

The NT Event Adapter allows the same capability for the NT platforms.

So far in our labs, in order to generate an event we use the CLI command wpostemsg. We need to automatically generate events from their source. Continuing with our previous example, you can get your application to write its event information into a flat file, as shown in the following graphic.

The existing Logfile Adapter can then be used to generate events.

The goal of the Logfile Format Editor is to convert entries in a flat file to events. For example, your application writes something like the following graphic.

However, you want something to correspond to each line, as shown in the following graphic.

The Logfile Format Editor allows you to specify mappings. Use this input:

Jan 1 14:39:46 bebop PerfPlus: SwapOut 7340

If you want this mapping:

The format statement is:

%t %s PerfPlus: SwapOut %s

Details (such as attribute assignments) will be explained later, but the critical point is that lines in the log file are compared against format statements until a match is found. When a match is found, the mapping from the ASCII text to a BAROC definition is made.

To invoke the Logfile Format Editor from the Edit Adapter dialog, click the Logfile Format Editor button.

Operations with Adapter: Starting, Stopping and Reloading

Running multiple UNIX logfile adapters

You can run multiple instances of the UNIX logfile adapter on a single system. It is recommended that additional adapters be run as non-TME adapters. To monitor different log files, each instance of the adapter must have its own configuration, format, class definition statement (CDS), and error files. If the adapters use event buffering (set using the BufferEvents keyword, which has a default value of YES), the adapters must also have their own cache files.

If you want to stop an adapter when multiple log files are running, you must specify the name of the adapter to stop. If you do not specify the adapter to stop, the default adapter without a name is stopped.

The syntax for the init.tecad_logfile command is as follows:

init.tecad_logfile [-s] {start | stop} [adapterID] &

If the -s flag (skip syslog) is specified, the adapter does not monitor the syslogd daemon.

If the -s flag is not specified, use & so that the command runs in the background while returning a command prompt to your session. Otherwise, because an adapter started without the -s option forks a child process to run the adapter, the process does not return to the command line until the child process ends.

Note:
If you start the adapter with the -s flag, you can also use the -s flag when you stop the adapter to avoid reconfiguring the syslogd daemon. You can also stop the adapter without the -s flag and it still works. However, do not stop an adapter with the -s flag if you did not start it with the -s flag.

If the -s flag is not specified, the UNIX logfile adapter startup script uses a UNIX pipe to monitor the syslogd daemon and the syslogd daemon is configured to write to the pipe, and the UNIX logfile adapter reads from that pipe. When the adapter ends, the startup script reconfigures the syslogd daemon to stop writing to the pipe before stopping the UNIX logfile adapter.

The following command starts a UNIX logfile adapter called syslog that monitors all syslog messages:

init.tecad_logfile start syslog &

Format file

The format file is described in detail in Format file.

Class definition statement file

The CDS file defines how an adapter constructs events. This file is derived from the format file using the logfile_gencds program. In general, you should never have to edit this file to add new mappings. The CDS file is described in detail in Class definition statement file and in Appendix C. Class definition statement file reference.

Error file

The error file is described in detail in Error file.

Events listing

The following table shows the class names and severities of all events defined for the UNIX logfile adapter. You can use the table to get a sense of how log file events are mapped to Tivoli Enterprise Console events and to determine if you want to make any changes. The events are defined in the BAROC file. See the IBM Tivoli Enterprise Console Rule Developer's Guide for more information about customizing BAROC files.

Event class structure

Event classes are defined hierarchically, with child classes inheriting attribute value defaults from the parent.

The adapter fills in the following attribute defaults. The attributes are used in event group filters.

The following events are defined for the UNIX logfile adapter in the tecad_logfile.baroc file.

Table 17. UNIX logfile adapter events
Event Class Default Severity
Logfile_Base WARNING
Logfile_Automounter HARMLESS
  Logfile_Amd WARNING
  Amd_Mounted WARNING
Amd_Unmounted WARNING
Logfile_Automount WARNING
Logfile_Bootpd WARNING
Logfile_Comsat WARNING
Logfile_Cron HARMLESS
Logfile_Date HARMLESS
  Logfile_Date_Set WARNING
Logfile_Ebbackupd WARNING
  Ebbackupd_Waiting WARNING
Logfile_Ebcatcomp WARNING
Logfile_Fsck WARNING
Logfile_Ftp WARNING
Logfile_Ftpd WARNING
Logfile_Gated WARNING
Logfile_Getty WARNING
Logfile_Halt WARNING
Logfile_Idi HARMLESS
Logfile_Inetd WARNING
Logfile_Init WARNING
Logfile_Innd WARNING
Logfile_Kernel WARNING
  File_Write_Error MINOR
  File_System_Full MINOR
NFS_Write_Error WARNING
Sendsig_Err CRITICAL
Kernel_Panic FATAL
NFS_No_Response WARNING
NFS_OK HARMLESS
Silo_Overflow MINOR
Logfile_Login WARNING
  Root_Login MINOR
  Root_Login_Failure WARNING
  Root_Login_Failure_From WARNING
Root_Login_Success WARNING
  Root_Login_Success_From WARNING
Repeated_Login_Failure WARNING
Repeated_Login_Failure_From WARNING
Logfile_Lpd WARNING
  Logfile_Lpd_Get_Hostname WARNING
Logfile_Lpd_Lost_Connection WARNING
Logfile_Lpd_No_File WARNING
Logfile_Mosaic WARNING
Logfile_Mountd WARNING
Logfile_Named WARNING
Logfile_Nfsd WARNING
Logfile_Nnrpd WARNING
Logfile_Oserv WARNING
  Oserv_Panic CRITICAL
Oserv_Graceful_Exit HARMLESS
Oserv_System_Error MINOR
  Oserv_Fork_Failed CRITICAL
Oserv_Exec_Failed MINOR
Oserv_Comm_Error WARNING
  Oserv_IPC_Dispatch_Failed MINOR
Oserv_Security WARNING
Oserv_Tmgr WARNING
  Oserv_Event_Method_Failed MINOR
Logfile_Passwd WARNING
Logfile_Pcnfsd WARNING
  Logfile_Printer WARNING
Printer_Connection_Abort WARNING
Printer_Error_Cleared HARMLESS
Printer_Door_Open WARNING
Printer_Offline WARNING
Printer_Output_Full WARNING
Printer_Page_Punt WARNING
Printer_Paper_Jam WARNING
Printer_Paper_Out WARNING
Printer_Powerup WARNING
Printer_Toner_Low WARNING
Logfile_Rarpd WARNING
Logfile_Reboot HARMLESS
Logfile_Rexecd WARNING
Logfile_Rftp WARNING
Logfile_Rlogind WARNING
Logfile_Routed WARNING
Logfile_Rquotad WARNING
Logfile_Rshd WARNING
Logfile_Rstatd WARNING
Logfile_Rtelnet WARNING
Logfile_Rwhod WARNING
Logfile_Sendmail HARMLESS
  Sendmail_Loopback WARNING
Sendmail_No_Space MINOR
Logfile_Snmpd WARNING
Logfile_Sockd WARNING
  Sockd_Connected HARMLESS
Sockd_Terminated WARNING
Sockd_Transfer WARNING
Logfile_Strerr HARMLESS
Logfile_Su WARNING
  Su_Failure WARNING
Su_Success WARNING
Logfile_Syslogd WARNING
  Syslogd_Nospace MINOR
Logfile_Talkd WARNING
Logfile_Telnetd WARNING
Logfile_Tftpd WARNING
Logfile_Xntpd WARNING
  Xntpd_Clock_Reset WARNING
Xntpd_Ntpdate WARNING
Logfile_YP HARMLESS
  Logfile_Ypbind WARNING
Logfile_Ypchfn WARNING
Logfile_Ypchsh WARNING
Logfile_Yppasswd WARNING
NIS_No_Response WARNING
NIS_OK HARMLESS
No_Permission WARNING
No_Resources CRITICAL
  No_Disk_Space WARNING
  File_System_Full MINOR
  LOCAL_File_System_Full WARNING
NFS_File_System_Full WARNING
SWAP_File_System_Full WARNING
Sendmail_No_Space MINOR
Syslogd_Nospace MINOR
No_Memory WARNING
No_Proc_Attributes WARNING
Server_No_Response WARNING
  NFS_No_Response WARNING
NIS_No_Response WARNING
Server_OK HARMLESS
  NFS_OK HARMLESS
NIS_OK HARMLESS

Default rules

The UNIX logfile adapter has a set of default rules that can be installed to enhance event server operation. Rules can enable the server to perform functions such as deleting events and sending e-mail to alert administrators of an unresolved problem. The rules are contained in the log_default.rls file and perform the following functions:

The event server also comes with some additional rules that you can install. The $BINDIR/TME/TEC/contrib/rules/security directory contains the security_default.rls file, which provides the following behavior to the event server:

See also Default rules


 

Old News ;-)

Tivoli-IBM TME10 FAQ -  How do I create a logfile adapter to read UNIX application logfiles?

8 Apr 03 | Tek-Tips

First, create a format file for your application.  Note that you must have a default Sun syslog or AIX errpt logfile adapter deployed before you can do this, and you will need to do some scripting to install your application LFA in a different location. If you fail to do this, the application LFA will break your syslog adapter.

Define a base event class if you have multiple event formats with common attributes:

FORMAT Nortel_IVR_Base
%s %t <%s> %s %s*
hostname "HSTNAME"
sub_origin "N/A"
adapter_host "EPLABEL"
date $2
origin DEFAULT
sub_system $3
error_code $4
msg $5
END


Then begin defining your events:

// Sat Aug 17 10:55:56 <KmnPin> 91003 Line 25 Host 1 Severity 8 Vps 1
FORMAT Nortel_IVR_Error FOLLOWS Nortel_IVR_Base
%s %t <%s> %s Line %s Host %s Severity 8 %s*
-line_number $5
-host_number $6
vps_number $7
msg PRINTF("<%s> %s Line %s %s", sub_system, error_code, line_number, vps_number)
END


Note the dashes "-" before some of the slots.  These signify event slots that are used locally and are not present in the BAROC file.

Create your Adapter Configuration Profile
Select the adapter type - a default AIX or Solaris type will do.

In the "Filters" tab, create your event filters.

000 Class=Nortel_IVR_Base
001 Class=Nortel_IVR_Error


In the "Environment" tab, set your keywords

AdapterErrorFile=/usr/Tivoli/lcf/bin/solaris2/TME/TEC/adapters/IVR_Logfile/etc/tecad_logfile.err
BufEvtPath=/usr/Tivoli/lcf/bin/solaris2/TME/TEC/adapters/IVR_Logfile/tecad_logfile.cache
BufferEvents=YES
MaxEventSize=4096
FilterMode=IN
LogSources=/opt/vps/vps33/log/alarm.log
PollInterval=30
ServerLocation=@EventServer


Note the use of the FilterMode keyword.  FilterMode=IN will only process events are present in the event classes in the "Filters" tab.  The default behavior is FilterMode=OUT.

If you have multiple logs, you must specify the full path to each logfile, separated by commas with no spaces in between.

In the "Actions" tab for the "Before" actions I do a little scripting in the adapter to create the LFA install directory and copy the necessary LFA binaries to the new LFA install directory.

TECADHOME=/usr/Tivoli/lcf/bin/solaris2/TME/TEC/adapters/IVR_Logfile
export TECADHOME
mkdir -p $TECADHOME/etc
mkdir -p $TECADHOME/bin
unset CHILD_OF_OSERV
cp $LCF_DATDIR/../../bin/solaris2/TME/TEC/adapters/bin/* $TECADHOME/bin
$TECADHOME/bin/init.tecad_logfile -s stop IVR_Logfile


Note:
You must use the "-s" switch in the adapter stop command to let the adapter know NOT to create a pipe to syslogd!  If you don't, you'll break your syslog adapter.

Ok, here's where it gets tricky.  In the "After" actions, we define some variables, then use sed to insert those variables into a temporary format file, then compile the temp format file and the finished format file:

TECADHOME=/usr/Tivoli/lcf/bin/solaris2/TME/TEC/adapters/IVR_Logfile
export TECADHOME
unset CHILD_OF_OSERV
REALNAME=`hostname`
export REALNAME
/usr/bin/sed -e "s/HSTNAME/$REALNAME/g" $TECADHOME/etc/tecad_logfile_IVR.fmt > $TECADHOME/etc/tecad_logfile.fmt.tmp
/usr/bin/sed -e "s/EPLABEL/${ACPEP_NAME}/g" $TECADHOME/etc/tecad_logfile.fmt.tmp > $TECADHOME/etc/tecad_logfile.fmt
logfile_gencds $TECADHOME/etc/tecad_logfile.fmt > $TECADHOME/etc/tecad_logfile.cds
$TECADHOME/bin/init.tecad_logfile -s start IVR_Logfile


Again, make sure you're using the "-s" switch on the adapter start command.

The reason for all of this scripting is to ensure that if the hostname differs from the endpoint lable, we can tell that when we look at the event.  This is helpful in clustered or HACMP configurations where you may have multiple endpoints installed.

In the "Distribution" tab, tell the adapter where it's going to get its format and error files from:

tecad_logfile.err=FQDN:/usr/Tivoli/bin/generic_unix/TME/ACF_REP/tecad_logfile.err
tecad_logfile_IVR.fmt=FQDN:/usr/easm/dev/adapter/IVR_Logfile/tecad_logfile_IVR.fmt


In the "General" tab, set the install location for the adapter's configuration files:

Install dir: /usr/Tivoli/lcf/bin/solaris2/TME/TEC/adapters/IVR_Logfile/etc

Config file: tecad_logfile.conf


Make sure your event class is added to a new or existing BAROC file.  Compile your rulebase and bounce the TEC server.

Save your adapter and distribute.  That is, after you've added your subscribers.
 

Recommended Links


In case of broken links please try to use Google search. If you find the page please notify us about new location
Google     

Adding a new message to a Logfile Adapters  Provides an explanation of how to add new messages to be processed by the Unix Logfile adapter or the NT Event Log Adapter.

Etc

 #Start the Tivoli Enterprise Console Logfile Adapter
if [ -f /opt/TMF/bin/lcf_bundle/bin/aix4-r1/TME/TEC/adapters/bin/init.tecad_logfile ]; then
    sleep 2
    /opt/TMF/bin/lcf_bundle/bin/aix4-r1/TME/TEC/adapters/bin/init.tecad_logfile  start  &
    echo "Tivoli Enterprise Console Logfile Adapter started."
fi



Copyright © 1996-2008 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. Submit comments This document is an industrial compilation designed and created exclusively for educational use and is placed under the copyright of the Open Content License(OPL). Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.

Standard disclaimer: The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.

Last modified: June 02, 2008