|
Softpanorama |
May the source be with you, but remember the KISS principle ;-)
|
There are many different types of log that are produced by modern OSes and applications. That's why Log analyzers are usually pretty specialized (e.g. web log analyzers, syslog analyzers, etc).
The main categories are as following:
While there is no sure thing in security but log analysis is pretty close. On
Unix syslog daemon permits remote logging. Generally you are better off sending
those logs to special loghost server to keep them system off that machine.
Sure the remote loghost could have been compromised as well but that's twice
as much work for attackers. Now you have not only to hack the machine but also the
loghost server to erase the intrusion event. And loghost server can be hardened
as tightly as bastion host.
The idea was to provide not only redundant logging but also provide an additional
work for hackers. Also a hacker might be not qualified enough to understand the
situation and be too busy doctoring up local logs to check to see if there is an
external logger.
The external log machine can very easily be almost perfect. first of all the logger can be put behind a NAT box, then non-local IPs can't even see it. Also the logging interface can be read-only or even an inline "stealth" box. It acts like a piece of copper wire, but records all the data going through it. You can also use serial interface to connect to the logger, to exclude network connections. In this case logger cannot be accessed for reading or resetting except through an interface not available via internal or external network.
IANAH (hacker), but isn't the first thing you do when you break into a system to 'fix' the logs?
Any high-profile site has (or should have) a box dedicated to syslog. To make it more secure you can install a firewall and limit services. In extreme case the only way to get logs is to burn a DVD and you need to logon to the console.
Dr. Nikolai Bezroukov
|
|||||||
check_logfiles 2.3.3 (Default)
Added: Sun, Mar 12th 2006 15:09 PDT (2 years, 1 month ago)
Updated: Tue, May 6th 2008 10:37 PDT (today)
About:check_logfiles is a plugin for Nagios which checks logfiles for defined patterns. It is capable of detecting logfile rotation. If you tell it how the rotated archives look, it will also examine these files. Unlike check_logfiles, traditional logfile plugins were not aware of the gap which could occur, so under some circumstances they ignored what had happened between their checks. A configuration file is used to specify where to search, what to search, and what to do if a matching line is found.
Kazimir is a log analyzer. It has a complete configuration file used to describe what kind of logs (or non-regression test) to be watched or spawned and the kind of regexp to be found in them. Interesting information found in logs may be associated with "events" in a boolean and chronological way. The occurrence of events may be associated with the execution of commands.Release focus: Initial freshmeat announcement
Cisco IP Accounting Fetcher is a set of Perl scripts that allows you to fetch IP accounting data from Cisco routers. It is capable of fetching this information from multiple routers. It summarizes this information on a daily and monthly basis. It optionally generates HTML output with CSS support, and it is able to ignore specific traffic.
Epylog is a log notifier and parser that periodically tails system logs on Unix systems, parses the output in order to present it in an easily readable format (parsing modules currently exist only for Linux), and mails the final report to the administrator. It can run daily or hourly. Epylog is written specifically for large clusters where many systems log to a single loghost using syslog or syslog-ng.
Author:
Konstantin Ryabitsev [contact developer]
About:
devialog is a behavior/anomaly/signature-based syslog intrusion detection system which can detect new, unknown attacks. It fits comfortably in a heterogeneous Unix/Linux/*BSD environment at the core of a central syslog server. devialog generates its own signatures and acts upon anomalies as configured by the system administrator. In addition, devialog can function as a traditional syslog parsing utility in which known signatures trigger actions.Release focus: Minor bugfixes
Changes:
Bug fixes include better handling of lines with some special characters. A timing error was fixed within alert generation: sometimes alerts would be sent inadvertently based on the timing of a new log arriving as an alert was sent out in specific high-volume log situations. Altered signature generation creates more exact regular expressions.
MultiTail lets you view one or multiple files like the original tail program. The difference is that it creates multiple windows on your console (with ncurses). It can also monitor wildcards: if another file matching the wildcard has a more recent modification date, it will automatically switch to that file. That way you can, for example, monitor a complete directory of files. Merging of 2 or even more logfiles is possible.
It can also use colors while displaying the logfiles (through regular expressions), for faster recognition of what is important and what not.
It can also filter lines (again with regular expressions). It has interactive menus for editing given regular expressions and deleting and adding windows. One can also have windows with the output of shell scripts and other software.
When viewing the output of external software, MultiTail can mimic the functionality of tools like 'watch' and such.
For a complete list of features, look here.
Net::Dev::Tools::Syslog version 0.8.0
=======================================
DESCRIPTION
This module provides functionality to:
- parse syslog log files, apply filters
- send syslog message to syslog server
- listen for syslog messages on localhost
- forward received messages to other syslog server
INSTALLATION
To install this module type the following:
perl Makefile.PL
make
make test
make install
DEPENDENCIES
This module requires these other modules and libraries:
Time::Local
IO::Socket
Sys::Hostname
Log file search and indexing software vendor Splunk Inc. announced Tuesday that it will soon add systems management host, network and service monitoring capabilities to its software through a partnership with the Nagios open-source project.The new capabilities, which will be added to Splunk's log file search and indexing applications over the next six months, will give systems administrators even more information to monitor and repair their networks, said Patrick J. McGovern III, chief community Splunker at San Francisco-based Splunk. Splunk has seen more than 25,000 downloads of its software since its 1.0 release came out in November, he said, while Nagios delivers about 20,000 downloads a month of its open-source network and service monitoring application.
The two working together is going to be a big win for both communities,Ф McGovern said.
McGovern is the son of Patrick J. McGovern, the founder and chairman of International Data Group Inc., which publishes Computerworld.
Splunk's free Splunk Server application runs on Linux, Solaris or BSD operating systems but is limited to searching and indexing 500MB of system log files each day. The full version, Splunk Professional Server, starts at $2,500 and goes up to $37,500 for an unlimited capacity. Both products search and index in real time the log files of mail servers, Web servers, J2EE servers, configuration files, message queues and database transactions from any system, application or device. Splunk uses algorithms to automatically organize any type of IT data into events without source-specific parsing or mapping. After the events are classified and any relationships between them discovered, the data is indexed by time, terms and relationships.
The integration of the Splunk applications and the Nagios features will take place in phases over the next six months, McGovern said.
The Nagios software allows monitoring of network services, including SMTP, POP3, HTTP, NNTP and PING, as well as monitoring of processor load, disk and memory usage, and running processes. Users can also monitor environmental factors in data centers such as temperature and humidity.
Other log file monitoring and management applications are available from competitors including Opalis Software Inc. and Prism Microsystems Inc.
Dana Gardner, an analyst at Interarbor Solutions LLC in Gilford, N.H., said Splunk's log file search capabilities bring a novel approach to IT management. Certainly on the theory basis, on the vision basis, it makes a lot of sense, Gardner said. Search can be a really useful tool to help systems administrators figure out a vexing problem amid a huge quantity of data.
It certainly gets at the heart of what keeps IT administrators up at night, managing complexity and chaos,Ф Gardner said.
SysLog & Event Log Management, Analysis & ReportingManageEngine EventLog Analyzer is a web-based, agent-less syslog and event log management solution that collects, archives, and reports on event logs from distributed Windows host and, syslogs from UNIX hosts, Routers and Switches. It helps organizations meet host-based security event management (SEM) objectives and adhere to demands of regulatory compliance requirements like HIPAA, SOX, and GLBA .
Sawmill is a world-class web log analyzer, but it's more than just that. Sawmill can analyze all of your log files, from all of your servers. Sawmill can analyze
- Streaming media logs: RealPlayer, Microsoft Media, and Quicktime Streaming Server.
- FTP logs from most FTP servers.
- Proxy logs from most proxy servers.
- Firewall logs from most firewalls.
- Cache logs from most cache servers.
- Network logs: Cisco PIX, IOS; tcpdump, and more.
- Mail logs: SMTP, POP, IMAP from most mail servers.
- And dozens of others!
Sawmill is a universal log analysis tool; it's not limited to web logs. You can use it to track usage statistics on your proxy and cache servers. You can use it to bill by bandwidth. You can use it to monitor all of the TCP/IP traffic on your internal network, or through your firewall, to see who's eating all that expensive bandwidth you just bought. As an ISP/ASP, you'll find dozens of uses for Sawmill beyond just providing your customers with the best statistics.
And that's not all! Sawmill lets you define your own log format, using a very powerful format description mechanism that can be used to describe just about any possible log format. Do you have your own special logs that you'd like to analyze? Have you written a custom server that no other tool knows anything about? In a few minutes, you can have Sawmill processing those files just as easily as it processes web logs (in fact, we'll happily write the log format descriptor file, and include it with the next version of Sawmill). All the advantages it has in web statistics, it has with all other logs -- you'll get unprecedented detail and power for your statistics, no matter where the log files came from.
"If you go into a room full of IT managers and ask how many are working on home-grown log solutions, half the room will raise their hands," said Stephen Northcutt, director of training and certification for the Bethesda, Md.-based institute. "Why is that bad? Because the guy who writes it leaves and doesn't document what he did or leave instructions behind. Then the person who takes over can't figure out how to interpret the logs or what to do if there's a problem."
Security experts have long advised that a clear audit trail is necessary to track suspicious network activity and quickly respond to security incidents. Northcutt agreed, and said companies that decide to take it seriously should "buy a commercial tool and pray that it works" or "get help from a MSSP."
Window dressing for compliance's sake
As part of the research, SANS polled 1,067 security-minded system administrators from a variety of industries. "Slightly over one fourth of the respondents stated that they retained logs for over one year. Almost half of the respondents [44%] don't keep logs more than a month," the report said. "Since many regulatory and accounting bodies are recommending or even requiring log retention of three to seven years, why do so many companies have such short retention times?"Those who answered the question cited three key problems: the amount of data to manage, the speed the log data comes in and the lack of a consistent format for the log data. "Closely related to all of these is a lack of manpower," the report said. "It takes people to maintain a logging system and more people to monitor it and, of course, man hours relates to money."
"That's not far off target," said Diane McQueen, systems engineer for Perot Systems, which manages IT security for the nonprofit Northern Arizona Healthcare hospital chain. "With the amount of paperwork auditing produces, a big problem is taking the time to look through those logs. It's a resource issue."
The report said many companies do nothing with their logs. At best, they look through them after an incident as they scramble to find the source of a problem. Another downside is that companies are often so zealous to satisfy the regulatory letter of such laws as HIPAA, Sarbanes-Oxley and Graham-Leach-Bliley that they hobble together half-baked logging systems.
The more diverse your environment, the more you need outside help.
Stephen Northcutt
director of training and certification, SANS Institute"For the smaller guys, it can be cheaper to pay the fine than pay for everything needed for full compliance," Northcutt said. "There are those who do window dressing to appear to be in compliance, but they're not really using their tools. They're not taking this seriously day to day."
The big picture may be worse than the survey suggests, said Adam Nunn, security and corporate compliance manager for a large U.S. healthcare organization. Nunn said his organization takes log management very seriously and that efforts are underway to further improve the system. But, he added, "Most of the smaller health care providers I am familiar with are seriously lacking logging capability" or they don't really review the logs they have.
Federal requirements boosting awareness
At the same time, the need to be in compliance with laws like HIPAA and Sarbanes-Oxley has helped IT managers understand the need to take log management more seriously."As computers become more numerous and regulation compliance becomes more a part of daily life, some system administrators are finding that log management is becoming a problem," the report said. "The scripts and manual processes that have historically been used by 80% of the market need to be upgraded. This has resulted in a relatively new log management industry. Log issues tend to snowball as the size of a company grows."
In a recent SearchSecurity.com report on organizations struggling with HIPAA's security rules, IT managers said regulatory demands had prompted them to improve their logging systems and invest in new tools.
"I have become a big advocate of the phrase 'trust but verify,'" Nunn said. "We must use the logging mechanism as a primary way to prevent unauthorized activity and enforce compliance of insiders and be able to track where our information is going and who accessed it."
Related information Why audit trails are critical to HIPAA complianceAre you using security technology effectively?
While he stressed the need for some companies to buy commercial tools or outsource their log keeping, Northcutt said the in-house programs are not always a bad thing.
"A locally-developed software solution isn't wrong per se. But if you go for the home-grown solution, your chances of success are better if you're an all-Windows or all-Unix shop," he said. "If you mix your operating systems, you're going to run into trouble. The more diverse your environment, the more you need outside help."
McQueen's advice to IT managers struggling with log management is this: "Set up your standards and adopt a tool that will alert you to any changes on the network," she said. "For example, if a new user comes on, the tool should alert you to its presence. That way, you don't have to spend time scanning the user directories every day to keep track of new users or other changes."
IPFC is a software and framework to manage and monitor multiple types of security modules across a global network. Security modules can be as diverse as packet filters (like netfilter, pf, ipfw, IP Filter, checkpoint FW1...), NIDS (Snort, ISS RealSecure...), webservers (from IIS to Apache) and other general devices (from servers to embedded devices).
Another way to explain IPFC : It's a complete generic Managed Security Services (MSS) software infrastructure. A graphical overview of the framework (to have a global overview of the IPFC software and how it's working).
The main features are :Centralized and unified logging of multiple devices (from server to firewall including special device) Dynamic correlation of logs Alarming Active evaluation of your security infrastructure Unified policy and configuration management Can be integrated into existing monitoring infrastructure Auditable source code available (under GNU General Public License) Scalability and Security of the framework Easy extendable
again, i'm cross posting to the log analysis mailing list cos, well, it's my list ;-)
On Mon, 3 Nov 2003, Eric Johanson wrote:
> Sorry for the IDS rant. I'm sure ya'll have heard it before. My only
> real exsposure to IDS is via snort.
oh, you know us shmoo, we have absolutely no tolerance for rants :-)
> It works on 'blacklists' of known bad data, or stuff that 'looks bad'.
> This doesn't really help when dealing with new types of attacks. Most
> can't deal with a new strean of outlook worm, or can tell the difference
> between a (new, never seen) rpc exploit, or a simple SYN/SYNACK.
this is true...ish. f'r one thing, lots and lots o' exploits are based on known vulnerabilities, where there's enough information disclosed prior to release of exploit that reliable IDS signatures exist. nimda's my fave example of this. especially for its server attacks, nimda exploited vulnerabilities that were anywhere from 4 months to 2 years old. all the IDS vendors had signatures, which is why nimda was discovered so quickly and so rampantly. in the limited dataset i had -- counterpane customers with network-based intrusion detection systems watching their web farms -- the message counts increased by an order of 100000 within half an hour of the first "public" announcement of nimda (a posting at 9:18 east coast time on 18 sept 2001, on the incidents at securityfocus.com mailing list) (not that i was scarred or anything).
so in that particular instance, signature based IDS was certainly highly effective in letting you know that something was up.
several of the signature based IDS vendors claim that they base their signatures on the vulns rather than on a given exploit -- i don't really know what that means, but i do know that several of them detected this summer's RPC exploits, based on the info in the microsoft announcement, and the behavior of the vulnerable code when tickled, as opposed to waiting to write a signature after an attack had been discovered. we got attack data from ISS far faster than could have been explained if they were strictly writing in response to new attacks.
> I suspect we'll start seeing products in a few years that are focused on
> 'whitelisting' traffic. Anything that doesn't match this pattern, kill
> it.
>
> While this is a pain with lots-o-crappy protocols, it is possible. Does
> anyone know of a product that functions this way?
>
there are a couple of IDS vendors that claim to do this -- they tend to call themselves anomaly detection systems rather than intrusion detection systems -- and i've yet to be convinced about any of them. the only one i
can think of is lancope, but rodney probably knows better than i do. the statistical issues are huge, because you've got to be able to characterize normal traffic in real time with huge numbers of packets.
it's more approachable IMNSHO for log data -- marcus ranum's "artificial ignorance" approach -- because you get far less data from a system or application log per actual event than you get numbers of packets. of course i'm not convinced that getting >less< data is an entirely good thing -- see below....
> The other big problem I see with IDS systems is the SNR. Most are soo
> noisy, that they have little real-world value.
>
> I think doing real time reviewing of log entries would have a much better
> chance of returing useful information - - but again, should be based on
> whitelists of know good log entries, not grepping for segfaulting apaches.
>
well, segfaulting apache is actually an important thing to watch for, cos it shouldn't ever happen in a production environment -- right, ben?
the HUGE MASSIVE **bleeping** PROBLEM with using log data for this is that general purpose applications and operating systems completely suck at detecting malicious or unusual behavior. where by "completely suck," i mean:
-- take default installs of solaris, winNT, win2000, cisco IOS, linux
-- get your favorite copy of "armoring solaris," "armoring windows" blah
-- do the exact >>opposite<< of what the docs say -- that is, make the
target systems as wide open and vulnerable as possible
-- take your favorite vulnerability auditing tool, or hacker toolbox, and
set it to "nuclear apocalypse" level (that is, don't grab banners, run the
damn exploits)
-- turn the logging on the boxes to 11
---> the best you'll see is 15% of the total number of attacks, AND what shows up in the logs is invariably >not< the attack that actually rooted your box. as ben clarified for me a lot when i saw him last month, a root compromise that succeeds generally succeeds before the poor exploited daemon has a chance to write anything in the logs. what you see in the logs are traces of what >didn't< work, on the way to getting a memory offset or a phase of the moon just right and getting a rootshell.
[this number based on a summer's worth of lab work i did at counterpane;
needs to be repeated with more aggressive hacker tools and rather more
tweaking of system logging...]
> Any comments?
>
> As for Rodney's comment: For Network based ISDs, I think we need to not
> look at attacks, but look at abnormal traffic. EG: my web server should
> not do dns lookups. My mail server should not be ftping out. Nobody
> should be sshing/tsing into the DB server in the DMZ. My server should
> only send SYNACKs on port X,Y and Z.
>
> I'm pretty sure that snort could be made to do this - - but it doesn't do
> this today without some major rule foo.
well, anywhere you can build your network to have a really limited set of rules, this approach is easy. hell, don't want my web server doing name resolution or web >browsing< -- so run ipchains or iptables or pf and block everything but the allowed traffic and get the logs and alerts as a fringe benefit.
doing it for the content of web traffic is a little trickier but still completely plausible because web logs are relatively well structured. for some really vague definition of "relatively."
doing it for a general purpose windows server or a seriously multi-user unix environment is a lot harder, cos the range of acceptible behavior is so much larger and harder to parametrize. but hey, that's >>exactly<< what the log analysis list (and web site) are working on...
and is also why i am continually pestering my friends in the open source development community to give me logging that is more likely to catch things like client sessions that never terminate properly (to catch
successful buffer overflows) (give me a log when a session opens and >then< when it closes)...and things like significant administrative events (like sig updates in snort, or config changes on my firewall).
had a fascinating talk with microsoft today about getting them to modify their patch installer (the thing that windows update >runs<, not windows update itself) so that in addition to creating a registry key for a patch
when it >begins< an install, it also generates a checksum for the new files it's installed when it >finishes<, and then put that into the event log where i can centralize it and monitor it and have a higher level of
confidence that the damn patch succeeded... not that i've been thinking about this stuff a lot lately or anything.
tbird
This is my attempt of compiling a 'top list' of audit trails that are being left after intrusions where the intruders try to cover their tracks but don't do a good job. To put it short, there are actually a lot of audit trails on a normal UNIX system, which can almost all be overcome, but with some effort, that most intruders evade.
"The syslog daemon is a very versatile tool that should never be overlooked under any circumstances. The facility itself provides a wealth of information regarding the local system that it monitors.
"However, what happens when the system it's monitoring gets compromised?
"When a system becomes compromised, and the intruder obtains elevated root privileges, he now has the ability, as well as the will, to trash any and all evidence leading up to the intrusion, on top of erasing anything else thereafter, including other key system files.
"That's where remote system logging comes in, and it's real super-easy to set up..."
Related Stories:
Linux Journal: Stealthful Sniffing, Intrusion Detection and Logging(Sep 16, 2002)
Crossnodes: Use Snort for Lightweight Intrusion Detection(Jul 15, 2002)
Linux and Main: Preventing File Limit Denials of Service(Jul 10, 2002)
LinuxSecurity.com: Flying Pigs: Snorting Next Generation Secure Remote Log Servers over TCP(Jun 06, 2002)
December 23, 2000 (SANS)
There may be more elaborate third party proprietary solutions for log consolidation, but the syslog capability within UNIX is simple and ubiquitous on UNIX platforms. This paper will specifically deal with the Sun Solaris environment when not noted otherwise.
This paper is intended to assist a data center manager in setting up a centralized syslog server. There are a variety of commercial packages that deal with security and troubleshooting; however the use of the syslog facilities is common to all UNIX systems and most network equipment. The configurations defined here are tested in the Solaris 8 environment.
References :
... ...[1] "Log Consolidation with syslog" by Donald Pitts, December 23, 2000 http://www.sans.org/infosecFAQ/unix/syslog.htm
[2] Understanding and using the Network Time Protocol by Ulrich Windl, et al. http://www.eecis.udel.edu/~ntp/ntpfaq/NTP-a-faq.htm Analysis
"Automated Analysis of Cisco Log Files", Copyright © 1999, Networking Unlimited, Inc. All Rights Reserved http://www.networkingunlimited.com/white007.html
An Approach to UNIX Security Logging, Stefan Axelsson, Ulf Lindqvist, Ulf Gustafson,
Erland Jonsson, In Proceedings of the 21st National Information Systems Security Conference,
pp. 62-75, Oct. 5-8, Crystal City, Arlington, VA, USA, 1998, Available in Postscript and PDF.A Comparison of the Security of Windows NT and UNIX, Hans Hedbom , Stefan Lindskog, Stefan Axelsson, Erland Jonsson, Presented at the Third Nordic Workshop on Secure IT Systems, NORD-SEC'98, 5-6 November, 1998, Trondheim, Norway, Available in PDF.
Download:http://members.xoom.com/chaomaker/linux/xlogmonitor.tgzHomepage: http://members.xoom.com/chaosmaker/linux/XlogMonitor is a tool for monitoring the Linux system logs. It offers monitoring of standard logfiles like /var/log/messages or /var/log/syslog as well as icmpinfo logfiles and memory consumption.
[July 16, 1999] Autobuse Grant Taylor
Autobuse is Perl daemon which identifies probes and the like in logfiles and automatically reports them via email. This is, in a way, the opposite of logcheck in that autobuse identifies known badness and deals with it automatically, while logcheck identifies known goodness and leaves you with the rest.Download: http://www.picante.com/~gtaylor/download/autobuse/
Intrusion Detection
Splunk, which approaches network management by helping IT staff find the proverbial needle in a haystack, says 35,000 people have downloaded its search engine since it launched in August 2005. The company's Splunk Professional search software filters through all the logs and other data generated by IT systems, devices and applications so problems can be found and fixed faster, according to the company. It is priced at $2,500 for an annual license.
LogLogic also attempts to enhance network troubleshooting by capturing logs from all of a corporation's hardware and software in what it calls a log-management intelligence platform. Delivered as an appliance, LogLogic lets customers analyze, store, generate reports on data for compliance and risk mitigation, company officials say. The LogLogic Compliance Suite starts at $10,000.
Log data more relevantWhile he doesn't see log management falling under the definition of network management, RedMonk's Governor says companies such as LogLogic, Splunk and others are making log data more relevant for network managers.
"Network management tends to be real time; log management is after the fact - it's more about looking at what happened and analyzing that," Governor says. "These companies are making log management more of a real-time function, and then it becomes more valuable. It's moving from being a subset of security management to more of an application-management function."
Another company, GroundWork Open Source Solutions, is positioning its IT monitoring tool as costing a fraction of what commercial products go for. GroundWork Monitor Professional, based on open source components, including Nagios, RRDTool and MySQL, gives customers a central point for monitoring applications, databases, servers and network equipment, officials say.
GroundWork Monitor Professional costs about $16,000 for an annual subscription and is installed at "hundreds of enterprises," according to company officials.
Recommended Links
In case of broken links please try to use Google search. If you find the page please notify us about new location
Open Directory - Computers Software Internet Site Management Log Analysis
- Commercial (121)
- Freeware
freshmeat.net Browse project tree - Topic Internet Log Analysis
[PPT] Unix Tools for Web log Analysis
Logwatch
www2.logwatch.org
Logwatch is a customizable log analysis system. Logwatch parses through your system's logs for a given period of time and creates a report analyzing areas that you specify, in as much detail as you require. Logwatch is easy to use and will work right out of the package on most systems
Splunk
[Apr. 17, 2006] Splunk Welcome
Splunk is search software that imitates Google search engine functionality on logs. Can be considered as the first specialized log search engine.
It can correlate some alerts:
[Feb 16, 2006] Splunk, Nagios partner on open-source systems-monitoring tools - Computerworldby Todd R. Weiss
Log file search and indexing software vendor Splunk Inc. announced Tuesday that it will soon add systems management host, network and service monitoring capabilities to its software through a partnership with the Nagios open-source project.The new capabilities, which will be added to SplunkТs log file search and indexing applications over the next six months, will give systems administrators even more information to monitor and repair their networks, said Patrick J. McGovern III, chief community Splunker at San Francisco-based Splunk. Splunk has seen more than 25,000 downloads of its software since its 1.0 release came out in November, he said, while Nagios delivers about 20,000 downloads a month of its open-source network and service monitoring application.
The two working together is going to be a big win for both communities,Ф McGovern said.
McGovern is the son of Patrick J. McGovern, the founder and chairman of International Data Group Inc., which publishes Computerworld.
Splunk's free Splunk Server application runs on Linux, Solaris or BSD operating systems but is limited to searching and indexing 500MB of system log files each day. The full version, Splunk Professional Server, starts at $2,500 and goes up to $37,500 for an unlimited capacity. Both products search and index in real time the log files of mail servers, Web servers, J2EE servers, configuration files, message queues and database transactions from any system, application or device. Splunk uses algorithms to automatically organize any type of IT data into events without source-specific parsing or mapping. After the events are classified and any relationships between them discovered, the data is indexed by time, terms and relationships.
The integration of the Splunk applications and the Nagios features will take place in phases over the next six months, McGovern said.
The Nagios software allows monitoring of network services, including SMTP, POP3, HTTP, NNTP and PING, as well as monitoring of processor load, disk and memory usage, and running processes. Users can also monitor environmental factors in data centers such as temperature and humidity.
Other log file monitoring and management applications are available from competitors including Opalis Software Inc. and Prism Microsystems Inc.
Dana Gardner, an analyst at Interarbor Solutions LLC in Gilford, N.H., said Splunk's log file search capabilities bring a novel approach to IT management. Certainly on the theory basis, on the vision basis, it makes a lot of sense, Gardner said. Search can be a really useful tool to help systems administrators figure out a vexing problem amid a huge quantity of data.
It certainly gets at the heart of what keeps IT administrators up at night, managing complexity and chaos, Gardner said.
Harmonious Splunk - Forbes.com
Splunk's corporate-network search tool, released this month, aims to reduce the amount of time it takes datacenter administrators to diagnose problems across a server system. And with playful, techie-familiar terms on its Web site--from "borked" to "whacked"--the company's marketing approach seems oddly appropriate.
"A bunch of us came together three years ago to build a search engine for machine-generated data and make sense of it in real-time--much the same way Web search engines make sense of data on the Internet," says Michael Baum, the company's "chief executive Splunker."
Sound like something Google (nasdaq: GOOG - news - people ) might do? It should--Splunk's core leadership team has worked for many of the Internet's most famous search companies, from Infoseek to Yahoo! (nasdaq: YHOO - news - people ).
When IT systems fail, troubleshooters are often forced to manually parse server logs to find the glitch. Depending on the complexity of the system--and the problem--this can take hours or days, costing companies both in lost revenue and IT staff time.
Many IT specialists call this "spelunking"--mining through cavernous log files across a server system that can potentially add up to millions of pages of poorly formatted text.
The answer, Baum says, is Splunk, a browser-based search tool that mines a virtually unlimited amount of data from log files in real-time and allows system administrators to find trouble points across a network of servers from a single point of entry.
Baum says Splunk differs from traditional log-file analysis software in that it can digest data from virtually any source, ranging from Web applications running on a Sun Microsystems (nasdaq: SUNW - news - people ) machine to database logs on a Dell (nasdaq: DELL - news - people ) server to corporate voice-over-Internet protocol phone networks--even systems that haven't been created yet.
"We realized that we needed to be able to take any stream of data--anything--and be able to index it and allow people to search it," Baum says. The result is universal event processing, which Baum calls a "much more intelligent way of cutting through the hundreds of gigabytes of data that people see in their datacenter every day."
Instead of copying log file data directly into Splunk's database, the software uses algorithms to analyze and standardize the information so administrators can search and link processes from systems the way they're used to searching for things on the Web. Baum says this is especially practical for e-commerce datacenters, where customers often seamlessly bridge dozens of servers on their way from the homepage to checkout and order fulfillment.
Baum says Splunk also embraces collaboration, from its open source code base to a network of "Splunkers," who share information about event analysis on the company's Web site. Corporate versions of the software can also be configured to communicate securely in a peer-to-peer manner, so partner organizations can share information to track incidents with no geographical boundaries.
Also nontraditional is Splunk's approach to marketing, using terms such as "haxx0rd"--a reference to computer hackers--that resonate more with technical-support specialists than a company's vice president of sales. Some material is even racy: Splunk T-shirts urge users to "Take the 'sh' out of IT."
"All of our marketing and imaging is targeted at trying to be the brand that can try to win the hearts and the souls of systems administrators," Baum says.
While Baum says he can't yet disclose any organizations that use the product, he says thousands of people have downloaded the software, and the response has been positive. Users can download Splunk for free and companies can purchase Splunk Professional, a premium service that includes technical support and advanced features.
"We're a company that's building our whole business from a grassroots standpoint, trying to get systems administrators to download and try our software, even at home, and then bring it into work--we're building from the ground up rather than from the top down," he says.
Baum says Splunk plans to grow the same way search companies like Google did, starting with a simple, effective tool and expanding it to include more features over time. And if Splunk works as well at de-"borking" as Baum says it does, his strategy might just work.Perl-Based Tools
Devialog by Jeff Yestrumskas
Perl-based.
About:
devialog is a behavior/anomaly/signature-based syslog intrusion detection system which can detect new, unknown attacks. It fits comfortably in a heterogeneous Unix/Linux/*BSD environment at the core of a central syslog server. devialog generates its own signatures and acts upon anomalies as configured by the system administrator. In addition, devialog can function as a traditional syslog parsing utility in which known signatures trigger actions.Release focus: Minor bugfixes
Changes:
Bug fixes include better handling of lines with some special characters. A timing error was fixed within alert generation: sometimes alerts would be sent inadvertently based on the timing of a new log arriving as an alert was sent out in specific high-volume log situations. Altered signature generation creates more exact regular expressions.Epylog by Konstantin Ryabitsev Perl-based.
Epylog is a log notifier and parser that periodically tails system logs on Unix systems, parses the output in order to present it in an easily readable format (parsing modules currently exist only for Linux), and mails the final report to the administrator. It can run daily or hourly. Epylog is written specifically for large clusters where many systems log to a single loghost using syslog or syslog-ng.
LooperNG Perl-based, by Mohit Muthanna
LooperNG is an intelligent event routing daemon. Primarily used for Network Management, this application can be used to accomplish a variety of tasks related to logging and alerting such as trap forwarding/exploding, event enrichment, converting event formats (syslog->SNMP, SNMP->flatfile, syslog->Netcool), etc. It uses a system of input and output modules to interface with the event sources/sinks and a "rules file" to control the flow of the events.
Logrep A logfile extraction and reporting system by Tevfik Karagulle
Logrep is a secure multi-platform framework for the collection, extraction, and presentation of information from various log files. It features HTML reports, multi dimensional analysis, overview pages, SSH communication, and graphs, and supports 18 popular systems including Snort, Squid, Postfix, Apache, Sendmail, syslog, ipchains, iptables, NT event logs, Firewall-1, wtmp, xferlog, Oracle listener and Pix.
- Changelog:
- Download: http://www.l0t3k.net/tools/Loganalysis/LogrepSource-1.4.2.tar.gz
- License: GNU General Public License
- Platform(s): Windows NT/2000, Linux
Swatch
Monitors and filters log files and executes a specified action depending of pattern in the log. BAsed on Perl but actually designed for users who do not know Perl. Does not makes any sense for users who know Perl.
Multipurpose plug-in based analyzers
Sawmill log analyzer; log file analysis; log analysis program
Sawmill is a world-class web log analyzer, but it's more than just that. Sawmill can analyze all of your log files, from all of your servers. Sawmill can analyze
- Streaming media logs: RealPlayer, Microsoft Media, and Quicktime Streaming Server.
- FTP logs from most FTP servers.
- Proxy logs from most proxy servers.
- Firewall logs from most firewalls.
- Cache logs from most cache servers.
- Network logs: Cisco PIX, IOS; tcpdump, and more.
- Mail logs: SMTP, POP, IMAP from most mail servers.
- And dozens of others!
Sawmill is a universal log analysis tool; it's not limited to web logs. You can use it to track usage statistics on your proxy and cache servers. You can use it to bill by bandwidth. You can use it to monitor all of the TCP/IP traffic on your internal network, or through your firewall, to see who's eating all that expensive bandwidth you just bought. As an ISP/ASP, you'll find dozens of uses for Sawmill beyond just providing your customers with the best statistics.
And that's not all! Sawmill lets you define your own log format, using a very powerful format description mechanism that can be used to describe just about any possible log format. Do you have your own special logs that you'd like to analyze? Have you written a custom server that no other tool knows anything about? In a few minutes, you can have Sawmill processing those files just as easily as it processes web logs (in fact, we'll happily write the log format descriptor file, and include it with the next version of Sawmill). All the advantages it has in web statistics, it has with all other logs -- you'll get unprecedented detail and power for your statistics, no matter where the log files came from.
Lire A pluggable log analyzer which supports over 30 log types.
Most internet services have the ability to log their activity. For example, the Apache web server adds for each web page request a line with information to a log file. Depending on the log format the line includes information like the page that was requested, the size of the page, which web browser was used, and much more. In case of your email server, a similar log file is made. It contains the emailaddress that sent the email, who received it, how large it was, etc. As a matter of fact, all internet services have this capability.
These log files contain a enormous amount of information, but the format is hard to interpret by hand. You need a tool that makes summaries of the data to help you analyze the content. In case of www services this converts to TopX lists for web browsers, domains and platforms, and a hits versus time plot. Most counters via third parties show these kinds of overviews.
For most log file type tools are available to analyze the content. Lire is such a tool. But Lire is different from most other tools. Lire is an integrated system which is able to analyze not just one type of internet service, but many. And the reports that summarize the interesting information from the log file are plugged in. You can add custom report types yourselves.
Lire can be used in different ways. You can run it from the command line or have a crontab job installed that send you reports by email. In the former case you can choose the output format for the report. Current output formats include plain text, HTML, DocBook, PDF and LogML. In case of the crontab the only format is plain text at this moment.
Currently, the log files for these services can be analyzed:
Lire is in full development, currently with three payed people working on it. Support is one of their tasks, so if you have a special request (new services for example) or general support questions please leave a message on LogReport's SourceForge site.
Commercial support Welcome to LogReport: Log Reporting for the 21st Century
The LogReport Project is a technology initiative launched nearly two years ago to gather information/best practices on log analysis and produce premium software to perform these tasks. Today the LogReport Project's flagship software product, Lire, provides reliable, extensible solutions to server and network operators around the world. In addition, the LogReport team, a group of international experts, provides world-class professional services, insuring that our clients have the absolute best solution for their needs.
Email Servers:
- Sendmail
- Postfix
- qmail
- exim
- nms (Netscape Messanger Service)
- ArGoSoft
Message Store:
- Netscape Message Store
- Netscape Messaging Multiplexor
WWW Servers:
- Common Log Format (Apache, IIS, etc.)
- Combined Log Format (Apache, Boa, etc.)
- Referrer
- Apache mod_gzip
- W3C Extended (Microsoft IIS 4.0 & 5.0)
DNS Queries:
- DNS Bind version 8
- DNS Bind version 9
DNS Zones:
- DNS Bind version 8
Firewalls:
- Cisco
- Cisco PIX
- ipchains
- ipfilter
- iptables
- WELF
- Watchguard
FTP Servers:
- xferlog (WU-FTPD, ProFTPD, etc)
- IIS FTP
Print Servers:
- CUPS
- LPRng
Proxies:
- Squid
- WELF
- MS ISA
Databases:
- MySQL
- PostgreSQL
Syslogs:
- BSD-like
- Netscape Messaging Server
- Solaris 8
- Kiwi Syslog Daemon
- Sendmail Switch Log
Dialup Products:
- ISDN Log
Recommended Papers
Sawmill log analyzer; log file analysis; log analysis program
- Easy To Use
- Extensive Documentation
- Live Reports & Graphs
- Package of Powerful Analysis Tools
- Attractive Statistics
- Database Driven
- Advanced User Tracking by WebNibbler(tm)
- Very Fast
- Easy To Install
- Highly Configurable
- Works With a Variety of Platforms
- Processes Almost Any Log File
[Nov 11, 2000] The OutRider Computing Journal: Creating a Log Class in Perl
"One recurrent theme in my job as a database administrator/assistant systems administrator/systems analyst is the need to keep track of what happened on the systems while I wasn't watching. What did the cron job do last night. What did all those spooler daemons do while I was at lunch? In other words logging. It bothered me that there was a lack of simple tools for doing such a simple, redundant job. So, I set out to do build some myself. My systems programming tool of choice is Perl, so, that is language I chose for the project. This journey took me out of my normal routine of straight-line Perl programming and dumped me in the land of Modules and Object Oriented Perl. I'm glad to say it didn't overwhelm me and in fact I found it rather easy to write."
"My first order of business was to take my old standby logging routines and objectify them. I had several concise routines that I would either import into the main package through a use statement or just simply copy/paste depending on my mood and what I was doing. They consisted of four routines: start_logging, stop_logging, restart_logging and log."
"This was quick and dirty code that, while it did the job, was not very simple to use. For example, if I needed to redirect the output of a sub-process to the log file I would have to say: stop_logging(), then run that process and redirect its output to the log then restart_logging() again. It was Rather clumsy and difficult to document. So, I set about to rewrite the routines in an object oriented manner. I followed the 'Three little rules' as formulated by Larry Wall in the objperl(1) man page and restated by Damian Conway in his book "Object Oriented Perl"..."
**** Perl for System Administration Chapter 9 Log Files
It's been said that if you work on any program long enough, that program will eventually be able to send electronic mail. It doesn't matter what the original purpose of the program was (if you can still remember)--if you develop it long enough, some day that program will send its first piece of email.
From the vantage point of a systems or network administrator, this means there are lots and lots of programs out there generating mail daily. Mail filters like procmail can help us with this deluge by sorting through the mail stream. But sometimes it is more effective to write sophisticated programs to actually read the mail for us. For example, we might write a program to analyze unsolicted commercial email (spam) or one that keeps long-term statistics based on daily diagnostic email from a server.
Web Techniques: Programming With Perl - Web Access Logs with DBI (Apr 16, 2000)
Etc
logcheck
a useful program for scanning system logs for unusual activity.
LogCheck works by scanning the various system log files (under Linux these are located in /var/log), and notifying the system administrator by e-mail if there is any unusual activity. Unusual messages in the log files can often be generated by intrusion attempts, or actual intrusions against your system.Installing LogCheck
LogCheck is available in RPM format from the Red Hat contrib archives, and from the same sources as PortSentry. Installing LogCheck from the RPM file or from the source code (read the INSTALL file provided with the source code) is relatively simple.
Configuring LogCheckLogCheck has four main configuration files. In the RPM version, these are stored in the /etc/logcheck directory. Normally, only the logcheck.ignore and the logcheck.violations.ignore files need modification. The normal process that I go through after installing LogCheck is as follows:
- Allow LogCheck to run once with the standard configuration files. This willl produce a large output file, which can be thrown away.
- 24 hours later, allow LogCheck to run again. This will detect any new entries in the log files since the last run, and will produce a smaller but still sizeable output file. Read this file carefully.
- For entries in the file that are of no great concern (use your judgement for this) find a specific identifying string in the entry. For entries that are in the "Security Violations" section, add the identifying string to the logcheck.violations.ignore file. For other entries (in the "Unusual System Events" section), add the string to the logcheck.ignore file.
- Repeat this process, once every 12 - 24 hours for approximately a week. By this stage, enough "bogus" entries will be filtered out by the strings that you have added to the .ignore files that the daily LogCheck report will contain only genuine system concerns.
Note that the RPM file specifies that LogCheck is to be run hourly, but normally I only run it daily except on critical systems that need regular monitoring. This is done by moving the /etc/cron.hourly/logcheck file into /etc/cron.daily.
Analyzes any text-based log files "on-the-fly" using contexts and executes a corresponding action.
Availability: anonymous ftp at ftp.cert.dfn.de
Web site: Logsurfer Homepage
Copyright © 1996-2008 by Dr. Nikolai Bezroukov. www.softpanorama.org was created as a service to the UN Sustainable Development Networking Programme (SDNP) in the author free time. Submit comments This document is an industrial compilation designed and created exclusively for educational use and is placed under the copyright of the Open Content License(OPL). Original materials copyright belong to respective owners. Quotes are made for educational purposes only in compliance with the fair use doctrine.
Standard disclaimer: The statements, views and opinions presented on this web page are those of the author and are not endorsed by, nor do they necessarily reflect, the opinions of the author present and former employers, SDNP or any other organization the author may be associated with. We do not warrant the correctness of the information provided or its fitness for any purpose.
Created: May 16, 1997; Last modified: December 12, 2008