Timezone Abbreviations

Oct 6, 2011 at 6:23 PM

Hello -

What a great tool.  I am very impressed.

I have been building and testing several format definitions for text-based log files that my organization has to regularly review for customer "Root Cause Analysis" requests.  The one that I cannot figure out yet is:

log file example:

 

zzz ***Sun Sep 18 14:03:44 EDT 2011 Sample interval: 5 seconds. All measurements in KB (1024 bytes)

14:03:45 This happened

14:03:46 The next thing happened

14:03:47 The thing after that happened

 

The first line is not the typical header, in that it does not appear at the beginning of each log entry "record".  Rather, It only appears as one, single occurrence at the very beginning of the entire log file.

Since each log entry "record" only has HH:mm:ss, I will have to come up with some way to pull month, day and year information from this one, single occurrence at the very beginning of the entire log file.

 

Any thoughts?

Coordinator
Oct 8, 2011 at 8:30 AM

Hello, 

Thank you for using LogJoint.

The problem you have encountered is actually quite common. Logs sometimes don't have "date" part in each log line. Instead date might be written at the beginning of log, or encoded into filename, or dumped in the log periodically. LogJoint doesn't provide a generic solution for this problem now. What you can do is to write a "format extension". Basically an extension is a way to inject your arbitrary code into different stages of log parsing. An example of an extension is what has been done to parse sysinternals log files.

It is not difficult to write an extension (if one knows c# :) ). However currently the interface between extensions and the main application is not defined precisely. Before fixing the interface I want to analyze common use cases. I would like you to provide me with sample logs and format definitions that you have already built . I want to write an extension for you for this thought log format. I am going to add this to the solution and cover with few automated tests. This will guarantee that your formats will work even is I change internal interfaces. Hope it is ok for you to put your format definitions and log samples to publicly available place like codeplex. You don't have to provide real logs. These might be samples without sensitive data.

Few questions about the format in question:

- Is date record "Sun Sep 18 14:03:44 EDT 2011" locale-dependent? In other words will it be the same if log is written on the system with different regional settings?

- Is there any handling of the situation when log lasts longer than 24 hours?

- Is there any handling of power events? Let me explain: the machine can be hibernated at 14:03 and waken up at 14:04 one week later. It there any way to know that time jump from log file?

- How big logfiles usually are?

Nov 3, 2011 at 1:47 PM
Edited Nov 3, 2011 at 2:00 PM
Sergeys –

Thank you very much for your kind response and offer to help. I apologize profusely for such a long delay. I simply could not squeeze getting the logs and responding into my work schedule.

    * The date record "Sun Sep 18 14:03:44 EDT 2011" is locale-dependent.   The EDT would instead be MDT here in the rocky mountain states, and would be PST on the Pacific coast.  Other time zones would vary similarly.
    * Each log covers a one-hour period. A new log file is started each hour. Logs “time-out” and are deleted after 7 days.
    * Because of the nature of the use of these systems, it would be very unlikely that hibernation would ever be used. More likely, might be for the machine to be powered down for a week. If it were, I believe that logging would go into a new file, because of the date-timestamp portion of the log file name.
    * The logfile that I have chosen is a common size. In some troubleshooting situations, I have seen logs (for the same one-hour period) that were almost twice the size.

As you suggested, I have compressed (7-zip) and attached a sample log file, FQDN_top_11.11.03.0800.dat.zip

I have also attached the work that I have done so far in LogJoint for this log file format as OSWtop.format.xml

At your service,

Dave Marshall

 

Coordinator
Nov 4, 2011 at 7:33 AM

Sorry Dave, I can't find your attachment in this discussion thread nor in my inbox. Can you send it directly to suhanovsergey gmail.com?

Coordinator
Nov 10, 2011 at 6:40 PM

Thank you, I got the archived data.

I just committed the changes I made for you. You have to check in and compile the latest version of logjoint by building logjoint.sln solution. Do you need help with that?