PowerShell – Splitting large log files

Split LogHave you ever found an issue with one of your systems and gone digging for the log file, only to find out that the file is too big to open for reading? There are a number of ways you can use in the *nix world, but for the Windows folks I’ve created a simple script to let you split the larger file into smaller chunks for easier reading and manipulation using PowerShell.

These are your assumptions for this script to work for you:

  • Must be run interactively
  • You know the full path and filename of the source file (use double quotation marks to surround the file/path if there are spaces)
  • You know the path to the destination (also, use double quotation marks to surround the file/path if there are spaces)
  • You have a decent amount of memory (the process loads the entire file…yes…the entire file into RAM while parsing the contents
  • You have some time to let it bake in the background. It’s better to wait than to not be able to read the file right?
  • You are perfect. There is no error handling (yet). If you make a mistake it will end the way that this gentleman’s IT experience did:

    What are you doing Dave?

    What are you doing Dave?

To start we set some initial counters. I’m going old school on this to generate the file names for the output chunk files.

# Set the baseline counters
#
# Set the line counter to 0
$linecount = 0
# Set the file counter to 1. This is used for the naming of the log files
$filenumber = 1

Now comes the interactive part where we prompt the user (that’s you) for the path and filename for the source file. I’m working on making this a GUI file picker, but for now let’s just stick with what works. I figure that if you are able to parse a file for content, then I’m assuming you know how to type a file path correctly.

# Prompt user for the path
$sourcefilename = Read-Host “What is the full path and name of the log file to split? (e.g. D:mylogfilesmylog.txt)”

# Prompt user for the destination folder to create the chunk files
$destinationfolderpath = Read-Host “What is the path where you want to extract the content? (e.g. d:yourpath)”

Because I’m a nice guy, I like to let the user (still you) know that the next process will take a little while.

Write-Host “Please wait while the line count is calculated. This may take a while. No really, it could take a long time.”

This is where the memory intensive part begins. I’m not an expert programmer, and you may have a better way to do this (feel free to add comments if you have any suggestions). I first read the file to capture a line count so that you can decide how many chunks you want to create. I’ve also assumed you don’t mind a little math.

# Find the current line count to present to the user before asking the new line count for chunk files
Get-Content $sourcefilename | Measure-Object | ForEach-Object { $sourcelinecount = $_.Count }

#Tell the user how large the current file is
Write-Host “Your current file size is $sourcelinecount lines long”

Now that you know the size of the original file, the next step is to ask how big the destination files need to be. This will request a number, but as a string, which is converted into an integer assigning a variable and using [int]$destinationfilesize to convert the string.

# Prompt user for the size of the new chunk files
$destinationfilesize = Read-Host “How many lines will be in each new split file?”

# the new size is a string, so we convert to integer and up
# Set the upper boundary (maximum line count to write to each file)
$maxsize = [int]$destinationfilesize

Let’s tell the user what our results are from the data collection up to this point and then we can begin the process.

Write-Host File is $sourcefilename – destination is $destinationfolderpath – new file line count will be $destinationfilesize

Here we go! This goes back to programming 101 by using a simple counter and a loop until we reach the threshold of that counter. The output files will be named splitlog(some number).txt and will each contain the number of lines of text that we’ve told the script from above.

Final warning! This loads the file into memory which will be pretty resource intensive but the end result is a happy, more easily managed set of files.

# The process reads each line of the source file, writes it to the target log file and increments the line counter. When it reaches 100000 (approximately 50 MB of text data)
$content = get-content $sourcefilename | % {
 Add-Content $destinationfolderpathsplitlog$filenumber.txt “$_”
  $linecount ++
  If ($linecount -eq $maxsize) {
    $filenumber++
    $linecount = 0
  }
}

Because we’ve eaten up lots of memory and processor we will need to free up some of the “garbage” we’ve left behind. It’s effectively a PowerShell poop-n-scoop.

# Clean up after your pet
[gc]::collect()
[gc]::WaitForPendingFinalizers()

 And there you have it! Hopefully you find it useful.

Here is the full script – http://gallery.technet.microsoft.com/PowerShell-Split-large-log-6f2c4da0

Logs

That's much better




PowerShell – Move Windows 7 Computers into OU structure by IP address (Active Directory)

Hi Folks!

I’f you’ve come to this article because of a permalink then I’ve got some news for you! The article has been selected to be posted on the Microsoft Scripting Guys blog as a guest article (insert applause).

You can now find the article in 2 parts here:

Use PowerShell to Move Computers Based on IP Addresses: Part 1

Use PowerShell to Move Computers Based on IP Addresses: Part 2 (coming 12/4/2011)

 




Nice Flash messages in Rails 2 and Rails 3

Ruby on RailsI’ve been doing some Rails work recently and one of the things that I’ve found to be a nice aesthetic add-on is putting some nicer flash messages in.

Over at dzone.com there is an example which I’ve worked off of – http://snippets.dzone.com/posts/show/3145

The example is as follows. Add this section to your application_helper.rb

def show_flash
[:notice, :warning, :message].collect do |key|
content_tag(:div, flash[key], :class => “flash flash_#{key}”) unless flash[key].blank?
end.join
end

In your CSS file you now add the settings to style your messages. Note that my examples assume some images are present so add images as you wish for within the message:

.flash_notice {

BORDER-RIGHT: #090 4px solid; PADDING-RIGHT: 10px; BACKGROUND-POSITION: 5px 50%; BORDER-TOP: #090 4px solid; PADDING-LEFT: 10px; BACKGROUND-IMAGE: url(/images/icon_success_lrg.gif); PADDING-BOTTOM: 10px; MARGIN: 2px; BORDER-LEFT: #090 4px solid; TEXT-INDENT: 40px; PADDING-TOP: 10px; BORDER-BOTTOM: #090 4px solid; BACKGROUND-REPEAT: no-repeat

}

.flash_warning {

BORDER-RIGHT: #c60 4px solid; PADDING-RIGHT: 10px; BACKGROUND-POSITION: 5px 50%; BORDER-TOP: #c60 4px solid; PADDING-LEFT: 10px; BACKGROUND-IMAGE: url(/images/icon_warning_lrg.gif); PADDING-BOTTOM: 10px; MARGIN: 2px; BORDER-LEFT: #c60 4px solid; TEXT-INDENT: 40px; PADDING-TOP: 10px; BORDER-BOTTOM: #c60 4px solid; BACKGROUND-REPEAT: no-repeat

}

.flash_error {

BORDER-RIGHT: #f00 4px solid; PADDING-RIGHT: 10px; BACKGROUND-POSITION: 5px 50%; BORDER-TOP: #f00 4px solid; PADDING-LEFT: 10px; BACKGROUND-IMAGE: url(/images/icon_error_lrg.gif); PADDING-BOTTOM: 10px; MARGIN: 2px; BORDER-LEFT: #f00 4px solid; TEXT-INDENT: 40px; PADDING-TOP: 10px; BORDER-BOTTOM: #f00 4px solid; BACKGROUND-REPEAT: no-repeat

}

In your view, which in my case is the views/layouts/application.html.erb (same for Rails 2 or 3) I have added this where I want my flash notices:

<%= show_flash %>

If you are running Rails 3 there are two additional steps you need to take. First you need to change the code on your view to display as follows:

<%= raw show_flash %>

Note that under Rails 3 the flash will render the html code as text rather than as the raw html. By adding the raw statement you fix this issue and it will render correctly.

The second change you want to make is for your scaffolds. By default the show.html.erb file already contains a message section. If you leave this in place you will end up with 2 flash messages on each render of the show action. To fix this, go into the show.html.erb under your app/view/scaffoldname/ folder and remove this line:

<p id=”notice”><%= notice %></p>

Here are the 3 image files that I’ve used as referenced in the CSS

That’s my hopefully useful Rails tip of the day!




Ruby on Rails tip – generating a project with specific rails version

Let me qualify that I am sporting the “noob” title with my Ruby on Rails development. That being said, I hope that I can offerRuby on Rails some small help to other new RoR folks as I learn the ropes myself.

This is a simple tip which I have found very handy since I’ve updated my rails to 3.0.0 this week. If you need to generate a new project but need to use a “legacy” version of rails then you simply include the version within your code using this syntax:

rails _version_ projectname

So my current need to use 2.3.5 for a site uses this command line:

rails _2.3.5_ mysite

Small tip but big help for those like me who have to manage and support multiple versions of Rails across environments.