Splitting CSV files in PowerShell with Split-Csv

In the course of my work, I am often times dealing with large CSV files that need to be broken up into smaller files, for whatever reason.  If the CSV file has a header, I can’t simply use something that breaks up the file by number of lines because I need to retain the header for use in each file.  So, I wrote up a script that I call Split-Csv.

Split-Csv takes a file path and a batch size as parameters, Path and BatchSize, respectively:

Split-Csv <InputFilePath> -BatchSize <int>

Script contents:

    [Parameter(Mandatory=$True,Position=1,HelpMessage="File path")]
    [Parameter(Mandatory=$True,HelpMessage="Batch size")]

If(!(Test-Path $Path -ErrorAction SilentlyContinue)) {
    Write-Error "$Path not found."

$inputFile = Get-Item $Path

$ParentDirectory = Split-Path $Path

$strFilename = dir $Path | Select-Object BaseName,Extension

$strBaseName = $strFilename.BaseName
$strExtension = $strFilename.Extension

$objFile = Import-Csv $Path

$Count = [math]::ceiling($objFile.Count/$BatchSize)

$Length = "D" + ([math]::floor([math]::Log10($Count) + 1)).ToString()

1 .. $Count | ForEach-Object {
    $i = $_.ToString($Length)
    $Offset = $BatchSize * ($_ - 1)
    $outputFile = $ParentDirectory + "\" + $strBaseName + "-" + $i + $strExtension
    If($_ -eq 1) {
        $objFile | Select-Object -First $BatchSize | Export-Csv $outputFile -NoTypeInformation
    } Else {
        $objFile | Select-Object -First $BatchSize -Skip $Offset | Export-Csv $outputFile -NoTypeInformation

Easy enough.

There are a couple of things of note.

1) I just drop these back into the same directory as the original; and,
2) I name the output files the same as the original appended with the number represented by their batch, zero-padded.

That’s it.


One thought on “Splitting CSV files in PowerShell with Split-Csv

  1. Nice script, worked as expected, took me a little while to figure out what the “$Length = “D” + ([math]::floor([math]::Log10($Count) + 1)).ToString()” was doing, but after debugging it became clear


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s