Marcus Lynn

← All posts

Post 8

Parsing Some Massive CSV Files

Recently I had to conduct an investigation that involved a 14GB CSV file, coming out to about 27.5 million lines. Unfortunately no program (Excel, NPP, etc) can handle that much D A T A, so to Python we go! I wrote this up in about 5 minutes and it took about another 10-15 to parse the entire file for what I was looking for. It will open up the CSV and go line by line, saving any line it finds to the “outFile” which can be reviewed in Excel.

import csv
count = 0
countLimit = 1000000 #Hard stop limit/base case for how many lines you want to parse. Set to zero or -1 if you want to parse the whole csv.
outFile = open("sample4.csv", "w", encoding='utf-8') #Outfile for results found. Name what you'd like.
searchByString = "Search by this string" #Term to search for in each line of the CSV.
currentRow = ""


with open('DeleteQuery.csv', 'r', encoding='utf-8') as file: #Replace DeleteQuery.csv with the name of your CSV file.
    reader = csv.reader(file)
    try:
        for row in reader:
            currentRow = str(row)
            if(searchByString in str(row)):
                outFile.write(str(row) + "\n") #Goes row by row checking for the search string.
            count = count + 1
            if(countLimit > 0 and count >= countLimit):
                break
    except:
        print("I broke on line: ", count) #Sometimes you might run into an error~

print("Final Row: \n", currentRow)
print("Total count: ", count)
outFile.close()

Pretty quick and dirty way of doing it but for the time invested not bad!

Cheers!

~Marcus

← All posts