python - read large csv into pandas by chunks

chunks=pd.read_table('filename',  chunksize=500000)
df=pd.DataFrame()
df=pd.concat((chunk==1) for chunk in chunks)

remove deplicates

To remove duplicated rows:


awk '!seen[$0]++' <filename>

To remove rows with duplicated field (say $1 is ID and need to remove the entire row if ID is duplicated):

awk '!seen[$1]++' <filename>

filter a file based on tokens in another file

BEGIN{

  FS="|"

  OFS="|"

  while ((getline < (“Token_list_file.csv")) > 0) {

  id[$1]=$1;

  }

}

{

  appid = $1;

  if(appid in id) {print $0;}

}

select records based on a list of indexies

BEGIN{

  FS="|"

  OFS="|"

  while ((getline < ("common_LoanNumber.pip")) > 0) {

  id[$1]=$1;

  }

}

{

  appid = $1;

  if(appid in id) {print $0;}

}

join multiple text files with same headers

If each file has a header line but in the output file you only want to have one header line:

awk '

    FNR==1 && NR!=1 { while (/^LoanNumber/) getline; }

    1 {print}

'  input_file1 input_file2 input_file3 ... > output_file

AWK splice text into chunks

This code slice the first 10K rows into F1, then next 10K rows into F2, and so on

awk 'NR%10000==1{x="F"++i;}{print > x}'  filename

plot a ROC curve

import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

from matplotlib.ticker import AutoMinorLocator

#plt.switch_backend('agg')

dt = pd.read_csv('cs_2017_roc_gain.csv', sep=",", header=0)

fig = plt.gcf()

plt.plot(dt.x_gain * 100.0, dt.y * 100.0)

plt.xlim([0, 100])

plt.ylim([0, 100])

ax = fig.gca()

ax.set_xticks(np.arange(0, 101, 10))

ax.set_yticks(np.arange(0, 101, 10))

minor_locator = AutoMinorLocator(2)

ax.xaxis.set_minor_locator(minor_locator)

ax.yaxis.set_minor_locator(minor_locator)

ax.set_xlabel(r'% of Transactions')

ax.set_ylabel(r'% of Chargeoff Accounts')

fig.suptitle('Consumer')

#ax.minorticks_off()

plt.grid(which='minor')

plt.grid(which='major')

plt.show()

fig.savefig('Consumer_2017.jpg', dpi=300)

print dt.head()