# How can I read a csv that has a size of 20 GB?

**URL:** https://community.konduit.ai/t/how-can-i-read-a-csv-that-has-a-size-of-20-gb/920
**Category:** DataVec
**Created:** [October 11, 2020, 9:46pm UTC](https://community.konduit.ai/t/how-can-i-read-a-csv-that-has-a-size-of-20-gb/920 "2020-10-11T21:46:56Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![yuniel-acosta](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/yuniel-acosta/32/105_2.png) [@yuniel-acosta](https://community.konduit.ai/u/yuniel-acosta)
#### Post date: [October 11, 2020, 9:46pm UTC](https://community.konduit.ai/t/how-can-i-read-a-csv-that-has-a-size-of-20-gb/920/1 "2020-10-11T21:46:56Z")

</div>

I need to read a CSV piecemeal to vectorize. The CSV is approximately 20 GB in size. I need to preprocess the file to remove certain columns and divide it into training data and test data. How can I do it without getting an out of memory error?

 ![image](https://canada1.discourse-cdn.com/flex035/uploads/konduit/original/1X/da1fa543bc8d36254cc5a0f385b8883e24a2c57a.png)

---

<div class="post-metadata">

### Author: ![agibsonccc](https://yyz1.discourse-cdn.com/flex035/user_avatar/community.konduit.ai/agibsonccc/32/697_2.png) [@agibsonccc](https://community.konduit.ai/u/agibsonccc)
#### Post date: [October 12, 2020, 12:56am UTC](https://community.konduit.ai/t/how-can-i-read-a-csv-that-has-a-size-of-20-gb/920/2 "2020-10-12T00:56:31Z")

</div>

@yuniel-acosta I would do it bits at a time if possible. There shouldn’t be a need to do everything in memory all at once. Could you share what code you’re using where you’re running out of memory?