When you have a big list and you need to remove items from that list.
Is it better to remove the items from that list or is it better to create a new list?
When you remove an item from a list, the list will shift all the elements after the removed item to the left to fill the gap. For example, if you have a list [1, 2, 3, 4, 5] and you remove the item at index 2 (which is the value 3), the list will become [1, 2, 4, 5]. The element at index 3 (which was previously the value 4) will now be at index 2, and so on.
This operation can be expensive, especially if the list is large and you are removing multiple items. It is usually more efficient to create a new list and add only the items that you want to keep, rather than removing items from an existing list.
On the other hand, creating a new list involves allocating a new block of memory and copying the elements that you want to keep into the new list. This can also be a time-consuming operation, but it is usually faster than removing items from a list, especially if you are only removing a small number of items.
Let’s take an example. I am reading a 10 MB Excel file that has 46 columns and 54,000 rows.
In the first method, I am passing the List of items and apply some filters. Based on those filters I am creating two lists. One list with valid records and another one with invalid records
In the second method, I am passing the same List of records when reading the file but in this case, I am removing the items from the first list and adding the removed records to a new list.
To measure the time that a method takes to execute in .NET Core, you can use the Stopwatch class from the System.Diagnostics namespace. Here’s an example of how you can use it:
using System.Diagnostics;
//...
Stopwatch stopwatch = Stopwatch.StartNew();
// Call the method that you want to measure here
stopwatch.Stop();
Console.WriteLine($"Elapsed time: {stopwatch.ElapsedMilliseconds} milliseconds");
This will start the stopwatch, run the method, and then stop the stopwatch after the method has completed. The ElapsedMilliseconds property of the Stopwatch class will give you the elapsed time in milliseconds.
Testing
Test 1: In this test 847 records are removed from the list that is 55143 in length.
In this example, we observe that creating 2 new lists are 7 milliseconds faster than removing the items from the list.
Test 2: In this test, no items are removed from the lists because. In this example creating two lists instead of removing the items from the list is 14 milliseconds slower. Showing that the operation of shifting the items is what causes the inefficiency when removing items from the list.
In general, it is a good idea to avoid modifying a list in place if you can, especially if you are working with large lists. Instead, you can create a new list and add only the items that you want to keep. This will usually be more efficient than modifying the original list.
David Mata blog 