To remove duplicates from a list in Python, you can:
Keep reading to see which one fits your case.
A set is a collection of unique elements. Converting a list into a set will remove all elements that are duplicated, i.e. appear the second time or more.
Remember that using this way can change the order of the elements in the list. If you want to keep the order, use the next method.
dict.fromkeys() is a method that creates a new dictionary from a list of keys. The keys are unique, so it will remove all duplicates.
This method is fast and keeps the order of the elements. However, it can only be used for lists of strings or integers. If you have a list of lists, you can't use this method.
This method is the most straightforward. You create a new list and add elements to it if they are not already in the list.
This method is not efficient. It has a time complexity of O(n^2) because of the for loop and the
in operator. If you have a large list, it will take a long time to run.
However, it can also be used for different data types inside the list. For example, you can also remove duplicate lists from a list using this method.
itertools.groupby() is a function that groups elements in an iterable. It groups elements based on a key function. If the key function returns the same value for two elements, they will be in the same group.
Also, the list must be sorted so that all duplicates are next to each other.
This method is also easy to implement. It can also handle different data types in the list.
But as I mentioned, the order of elements is not kept so use it with caution.
To recap, there are 4 ways to remove duplicates from a list in Python:
- Use a set:
list(set(names))(fastest, order ignored)
list(dict.fromkeys(names))(fast, order kept)
- Create a new list (slow, order kept, can be used for different data types in list)
[k for k, g in itertools.groupby(sorted(names))](fast, order ignored)
Each method is a fit for different cases. Pick one that fits your case the most.
If you have any questions, feel free to leave a comment below.