Python Concatenate Lists
5 minute read Updated: July 6, 2021
Get The Newsletter
Subscribe to our newsletter for the latest tips and tricks!
We won't spam you.
Concatenate Two Lists in Python
Problem: You have two lists and youd like to join them into a new list. Solution:
TLDR: Use +
In almost all simple situations, using list1 + list2 is the way you want to concatenate lists.
The edge cases below are better in some situations, but + is generally the best choice. All options covered work in Python 2.3, Python 2.7, and all versions of Python 31.
Combine Lists In Place In Python
Problem: You have a huge list, and you want to add a smaller list on the end while minimizing memory usage.
In this case, it may be best to append to the existing list, reusing it instead of recreating a new list.
As with any optimization, you should verify that this reduces memory thrash in your specific case and stick to the simple idiomatic x + y otherwise.
Lets use the timeit module to check some performance numbers.
In this example, where x is 3000 elements, extend is around 50x faster.
Concatenating Lists With Large Elements is Fine
If the elements in your list are huge [million character strings], but the list size is less than a thousand elements, the previous solution x + y will work just fine. This is because Python stores references to the values in the list, not the values themselves. Thus, the element size makes no difference to the runtime complexity.
In this case, extend does not have an advantage.
Avoid Chain From itertools For Two Lists
It is possible to use chain from itertools to create an iterable of two lists.
We can check the performance of using chain:
Using chain with two lists is slower in all cases tested, and x + y is easier to understand.
Combining N Lists in Python
If you need to add three or even ten lists together and the lists are statically known, then + for concatenate works great.
Flatten a List of Lists in Python
However, if the number of lists is dynamic and unknown until runtime, chain from itertools becomes a great option. Chain takes a list of lists and flattens it into a single list.
chain can take anything iterable, making it an excellent choice for combining lists, dictionaries, and other iterable structures.
Performance of Flattening a List of Lists
Performance doesnt always matter, but readability always does, and the chain method is a straightforward way to combine lists of lists. That said, lets put readability aside for a moment and try to find the fastest way to flatten lists.
One option is iterating ourselves:
Lets check its performance vs chain:
This shows that chain.from_iterable is faster than extend.
Flattening and Merging Lists With One Big List
What about adding a list of lists to an existing and large list? We saw that using extend can be faster with two lists when one is significantly longer than the other so lets test the performance of extend with N lists.
First, we use our standard chain.from_iterable.
We then test its performance:
Next, lets try concatenating by adding everything onto the long list:
Performance Test:
There we go, extend is much faster when flattening lists or concatenating many lists with one long list. If you encounter this, using extend to add the smaller lists to the long list can decrease the work that has to be done and increase performance.
Summary
These are the main variants of combining lists in python. Use this table to guide you in the future.
Also, if you are looking for a nice way to standardize the processes around your python projects running tests, installing dependencies, and linting code take a look at Earthly for Repeatable Builds.
2 lists | x + y | No |
1 large list, 1 small list | x.extend[y] | Yes |
Known number of N lists | x + y + z | No |
Unknown number of N lists | list[chain.from_iterable[l]] | No |
List of Lists | list[chain.from_iterable[l]] | No |
1 large list, many small lists | for l1 in l: x.extend[...] | Yes |
I did all the performance testing using Python 3.9.5 on MacOS BigSur.︎
If you dont have a performance bottleneck, clarity trumps performance, and you should ignore the performance suggestions.
Spreading the word about Earthly. Host of CoRecursive podcast. Physical Embodiment of Cunninghams Law
Categories: Tutorials
Author: Adam Gordon Bell
Updated: July 6, 2021
Previous NextGet notified about new articles
We won't send you spam. Unsubscribe at any time.