Optimize performance of flatten function in python

One avenue to try improve your speed is to use the Pythonic principle that it's Easier to Ask Forgiveness than beg Permission. This means that instead of testing your data every single time, you should just assume that it's in the correct form and catch the errors when you've gotten it wrong.

For instance, you test your dictionary here:

if isinstance(v[0], dict):
    counter = 0
    for entry in v:
        new_count_key = new_key + sep + str(counter)
        items.extend(self.flatten(entry, new_count_key, sep=sep).items())
        counter += 1
else:
    items.append((new_key, v))
    if new_key not in self.key_list:
        self.key_list.append(new_key)

What if instead you just assume that you have a list, and then work with the dictionary if you don't:

try:
    items.append((new_key, v))
    if new_key not in self.key_list:
        self.key_list.append(new_key)

except AttributeError:  # Is a dict, not a list
    counter = 0
    for entry in v:
        new_count_key = new_key + sep + str(counter)
        items.extend(self.flatten(entry, new_count_key, sep=sep).items())
        counter += 1

What you're doing here is attempting to append to items with the assumption that it's a list. If items is a dictionary and not a list, then your except catches the AttributeError that's raised and it will run the code for the dictionary format instead.

The advantage to this, is that you're not detecting the type, you're just catching when it's not a list. That means you spend no time calculating the datatype, you only need to spend time catching the exception when it comes up. This works best if the except catches the rarer datatype. So with my proposed solution, if most items will be lists then you'll save a lot more time than if most items end up being dictionaries.