I've always heard from people that Colin Baker was the least liked and with that coat it helped reinforce that idea.
He had a horrible costume, but I really don't think it was to blame. He looked a right ****, don't get me wrong, but you can't judge the character based on that. I remember the meltdown fandom went into when the first pictures of Matt in his bow tie leaked. Everyone was so ready to hate on it, but now I couldn't imagine him wearing anything else. If anything, it's probably turned into the most popular costume of any Doctor. The writing of the character and the poor quality stories were the let down. Certainly not the coat.
