It really depends how long we have Matt's Doctor, right now I think Pattinson is too young but who knows once he gets to Matt's age? Trouble is, no matter how good he may be in the role, there will always be a) Anti-Twilight brigade hating him for that and b) the more, um, 'involved' Twilight fans hating him for not being a vampire.
Honestly, as long as people continue to associate Robert with Edward, that's always gonna be an issue. Part of it is that people have a problem seeing past typecasting. If twilight fans/haters could see past that, then they would appreciate his take on the role a lot better.