This is a common misconception, that all Swedes speak English so there is no point in learning the language if you speak English. While it is true that many Swedes can speak English, most learned English at school and do not regularly practice speaking it.
This comment comes from a native English speaker, who now lives in Sweden (found in The Local’s forum):
I don’t know where people keep getting this idea that all Swedes speak English as a matter of course during their days. Any Swede under the age of 60 learnt English while they were in school. 90% of them have never used it again since the day they walked out of the school gates. Most Swedes CAN speak some English, if they need to. Do they all walk around town speaking English ? No. Oddly enough, Sweden’s medium of communication is Swedish.