Understand all the words but still confused

I often understand all the words in a sentence but I don't understand the meaning of the whole sentence. An example from today:
When I arrived at church one of the youth said these words "look don't know confused". I thought he meant I looked confused and didn't know where to go. I rode my bike to church for the first time, usually I sit in the car and Soeun drives. So I was a bit confused, I couldn't remember which house the church was in, they all look the same.
But he actually meant that we was confused when he looked at me. He thought he didn't know me because I looked different. I was wearing my bike helmet and mask to keep off the sun and dust.