That's interesting. Do you feel like it's "performance pressure"? Do you find you are distracted by the thought of people watching you? For me, it started when my partner showed an interest in watching me jerk off. It turns her on to watch me pleasure myself and she can alternate between kissing and watching. She likes to hold my balls while she wat. Anyway, I found that being watched along with the appreciative comments my penis inspires is a huge turn on. The thought of other people (men and women) watching me and touching themselves is an incredible trn on,and I find it easy to stay hard and even put on a show edging the tip till I bust a juicy nut!
If it helps, think about the pleasure you bring others by pleasuring yourself.