Patricia L. Liberty has written an excellent short paper on why sexual contact between a pastor and a parishioner is never justified.
Since the ministerial relationship is professional in nature, it is inappropriate to call a sexual encounter an affair. Affair is a term used to describe a sexual liaison between peers, or equals. In addition, the term affair focuses attention on the sexual nature of the behavior rather than the professional violation. It also places equal responsibility for the behavior on the congregant. Since clergy have a responsibility to set and maintain appropriate boundaries, those who are violated by clergy’s inappropriate sexual behavior are not to be blamed even if they initiated the contact.
via Why It’s Not An Affair by Patricia L. Liberty.