19 Do not entertain an accusation against an elder unless it is brought by two or three witnesses. 20 But those elders who are sinning you are to reprove before everyone, so that the others may take warning. 21 I charge you, in the sight of God and Christ Jesus and the elect angels, to keep these instructions without partiality, and to do nothing out of favoritism. 1 Timothy 5:19-21
The responsibilities and charges of an elder or a pastor are great, but so are the consequences of flagrant sin. There’s a reason why Paul told Timothy that accusations needed to be brought by at least two witnesses. Otherwise, any person who didn’t like he pastor could bring accusations against him without any corroboration. It’s not a small thing to make an accusation, nor should it be.
Again, pray for your pastor to be so far removed from any such accusations. Pray that he will safeguards in place to prevent himself from getting in compromising situations.