He can charge you what he wants- you either pay or don't
If he takes you to the Housing and Property Tribunal HE has to prove he is right and you are wrong
He may have a hard time doing this.

You either capitulate or call his bluff- I suspect he may he all talk (or email link)...