Boris Johnson has stood by his remarks about the burka after the Conservative Party chairman told him to apologise, reports BBC.
The former foreign secretary has been criticised for saying Muslim women wearing burkas "look like letter boxes" and comparing them to "bank robbers".
PM Theresa May has backed calls for Mr Johnson to apologise, saying the remarks have "clearly caused offence".
But a source close to Mr Johnson said he "won't be apologising", adding it was "ridiculous" to attack his views.
"We must not fall into the trap of shutting down the debate on difficult issues," the source added.
"We have to call it out. If we fail to speak up for liberal values then we are simply yielding ground to reactionaries and extremists."
Mr Johnson's remarks, in a Daily Telegraph article, have provoked criticism from Muslim groups, some Tory MPs and opposition parties.
In the article, Mr Johnson said full-face veils should not be banned but looked "ridiculous".
However, the founder of the Conservative Muslim Forum said the remarks would harm community relations.
Conservative Party chairman Brandon Lewis said he agreed with Foreign Office Minister Alistair Burt that there had been degree of offence in Mr Johnson's comments, and called on the former London mayor to apologise.