Libya's PM has suspended his foreign minister after she met informally with her Israeli counterpart.
Libya does not recognise Israel, as Tripoli backs the Palestinian cause, and the meeting has sparked protests, reports BBC.
Israel's Eli Cohen described the meeting with Najla al-Mangoush as a historic first step in establishing relations.
Israel is trying to build closer ties with more Arab and Muslim-majority countries, such as oil-rich Libya.
However Libya's presidential council, which represents its three provinces, said it was illegal to normalise relations with Israel.
The Speaker's Office in parliament has accused Ms Mangoush of grand treason, and Prime Minister Abdul Hamid Dbeibah has referred her for investigation.
Mr Cohen said he met Ms Mangoush by chance last week on the sidelines of a summit in Rome, and they discussed "the great potential for the relations between the two countries".
He said they talked about Israeli aid in humanitarian issues, agriculture, water management and the importance of preserving Jewish heritage in Libya, including renovating synagogues and cemeteries.
Libya's foreign ministry said Ms Mangoush had rejected a meeting with representatives from Israel, and what had taken place was "an unprepared, casual encounter during a meeting at Italy's foreign affairs ministry".
A statement also said the interaction did not include "any discussions, agreements or consultations" and the ministry "renews its complete and absolute rejection of normalisation" with Israel.
Libyan Foreign Minister Najla al-Mangoush has been accused of grand treason and referred for investigation
Protests broke out in the capital Tripoli and some other cities following news of the meeting. Roads were blocked, tyres burnt and demonstrators waved the Palestinian flag, though the protests appear to have been relatively small.
Libya has been in turmoil for years, with the country split between the interim, internationally recognised government in Tripoli and a rival one in the east.
Should any deal between Israel and Libya be brokered, it would be complicated by that political division, which has existed since the overthrow of long-time dictator Muammar Gaddafi 12 years ago.
Gen Khalifa Haftar of the Libyan National Army (LNA) runs the rival government in the eastern coastal city of Tobruk.
Israel's charm offensive began under the 2020 Abraham Accords, which seek to get countries which are hostile to Israel to recognise its sovereignty and establish diplomatic relations.
So far, Israel has done this with the United Arab Emirates, Bahrain, Sudan and Morocco. However, the government of Benjamin Netanyahu has been criticised for West Bank settlement construction and military raids on suspected militant strongholds in the occupied Palestinian territories.
On Sunday evening, Libya's Presidential Council requested "clarification" from the government over what had happened. The Presidential Council carries out the functions of head of state and is in charge of the country's military.
A letter from the body said the meeting between the two foreign ministers "does not reflect the foreign policy of the Libyan state, does not represent the Libyan national constants and is considered a violation of Libyan laws which criminalise normalisation with the 'Zionist entity'".
It also asked Mr Dbeibah "to apply the law if the meeting took place".
Under Gaddafi, who was a strong supporter of the Palestinian cause, thousands of Jews were expelled from Libya and many synagogues were destroyed.