Addressing parliament in Kiev, he said: "There will be a victory parade... in Ukraine's Sevastopol."
Russia annexed the peninsula - which has a Russian-speaking majority - in March after a controversial referendum.
Are the new Defence Minister's remarks just bluster, or will he actually try to retake the Crimea? I think that would be a big mistake that would put all of Ukraine at risk. President Putin is not going to give the Crimea back and the Ukraine doesn't have the fire-power to take it, so what is there to gain. Why open that door when you have no idea what might come through it?
In eastern Ukraine, a government offensive against pro-Russian separatists is continuing.
German Chancellor Angela Merkel and French President Francois Hollande have urged Russian President Vladimir Putin in a conference call to use his influence to put pressure on the rebels in the Donetsk and Luhansk regions.
Mr Putin said he was "deeply concerned about the rise in deaths among the civilian population and sharp increase in refugees" entering Russia from south-eastern Ukraine, according to the Kremlin.
All three leaders agreed that the Organization for Security and Co-operation in Europe (OSCE) should play a more active part in monitoring the situation in the conflict zone.
Ukrainian President Petro Poroshenko later said he was ready to return to a ceasefire provided it was observed by both sides, all hostages were freed and borders secured by government forces.
His statement came after he spoke on the phone with US Vice-President Joe Biden. Mr Poroshenko called off a unilateral truce on 30 June, accusing the rebels of staging deadly attacks on Ukrainian government troops.
Lt Gen Heletey, 46, was approved by MPs in Kiev after being recommended by Mr Poroshenko as someone who would work day and night to restore the military capability of the country's armed forces.
|Lt Gen Heletey, Ukrainian Defence Minister|
The status of the city, home port of Russia's Black Sea Fleet for centuries, was disputed by Russia long before it annexed Crimea.
A new chief of the general staff, Viktor Muzhenko, was also appointed on Thursday.
Since the re-launch of the "anti-terrorist operation", at least five government soldiers have been killed and 28 wounded.
At least 250 civilians killed in eastern Ukraine since April, according to Ukrainian and rebel reports
Nearly 200 soldiers killed and 619 wounded since April, according to Ukrainian defence ministry
At least 800 rebels killed since April, according to rebels
At least 110,000 people have left Ukraine for Russia this year to date, most of them from eastern regions, according to the UN
Some 54,400 people have been displaced within Ukraine itself, the UN says
Rebels in Luhansk accused government forces of killing civilians in the village of Luhanska on Wednesday.
District mayor Volodymyr Bilous told Ukrainian news agency UNN that warplanes had bombed the area, killing nine and injuring 11. Another report spoke of 12 deaths.
As amateur videos emerged of burning houses and dead bodies, the Ukrainian military denied attacking the village with artillery or aircraft and blamed the rebels instead.
Rebels in Donetsk reported fresh heavy fighting with troops backed by tanks and planes in the village of Mykolayivka on Thursday, with an unknown number of casualties.
Three traffic policemen were shot dead overnight in the city of Donetsk.
Separately, Ukraine's border guards said Russian army helicopters violated Ukrainian air space in the Luhansk region.
The violence erupted in April when separatists declared independence in the Donetsk and Luhansk regions.
In November, Ukraine's President Viktor Yanukovych decided, under pressure from Russia, not to sign an agreement with the EU, leading to street protests in Kiev and his eventual overthrow.
President Poroshenko signed the free trade part of the EU deal in Brussels last Friday, after earlier signing the political co-operation clauses.