Prime Minister Manmohan Singh will hold talks with leaders of the separatist Hurriyat Conference in New Delhi on May 3.