Yes, we do believe in it.
عَنْ عَائِشَةَ، زَوْجِ النَّبِيِّ صَلَّى اللهُ عَلَيْهِ وَسَلَّمَ، أَخْبَرَتْهُ: ” أَنَّ رَسُولَ اللَّهِ صَلَّى اللهُ عَلَيْهِ وَسَلَّمَ كَانَ يَدْعُو فِي الصَّلاَةِ: اللَّهُمَّ إِنِّي أَعُوذُ بِكَ مِنْ عَذَابِ القَبْرِ
(Sayidah) Aishah the wife of the Prophet ﷺ narrated that, Allah’s Messenger ﷺ used to supplicate in the prayer saying, ‘O Allah, I seek refuge with You from the punishment of the grave.’ (Sahih al-Bukhari, 832).
There are so many evidences proving the punishment of the grave that scholars have written entire works on the subject. Some sects, like the Mu’tazilah, rejected punishment in the grave. However, the Ahl Sunnah strongly opposed them due to the numerous evidences proving it.
In the Hadith above we find the Holy Prophet ﷺ seeking protection from the punishment of the grave, the Prophet ﷺ would not seek refuge from something that does not exist or is not a possibility. Therefore it is a clear evidence in support of the existence of punishment in the grave.
The proofs for punishment in the grave are so numerous that we find Imam al-Ghazali, claiming it is Mutawatir.
وأما عذاب القبر فقد دلت عليه قواطع الشرع إذ تواتر عن النبي صلى الله عليه وسلم وعن الصحابة رضي الله عنهم
As for the punishment of the grave, indeed definite evidence from revelation has proven it, because it is Mutawatir from the Prophet ﷺ and the Sahabah . (Imam al-Ghazali, al-Iqtisaad fi al-Itiqaad).
And Allah (Most High) Knows best.
– Answered by Shaykh Noorud-deen Rashid (04.05.2021)
See also (video):