Date fruit, which comprises nearly 80% of fruits produced in the Sultanate of Oman, is the leading agricultural crop in the country. Sophisticated technologies are utilized to improve the date fruit production in the country, but date sorting and grading after post-harvesting still create problems for the date cultivators. Manual date sorting is a time-consuming process and raises ambiguity about the accuracy and consistency of the grading system. This paper presents a date fruit classification system based on deep learning. It categorizes dates into four different classes based on their physical attributes. Dates fruit classification in automatic mode is challenging due to its high visual similarity, and getting better grading accuracy and performance evaluation, numerous combinations of hidden layers and epochs have been done. The performance of the model is evaluated based on its accuracy and losses. The model performance is appreciable, with a validation accuracy of 97.2%.