The application of deep learning to detect an object has been widely applied in various fields, one of which is detecting the gender of the object being tested. Gender detection is to determine the image of a man or woman's face. In this case, the Convolutional Neural Network (CNN) method has been able to recognize the existence of this gender difference. However, in some cases, objects detected in conditions wearing accessories; such as hats, bandanas, scarves, hair ribbon, hijab (headscarves for Muslim women), and so on; to cover their heads so that some of their faces are covered too. The partial closure of the face on the object is one of the obstacles in determining the symmetrical shape between men and women on the object being detected. Gender detection on objects wearing accessories is a challenge for research in the field of deep learning. In this study, we propose a very deep learning method using VGG16 to detect gender in the case of objects wearing accessories. We change the Fully Connected Layer (FC Layer) on VGG16 with the number of layers we propose. VGG-16 has three of fully connected layers with a large number of layers, that is 4096 layers, while the fully connected layer we propose has 9 layers with the largest number of layers 128 and the smallest 16. In addition, until now there are no data sets for women using scarves, we have built our own datasets and we use transfer learning added with data augmentation techniques. CNN can predict the genders only get 77% testing accuracy. But if the CNN combined with our proposed modified VGG-16 as a feature extraction layer, the test accuracy increased significantly by 13.56%, that becomes 90.56%.