This post was inspired by a YouTube video I published on my channel some time ago. We will do Face Recognition in JAVA.
We are partnering up with TechInDeep to bring you a full course on Deep Learning in JAVA.
Before continuing with this post make sure to check out some other articles as well:
If you want to learn how to build Neural Networks from scratch you can check:
Demonstration Video
Source Code Download
Programming Language:
- JAVA
What libraries are we going to use:
- OpenCV
- ND4J
- DL4J
Who is this article for:
- A developer who already is somewhat familiar with the Face Recognition problem. For all of you that are struggling with this solution, I will be posting a very detailed tutorial on each workflow step described bellow backed up with code files and many examples. For the more experienced ones we are still not done. This application can be improved by a lot.
Additional Questions:
- I will be answering all additional questions here.
Workflow
- Detect Face (Phase I)
- Extract face feature vector (Phase II)
- Find the most similar face from our data set (Phase III)
So let’s get started…
Face Detection
In order to create a face detector I used OpenCV. This library provides us with a great real time face detection algorithm using Haar Cascades. OpenCV for JAVA has a class called CascadeClassifier that implements the algorithm. At object creation time of the class, you will need to provide the location of the xml file that CascadeClassifier needs in order to detect faces on the image. The xml file that I actually use is a pre-trained file that is dispatched with OpenCV. The file is called “haarcascade_frontalface_alt.xml” and you will find it included in the project. Detecting faces is as simple as calling the function:
detectMultiScale(image,matFaceList,_scaleFactor,_minNeighbours,0,_minFaceSize,_maxFaceSize);
If you want to know more about this function I suggest to read the following documentation on OpenCV. More in depth on this problem and OpenCV API will be covered in another tutorial.
The output from detectMultiScale is a MatOfRect type that contains an array of type Rect. Now this array actually contains the X and Y coordinates, as well as the Width and Height of the area where the face is located on the image. Now given that rectangle we can actually extract the faces individually from the image. Since the image is loaded in a Mat, we can extract the face pretty easy just by calling the submat function and passing the face Rect as a input parameter. Next what I do is I save the face as an image. And that is pretty much it.
The whole process can be seen in ExtractFacesFromImageTask.java file.
With this we finish the task of Face Detection.
As I said more on this topic in another tutorial…
Face Recognition in Java
Once we have extracted the faces we can continue with the next item defined in our workflow and that is face recognition. Since the goal is to create an app that can recognized undefined number of faces, we need to split this section in two. The first part will explain how to obtain facial features and the second one would measure the similarity between the extracted facial features and the features we have in a data set.
Data Set in our project is a file that contains list of face feature vectors and face labels. So every single face feature vector would contain a label as well.
Creating a facial features vector
In order to recognize faces we would need to extract face features from a given image. We can either train our own neural network to get good quality facial features or we can use a pre-trained one. In this project I do use a pre-trained one.
Pre-trained network is just a model that was already trained and has calculated weights in it. That’s all…
Usually people like using the first couple of layers of such network because they tend to generalize better. The main advantage is that I don’t have to train the network myself. Sometimes I don’t have big enough training set or I don’t have the necessary computing power to do the training. So I opt out for using these pre-trained models.
Creating the facial features vector boils down to using a pre-trained neural network model that we can obtain from the DL4J library. In order to do that we would need to load the “VGGFace” Computational Graph. The idea is to pass the face image into the “VGGFace” computational graph to process it and we would use the output of layer “pool4” as our input into the classification method we will use later. Since this is a pre-trained model it would extract excellent features for the classifier. I chose to get the outputs out of the layer named “pool4” which I think it is OK for this demo.
So why this approach?
In order to create a face recognition in java, that can be trained on unknown amount of faces, I could not use the whole pre-trained network, since I don’t know how many faces I would enter.
Instead I decided to take the output from the neural network and write it as a n-dimensional vector. The logic behind this is the following: Since similar faces would yield similar vectors they will be grouped in a cluster. Or simply put, similar faces would end up having similar vector elements.
So if I want to classify a new face, I would run the same procedure, get the output vector from the neural network and calculate the distance between the vectors that are already in my data set. My data set contains a vector and a label, so If I find which vector is most similar to the one I am processing at the moment I will also find the label. I do that using Euclidean Distance but even better solution would be KNN algorithm.
Face Recognition implementation workflow
So the code starts with loading in the weights of “VGGFace” into our Computational Graph. We read in the image using NativeImageLoader to transform the image into the correct size for the “VGGFace” neural network. Then we use the function Featurize from our TransferLearningHelper object. Next we would like to “flatten” the matrix, into a vector, so we use the function reshape from ND4J library.
Finally we have our vector that represents the facial features from the person and we have a label. We combine them and put them in a data set. When a new face comes along, we use the same procedure as described above to get the feature vector, flatten it and compare it to the rest of our data set. The vector that has the closest distance to it (most similar) wins and we display the label (the name of the person).
If the presented label is wrong we can correct it by entering the new label. Please note that we already have the feature vector for the face so all we need to do is insert this record into our data set. We use this method to correct miss classifications and increase the accuracy.
Final Words
A basic Face Recognition in Java technique is explained. This is it for now…
We will cover every topic described here in more detail very soon. Also please note that this is not the most optimal and best solution for this problem. So how to improve accuracy and optimize for speed will be covered in another post.
Complete source code: Face Recognition in Java Project
kamila
July 9, 2019 at 7:12 amHello sir,
Can we develop a face recognition system using java???? but without using any libraries.
vanco
July 10, 2019 at 8:53 amYes you can. And it is not even that hard. All you need to implement is a Convolutional Neural Network and K-Means. But at least use some fast and reliable Linear Algebra library for the matrix computation. I don’t recommend you to write the linear algebra operations yourself because there is a good chance of ending up with a very slow performance code.
Good luck on the project 🙂
hakan
July 17, 2019 at 10:41 amError ı’m getting what ı should ı do?
Exception in Application start method
Exception in thread “main” java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
vanco
July 25, 2019 at 10:33 amPlease check if you have correct libraries to run the application.
hakan
July 17, 2019 at 1:11 pmWhat is a Problem?
Exception in Application start method
Exception in thread “main” java.lang.reflect.InvocationTargetException
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at java.base/sun.launcher.LauncherHelper$FXHelper.main(LauncherHelper.java:1051)
Caused by: java.lang.RuntimeException: Exception in Application start method
at javafx.graphics/com.sun.javafx.application.LauncherImpl.launchApplication1(LauncherImpl.java:900)
at javafx.graphics/com.sun.javafx.application.LauncherImpl.lambda$launchApplication$2(LauncherImpl.java:195)
at java.base/java.lang.Thread.run(Thread.java:835)
Caused by: java.lang.IllegalAccessError: superclass access check failed: class com.sun.javafx.scene.control.ControlHelper (in unnamed module @0x698004e4) cannot access class com.sun.javafx.scene.layout.RegionHelper (in module javafx.graphics) because module javafx.graphics does not export com.sun.javafx.scene.layout to unnamed module @0x698004e4
at java.base/java.lang.ClassLoader.defineClass1(Native Method)
at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1016)
at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:151)
at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:802)
at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:700)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:623)
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at javafx.scene.control.Control.(Control.java:86)
at View.MainView.(MainView.java:36)
at View.ApplicationController.showMainView(ApplicationController.java:29)
at App.start(App.java:10)
at javafx.graphics/com.sun.javafx.application.LauncherImpl.lambda$launchApplication1$9(LauncherImpl.java:846)
at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runAndWait$12(PlatformImpl.java:455)
at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runLater$10(PlatformImpl.java:428)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:389)
at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runLater$11(PlatformImpl.java:427)
at javafx.graphics/com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:96)
at javafx.graphics/com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at javafx.graphics/com.sun.glass.ui.win.WinApplication.lambda$runLoop$3(WinApplication.java:174)
… 1 more
Tony Nguru
August 14, 2019 at 8:34 pmHi Vanco.
Thanx a lot for this project. I did manage to have it run without using maven. Had to compile opencv 4.0 and cuda 10 though and went through a few challenges especially coz of using different versions of bytedeco-javacv and bytedeco-javacpp but i finally figured it out in a normal netbeans project.
Now for the next step am trying to see of i can save the trained data into a new haar cascade model or some other supported standard using DL4J. I saw you save only at runtime in a trainedList. Which disappear after the program terminates. Is it possible you also try it out once you get some time.
vanco
August 16, 2019 at 6:50 amHi Tony,
Yes it is possible to save it. In my original project instead of working with memory I work with HDF5 file type. It behaves as my face database. So inside I would save the INDArray I get from the Neural Netowork. That would create a cluster of similar faces. All you have to do is assign centroids to those clusters in order to describe them with one central point.
Good job Tony,
Thank you for your comment.
zie
October 15, 2019 at 3:28 pmException in thread “main” java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.launcher.LauncherHelper$FXHelper.main(LauncherHelper.java:767)
Caused by: java.lang.RuntimeException: Exception in Application start method
at com.sun.javafx.application.LauncherImpl.launchApplication1(LauncherImpl.java:917)
at com.sun.javafx.application.LauncherImpl.lambda$launchApplication$159(LauncherImpl.java:182)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: Pretrained model file failed checksum.
vanco
October 30, 2019 at 6:59 amOK, so from the error I can see that the pretrained model I use has not downloaded successfully. The first time you run the application there is a big file for the VGG model that the app automatically downloads. Please wait for the app to properly finish.
zie
October 15, 2019 at 3:30 pmHow to deal with the problem I met above,thanks!
Nora Kovacs
February 17, 2021 at 7:48 pmhi can not download source code the link does not work. is it still available?
many thanks,
Nora
vanco
February 17, 2021 at 8:30 pmHi Nora,
I just checked the link and it is working as expected. Let me know if you still experience download issues
Simran
February 23, 2021 at 7:16 amHi Vanco.
So, honestly I’m pretty much a novice here. Although I’ve been doing java programming. Question is, OpenCV, ND4J, DKL4J, are they just downloadable?? Also, the pretrained model….is it supposed to download the first time I run the program? Or I can download it outside the program?
vanco
February 23, 2021 at 9:06 amHi Simran,
OpenCV, ND4J and DL4J are packages that you can include in your project. Those are dependencies that you must include in your project. They are downloaded and included. The pre-trained model is also downloaded automatically at first run. So you have to wait (depending on your connection) for maybe 20 mins to download the model. On the dependencies and how to include them please check out the Quick Start for Java using Maven: https://deeplearning4j.konduit.ai/getting-started/quickstart
Good luck
vpn special code
March 31, 2024 at 9:14 pmThis is a topic which is near to my heart… Best wishes!
Where are your contact details though?
Also visit my webpage … vpn special code
vpn coupon 2024
March 31, 2024 at 10:55 pmHi there, I want to subscribe for this weblog to take most recent updates, therefore where can i do it please assist.
Feel free to visit my homepage vpn coupon 2024
vpn special coupon
April 1, 2024 at 5:33 pmThe other day, while I was at work, my sister stole
my apple ipad and tested to see if it can survive a twenty five
foot drop, just so she can be a youtube sensation. My iPad is now broken and she has 83 views.
I know this is totally off topic but I had to share it with someone!
Feel free to visit my blog post … vpn special coupon
vpn special coupon code 2024
April 7, 2024 at 11:34 amSuperb site you have here but I was curious about if you knew of any community forums that cover the same topics
discussed in this article? I’d really love to be a part
of group where I can get comments from other experienced individuals
that share the same interest. If you have any recommendations,
please let me know. Appreciate it!
my web-site :: vpn special coupon code 2024
vpn special code
April 9, 2024 at 10:07 pmVery great post. I just stumbled upon your blog and wanted
to mention that I have really enjoyed surfing around your weblog posts.
In any case I will be subscribing to your rss feed and I’m hoping
you write once more soon!
My web site: vpn special code