Abstract
Tattoo matching is a compelling problem at the intersection of biometrics and vision. Tattoo-matching systems offer potential aid in identifying groups or individuals who associate with or can be identified by their tattoos. Because tattoos of the same class (e.g. tattoos of hearts) can appear very different, many computer vision algorithms provide helpful partial solutions to the problem, but finding the optimal combination of those algorithmic tools is not at all trivial. This paper describes a software framework designed to compare a large set of computer vision algorithms as efficiently and thoroughly as possible. It provides one-to-one, one-to-many, and many-to-many comparisons of performance, while being as simple as possible to run. Although the original motivation served tattoo-matching, Although designed with tattoo-matching in mind, the framework and its architecture also provide a foundation for efficiently exploring other vision-based biometric challenges.
1. Introduction
More and more common, tattoos are of increasing interest and importance to the biometrics community. Depending on the specific application at hand, systems for matching tattoos can leverage a wide variety of pixel-processing approaches. Figure 1 (top) hints at the variety of tattoos that might be considered "matching": the right pair are the different images of the same tattoo and the left pair are distinct renderings of the same figure. Resources such as OpenCV [13] and others offer collections of algorithms for exploring this challenging matching problem, but they provide less structure for composing and comparing those algorithms for specific tasks. The bottom panel of Figure 1 shows two very different combinations of feature detection, extraction, matching, and scoring for the two tattoo pairs above.
Because challenges such as tattoo-matching potentially benefit from many pixel-processing approaches, this work presents the design, deployment, and results of from a software framework that combines computer vision algorithms to gain insights into the potential and challenges involved with using tattoos for biometric purposes. The system allows users to test out and analyze the performance of different keypoint detectors, descriptor extractors, descriptor matchers, match scorers, and other algorithms, described in section 3.
Many of these algorithms have been implemented in OpenCV, but several others do not; the framework's interface enables easy incorporation of almost any off-the-shelf resources. Since tattoo matching is a relatively unexplored problem, our system's structure is designed to thoroughly test all variations and combinations of existing computer vision algorithms and can compare multiple metrics for the system's performance. Such a framework may serve researchers of other image-based biometric tasks, as well.
2. Background
Tattoos could be very important for the field of biometrics. While there exist techniques that are able to match two dimensional image patterns, tattoos have a number of specific issues that make them difficult to match using traditional computer vision algorithms. Issues include lighting, skin imperfections, the changing surface they exist on, artistic ability, etc. We would like images to have a better score based on their similarity. For example, two snake tattoos should be a very close match, a snake tattoo should be a match with a different snake tattoo, and a snake tattoo should not be a match with a dragon tattoo.
2.1. Tattoos and Biometrics
Tattoos have a high potential for use in identification since they are larger and more accessible than other physiological characteristics such as fingerprints or irises. They could be used to distinguish a gang member or identify a person after a natural disaster, along with many other security and safety applications.
2.2. Problem space for tattoo-matching
Because tattoos are on bodies, they can appear stretched or warped in different images of the same tattoo. Perspective changes can also change a tattoo’s shape in a two dimensional image. In order for a computer vision algorithm to perform well, it has to account for these visual differences by using algorithms that are invariant to many geometric transformations, including shearing, translation, rotations, and scaling; and fortunately, many feature comparison algorithms are invariant to these transforms. Tattoos are also affected by lighting and context changes that are common problems in image processing. Images taken with surveillance cameras may not be high quality and are unlikely to be taken under good lighting conditions. Finally, tattoos are often partially obscured by clothing or may be cut off when they are not the main focus of the image.
Beyond image processing challenges, tattoos carry their own set of problems. Varied skin tones can make it difficult to detect where skin ends and the tattoo begins, especially when skin is close to the color of a tattoo that is only an outline as opposed to a fully filled in design. Also, a tattoo artist’s skill or different interpretations of the same image can greatly impact the kind of tattoo produced. Thus, due to differences in the design, interpretation, and production of a tattoo, tattoo images that are of the same class (e.g. a tattoo of a heart) can appear very different.
We wanted our system to be accurate no matter what database of images it is used on. It would not be possible to create a single program that takes two images and gives them a score because images taken under one set of conditions would have different results than another set of images taken under different conditions. Since the actual use and types of images of tattoos the system will be used on is unknown, we created a system that allows users to swap out algorithms and parameters to find the best combination for their particular set of images.
3. Algorithms
Many algorithms can serve tattoo-matching. Most of them start by seeking corresponding keypoints between images. Keypoint-matching generally has three stages: feature detection, descriptor extraction, and descriptor matching. These can overlap - some algorithms can both detect and extract keypoints - but are distinct enough to form the backbone of our system design, Figure 2: our current system uses the OpenCV implementations of the algorithms noted. The final stage, match-scoring, quantifies the overall match results. The framework allows easy combination and comparison of various detectors, extractors, matchers, and scorers in order to determine the combination with the best precision, accuracy, recall, and/or performance.
3.1. Keypoint Detectors
The most common type of algorithms implemented, detectors locate all areas of interest in an image. These areas of interest can include keypoints, curves, or regions. Keypoints designate regions where the image gradient is changing in a locally distinct way, e.g., at a corner. Often this means that such a point would be re-identifiable regardless of modest rotation, warping or other simple transforms. The detector algorithms return keypoint locations for use in the following extraction stage. The current system uses SimpleBlob, BRISK[1], Dense, FAST[2][3], GFTT[4], HARRIS, MSER[5], ORB[6], SIFT[7], STAR[8], and SURF[9] detectors; others would be straightforward to incorporate, as well.
3.2. Descriptor Extractors
After finding the locations of an image’s keypoints, the system passes those locations along to the designated extractors. An extractor algorithm creates a vector characterizing the image region around a keypoint. For example, if the descriptor has found a point where the gradient changes in a useful way, the extractor might include a histogram of the orientations along which the gradients change. Implemented extractors include BRIEF[10], BRISK, FREAK[11], ORB, SIFT, and SURF. In addition to returning a descriptor vector, some extractors also provide a scale and orientation reference: this information can increase the set of conditions for which matching, the next pipeline stage, will succeed.
3.3. Descriptor Matchers
The extracted vectors are matched by the algorithms in this stage of the pipeline. Matches are made by pairing keypoints with the most similar visual characteristics. For instance, typical matchers will compare the gradient-orientation histograms, and some methods will also account for characteristics such as the difference in pixel location. They can be augmented by the geometric consistency of a set of correspondences, e.g., through RANSAC. Matchers currently implemented are variations on brute force (Hamming, L1), radius matching, KNN, K-Nearest Neighbors, and a fast KNN approximation known as FLANN[12].
3.4. Scorers
In the final stage of the system, the list of keypoint matches is passed along to a scorer algorithm that quantifies the matching of the two images. Our current tattoo-matching task uses two scoring algorithms, known as Simple and Distance scoring.The Simple scorer consists of the ratio of the number of matches found to the number of possible matches that could have been made, i.e., the number of keypoints in the sparser of the two images. The Distance scorer yields the average vector distance between the matched descriptors, i.e., the visual similarity found.
3.5. Flexibility in handling other algorithms
While most algorithms fit into this set of stages, some do not fall into precisely one of these categories. Our framework can also incorporate approaches that work outside this basic pipeline. Rather than finding keypoints and passing information along from stage to stage, many algorithms provide a global descriptors of an image, already prepared for matching against the corresponding descriptors from other images. In this vein the current system includes contour-matching by ShapeContext [16], GIST [14] and local binary patterns, LBP [15]. In each of these cases the upstream processing is idiosyncratic, but all of them take in two images and output a match score between them. From a high-level perspective, then, these algorithms add three more scorers to the two noted above.
3.6. Ensemble Learning
The framework's primary strength is in its organization of these component algorithms and its facility for creating combinations of them. Ensemble learning naturally builds from this foundation, and our system includes an implementation of AdaBoost [17], an adaptive boosting algorithm. To use AdaBoost, the user simply appends a flag to the algorithms they want to run (see Section 5). The AdaBoost algorithm will then use each pair of images as an instance and each specified computer vision algorithm as a weak learner. The result is a composite, learned algorithm whose prediction is a weighted sum of all of the weak learners’ predictions.
3.7. Tattoo Segmentation
Thus far, we have assumed that the images are primarily of tattoos and nothing else, but that is generally not the case when images are pulled from surveillance cameras or other realistic scenarios. Most images that contain tattoos are not solely of tattoos. For example, there could be an image of a man who has a tattoo on his arm. Certainly, there might be a Frisbee, a dog, a slide, birds, or any other various things in the image, but we only care about the tattoo on the man’s arm.
Thus, we have incorporated tattoo-segmenting software into our system that is run before any of the other steps. The segmentation software generates a mask, i.e., a grayscale image in which intensity represents the probability of that pixel being a part of a tattoo. When specified, the framework uses a threshold to binarize the image and extracts the the convex hull of the tattoo region. Finally, a bounding box of the convex hull is created and the corresponding area of that bounding box in the original tattoo image becomes the image that the computer vision algorithms will operate on. (Figure 3.) As with ensemble learning, this is an option that can be included – or not – at run time in order to determine the performance of the matching and/or segmentation portions of the system.
4. Framework
The framework seeks to balance the conflicting demands of being both user-friendly and extensible. Although OpenCV has a wide array of vision algorithms, it is not as user-friendly in providing support for testing combinations of those algorithms. Our system, built alongside OpenCV, simplifies the task of running "contests" among component algorithms in order to determine which pieces work best for a particular vision-based biometric task. Thus, though initially tattoo matching, the framework is not limited to that domain – it will support a wide variety of situations in which a user needs to quickly compare the performance of various combinations of vision algorithms.
4.1. "Contest" support
The framework efficiently runs many different image comparison "contests," where the contest winner is the most accurate combination of algorithms for a particular set of images. Such contests to be one-versus-one, one-versus-many, and many-versus-many. In terms of tattoo-matching, this translates to one image versus one image, one image versus a folder of images, and a folder of images versus a folder of images. We foresee several possible use cases for each of these contests. For example, one versus many may be used to determine the closest matches for one image against a database. The one versus one, on the other hand, may be used for finding the match score between two particular images. A winner for one contest may not be the same for a different contest with different images.
4.2. Algorithm Specification
The system does not take on the constraints (and dependencies) of a graphical interface; Figure 2 shows an example invocation of the Python-based framework. For flexibility, command line flags are used to specify the algorithms to be compared. For example, a user might enter -d SURF SIFT as an option. The -d specifies the stage of matching of concern – keypoint detection in this case, and SURF SIFT indicates that the matching should use both SURF and SIFT algorithms for detecting keypoints. This will not combine the algorithms (yet), but will calculate each algorithm's match scores separately. Similar flags for extraction and scoring steps. Algorithm specification is designed to be extensible so that it is easy to add algorithms to the system.
4.3. Parameter Specification
When finer tuning of the algorithms is desired, there is a system in place to tweak the values of the parameters for each algorithm that can be specified as previously described via the command line flags. This system consists of text files containing key value pairs for each of the algorithms. The user of the system can edit these files to tweak the parameters.
4.4. Serialization