Starting a project in digital humanities often follows a standard procedure which consists of gathering a large collection of data and then cleaning and processing it with computational models for information extraction and statistical analysis. It is the data processing pipeline.1 From a digital art history perspective, the process also implies adapting to specificities of the field of art history through the definition of research questions, with a core understanding of the corpus of study, as well as the development of methodological solutions to adapt computational methods.

It is within this context that this research on the understanding of hand gestures in early modern paintings was conducted. Painted hands represent a complex object of study from an art historical perspective,2 and constitute a challenging field of exploration from a digital perspective3 with the recent development of performative computer vision methods for body-pose estimation.4 At the intersection of art history and computer science, the research project primarily aimed to outline the evolution and use of specific hand poses in time and place through the creation of interactive tools for visualization. It explored the possibility to reach a comprehensive understanding of hand poses in the early modern period, with the analysis of their specific codification and potential gestural systems. Quickly reduced to a more humble scope after the first curatorial steps towards the creation of a corpus of hands, the project had to align with the reality of working with artistic and art historical material, and address important and too often hidden issues of the framework imposed by the digital environment. Above all, it raises questions about the possibility of globally studying hand gestures on a corpus limited by its metadata and poor computer vision results, as well as the way results can be properly addressed when built upon curatorial biases. These two primary factors also outline fundamental issues in the way we conduct projects in digital art history and how some assumptions on both the digital and art historical realm impact research.

The dataset and its curation represent a foundational layer to research conducted in the digital realm. The very first step consists of understanding the material, and to perform adequate cleaning in order to further process it with computational models. In our case, the dataset consists of the photographic collection from the Bibliotheca Hertziana:5 Digitized photographs of paintings broadly ranging from the 14^th to the 17^th Century, and their corresponding metadata. This metadata includes the titles, authors, and dates of creation of the artworks, and were inconsistently produced in German, English, French, and Italian. ‘Inconsistency’ is a keyword in such an interdisciplinary context as it expresses the diverging reality of information representation between art history and computer science: The transcription of the knowledge produced in art history does not naturally meet the digital requirements. Still, the photographic collection of the Bibliotheca Hertziana is an extreme case: Titles come in different languages and are not standardized according to specific iconographies, the names of artists are not consistent from one annotator to another and may be referred to with a nickname or surname, dates come with many question marks and with non-numerical formats and broad formulations, and the place of creation of an artwork is usually missing.

The inspection of this metadata established the initial curatorial steps, where titles and authors had to be standardized with the help of the Integrated Authority File6 and Iconclass norms,7 question marks removed, and dates formatted into brackets of times. In the process, information was lost, such as the markers of uncertainty of some data, as well as the typical model of knowledge produced by art historians. The example of the question mark used to address the uncertainty of date attributions is strongly evocative of this loss. Based on the favored type of representation system, the information becomes right or wrong, and the removal of a question mark at the end of a year means shifting the data from the unsure into a true and established fact.8

The next step consisted of extracting hands from the paintings with the help of a machine learning model for human body-pose estimation. The process involved a model called OpenPose,9 which uses a system of keypoints to describe the position of the bodies detected on any given image. As we can see in fig. 1, based on these keypoints, the hands were automatically extracted from the original images in order to build a collection of hands. Additionally, a set of features were created from the keypoints to better characterize the pose of the hands, such as the use of unit vectors for the direction of the fingers, and the angles shaped by the joints. Such geometric characterization better helps the comparison of different poses, regardless of their formal aspect.10

As it has been shown and discussed before,11 paintings are difficult material from a computer vision perspective. OpenPose was pre-trained on natural images12 and gives a lower accuracy when applied to artworks. Hence, many bodies were not found in images, and consequently not all the hands are present in the final collection. Furthermore, many hands detected were not correctly estimated. Based on the visual information provided by the keypoints, the collection of hands had to be manually cleaned, reducing the corpus of study from 18,641 detected hands to a total of 5,993 correctly estimated hands.

These two simple curatorial steps — a preamble of further research on hands — show important issues that impact potential results of the work and the way research questions should be addressed from a digital art history perspective. First, there is the question of uncertainty induced by the results from the machine learning model. The material produced, a total of 5,993 hands, is not only a fairly small amount of data from a computational perspective but it is also the expression of missing information when put back in the context of their original artworks. Indeed, it is difficult to speak about gestural patterns within a composition when barely two hands out of five bodies were detected on a single painting. Consequently, the primary question about outlining gestural systems has to be rephrased to interrogate the way a narrative can be properly built from the lack of accuracy of the computer vision system. It now becomes an acknowledgement of what the digital methods ‘obscure’13 rather than what they develop.

Furthermore, the missing metadata and the re-formatting of the latter introduce another layer of uncertainty regarding its exactness, provenance, and certainty. A phenomenon observed in multiple projects that is now being progressively addressed.14 These issues, brought by the use of computational processes on art historical material, shed light on the important notions of bias and subjectivity. Whether it is the description of the hand pose through keypoints, the computer scientist implementing the machine learning model, the art historian registering data in an archival system, or the digital humanist processing metadata, every decision represents a bias that has to be made visible and taken into account when addressing potential results of a digital project. As further explained by Frédéric Kaplan “data are never given, but taken and transformed […] at each step of the data processing, choices are made and biases apply. Understanding these technical choices is crucial to develop new interpretive theories.”15 In our context, a proper trade-off has to be found between the right amount of cleaning processes and the preservation of the original content in order to make claims and outline tendencies in the dataset. Yet, no ground truth can be established from the results, and more data resolutely does not mean a more objective truth.16

This deceptive aspect of digital methods not only results in a current difficulty to build more art historical knowledge but it is also a matter of the fundamental assumptions made in both the field of computer science and art history. Indeed, the peculiar context requires some expertise on two complex fields that have different approaches in the treatment of data and the establishment of new knowledge. Therefore, we do not only need a better understanding of the subjective choices that occur in the curation process, but also in the way we perceive and apprehend the field of art history, how computational approaches can or should be integrated to it, and the novelty they may bring.

The assumptions made about the practice of art history and the simplification of concepts this entails are one of the main issues that prevent the successful development of a field of digital art history. In our case, the research did not start by questioning the notion of influence and potential artistic intentions in an art historical context, nor the computational approach itself which implies a strong formal approach to art history, nor the actual implications of working with a digital image. Yet these three assumptions impact the way we shape research questions and engage with art historical material in the digital environment. There is an actual need to refine our general understanding of the field of art history and its concerns, as well as the digital material used, in order to better address them in computational perspectives.

First, as we have seen from many digital art history projects,17 research very often presupposes an artistic influence of one artist on another. They are looking for similar patterns within a dataset, trying to make visual connections as if the images were part of a giant network, without questioning the meaning and implications of such an influence. In art history, the notion raises various positions and arguments. The one of connoisseurship involves the possibility to recognize an artist behind an artwork through the reconstruction of potential external factors that could have led to its creation. These factors involve a practice in a workshop, a regional or time period production, a commission, or the requirements of the art market.18 Nevertheless, contemporary practice of art history tends to devalue connoisseurship 19 as it involves formal and iconological approaches that do not align with the vision of a ‘new art history.’20 Assuming visual connections amongst artworks is assuming a world made of influences, which underrate the artistic intention.21 Hence, in order to align with actual art historical concerns, the question whether artists are actually influenced by other artists and their cultural context or if a work of art is the result of an artistic intention should be acknowledged, addressed, and possibly further investigated from a computational perspective.

Another question brought by the digital promise is the actual possibility of offering new developments and methods for the field of art history, knowing that most computational approaches imply formalism, an analysis based on the sole apprehension of the visual aspect of the artworks. It seems that the field of digital art history is bound to perform some type of pure formal connoisseurship, bringing back to the table issues and concerns from the past century, when the field of art history attempted to move on from these methods. Yet, as questioned by Rodriguez-Ortega on the neo-formalist perspective and the scope of analysis afforded by computational methods, “is it possible to do something different?”22

The ambition of a computational practice has, in fact, more to offer than a digital mimicking of traditional art historical practice at a large scale, and, in order to shape new narratives, there is a critical need for a better understanding of the computational environment. This awareness starts with what we are attempting to analyze, our object of study, that is, at a primary level, a digital image. As already pointed out a decade ago,23 in most cases we are treating these images as works of art, when they are in fact digital copies, a non-tangible file made of binary encodings, preserved in the black box of a server. While working with a photographic collection, we actually find ourselves in the lineage of an unresolved problem of objectivity of photography, “the main tools of art historians,”24 and a long tradition of image manipulation for rhetorical convenience. As summarized by Lum on the consideration of a photographic truth: “When an image is needed, it does not matter the means.”25

Nevertheless, the digital layer added to these photographs consists of a mathematical transformation, the ‘mathesis’ of Drucker,26 from which a computational interpretation of the painted hand is performed and a further layer, made of keypoints to express the pose of the hand, is added. The novel definition of the image represents a turning point in the formulation of new methodological propositions in art history. Based on a computational vision of the hand and the creation of features for the comparison of poses, we define the framework of a postural approach in opposition to the formal one. In this context, we shift from aesthetic values of the image to a synthetic and geometric vision of the pose of the hand, where we do not focus on the form anymore, neither essential nor inessential,27 but rather on an interpretation of its sign.28 Although we cannot escape comparative methods that have driven the field of art history since the arrival of photography,29 and which are now an essential aspect of computational approaches,30 it is based on such new conception of the ‘voir’ in the creation of the savoir,31 along with a proper formulation of the research intention, that new methodologies and research questions can be established regardless of the uncertainties inherent to a digital art history.