Wavelet Analysis of Paintings by
Goswin van der Weyden and Paul Gauguin
Adviser: Robert Calderbank
Josephine C. Wolff
“The opportunity to apply math to an unsolved problem, to produce results that hadn’t been seen before and evaluate an unused data set, was a radically new one for me.”
What is a math thesis, anyway? For pretty much the entirety of my first three years at Princeton, I could offer no authoritative answer to that question. In my head, possibilities loomed threateningly, ranging from the merely intimidating to the utterly unattainable. Would it have to involve the invention and proof of some brilliant new idea—the Wolff theorem, perhaps? And if I failed to prove it, would they actually withhold my diploma? And if so, wouldn’t I be better off majoring in something else?
Fortunately, the series of prerequisite math classes for the major left very little time to worry about such distant prospects and I somewhat recklessly declared my major in the spring of my sophomore year, confident that I would figure everything out, eventually.
Needless to say, I did not prove a Wolff theorem for my senior thesis. Not even close. I did, however, do a mathematical analysis of a set of paintings by Dutch painter Goswin van der Weyden using wavelet transforms, hidden Markov models, and machine learning algorithms. To me, this seemed nearly as impressive because I didn’t know what half of the words in the previous sentence even meant at the beginning of my freshman year. The project focused on a set of 15 paintings by van der Weyden, each of which had an underdrawing, or sketch, beneath the paint that had been uncovered using infrared reflectography technology. The sketches fell into four different categories, based on how detailed they were, and the art historians who provided us with the data set were interested in whether these categories corresponded to visible differences in the finished paintings.
Van der Weyden ran a painting workshop in Antwerp at the very beginning of the 16th century, and he employed many students and apprentices there who worked on the paintings. Because he had so many students, it is possible that the detailed underdrawings beneath the paintings were intended to guide the work of amateur apprentices only just beginning to learn their craft. The looser or sketchier underdrawings may, in turn, have been meant to assist more experienced, advanced students. My thesis project was to analyze each painting using a mathematical technique called wavelet transforms and then classify those results with different machine learning algorithms to see if there were clear similarities between the paintings that had similar styles of sketches.
My primary adviser, electrical engineering and applied mathematics professor Robert Calderbank, suggested the topic when I met with him at the very beginning of my senior year. There was already an active painting group in the math department that had done previous work on mathematical analysis of paintings, and Professor Calderbank recommended that their work might provide good background and guidance for my own research. The project was an entirely new one for me and didn’t build directly on work I had done for either of my junior papers, but the earlier and ongoing research of the Princeton painting group was invaluable in helping me figure out where to start my research. While it didn’t directly relate to my earlier independent work, my thesis certainly utilized many of the skills I had acquired in my college math and computer science classes, especially when it came to parsing and digesting math articles and puzzling through dozens of computer programs.
During the fall semester, I mostly focused on researching the areas of math and computer science most relevant to the project, including wavelets, hidden Markov models, and machine learning. I tried to acquire as much background knowledge as I could and started drafting out the early sections of my thesis that explained the important math and computer science topics and laid the groundwork for the later research. During intersession, in January, I spent a week in Brussels, Belgium, where my second reader (math professor Ingrid Daubechies) was on leave at the electronics and informatics department of the Vrije Universiteit. There, I got to meet and talk with several other students and professors who were working in similar fields of image processing. I also had the chance to explore a little of Brussels—the city where van der Weyden was born in 1465.
Soon after I returned to Princeton for the spring semester, the Musée Royal des Beaux-Arts d’Anvers in Antwerp and the Musées Royaux des Beaux-Arts de Belgique in Brussels generously provided us with the scans of the van der Weyden paintings, and I was able to begin my own analysis of them, drawing on the techniques I had researched during the previous semester. In this sense, my topic became increasingly narrow, or focused, as the year went on. For the first few months, I was researching very broadly the question of how mathematical concepts could be applied to images and artwork. Gradually, I focused more specifically on the certain types of wavelet transforms and hidden Markov models that had been most useful to other researchers doing similar projects and then, in the second semester, I narrowed my focus to one specific set of paintings to apply those techniques myself.
Because the data depended on curators at the different museums we partnered with being willing to provide us with high-quality scans of the artwork, I did not know when I began my thesis what data exactly I would have to work with that year. It was necessary to determine the central research question in harmony with the museums that provided the data, an experience that was both immensely rewarding, because it meant that real art historians were interested in the questions we were studying, and also, at times, frustrating and slow.
Throughout the year, I met regularly with my adviser—usually once every week, or every two weeks—and also talked with some of the graduate students and other professors in the department who were working on related research. Professor Calderbank suggested relevant readings and provided guidance at every step of the way; he was very patient during the periods of the year when my mind wandered to other things, but also reminded me periodically of the various markers I should use to make sure I was still on track. Particularly helpful was his insistence that I turn in some writing (not a full draft—just a couple of chapters) before the end of the first semester so that I would have something concrete to work on throughout the later months. I know this is nothing new—everyone will tell you the key to writing a good thesis is to start early. Most of the thesis-writing advice I have to offer is in this vein: Start early, choose a topic you’re really excited about (anything else will get very, very old very, very quickly), familiarize yourself with the background research well in advance, have confidence in your ability to put together valuable, new analysis, and start writing it up as soon as possible. These are all things you already know, both because they’re largely common sense and because you’ve heard them a million times before. I reiterate them here only because they’re absolutely, entirely true.
Writing a thesis, in many ways, was nothing like taking a regular math class. In the humanities, it can perhaps be compared to embarking on a very long, self-guided research paper, comparable to the final projects for many English and history classes. In math classes, work typically involves solving regular problem sets and solving problems that have already been solved before to fully grasp a new area of math. The opportunity to apply math to an unsolved problem, to produce results that hadn’t been seen before and evaluate an unused data set, was a radically new one for me.
Several parts of the thesis process were immensely challenging for me: wading into long math texts on topics I felt unsure about, tackling new programming languages, and writing about math in a technical manner while still trying to make it accessible and even appealing to a wider audience of art historians, to name a few. But it is not at all difficult for me to identify the single most rewarding part of the experience. Conducting my own research, producing new, original results, and completing an independent undertaking underscored that though the foundations of geometry are more than 2,000 years old, and the fundamentals of calculus date back hundreds of years, the field of mathematics is still alive and growing, with still plenty of unsolved problems. And even a 20-year-old college student with no theorems to her name and no innate genius for mathematics can make a contribution.
Wavelet Analysis of Paintings by
Goswin van der Weyden and Paul Gauguin
Josephine C. Wolff
Formerly Professor of Electrical Engineering,
Mathematics, and Applied and Computational Mathematics
(Now Dean of Natural Sciences, Duke University)
“I discovered that she is a gifted writer, and I liked the way she used paintings to bring to life the mathematical techniques used for forensic analysis.”
Is there something intrinsic about a piece of mathematics that renders it pure or applied? I know there are mathematicians who believe they can separate the two, but exactly how they make the distinction is beyond me. I would say instead that what is intrinsic to applied mathematicians is their interest in a dialogue with another discipline.
Last year, Josephine Wolff was part of a conversation between mathematics and art history. It was a conversation in which art conservators and art historians communicate the characteristics of different painters and how their work evolved over time. It is a conversation in which mathematicians start by listening and then think about how to capture characteristic features in a way that has predictive value.
Josephine learned how to capture brushstrokes at coarse and fine scales using wavelets. She learned firsthand the advantages of dual-tree complex wavelet transforms for image analysis. Then, she used machine learning techniques from computer science to look across scales to create features that are able to capture the style of an individual painter.
These preparations took most of fall semester, but toward the end of the semester Josephine started to write. I discovered that she is a gifted writer, and I liked the way she used paintings to bring to life the mathematical techniques used for forensic analysis. Her experience as managing editor of the Daily Princetonian meant that she was serious about deadlines, a quality that is much appreciated by every adviser.
It was only at the start of spring semester that we narrowed in on the topic of Goswin van der Weyden underdrawings. We were given 15 painting details, and for each image the Musée Royal des Beaux-Arts d’Anvers provided a scan of the underdrawing or sketch that had been revealed by infrared reflectography technology. Each of the underdrawings fell into one of four categories identified by art researchers, based on the materials used to draw the sketches as well as the level of detail they exhibited.
We were told that the authorship of the paintings was not in doubt, but that the museum was interested in exploring the stylistic heterogeneity of the paintings as well as their underdrawings. Art historians have hypothesized that this stylistic heterogeneity may be related to the intervention of workshop assistants, the development of the artist’s style throughout his career, or the function of the underdrawings. For example, some of the underdrawings are much clearer and more detailed than others, suggesting that the students working on them may have required more guidance in their work.
During intersession, Josephine traveled to Belgium, where my colleague Ingrid Daubechies introduced her to signal processing researchers at the Vrije Universiteit who are using characteristics of the rolls of canvas used by Vincent van Gogh to better understand the progression of his style. Everyone met up again just before graduation when we presented our results in a conference organized by Program in Applied and Computational Mathematics (PACM) alumna Shannon Hughes at the Museum of Modern Art in New York.
My computer science graduate student Sina Jafarpour helped Josephine navigate the data set, the unfamiliar world of MATLAB, and the universe of machine learning algorithms. He was excited by the problem and by the experience of guiding undergraduate research. I know that he fielded lots of e-mail while I was sleeping. Together we showed that classification of the labeled data into four categories using the LogitBoost machine learning algorithm achieved 90 percent accuracy. This is remarkably good, and we declared ourselves ready for unlabeled data.
In some cases the art historians knew the labels, and in some cases they were undecided. There were 10 test data sets, and in all cases except one our classification agreed with the most- or second-most-probable attribution. We did, however, get one of the data sets (Vis-6) badly wrong. This caused quite some excitement, and according to Maximiliaan Martens from the University of Ghent:
Your algorithms “see” connections that art historians, as we, haven’t been able to distinguish. Again, all data of set 1 are attributed on historical evidence to Goossen van der Weyden. Cat. 1 which is the so-called Tsgrooten Triptych is the only work for which an archival document exists to confirm the attribution. Vis-6 is a detail of the right panel of the very small Tsgrooten Triptych (right panel approx. 34 x 11 cm)
Cat. 3, e.g. is attributed to Goossen because it’s a major altarpiece in Lier and he is the only artist known to have stayed in that town at that time. In other words, in the latter case, we only have rather faint circumstantial historical evidence.
Discovering a high concordance between classes 1 and 3 is, in this case, a major step in confirming that the art historical grouping is indeed less heterogeneous than previously thought.
Josephine won the Gregory T. Pope ’80 Prize for science writing, and we all hope to end up with an art history publication in the near future. This week, she is creating a virtual poster that will rotate on the PACM website. The physical poster we plan to hang next to the classrooms in Fine Hall to encourage another student to continue our conversation with art history.