Words to Numbers

When empirically studying questions in the social sciences, we often need to find numeric measures for “soft” constructs such as culture, trust, individual traits, strength of institutions, etc.¹ However, most human interaction is not numeric, but verbal. Only recently, computer-based analytical methods have become available to study large amounts of text and speech data, in a stringent, more objective way. This computer-linguistic revolution has a great potential impact for all the social sciences, and economics in particular, because we may now be able to apply these methods to distill complex concepts such as “trust” down to numbers.

However, economic research has not yet realized the full potential of these new tools of text analysis. Most applications have focused on predicting asset prices in finance, measuring media slant or public sentiment in political economics, or applying Latent Dirichlet Allocation to extract latent topics from high-dimensional data (Gentzkow et al. 2019). The CAS Research Group aimed, in dialogue with the research frontier in computational linguistics, to broaden the scope of both the economic questions that these new tools of text analysis might be applied to, and the methods used in this field.

In particular, we aimed to address substantive questions in three project areas: (I) Understanding knowledge production, (II) Measuring organizational culture and management practices, and (III) Adapting to institutional and sectoral change. Against this backdrop, let us briefly highlight three studies we pursued during the year.

Analysis of communication in experiments using Natural Language Processing (NLP) techniques

Teamwork plays a crucial role in knowledge work. Given its importance in achieving favorable outcomes, cooperation has been the subject of extensive study in many laboratory experiments. Many studies have varied on whether either closed-form or free-form communication was available for subjects to achieve cooperation (Kooper and Kagel 2023). However, so far this communication has been—if at all— only rudimentarily analyzed, either because the presence or absence of communication was treated as a binary variable, or because reliance on human coders has limited the use of modern text analysis. We are the first to systematically apply NLP approaches to study the content and style of communication and its effectiveness in cooperative games. For our study, we (Florian Englmaier, Nicola Lacetera, Matylda Trocinska) did not collect primary data but teamed up with David Cooper (University of Iowa), who has run numerous classical studies that collected text data. Together we have collected the raw data from a comprehensive compilation of existing papers, with access to the raw data, including communication logs. The analysis of team dialogues will enable us to understand the thought process and identify the reasons behind certain strategies being adopted by the teams, which has not been possible using other common methods, such as response times. NLP algorithms allow us to look for common topics present across a range of games and analyze the difference (or similarity) between coordination and cooperation in laboratory games. The significant amount of text data collected from past experiments enables us - hopefully - to draw general conclusions that are applicable to different experimental settings, including some real-world situations.

Patterns of communication in non-routine analytical team tasks

In LMU’s Organizational Economics group we have a major research program studying productivity in non-routine analytical team tasks, a very prominent yet understudied part of modern labour markets. In order to have access to one of the settings in which we study such problem solving teams, we have developed a close collaboration with a provider of an Escape Room (Englmaier et al. 2024). In these Escape Rooms we had the possibility to collect very detailed data for roughly 300 student teams that we equipped with an indoor localization system, providing us with precise second-by-second data on the location and motion of participants within the rooms. In addition, we collected individualized voice recorder data to record team communication. These data are unique, as so far no measurements with such a level of detail have been available for controlled settings. The location data give us measures of effort and search patterns, and complement the voice recorder speech data, as physical proximity is likely correlated with actual information exchange. By applying NLP algorithms to the transcribed speech data, we hope to be able to see whether the content and modes of communication (horizontal vs. vertical) are correlated with team success. Working together intensively in the Research Group helped dramatically, first with transcribing the data in a meaningful and workable way, and in particular because input from computer linguists clarified the path ahead in data analysis. We hope that by the end of 2024 we will have completed the paper with the analyzed speech data.

Assessment of the benefits of new medicines

Another prominent study, pushed as part of the Research Group, concerns the digitized textual analysis of the documents that pharmaceutical companies submit to the German Federal Joint Committee (G-BA) for the assessment of the benefits of new medicines. On the one hand, it is reasonable to expect that the decision to approve or not approve a new drug is based on “hard” information, such as the quantitative findings (and their reliability) from the clinical trials. The dossier that applicant companies present to the committee, however, contains several sections with long, text-based descriptions and considerations that far exceed the scope of the statistical analyses. The goal of this project is to determine whether “softer” components of these dossiers can predict GBA approval, due to the manner in which the case for a drug is presented in the documents. Nicola Lacetera is working on this project with Paul Hofmarcher (University of Salzburg) and Simeon Schudy (Ulm University).