AI Critique is an online technical report series that features artworks alongside critiques generated by large language models (LLMs). The present article outlines the purpose of its publication.

1. The Paradox of Generative AI

The development of artificial intelligence (AI) has been a remarkable phenomenon in recent years, particularly in the domain of Large Language Models (LLMs). These entities are evolving beyond the status of mere tools and are exhibiting the potential to challenge humanity's very identity. While the impact of LLMs undoubtedly encompasses both positive and negative aspects, it is the contention of this paper that, specifically regarding art (though this paper addresses only one facet), the resultant interplay of influences across several dimensions is profoundly positive.

In the realm of AI and art, the predominant perspective is oriented towards the generation of artworks by generative AI, such as LLMs. The remarkable capacity of these models to produce text, images, and other forms of expression is advancing at an unprecedented rate, surpassing the bounds of imagination. In numerous genres, the output of these generative AI models is now indistinguishable from human creation, and in some cases, it can be argued that the quality of their output surpasses that of humans. The profound impact of generative AI on the artistic landscape is comparable to the revolutionary transformation brought about by the invention of photography in the world of painting. It can be posited that generative AI will catalyse an even more significant metamorphosis across the broader spectrum of artistic disciplines.

However, the present discussion pertains to the consideration of whether the capacity to produce prose, paintings, or videos of remarkable quality is indicative of genuine comprehension and expression. It is a commonly acknowledged distinction from humans in this regard. This phenomenon may be referred to as the ‘Generative AI Paradox’ (Fig. 1). The crux of the paradox lies in the fact that ‘humans generate based on proper understanding, but generative AI can generate without understanding.’

The present author takes issue with this statement, declaring, ‘That's not the case!’ Rather than entering into a debate starting with questions like ‘What is understanding?’ or ‘What defines humanity?’, a different strategy is proposed here. This strategy is to silently present something so compelling that it can only be interpreted as understanding, thereby compelling acceptance. In essence, rather than generating artworks, it is proposed that this paradox be challenged by having the AI write critiques of artworks and demonstrating the depth of those critiques.

2. Methodology for Generating Artwork Critiques

A fundamental question to consider in this regard is the theoretical basis that should underpin the generation of art criticism. A crucial point to consider here is whether to venture into the domain of evaluating the work itself, or to confine oneself to interpretation. Regarding this, the mainstream approach in the latter half of the 20th century was to avoid evaluation and to place emphasis on interpretation. In contrast, Noel Carroll [Carroll, 2009] proposed that the paramount task of the critic is evaluation, that is to say, value judgement. Moreover, he placed considerable emphasis on success value as opposed to reception value. By this, he means the value derived from the emotions, interpretations, and experiences that the work imparts to the viewer. This evaluative criterion focuses on the extent to which a work achieves its objectives in light of the author's intent or concept.

We adopted Carroll's proposed framework of reasoned judgements as the theoretical basis for our art generation system, considering it to be exceptionally well-suited to the strengths of Large Language Models (LLMs). In the initial phase of our inquiry, we embarked on the task of delineating the components of criticism. Through a rigorous analytical process, we established the following components: description, classification, contextualisation, elucidation, interpretation, analysis, and evaluation. Subsequent to this preliminary phase, we proceeded to identify “evaluation” as the indispensable, crucial element that constitutes criticism. The remaining components were thus positioned as foundations to derive “evaluation”, meaning their presence is not strictly necessary.

Furthermore, To enable the generation of multifaceted criticism, we comprehensively presented the LLM with knowledge encompassing theories applicable to each constituent element. These are the following 15 critical theories: Structuralist Criticism, Narratology Criticism, Reception Theory Criticism, Deconstructionist Criticism, Psychoanalytic Criticism, Thematic Criticism, Feminist Criticism, Gender Criticism, Genetic Criticism, Marxist Criticism, Cultural Materialist Criticism/New Historicist Criticism, Socio-criticism, Cultural Studies, Systems Theory Criticism, and Postcolonial Criticism/Transnationalism [Ogura, 2023].

In this way, we presented the broad framework (constituent elements) of criticism to the LLM, while also indicating the critical theories applicable to each constituent element. Furthermore, to teach the LLM the style of criticism, we also input a dataset containing short descriptions of artworks. The overall structure is shown in Fig. 2.

By inputting an image of the work to the LLM in this manner, a critique consisting of seven components is generated. Subsequently, compressing this into several paragraphs enhances the overall consistency, stylistic naturalness, and readability of the critique. As a final step, a one-sentence critique that minimally captures the essence of the work is generated. This stepwise process is a prompt generation technique called Chain-of-Thought (CoT).

3. Evaluating Criticism via the Turing Test

We conducted a Turing test (Fig. 3) to assess how well the AI-generated criticism measured up against critiques by human experts. Subjects were shown both expert criticism and LLM-generated criticism of artworks and asked to determine which was human-made (or LLM-made), with evaluation based on their accuracy rate.

The selection and generation of both AI and human critiques of artworks (Fig. 4) necessitated considerable deliberation. The initial step entailed the selection of artworks, for which the LLM would have learned the text if a significant amount of criticism or commentary on a work already existed online. Despite the absence of any dedicated research on comprehensive art criticism (as far as we are aware), studies that generate image captions generally employ a strategy of intensive training on knowledge bases such as Wikipedia. The objective of the present study was to evaluate whether the model could consistently understand, value and produce high-level criticism for unknown works, rather than merely aggregating human-written material. In order to address this objective, the opposite approach was adopted.

With this policy, works were excluded from consideration if an LLM correctly identified the title or artist when presented with an image. Once the target works had been determined, a refined, three-paragraph version of the second-stage critique from the aforementioned generation process was adopted – specifically, a detailed critique faithful to Carroll's framework.

Finding appropriate human critiques, however, proved to be a challenging endeavour. Datasets employed in AI research are fundamentally of the caption variety; to the best of my knowledge, none contain criticism of sufficient aesthetic depth. Consequently, selections were made primarily from the non-profit art history organisation Smarthistory (https://smarthistory.org/). In undertaking this endeavour, we excluded any descriptions containing facts that could not be derived exclusively from the artwork itself, as such elements would immediately reveal the involvement of a human author.

What were the crucial results? The most fundamental findings from 60 subjects are shown in Fig. 5. The graph illustrates the percentage of times subjects accurately differentiated human-made descriptions from those generated by LLM for five artworks. As a baseline, responding randomly without reading the critiques would result in a half chance, or 0.5. At first glance, the graph shows a distribution close to 0.5, and upon closer inspection, it ranges between 0.5 and 0.6. In other words, it is reasonable to conclude that the distinction between the two was virtually impossible to make.

It is interesting to note the extremely low correct answer rate of 0.3 for the fourth problem. This can be attributed to the inclusion of a trick question, which was included by the author as a result of a slight sense of mischievousness. This study employed a unique pairing of LLM-generated one-line critiques, utilizing this format exclusively for this work, with expressions of irony or satire. The failure to recognize the ironic intent inherent in these expressions can lead to misinterpretations, rendering the LLM's critique seemingly more reasonable and, consequently, resulting in errors.

The Turing Test was originally conceived as a means of assessing the extent to which an AI could convincingly emulate a human conversational partner. Consequently, the necessary and sufficient condition for passing the conventional Turing Test was the degree of human-like behaviour exhibited by the AI. Typically, even somewhat underhanded techniques, such as deliberately introducing typos, were considered significant.

 The analysis demonstrated that a limited number of subjects cited the style of the critical writing as a reason; instead, the predominant reasons were accurate content, detailed background knowledge, profound interpretation, and persuasiveness. This finding indicates that LLMs are capable of generating criticism that matches the level of human experts, surpassing the confines of mere surface expression style and encompassing aesthetic perspectives. Consequently, these results offer empirical support for the position that the previously mentioned paradox of generative AI may not hold true.

It is noteworthy that despite the absence of augmented LLM development, LLMs are posited to inherently possess the capacity to generate criticism that surpasses that produced by human agents. In methodologies aimed at elucidating this potential, the crux lies in the human element's aesthetic discernment and the articulation of the desired nature of criticism. Consequently, the very delineation of the problem, namely the distinction between LLM-generated criticism and that authored by humans, becomes moot.

4. Operational Policy and AI-Centered Ecology of Artistic Feedback

This report will disseminate the critiques that have been generated by the generative system delineated in Sec. 2. In instances where the output is in English, the equivalent in Japanese will be generated; conversely, where the output is in Japanese, the equivalent in English will be generated. As a general rule, the output will be published in its original form. However, if the output contains explanatory notes in brackets that respond to prompt instructions, these will be removed.

Hereafter, several points worthy of consideration will be addressed. Firstly, the detailed critique generates responses covering all seven components of criticism proposed by Noel Carroll, and furthermore, it is compelled to describe based on 15 critical theories. This aims to maximise the diversity of perspectives, and consequently, it may include elements that appear somewhat forced. Nevertheless, in the ensuing phase, the refined version is directed to refine and compress the detailed version to ensure consistency, thereby mitigating such aspects. Indeed, four of the five tasks that passed the Turing test employed this version (the remaining one used a single-sentence critique).

It is important to note that the following issue, which is inevitable when working with LLMs, persists: they are prone to committing errors known as “hallucinations” (with confidence). Critiques containing such errors, particularly those potentially diminishing the value of authors or works, may raise questions of accountability. This connects to fundamental ethical issues in AI application. The issue at hand is that of establishing the entity or entities responsible for the consequences of actions undertaken by artificial intelligence (AI), and determining the manner in which this responsibility is to be assumed.

GALLABO TOKYO seeks to address this issue in a creative manner as part of an adopted approach that integrates both physical and online dimensions. This approach is designated as the ‘AI-Centred Ecology of Artistic Feedback’.

This constitutes an ecological creative environment in which the artistic meaning and value of a work are continually updated. It is based on criticism generated by artificial intelligence (AI) agents, through repeated interaction mediated by the work, criticism, and comments, involving the artist, the viewer, the AI agent, and the AI designer. This approach can be positioned as a new development in the AI era, building upon Nicolas Bourriaud's “Relational Aesthetics” and Claire Bishop's “Aesthetics of Adversity”.

Note)

For details on the generation of art criticism using LLM introduced in this article, the subject experiment (Turing test) involving 60 participants, and the proposed “theory of mind” task related to art criticism (Fig. 6) along with its evaluation using 41 LLMs, please refer to two papers published in 2025 (References [1][2], Fig. 7).