AI in Neuromedicine
The term “artificial intelligence (AI)” refers to computer systems and programs that use various techniques to model and simulate human cognitive abilities. For this reason, AI research is considered a subfield of cognitive science, and sometimes even the heart of cognitive science. Against this backdrop, the two fundamental general technical approaches to AI research can be assigned to cognitive science paradigms. A rough distinction can be made between two paradigms: the symbolic and the connectionist paradigm.
According to the symbolic paradigm, cognition consists of the logical manipulation of symbols, which become meaningful by referring to specific objects. With the help of logical inference rules, algorithms can be written that simulate certain cognitive abilities and are thus able to solve specific tasks. The symbolic paradigm is the basis of the classical form of AI. Examples of classic AI models of this type include medical expert systems (XPS), which use decision trees to support medical decision-making. AI systems modeled on the classical symbolic paradigm are characterized by their explainability. Because the inference rules, symbols, and algorithms are designed and implemented by programmers, they can in principle predict and understand the program's mode of operation and results. While systems of this type can perform many tasks, they have significant limitations. For example, it is almost impossible for such systems to perform speech or image recognition tasks. Recognizing speech and images would require programmers to incorporate an unmanageable number of explicit rules in order to capture the diverse spellings of letters, for example. The ability to correctly identify and classify patterns is important in medical practice, as part of a physician‘s work involves recognizing patterns, for example in radiological images.
The other approach to cognitive capabilities is the connectionist paradigm. This approach is based on the neural structure of the brain and the way brains process information. Instead of concrete symbols and logical links between these symbols, the connectionist paradigm generally views information processing and cognition as located throughout the entire neural system. Instead of concrete symbols and logical connections between these symbols, the connectionist paradigm generally understands information processing and cognition as being located in the entire neural system, so that specific information cannot be clearly located in and thus read off of a system. Because of that, unlike in the symbolic paradigm, it is impossible even for the designers of these AI systems to predict or understand the exact functioning – i.e., the information processing processes – of the system. For this reason, neural networks are so-called black boxes, i.e., systems whose functioning is opaque to both designers and users.
AI systems based on the connectionist paradigm do not require programmers to input explicit rules; rather, such systems are “self-learning.” Self-learning AI systems – the term commonly used here is “machine learning” – are those that, without explicit programming, are able to identify and classify patterns in large amounts of data and produce specific outputs based on this identification. While there are a number of different forms of machine learning, for the ethical considerations discussed here it is only important to note that self-learning AI systems are supplied with input data, such as a series of images. As part of the training process, these systems then “learn” to generate specific outputs, such as the classification of image material. Due to this mode of operation, neural networks have a significant advantage over classical AI when it comes to identifying complex patterns. Systems of this type can identify and classify images or recognize and process language. These special features of neural systems have made contemporary AI usable in areas of application for which classic AI systems were not suitable. Machine learning (ML) programs have proven to be particularly promising tools in medical practice. Such systems are already in use today, especially in radiology, for example in the identification of diseases in radiological images.
When discussing the ethical aspects of using AI in medicine, it is helpful to start from a pluralistic, principle-based ethical perspective. Such an ethical approach differs from both utilitarian and virtue ethics in that it is based on a set of equal-ranking principles, i.e., principles that are located on the same level and may conflict with each other. According to principlism, ethical principles have only prima facie validity: they only guide action once they have been weighed against other principles relevant to the situation in question. Furthermore, ethical principles are underdetermined in their general and formal presentation and must be concretized in their application to specific cases. For example, if the principle of beneficence stipulates that physicians should promote the well-being of their patients, it must be specified how this general principle applies in a specific situation. In a medical context, such concretization can simply consist of examining the specific needs of a patient. Last but not least, it should be noted that when applied to individual cases, it may become clear that principles in their original formulation are not convincing or even need to be abandoned altogether. Nevertheless, principles offer helpful guidance by providing a rough initial indication of the moral considerations that need to be taken into account. The ethical principles central to the use of AI in medicine are beneficence, justice, transparency, responsibility, trust, and respect for autonomy.
Beneficience
The ethical analysis of technologies used in medical practice serves not only to reveal limitations, prohibitions, and problematic aspects, but also to explain under what conditions the use of technological aids may be morally imperative. Thus, in the case of AI technologies, the impression should not be given that the ethical debate emphasizes only their problematic aspects. Given that the primary duty of physicians is to ensure the well-being of their patients, it is important to examine whether and to what extent AI supports the fulfillment of this duty.
The use of AI in medicine promises a number of advantages for patients, medical professionals, and society as a whole. For patients, it is conceivable that certain diagnoses will be made more quickly, more accurately, and with fewer errors. It is already apparent—and it can be assumed that this impression will be reinforced in the future by the further development of ML algorithms—that AI systems are sometimes better at identifying certain disorders in images than even experienced physicians. In general, the use of novel technologies that can make medical procedures faster, more accurate, and less prone to error, thereby promoting the well-being of patients, is ethically imperative, at least prima facie, for this very reason.
In addition to identifying possible diseases in images, self-learning AI systems can also be used for administrative tasks. With their ability to process language and classify handwritten documents, AI could relieve doctors of administrative tasks. This would not only benefit the well-being of doctors, but in the best case scenario, it would also give them more time to care for their patients. Last but not least, there is hope that AI systems will promote more individualized, personalized medicine by processing and analyzing large amounts of data. However, it is not only patients and doctors who could benefit from the use of AI in medicine. Provided that AI can make a number of medical processes more efficient, there is also hope that costs can be saved, resulting in benefits for society as a whole through more affordable medical care. If these hopes associated with AI can be fulfilled, this would be a strong moral argument for the use of AI in medicine from a beneficence perspective.
Justice
The two interconnected and co-arising central aspects of justice raised by the use of AI in medicine concern distributive justice and issues of discrimination. From the perspective of distributive justice, there is a debate about the extent to which AI systems can influence access to the best possible healthcare. This issue results from the specific training methods used to train self-learning systems. For example: the choice of training data and the way in which this data is labeled and pre-sorted in certain learning processes can lead to varying levels of performance of the system among different population groups. If, for example, a system for detecting skin diseases is trained exclusively with images of light-skinned people, this can result in it being less able or even unable to identify diseases in images of dark-skinned people. In this way, structural inequalities between groups of people can be exacerbated by limiting their access to health technologies. It is important to emphasize that the discrimination in question can exist even if there is no intention to discriminate on the part of the programmers. To prevent this problem, AI systems used to identify diseases and make diagnoses must be tested to determine how reliable their results are for different population groups. If it turns out that an AI produces inadequate results for certain groups, technical measures should be taken—such as reselecting the training data—to improve performance. In any case, patients should be informed about how reliable AI is in their case.
While the ethical debate on aspects of fairness sometimes focuses on technical and mathematical definitions and solutions to fairness problems – the basic concept here is bias, which in this context refers to the systematic unequal treatment of things that are, at least at first glance, equal – such approaches should be supplemented by a normative ethical analysis. From a technical-mathematical perspective, bias refers to a type of statistically unexpected result, such as when an AI application is not as successful as intended in identifying clinical pictures in the case of a specific group of people. Precisely because not all forms of unequal treatment carry equal moral weight, as the debate on so-called desirable biases shows, an ethical analysis of the problematic aspects of unequal treatment is necessary.
Trust
Trust is a fundamental concept in medical ethics. Ensuring the trustworthiness of those involved in the healthcare system is a moral imperative. Trustworthiness is a normative category that is independent of the question of whether and whom individuals actually trust. Not every person who is trusted is trustworthy, and not every person who is not trusted deserves this reaction. While it is the task of psychology and sociology to use empirical research to reveal which characteristics lead people to trust other people or institutions, it is the task of philosophy to analyze the characteristics of trustworthiness from a normative-ethical perspective. The fundamental philosophical question is therefore: Who or what should people trust, and why?
A second important distinction concerns the difference between trustworthiness and reliability. In philosophical discourse, trust is considered to be the attitude we adopt toward other people, in which we assume that the person we trust is fundamentally willing and able to justify the trust placed in them. For example, if a patient trusts a doctor before an operation, they assume that the doctor is willing to perform the operation successfully and also has the relevant skills to do so. However, not all forms of trust necessarily have these characteristics. In personal relationships, such as the relationship between parents and children or between psychiatrists and their patients, it is possible that trust is given in order to either improve the relationship between the individuals or to strengthen the self-esteem of the individuals who are trusted. As these examples show, individuals can build a relationship of trust with each other for different reasons and motives. Furthermore, relationships of trust are linked to interpersonal reactive attitudes. If trust is broken, people are justified in showing disappointment, anger, or other negative reactive attitudes.
The characteristics mentioned above distinguish relationships of reliability from relationships of trust. A relationship of reliability exists, for example, when a person relies on a bridge they are crossing not collapsing or a vehicle they are traveling in not suddenly stopping. As these examples show, people can have relationships of reliability not only with other people, as is the case in relationships of trust, but also with tools. However, this does not mean that relationships of reliability cannot also be established with people. For example, if you rely on a person to behave in a certain way without that person knowing about it, there is no relationship of trust, but rather a relationship of reliability. According to a widely known anecdote, the inhabitants of Königsberg in the 18th century relied on Immanuel Kant going for a walk at a certain time every day. If Kant did not do so on a particular day for some reason, the inhabitants had no reason to be angry with Kant because he behaved differently than they expected. In a relationship of reliability, it is therefore not appropriate to react with disappointment or anger toward the person who does not behave in the expected manner. Of course, one can be annoyed that a vehicle no longer works, but it would be irrational to be angry at the vehicle. Similarly, it would be inappropriate to be angry at a person who does not behave as we expect if that person was unaware of our expectation and had no reason to behave in the manner in question. Furthermore, in the case of a relationship of reliability, there is only one basis that justifies our relying on something or someone: namely, the probability that the expected behavior will actually occur in the case of persons, or function in the case of artifacts. The reason for relying on a bridge, then, lies solely in how likely it is that the bridge will not collapse.
This highlights the two key differences between reliability and trust relationships: First, a trust relationship can be established for various reasons and motives, whereas a reliability relationship can be justified solely by the likelihood of expected behavior or functioning. Second, in the case of a relationship of trust, but not in the case of a relationship of reliability, it is appropriate to adopt certain reactive attitudes. This makes it clear that the relationship between people and AI can only be a relationship of trust in the true sense if AI is understood as a subject in the strong sense.
While it is generally controversial in the AI debate whether and to what extent self-learning AI possesses the relevant capacities to be understood as subjects and even whether and to what extent AI is therefore worthy of moral consideration, it should be clear, at least in the case of the form of AI used in medicine, that such ambitious attributions of agency and subjecthood are not justified. AI models used in medicine are technical tools designed for narrowly defined purposes—such as the identification of diseases in radiological images—but they are not moral agents in the sense relevant here. If such AI systems are not subjects, it follows that it is pointless to establish a relationship of trust with them. Thus, it makes no sense to be angry at an unreliable AI system, nor does it make sense to trust such a system in order to improve the interpersonal relationship with it. So when we debate what it means for an AI to be “trustworthy”, this should be understood as asking how reliable an AI is. As stated at the outset, the normative basis for relying on an artifact is solely the expected probability that this artifact will perform the function for which we developed it. So while the term trustworthy AI is used in ethical discourse, it should be understood in the sense of trust as reliance, i.e., trust as reliability.
When these considerations are applied to the use of AI in medicine, it becomes clear that such AI is trustworthy in this sense of reliability precisely when there are good reasons to assume that it is a reliable tool. Since different AI applications are used for different purposes—such as diagnosis, image classification, or administration—the criteria for their reliability depend on area-specific tasks. For example, AI that only classifies images correctly in about half of all cases can hardly be considered a reliable tool for identifying diseases and therefore cannot be considered trustworthy AI.
However, whether or not AI is trustworthy in the sense described above is not the only dimension in which trust is relevant in the context of AI use in medicine. AI systems are developed, sold, and used by individuals and institutions. As with all tools, the central relationship of trust exists between the individuals involved in these interactions. In the case of AI used in medicine, this relationship includes the individuals and groups who develop and sell AI, the doctors and institutions—hospitals, research facilities, etc.—who use AI, and the patients for whose medical care AI was designed. When these interpersonal relationships are taken into account, referring to reliability as a basis for trust is no longer sufficient. This does not mean that reliability is irrelevant to interpersonal relationships of trust, but rather that other aspects must be considered in addition to reliability.
In the relationship between patients and physicians, the basis of trust is the medical profession's code of ethics and the character of the medical staff. According to the medical profession's code of ethics, physicians make decisions that are primarily focused on the well-being of their patients, However, they should not do so in a paternalistic manner, but rather respect the autonomy of their patients by obtaining informed consent prior to any procedures and disclosing all relevant information that contributed to the diagnosis in the context of the doctor-patient consultation. If AI is used in this context, it may be necessary to pass this information on to patients. This is particularly true if a diagnosis was made using an AI system whose functionality is limited for groups of people to which the patient belongs. But even apart from such cases, it may be useful to address the use of AI. In the case of AI systems whose use is not yet standard medical practice, such disclosure is mandatory.
Furthermore, it is important to note that decisions regarding the use of AI should be made by trustworthy decision-makers. Those physicians responsible for the use of AI in medicine should, in order to be considered trustworthy actors, firstly ensure that AI is not used solely to save costs while simultaneously compromising medical care, and secondly, examine whether and to what extent the hopes associated with the use of AI in medicine have actually been realized. If, for example, it turns out that the use of an AI system leads to frequent errors, which in turn have to be corrected by medical staff, the use of AI should be reconsidered. In order to ensure the trustworthiness of the institutions responsible for healthcare, it is therefore necessary to thoroughly examine whether AI systems prove themselves to be capable of sustainably improving the well-being of patients.
With regard to the relationship of trust between doctors and patients and the developers of AI, the establishment of independent auditing agencies is a key consideration. On the one hand, these agencies can identify and name any structural discrimination issues that arise in the development of AI; on the other hand, they can scientifically validate whether the promises made by developers regarding the performance of AI are actually correct. Finally, the relationship of trust between the parties involved in this trust relationship can be improved by, first, giving doctors a central role in the development of AI, allowing them to familiarize themselves with the programs and contribute their knowledge to the development process and, second, taking the moral dimensions of the respective AI system into account when allocating research funds. In this way, structural incentives can be created to develop AI systems that do not merely serve to maximize economic profit, but actually benefit patients.
Transparency
Issues of transparency are linked in various ways to both the principle of respect for patient autonomy and trustworthiness. Particularly given the novelty of AI systems in medicine, it seems reasonable that both the development and use of AI in a medical context should be subject to transparency requirements. However, in the case of AI, this requirement for transparency must be limited insofar as – at least at the present time – the performance of many AI systems declines as their explainability increases. Since AI systems used to identify diseases are black-box systems, the explainability of such systems is a problem that must be weighed against other ethically relevant considerations. If, for example, an AI system is used to identify diseases, it must be weighed up whether and to what extent the possibility of being able to explain the output of the AI outweighs the disadvantage of less precise AI performance. The principles underlying this are the principle of beneficence on the one hand—precise AI benefits patients by enabling clinical pictures to be identified quickly and with a high degree of accuracy—and the principle of transparency on the other, which requires the necessity of medical interventions to be explained and justified by the treating physicians. Which of these two principles carries more weight in a given situation depends on the specific characteristics of the situation. At the same time, work should continue on technical solutions that mitigate the trade-off between performance and explainability of AI systems. Furthermore, it should be noted that transparency and explainability allow for differentiation, so that varying degrees of transparency are possible depending on different considerations. Against this background, it is conceivable that at least certain forms of transparency that do not overly compromise the performance of an AI could be ethically required in the context of informed consent.
In addition to the question of balancing performance and transparency, a second aspect of the transparency debate concerns the question of what kind of transparency should be required and to whom. It is relatively undisputed that the technical tools used by physicians must be reliable and that the degree of reliability should be disclosed to patients, at least upon request. Particularly in light of the potential problem of discrimination, disclosing the accuracy of an AI system to patients is ethically imperative. Less relevant, however, are technical details of the system, which usually do not help patients in their medical decision-making. Such details may be relevant, however, if the question of transparency is posed not to patients but to physicians. Since this group of people has extensive medical knowledge that patients generally do not have, and since doctors use the AI systems in question as tools to help them make decisions for which they themselves are responsible, different transparency considerations apply in this case than in the case of patients. Phyisicians must be able to assume that their tools take into account the current state of scientific knowledge and will be improved in the event of new findings. In this context, too, it may be useful to involve doctors in the development of AI systems.
Depending on who is required to provide what kind of transparency, it is ultimately important to ensure that the disclosed information is useful for the respective group of people. In the debate on transparent AI, a distinction is made here between the concepts of interpretability and explainability. Interpretability refers to the kind of understanding of AI that requires extensive technical knowledge and the ability to comprehend causally and functionally how AI works. This type of transparency is not relevant for doctors or patients. Explainability, on the other hand, refers to a type of understanding that is oriented toward specific questions. In a medical context, these questions primarily concern medical decisions. For example, it may be useful for a patient and a physician to understand that a self-learning AI is able to recognize patterns and thus identify diseases based on specific training data. Such an understanding of how AI works is not overly demanding, but can nevertheless be decisive for medical decisions. These considerations suggest that the problem of black-box AI requires a differentiated analysis in each individual case. It is important to examine what kind of transparency we should demand, to whom we should be transparent, and how the relevant information should be disclosed.
Responsibility
Both medical professionals and developers of medical technology bear responsibility for the use of the tools they use or create. In the case of AI systems, however, a debate has arisen in which the question of assigning responsibility in the event of defective AI is being discussed. The background to this is the idea that the black-box nature of self-learning AI systems could undermine the usual attribution of moral responsibility. This is because, according to a widely held view, moral responsibility requires that the consequences of an action must be essentially foreseeable. Only what we can reasonably foresee, according to this view, is subject to our control and can be part of our intention to act. However, the argument goes, the fundamental unpredictability of AI outcomes prevents the attribution of moral responsibility for precisely this reason. If neither the developers nor the medical professionals using the AI can predict what outcomes an AI will produce, neither of these groups of actors can be held morally responsible for the failures of AI systems. The danger expressed in this consideration is that gaps in responsibility could arise when AI is used in medicine.
A number of objections have been raised against this argument, two of which are particularly relevant. First, it is generally accepted that the producers of medical devices are responsible for ensuring that the aids they develop actually fulfill the advertised function. Furthermore, those who use AI systems, i.e., those responsible for the healthcare system, must ensure that the systems they use can be relied upon to work properly. If this is not the case, responsibility for errors can still be attributed even if the functioning of an AI is not transparent. From this perspective, the focus shifts from the direct attribution of responsibility in the event of errors to the responsible use of AI in general. Responsibility, according to this line of thinking, is more complex than the idea of a direct attribution of responsibility based on individual actions. It may be true that we can only attribute an action to a person if they can foresee the consequences of their actions. It may also be true that this is precisely what is not possible in the case of AI development and the use of AI. However, this does not mean that developers and doctors who use AI are not responsible for the use of AI. Rather, it should be emphasized that developers and users of AI are just as responsible for the use of AI in various respects as dog owners are for their animals, over whose behavior the owners have just as little control and which they can predict with just as little absolute accuracy as the results of AI. In order to take this responsibility into account, it can be required that AI systems be tested and validated by an independent body. It can also be required that certain aspects of AI be made transparent, such as the selection of training data and the success rate of AI in different fields of application.
A second form of criticism of the idea that gaps in responsibility arise because the functioning of AI is not fully understandable and therefore fundamentally neither controllable nor predictable can be derived from the observation that even people sometimes represent black boxes. For example, in the case of neuroradiology, it has been observed that doctors are by no means always able to disclose exactly how they arrived at certain results because their training does not provide them with clear and unambiguous rules for every clinical picture that they can use to clearly classify individual cases. Instead, they are often confronted with a certain number of images and gradually learn to identify diseases through this process. The opacity of medical decision-making does not prevent them from being responsible for their decisions. It could also be argued that there is no need to be completely transparent about how a particular result is achieved when assigning responsibility.
Even if AI systems have characteristics that are relevant to the practice of assigning responsibility due to their black-box nature, it does not follow that individuals therefore bear no responsibility for the use of these systems. Rather, attribution of responsibility is a complex phenomenon in which individuals and groups of individuals can and should be held responsible for the production and use of technologies even if they cannot explain or anticipate the functioning of the technologies in question in every detail.
Respect for Autonomy
The principle of respect for autonomy in the context of medical ethics is usually understood as an ethical imperative that focuses on respect for patient autonomy. Given historical experiences where a paternalistic doctor-patient relationship has led to ethically questionable imbalances in decision-making and respect for patient autonomy, this focus is not surprising. However, particularly when discussing the use of AI in medicine, ethical analysis cannot be limited to this perspective alone. Rather, it is necessary to examine whether and to what extent AI in medicine could also restrict the autonomy of medical professionals in problematic ways.
To clarify this question, it must first be pointed out that a complex and differentiated philosophical debate has developed around the concept of autonomy, from which a number of different conceptions of autonomy have emerged. Central to the use of AI in medicine is the understanding of autonomy as negative freedom, autonomy as positive freedom, and autonomy as an individual decision to consent to or refuse medical interventions. The pertinent form of autonomy here is fundamentally an individualistic one. This does not mean that relational concepts of autonomy are irrelevant, but only that the individualistic concept of autonomy provides a good starting point for the ethical debate. Finally, it is important to note that respect for autonomy as an ethical principle does not encompass all conceptions of freedom and self-determination that are discussed in the discourse. For example, it may be a form of freedom when a person, free from external influences, pursues their inclination to inflict pain on others. However, this form of freedom is obviously not morally worthy of respect. So while respect for human freedom and self-determination is fundamentally a moral imperative, it is necessary to examine the type of freedom involved in each individual case.
The influence of AI systems in medicine on medical professionals primarily concerns the question of the extent to which such systems restrict the negative and positive freedom of staff. The question of negative freedom primarily concerns freedom from external influences on the decision-making of staff. This issue affects not only automated forms of AI-supported disease diagnosis, but also the automation of work processes in everyday medical practice. AI systems that identify diseases and those that structure workflows by automating administrative tasks or planning working hours, for example, could, according to this line of thinking, externally dictate to medical professionals how they should organize their daily work and intervene in everyday work processes. Medical staff can therefore be considered restricted in their negative freedom because they have to follow the guidelines provided by such AI systems. However, such external guidelines and workflows are not a new development that has only come about with AI systems. The fact that medical professionals are subject to external constraints is a basic requirement for the organized functioning of a complex system such as a hospital. The restriction of negative freedom in question is therefore not ethically objectionable if the AI systems used reliably perform administrative tasks.
The problem associated with the possible restriction of the positive freedom of medical professionals primarily concerns the worry that doctors could lose medical skills if AI systems take over certain tasks. In this understanding, the relevant positive freedom therefore concerns the conditions that enable actions related to the skills of medical staff. One concern here is that doctors will lose the ability to recognize certain diseases in images, for example, if AI systems take over this task completely due to their greater accuracy and efficiency. The term raised in this context is that of de-skilling, which describes the loss of human skills in the course of technological solutions. In this case, too, it seems reasonable to assume that doctors will lose certain skills over time if these are reliably taken over by AI systems. The relevant skills are unlikely to be part of medical training in the future, and the lack of daily practice suggests that even those doctors who once possessed the relevant skills will lose them over time. So while this kind of loss of positive freedom in terms of certain skills is obvious, it is also true in this case that such processes of de-skilling are neither new nor necessarily problematic in everyday medical practice. Throughout medical history, the emergence of new technical possibilities has always led to a shift in the demands placed on doctors. Instead of training the ability to detect diseases by hand scanning, for example, doctors today must be able to operate an MRI machine and interpret the images it provides. Since MRI can provide more accurate information than a physician could obtain by hand scanning alone, the loss of the ability to detect diseases by hand scanning is not problematic. Similarly, in the case of the use of AI in medicine, physicians will lose certain skills over time if AI provides better information in certain areas of responsibility than physicians can obtain without the help of AI. However, this does not restrict the positive freedom of doctors in any problematic way and leads more to a change in their job profile – a re-skilling or up-skilling – rather than a complete loss of relevant medical skills.
More ethically problematic is the influence of AI systems in medicine on patient autonomy. In the medical context, patient autonomy is respected by making it mandatory to obtain informed consent prior to medical interventions. Informed consent must meet certain criteria. It must be given intentionally, free from manipulative and coercive influences, and the person giving consent must have a sufficient understanding of what they are consenting to or what they are informedly refusing. The use of AI in medicine could undermine the latter point, especially in the case of AI systems used to identify diseases. Since systems of this kind are black boxes, the question has been raised whether and to what extent physicians can explain and justify their decisions for or against a medical intervention made with the help of such AI systems. The argument is that, since sufficient understanding on the part of patients requires their doctors to be able to disclose in a comprehensible manner how they arrived at a particular diagnosis, the use of black-box AI in medical decision making could undermine patient autonomy.
The fundamental question raised in this context therefore primarily concerns the aspect of understanding autonomously given, informed consent. The question is, accordingly, what kind of information must be disclosed to patients and understood by them in order for informed consent to be considered an expression of autonomy. Obviously, patients cannot and should not be expected to understand all the technical medical details that led a physician to a diagnosis. If such a high standard of understanding were applied, hardly any informed consent would meet the criterion of understanding. Similarly, it is not convincing that patients must be told how an AI system works in detail and why it has identified a particular disease. Furthermore, as already explained above, human doctors are also black boxes in certain respects. Doctors are by no means always able to reconstruct their own decision-making in such detail that they could disclose all relevant justifying reasons. Particularly in the case of identifying diseases in images, radiologists are often unable to say exactly which features of an image indicate a particular disease. Nevertheless, radiologists' assessments are often correct. Against this background, it seems unconvincing to consider informed consent appropriate in one case and to argue in the other that the essential features of morally justifiable informed consent are not fulfilled. Of course, this does not mean that physicians should not be able to explain the basis for their decision to patients as part of their duty to disclose information, but it does mean that the reasons that led to the decision do not require a detailed explanation of all aspects of the decision making process. If AI has been proven to be reliable in identifying diseases, pointing out that AI is usually reliable in such cases may be sufficient for the understanding relevant to informed consent. However, this presupposes, first, that the AI actually works reliably and, second, that the physician has no independent reasons to assume that it has made an incorrect identification.