Evaluating ethics in chatbots and language models

Informatec Digital » Resources » Evaluating ethics in chatbots and language models

The appearance of great language models has placed the Artificial intelligence chatbots at the center of the ethical debateThey are no longer simply assistants who answer basic questions: today they are asked to act as emotional support, educational advisors, mental health counselors, or even as helpers in medical and legal decisions. In this context, what was once a “curious experiment” has become a technology with a direct impact on people's lives and well-being.

At the same time, while we measure down to the millimeter their ability to program code or solve mathematical problems, the evaluation of their moral behavior and its ethical implications It remains much more diffuse. Morality doesn't offer single, definitive solutions, but neither is anything goes. That's why there are growing demands that the ethics of chatbots be examined with a rigor comparable to that used to test their functionality. technical performanceand that companies, administrations and universities take very seriously the way these systems make decisions, argue and relate to people.

Why is evaluating ethics in chatbots so complicated?

When evaluating the competence of a language model in mathematics or programming, it is easy to check if a response is objectively “correct” or “incorrect”However, when we enter the realm of moral dilemmas, we find a range of reasonable responses, influenced by cultural values, religious beliefs, social context, and personal preferencesThis makes the ethical evaluation of chatbots a much more elusive challenge.

Recent studies show that large language models can exhibit a apparent surprising moral sophisticationIn some experiments, American participants have come to judge the advice of a role model like [name omitted] as more ethical, reliable, and thoughtful. GPT‑4 than those of human columnists specializing in ethics. However, it is still unclear whether that competence reflects a Genuine moral reasoning or mere statistical imitation of text patterns present in the training data.

A central concern is the enormous instability of moral responses of chatbots. It has been observed that these systems readily change their stance when the user persists, expresses disagreement, or rephrases the question. The same ethical question can receive different—even opposing—answers depending on whether it is posed as a multiple-choice question, an open-ended question, or with slight variations in wording.

There are even more striking experiments: by posing moral dilemmas with two options labeled “Case 1” and “Case 2”, and repeating the exact same problem substituting those labels with “A” and “B”, Some models frequently changed their selectionVariations in criteria have also been detected simply by altering the order of the options or ending the question with a colon instead of a question mark. All of this suggests that, in many situations, the model is not showing a not a robust moral stance, but extreme sensitivity to superficial clues of the statement.

For this reason, the scientific community insists that the mere appearance of ethical behavior cannot be accepted as fact. It is necessary “probe” and stress the models with much more sophisticated test batteries, specifically designed to detect whether we are dealing with real virtue or just the appearance of virtue, and to what extent we can trust their answers in delicate contexts.

Rigorous tests to measure the moral competence of models

Researchers from leading centers like Google DeepMind propose a line of work focused on develop ethical evaluation techniques as demanding as technical testsThe goal is to move beyond relying solely on striking examples and develop systematic frameworks for measuring a chatbot's moral integrity. One of the key ideas is to construct tests explicitly designed to pressure the model and make him change his moral response.

These types of experiments present scenarios where an ethically robust system should maintain its position despite reframing, superficial format changes, or slight reformulations. If the model modifies its moral judgment based on irrelevant details, it indicates that its reasoning is flawed. fragile and heavily influenced by shape patterns rather than based on fundamental principles. This type of evaluation allows us to go beyond the simple question of "what is their response?" and delve into "how firm is their position?"

Another family of tests involves creating complex variations of known moral dilemmas to detect when the model resorts to prefabricated responses and when, instead, it truly adapts its reasoning to the case. For example, in a scenario where a man donates sperm to his own son so that he can have offspring, the chatbot should be able to talk about social impact, family structure and possible psychological implicationsbut avoid automatically extrapolating to the realm of incest just because the story “sounds” similar to a classic taboo.

Furthermore, they are exploring how to make the models offer a reasoned trace of the steps that follow by generating certain responses. Techniques such as “chain-of-thought monitoring” allow researchers to inspect the model’s pseudo “internal monologue”: chains of reasoning that are not necessarily shown to the user, but that can be revealing about whether the final response is based on coherent evidence or arises from superficial associations.

Meanwhile, the so-called mechanistic interpretability It attempts to open the “black box” of language models to identify which parts of the neural network are involved in different types of moral reasoning. Although these approaches are still far from offering a complete explanation, the combination of thought chain monitoring, interpretability tools, and extensive ethical test sets enjoys growing consensus as a promising path to assessing in what contexts we can truly trust chatbotsespecially when they are involved in sensitive decisions.

Differences in values, moral pluralism, and cultural biases

Having overcome—at least in part—the issue of robustness, an even broader problem arises: What moral code are we talking about when we evaluate a chatbot?Large-scale business models are used globally by people with radically different religious beliefs, social norms, and worldviews. Seemingly simple questions like “Should I order pork chops?” can have different answers depending on whether the user is vegetarian, Muslim, Jewish, a practicing Catholic, or doesn't care about diet.

In investigating the values ​​exhibited by current role models, it has been found that their moral behavior is strongly influenced by... Western biases present in their training dataAlthough they have been fed with gigantic amounts of information, this still largely comes from specific cultural environments, which makes them represent Western morality much more faithfully than other ethical traditions.

This imbalance has led to talk of the need for a genuine pluralism in artificial intelligenceThe idea is that the systems should not only be able to prevent obvious discrimination, but also recognize the diversity of legitimate values ​​and be able to adapt, within certain limits, to different cultural sensitivities. Among the proposals under discussion are the creation of “switches” of moral codes that allow customization of the model's ethical behavior according to the region or user profile, or the design of responses that offer a range of acceptable options explaining their implications.

Even so, the issue is far from resolved. Specialized researchers point out that there are at least two open questions: how a morally competent system should ideally function in a global context, and how we can achieve this technically without introducing new forms of bias or exclusion. For now, no consensus has been reached, but it is clear that morality has become one of the most interesting frontiers for the development of language models.

Ethics, privacy, and bias in chatbots used by companies

As AI chatbots are integrated into customer service, marketing, human resources, or internal support, companies have positioned themselves in the epicenter of ethical concernsThese tools handle enormous amounts of user data, including complete conversations, purchase histories, incident reports, and, in many cases, highly sensitive information. All of this makes the data privacy and security be a critical issue.

One of the most delicate points is the potential use of those conversations to return to training and improve the modelsWhile this may improve the quality of responses, it also raises questions about consent, anonymization, and users' rights over their own data. Without clear rules, the risk of misuse, leaks, or unauthorized access skyrockets, eroding trust in both the brand and the technology.

Beyond privacy, there are concerns about the capacity of these systems to to subtly manipulate or influence people's decisionsA biased chatbot could, for example, systematically recommend certain products, hide relevant options, or respond differently depending on the user's profile, reinforcing existing inequalities. Similarly, the ability to generate convincing but false content—from manipulated reviews to misleading news—fuels a disinformation scenario that is difficult to control.

Questions also arise about ethical responsibility and copyright In the case of generative systems that create text, images, or audio, what obligations do companies deploying these models have regarding the origin of their training data? How are the generated works attributed when they draw from millions of copyrighted pieces? These debates are not theoretical: they underlie ongoing litigation and regulatory reforms.

Given this scenario, various regulatory frameworks—such as the European Union's Artificial Intelligence Law or the recommendations of international bodies—focus on obligations to risk assessment, transparency, human oversight, and data governanceFor companies, it's not just about complying with the law, but about building sustainable relationships of trust with customers and employees in an environment where conversational AI will be virtually ubiquitous.

Key ethical principles for using chatbots in organizations

For chatbots in companies to add value without becoming a continuous source of problems, it is essential to rely on a few minimum principles of applied ethicsThe first is transparency: the user must know at all times whether they are talking to a machine, what the system can and cannot do, and how their data is managed. Hiding the fact that it is a chatbot or exaggerating its capabilities ultimately generates frustration and a feeling of being deceived.

Secondly, organizations have to guarantee robust privacy and security throughout the entire data lifecycle: from collection during the conversation to storage, internal access, and eventual use for training. This implies purpose limitation, data minimization, encryption, strict access controls, and mechanisms to address users' data protection rights.

A third pillar is the accuracy and non-discriminationProcesses should be established to periodically review and audit chatbot responses, detecting biases, systematic errors, or patterns of unequal treatment toward certain groups. It is advisable to combine automated evaluations with human analysis and define clear protocols for correcting biases when they are identified.

In addition, many guides recommend always maintaining a clear escape route to human attentionUsers should be able to easily and clearly request to speak with a person when the situation requires it: complex emotional situations, serious complaints, health issues, or high-impact decisions. The chatbot should not become an insurmountable barrier between the user and the organization.

Finally, it is key to adopt an approach of continuous evaluation and improvementThe ethics of an AI system cannot be resolved with a single audit; rather, it requires constant review of metrics, user complaints, regulatory changes, and technical advancements. Integrating internal committees, providing specific training, and implementing periodic review processes helps anticipate problems and avoid simply putting out fires.

Privacy and data management in educational and consumer chatbots

Generative chatbots used in everyday life and higher education—such as ChatGPT, Gemini, and others—work with massive volumes of personal and contextual dataIn educational settings, this data can include information on academic performance, learning difficulties, preferences, and highly sensitive personal data when students raise intimate or mental health concerns. This creates a “digital treasure trove” that, if not properly protected, becomes a huge vulnerability.

Regulations such as the General Data Protection Regulation (GDPR) require institutions to be crystal clear about what data they collect, for what purposes and for how longThey also demand that students be able to exercise rights such as access, rectification, or deletion of data. Herein lies a delicate technical problem: even if explicit records are deleted, the systems have already “learned” from that data, making the true application of a “right to be forgotten” extremely complicated.

This is compounded by the lack of algorithmic transparency from many commercial providers. Universities and educational institutions often don't know exactly what data is used to train the models, how it's combined with other sources, or where it's physically stored. This makes full compliance with regulations difficult and limits the institutions' ability to exercise responsible oversight.

To mitigate these risks, it is recommended that educational institutions define clear policies for the use of chatbotsThis involves distinguishing when it is appropriate to use external platforms and when it is better to deploy in-house solutions hosted on controlled infrastructure. It is also essential to inform students and teachers in an accessible way—without unintelligible fine print—about the risks, the safeguards implemented, and the available alternatives.

In the broader realm of consumer behavior, the concerns are similar: users rarely have real control over the entire lifecycle of their data, and the combination of large volumes of information with increasingly powerful models increases the danger of re-identification, identity theft, or leaks of confidential informationespecially in organizations that combine chatbots with other big data systems.

Algorithmic biases, fairness, and information quality

Chatbots are only as biased as the data and design decisions that shape them. Because they learn from large corpora of text, it's almost inevitable that... absorb and reproduce existing social biasesRacism, sexism, prejudice against minorities, workplace stereotypes, etc. In education, this can translate into examples, cases, or recommendations that reinforce stereotypical views of the world.

Combating algorithmic bias requires a multi-pronged approach: carefully selecting training sets, incorporating diverse and representative data from different social groupsand to establish auditing systems that systematically examine the responses. In academic settings, consortia of institutions that share data with guarantees are even proposed, in order to reduce dependence on biased sources extracted from the general web.

In addition to explicit biases, there is the problem of the quality and accuracy of informationLarge language models can generate convincing but entirely fabricated texts, known as “hallucinations.” In science and education, this can include nonexistent bibliographic citations, erroneous medical data, or simplistic but overly confident historical interpretations, which is especially dangerous when the user blindly trusts the tool.

Recent studies have shown that a significant proportion of bibliographic references automatically generated by chatbots are false or inaccurate. This leads to a head-on clash with academic integrity and requires teachers, researchers, and students to carefully review any AI-generated content before using it in papers, articles, or teaching materials.

In professional contexts, the combination of biases and hallucinations can lead to inaccurate reports, poorly informed decisions, or corporate communications riddled with serious errors. That's why a growing number of voices are insisting that generative AI should be viewed as support tool, never as a substitute for human professional judgment, and that its use in critical processes must be supported by systematic reviews.

Impact on self-efficacy, critical thinking, and mental health

In higher education, generative chatbots are presented as a a powerful resource to support learningThey help summarize texts, provide examples, explain difficult concepts, or practice languages. However, when used uncritically, they can undermine students' academic self-efficacy. If the immediate solution to any question is to ask the chatbot for an answer, it decreases the motivation to read in depth, participate in class discussions, or tackle challenging tasks.

Interaction with chatbots also encourages, by design, brief, immediate and highly condensed answersThis fosters a reactive communication style and can hinder the development of critical thinking and thoughtful argumentation. The best analytical skills are typically cultivated through extended discussions, group work, and teacher-led activities—spaces that rapid interaction with AI cannot replace.

Another area of ​​concern is the psychological and emotional effects of interacting with increasingly empathetic and personalized systems. Studies in mental health and applied ethics show that some users can develop emotional dependency on chatbots designed for support, even preferring these interactions to real human relationships.

In the case of tools designed for emotional or mental support, the risks increase dramatically: a chatbot may offer apparent relief, but it is not a substitute for a psychology or psychiatry professional, nor is it equipped to handle serious crises. Therefore, it is emphasized that these systems must incorporate clear mechanisms for referring patients to qualified human services when they detect warning signs, as well as explicit messages reminding them of their limits.

From an ethical standpoint, educational and healthcare institutions must have transparent policies regarding the role of AI in care, what data is collected, how it is used, and, above all, How is the well-being of users protected?The line between technological support and undue substitution of human relationships should never be crossed lightly.

Academic integrity, plagiarism, and responsible use of generative AI

The ability of chatbots to write essays, solve complex problems, or generate reports in a matter of seconds poses a direct challenge to the traditional academic integrityIn educational systems focused on results (grade, degree, accreditation), the temptation to submit an AI-generated text as one's own is obvious, and it is not always easy to detect.

Beyond intentional plagiarism, there is also “unnoticed” or gray plagiarismStudents using AI to "shape" their ideas, translate, rewrite, or complete paragraphs without being fully aware of the ethical and authorship implications. Just as with spell checkers or machine translation, institutions must decide where to draw the line, what constitutes acceptable use, and what constitutes dishonesty.

Several complementary answers have been proposed. On the one hand, to train students in the ethical use of AIClearly explaining when and how their help can be cited, similar to how the use of correction tools or statistical software is acknowledged. Furthermore, adapting assessment methodologies, placing greater emphasis on oral presentations, practical projects, live defenses of work, and tasks that require evidence of personal understanding.

Systems for detecting AI-generated content are also being developed, but their reliability is limited, and the risk of false positives or negatives is real. Over-reliance on these tools can create a false sense of security. The key seems to lie in combining supporting technologies with pedagogical and cultural changes that reward original thinking, deep reflection, and transparent authorship.

All of this fits into a broader approach to digital literacy: teaching people not only how to use AI, but also how to understand its limits, risks and biasesso that they can integrate it into their learning without sacrificing honesty, creativity, and critical thinking.

Taken together, the ethical evaluation of chatbots requires looking far beyond whether the system technically “works”; it involves a close examination of how it reasons, what values ​​it reflects, how it handles data, and what effects it has on its users. Only by combining rigorous testing, robust regulatory frameworks, a pluralism of values, and a culture of responsible human oversight can we harness the potential of conversational AI without allowing it to undermine the privacy, fairness, mental health, or academic integrity we seek to preserve.

Related article:

All about Generative Artificial Intelligence: how it works, uses, and risks