In a recent conversation with Professor Fabio Roli, one of Europe’s leading experts in AI Security and Trustworthy AI, the following shift emerged clearly: the key issue is no longer only how powerful AI systems are, but whether their performance can be verified, controlled, and trusted in real-world conditions.
Artificial Intelligence has rapidly moved from research laboratories into business processes, products, public services, and everyday decision-making.
For many years, the central challenge for AI was performance: AI systems were often too limited, too brittle, or too inaccurate to be trusted in real applications. At that time, the key question was simple: does the system work well enough?
Today, the situation has changed. In many tasks, AI systems achieve impressive, sometimes even superhuman levels of performance. But this success raises a new and more difficult question: if AI performs so well, can we trust it?
It is worth noting that the history of technology shows that performance is the starting point of trust, not its destination.
"We do not trust an aircraft simply because it can fly fast or far; we trust it because it has been tested, certified, monitored, and designed with safety, redundancy, accountability, and clear operating limits."
The same is now becoming true for AI. Once a technology works, trust depends on additional factors: reliability, security, robustness, transparency, fairness, governance, and the ability to understand when and why it may fail.
This is where the debate on Trustworthy AI begins, and where projects such as InfoAIcert become essential.
The Cow Experiment: When AI Learns Shortcuts
One of the clearest examples of why high AI performance can be misleading comes from a well-known image recognition problem, often referred as “the cow on the beach”.
Imagine training an AI system to recognize cows. If most of the training images show cows standing in green fields, the AI system can achieve excellent accuracy to recognize cows standing in green fields. But what has it learned?
A human would normally recognize a cow by its shape, body structure, head, legs, and other meaningful visual features. The AI system, however, may learn an easier strategy: instead of learning the concept of “cow,” it may rely heavily on the background. In other words, it may learn that green grass is a strong “signal” for cow.
This is not simply a funny mistake. It is a real and serious scientific problem known as “shortcut learning”.
Shortcut learning happens when an AI system finds a rule that works well on the training data and on standard test data but fails when the real world changes. The model has not necessarily learned the intended solution; it has learned a convenient correlation that was useful in the data it saw during training.
In the cow example, the shortcut is: grass means cow. Therefore, the system may correctly classify many cows in fields but fail when the same animal appears on a beach, in a street, or in another unusual context different from training data.
This is why accuracy alone can be deceptive. AI system can be right for the wrong reason.
"This is the key lesson: modern AI systems do not simply learn the world. They learn from data. If the data contains biased, incomplete, or misleading regularities, the system may learn those regularities as shortcuts."
For companies and institutions, this has a practical consequence: testing AI only on data like the training data is not enough. To build trust, we must test whether the system generalizes to new, realistic, and unexpected conditions, the kinds of conditions it will face in the real world.
This is where trustworthy AI begins: not only asking whether the system gives the right answer, but also whether it gives the right answer for the right reasons.
The Big Bet Behind Large Language Models
Since 2022, Large Language Models have changed the public perception of Artificial Intelligence.
For the first time, millions of people have been able to interact with AI systems through natural conversation: asking questions, writing documents, translating texts, generating code, summarizing information, and exploring ideas.
This progress is largely based on one of the central ideas behind modern AI development: scaling laws.
"The basic intuition is simple: if we train larger models, with more data and more computational power, performance tends to improve. In recent years, this strategy has produced extraordinary results and has made systems such as ChatGPT possible."
In this sense, modern AI is built on a major technological bet: that by increasing scale, we can continue to obtain more capable systems.
But this bet has an important limit.
Scaling can improve average performance, fluency, and the range of tasks that an AI system can handle. However, it does not automatically eliminate the deeper problems of AI reliability. A larger model may still make unexpected mistakes, reproduce biases in the data, follow misleading correlations, or fail in situations that differ from those it has seen during training.
"This is why the key question is not only whether larger models will become more powerful. The real question is whether they will also become more reliable, secure, explainable, and trustworthy."
In other words: scale is necessary for today’s AI progress, but scale alone could be not sufficient to build trust.
The challenge for the next phase of AI is therefore not only to make models bigger, but to make their behavior more verifiable. We need to understand when they work, when they fail, why they fail, and how their risks can be assessed and mitigated.
That is the point where performance becomes a trust problem, and where the discussion on Trustworthy AI becomes essential.
Accuracy Does Not Mean Reliability
One of the biggest misconceptions in the AI debate is the assumption that high accuracy automatically means reliability.
It does not.
"Accuracy tells us how often a system gives the right answer under a given set of test conditions. Reliability is a broader question: can we depend on the system when conditions change, when the stakes are high, or when errors have real consequences and can damage people?"
This distinction is not unique to AI. It appears throughout the history of technology.
At the beginning of a technology’s life, performance is essential. If a machine does not work, no one will trust it. But once the technology becomes powerful and widely adopted, trust depends on much more than performance: transparency, accountability, safety, control, robustness, and alignment with human and social values.
A fast car is not trusted only because it is fast. It is trusted because it has brakes, safety standards, inspections, traffic rules to be respected, liability frameworks, and a clear understanding of who is responsible when something goes wrong.
AI is now entering the same phase.
Modern AI systems can achieve impressive results, but they can still make critical mistakes, reproduce biases, generate plausible but false outputs, or fail in situations that are different from those used during training. The problem is not only that AI can be wrong. The deeper problem is that we often do not know in advance when it will be wrong, why it will be wrong, and how serious the consequences may be.
This is especially important for Large Language Models. There is an active scientific debate on whether these systems merely learn statistical regularities in language or whether, at least in some cases, they develop internal representations that resemble “world models”: structured ways of representing aspects of the external world.
This debate should not be reduced to slogans. Saying that LLMs “only predict the next word” is technically true at one level, but it does not fully settle the question of what kinds of internal representations may emerge during training. At the same time, even if these models do learn useful representations of the world, that does not automatically make them reliable, safe, or trustworthy.
For organizations adopting AI, the practical question is therefore not only: how accurate is this system?
But also: can we verify its behavior? Can we understand its limits? Can we detect failure modes before they cause harm? Can we assess and mitigate the risks? Can we explain who is accountable when the system fails?
This is why accuracy is only the beginning of AI evaluation. Reliability requires testing, monitoring, transparency, risk assessment, and governance.
In short: high accuracy can make AI useful, but only reliability can make it trustworthy.
What Does “Trustworthy AI” Really Mean?
“Trustworthy AI” is one of the most important concepts in the European approach to Artificial Intelligence.
But it is also often misunderstood.
"Trustworthy AI does not simply mean that an AI system performs well. It does not mean that it is impressive, powerful, or accurate on a benchmark. In the European vision, Trustworthy AI means that an AI system should satisfy three fundamental conditions: it should be lawful, ethical, and robust."
- Lawful means that the system respects applicable laws and regulations.
- Ethical means that it respects fundamental rights, human values, and ethical principles.
- Robust means that it is technically reliable and safe, while also considering the social environment in which it operates.
To make these principles concrete, the European guidelines identify seven key requirements for Trustworthy AI:
- human agency and oversight
- technical robustness and safety
- privacy and data governance
- transparency
- diversity
- non-discrimination and fairness
- societal and environmental well-being
- accountability.
This is why Trustworthy AI should not be understood as a single technical feature. It is not only about cybersecurity. It is not only about explainability. It is not only about legal compliance.
It is a multidimensional framework for building AI systems that people, companies, and institutions can rely on.
AI security is therefore a crucial part of the picture, but it is only one part. A system may be secure against attacks, but still not transparent. It may be accurate, but still unfair. It may comply with some technical requirements but still lack adequate human oversight or accountability.
This is also why AI requires new forms of evaluation. Unlike traditional software, AI systems are not only programmed; they are trained on data. Their behavior can therefore depend on statistical regularities, biases, hidden correlations, and failure modes that may not be visible from the code alone.
For this reason, Trustworthy AI must be built through a combination of technical testing, transparency, risk assessment, governance, and accountability.
In practical terms, the key question becomes: can we show, with evidence, that this AI system is lawful, ethical, robust, and aligned with the requirements of Trustworthy AI?
That is the bridge between the European vision of Trustworthy AI and the practical challenge of AI assessment, certification, and compliance.
Democratizing AI Compliance
One of the major challenges in the AI ecosystem is making Trustworthy AI accessible not only to large technology companies, but also to small and medium-size enterprises.
The EU AI Act introduces new requirements for many AI systems, especially high-risk ones. Companies will increasingly need to demonstrate transparency, traceability, risk management, security, robustness, fairness, privacy, and accountability. But for many organizations, especially SMEs, this is not easy and could be a big “barrier”.
This is where InfoAIcert project comes in
InfoAIcert is a research and industrial innovation project funded by the Italian Ministry of University and Research through the Fondo Italiano per le Scienze Applicate. Its goal is to develop the first made-in-Italy platform for conformity assessment of high-risk AI systems under the EU AI Act.
The project brings together two complementary strengths: the scientific leadership of the University of Genoa’s sAIfer Lab in AI security and Trustworthy AI, and the industrial expertise of Tinexta InfoCert as a major European trust service provider.
The ambition is practical: to transform frontier research in AI security into an operational framework that companies can use.
The InfoAIcert platform is designed to evaluate AI systems through a structured process: testing security and robustness against real-world threats, assessing trustworthiness properties such as fairness, transparency and privacy, performing risk analysis and mitigation assessment, and generating standardized technical documentation for conformity assessment.
In this sense, InfoAIcert aims to enable a Certification-as-a-Service model: organizations submit their AI systems to a structured evaluation pipeline and receive evidence that can support conformity assessment, certification processes, and interaction with Conformity Assessment Bodies. This is what “democratizing AI compliance” means in practice. It means reducing the gap between regulation and implementation. It means helping organizations that do not have large internal AI security teams. It means making advanced testing, documentation, risk assessment, and governance tools accessible beyond big tech companies.
The final goal is not only to help companies become compliant. It is to help them build AI systems that are more reliable, more transparent, more secure, and more trustworthy. In short: InfoAIcert aims to move AI compliance from a complex regulatory burden to a practical pathway for building trust.
The AI Act as a Competitive Advantage
Regulation is often perceived as a burden: something that slows innovation, increases costs, and creates complexity.
But the European AI Act can also be seen from a different perspective.
In an economy where AI systems are becoming more powerful, more autonomous, and more widely used, trust will become a decisive market factor. Companies will not only compete on model performance, speed, or cost. They will increasingly compete on their ability to demonstrate that their AI systems are reliable, secure, transparent, fair, and well governed.
The AI Act as a Competitive Advantage
The InfoAIcert project wants to create AI systems that are competitive, compliant, and certified. The message is that compliance is not only about respecting rules; it is also about becoming more credible, more transparent, and more attractive to customers, partners, regulators, and investors.
Organizations that will be able to meet these requirements early will not simply reduce legal risk; they will also be better positioned in a market where customers increasingly ask for evidence that AI systems can be trusted.
This is particularly important in critical sectors such as finance, healthcare, public services, identity management, autonomous systems, and industrial automation. In these domains, users and institutions will not be satisfied with the claim that an AI system is “powerful.” They will need evidence that it is safe, robust, accountable, and aligned with human values.
In this sense, the European approach is not only about regulating AI. It is about creating the conditions for a trustworthy AI market.
This is like what happened in other technological domains: safety standards, certification processes, and quality marks did not stop markets from growing. In many cases, they made growth possible, because they gave users and companies the confidence needed to adopt new technologies at scale.
AI is now entering the same phase
Being among the first companies able to show that an AI system is compliant and certifiable may become a strategic advantage. It can strengthen customer confidence, reduce uncertainty, support access to regulated markets, and differentiate responsible AI providers from competitors.
This is why the AI Act should not be seen only as a legal hurdle. It can become a framework for industrial leadership.
In short: the future AI market will not reward only the most powerful systems. It will reward the systems that can be trusted.
Key Takeaways
Artificial Intelligence is becoming more powerful, more pervasive, and more useful across business and society.
But performance alone is no longer enough.
High accuracy can make an AI system impressive, but it does not automatically make it reliable. A system may produce excellent results in standard conditions and still fail when the context changes, when the data is biased, or when it has learned shortcuts instead of the intended solution.
This is one of the central lessons of modern AI: we must not only ask whether a system gives the right answer, but whether it gives the right answer for the right reasons.
This is why the European vision of Trustworthy AI is so important. Trustworthy AI is not a single feature: it means building systems that are lawful, ethical, and robust, supported by transparency, accountability, human oversight, privacy, fairness, security, and risk management.
For companies, this marks a strategic shift. The key question is no longer only: what can this AI system do?
The key questions are also: can we verify how it behaves? Can we understand when it may fail? Can we assess and mitigate its risks? Can we document and demonstrate its compliance? Can users, customers, and regulators trust it?
This is the challenge that projects such as InfoAIcert are designed to address: transforming advanced research in AI security and Trustworthy AI into practical tools for assessment, certification, and compliance with the EU AI Act.
In the next phase of AI, trust will not be a secondary issue. It will become a condition for adoption, a requirement for compliance, and a source of competitive advantage.
The future AI market will not reward only the most powerful systems. It will reward the systems that can be trusted.
