Introduction
Recently, the Supreme Court of India in Pooja Ramesh Singh v. Jammu & Kashmir Bank Ltd.1 made strong remarks against the National Company Law Tribunal (NCLT) Bench that passed an order relying on an artificial intelligence (AI)-generated hallucinated judgment. Such a remark came from the court after multiple exposures to such hallucinated judgments and the trigger was a repeat instance from the NCLT. Sometime before this order came to be passed, the court had floated a draft regulation for “regulating the use of AI in courts” calling for feedback from the experts in the field and the public.
In light of the guideline this article briefly throws light on hallucinations, the strength of the draft guideline in managing the use of AI in courts and suggests some ways to mitigate or address such hallucinations.
Highlights of the draft guideline released by the Supreme Court and opinion of the authors
There are many salient aspects in the regulation that will aid in the deployment and use of AI in courts as mentioned below:
1. List of acceptable uses and prohibited uses of AI in legal framework.
2. Requirement for disclosure/transparency wherever AI is used in the legal process.
3. Requirement of adopting explainable AI systems into the framework along with its responsible use.
4. Proportional use and adoption of AI necessary for the task and human-in-loop oversight.
5. Setting up a regulatory body to supervise the implementation and functioning of AI systems.
6. Transparency in auditing process, disclosure of incidents and an internal mechanism to address them.
7. Training to all individuals working with AI systems in courts.
There are also a few “difficult to implement” rules in the draft guideline which will be briefly discussed. One of them is the prohibition on the use of “opaque and unexplainable AI systems that may materially affect the lawful rights or personal liberty of any party”2. An AI model whose design allows for gaining insight into the decision-making process is called a transparent and an explainable model. Such models are referred to as white box models. Those models that are not transparent or give insight into the decision-making process are called “black box” models. Large language models (LLMs) are black boxes by design. The prohibition is difficult to implement because there are not many white box models available in the market that can be usefully deployed in legal settings like courts. Most of the LLMs that are currently used for research work and drafting legal documents are black boxes which offer no transparency into their decision-making process or are explainable. The lack of definition on “materially affect the lawful rights or personal liberty of any party” confounds the prohibition imposed. It is true that such a phrase is difficult to define and better to be left open to accommodate unexpected cases, but at the least an open-ended definition is required to understand and further the intent of the rule.
Another such rule is Rule 20(1)(a) which states “no personal data of any person shall be used to train, test, or refine any AI System without the prior approval of the appropriate authority and, where applicable, in compliance with applicable data protection law”. When deploying off-the-shelf models, which will mostly be the case, it is difficult to implement such a rule as almost all such models are pre-trained and fine-tuned with personal data. Further, the training also depends on the use cases that the AI will be deployed for. The draft rule does define “anonymisation”3 but is silent on how it should be done or the standard that should be adopted. Unless a robust and consistent anonymisation4 methodology is implemented (specific to court setting) it is unlikely for this rule to complied with.
The definition of “harm” in Rule 3(1)(za)5 seems to be narrow in the context of the use of AI in courts. The most significant impact that AI in legal field can have is “miscarriage of justice” in some form. Say, AI is deployed for urgent listing of cases before the right portfolio Judges and the system fails to function as expected. The party whose case does not get listed on time is denied speedy remedy due to failure of the AI system. Under these circumstances, the immediate financial and physical harm might not be apparent until the case reaches the right Judge, but at that instance, the party is denied speedy remedy. In the opinion of the authors, the scope of the definition requires enlargement. The definition should take into consideration the language adopted in Rule 20(1)(e).
Then there is Rule 166 which is worded in way to actively promote adoption of AI into the judicial system. The use of the word “shall” denotes a mandate for the adoption of AI failing which there needs a good reason for not adopting it. Such a language gives an impression of forcing the adoption of AI into legal system rather than exploring the possibility of streamlining process with AI system’s assistance. Ideally, pilot studies need to be carried out before adopting AI into any process to test its performance. The draft rule does state that “Technical and Ethical Impact Assessment” must be carried out along with controlled environment testing for deploying AIs in courts. However, a full-fledged AI governance program will be the need of the hour to manage the complete life cycle of AI until decommissioning.
While a few amendments to the language in the draft regulation will take us a step close to better regulation of AI, it is important to understand the underlying reason for hallucinations.
What is hallucination?
The term “hallucination” in general means refers to “a false perception of objects or events”7. Its meaning in the context of AI has been debated for long and there is no universally accepted definition. However, researchers in the field have come up with a workable definition — “ ’AI hallucination’ refers to the phenomenon where AI generates distorted information. Scholars have defined distorted information as ‘false or inaccurate information regardless of intentional authorship’ ”.8 The draft guidelines define “hallucination” in Rule 3(z) in similar lines.
The simplified way to explain hallucination is — when an LLM is used for analysing a scenario (or problem) which it has not been trained on, it produces results with its existing training data which may not be the correct response to the scenario.9 This it does by putting together a string of acceptable words (based on the training data and weights/parameters) which ultimately gets displayed as output irrespective of the existence of such string of words in the real world. As a result, the AI may give an output (case laws for example) which will look reasonable but such an output may not exist in reality.10 It is difficult to ascertain the truth value of such outputs without independent cross-verification.
For every output generated by LLM, in the background the model attributes something called as a “confidence score”. When the output of a system correlates with the training data, the output so generated is given a high confidence score (by the LLM itself), meaning the output has a higher possibility of being accurate. The outputs having a low “confidence score” might not be accurate due to lack of the LLM exposure to the unique set of data (training) required to process that problem/scenario. In real life use cases, these scores to the output so produced are never made available to the end-users to allow them to make an educated decision to rely on the result or not.
Be that as it may, even the confidence score is not a good indicator of the efficiency of the model as these scores are again a result of the training itself. An LLM trained with improper methodology (or bias in data) will end up attributing high confidence score even to inaccurate outputs.11 This is how hallucinations creep into the output and go unnoticed due to over-reliance on the AI system. It is a common problem with all AI systems and more rampant in generative AIs, foundation models and LLMs.
It is not just technical reasons that hallucinated judgments find place in legal documents before courts and judgments, but also a lack of human scrutiny of the output produced by such AI systems. For measures to be effective in the context of adopting AI in courts, they should cover both technical and non-technical aspects of deployment. With this understanding of what causes hallucinations different measures to address them can now be explored.
Possible measures to combat hallucination and regulate AI
As discussed above, a multifold approach must be adopted to address the issue of hallucinated judgments considering the following:
Restricted use of foundation models/LLMs: Though the draft guideline lists prohibited area of use of AI, this measure would still leave doors open for adoption of LLMs for multiple tasks in courts, more so because foundation model is built for such varied applications. So the “proportionality principle” in Rule 12 must impose restrictions on use of such heavy models to make the rule effective. For a use case that can be achieved with a simple automation algorithm, using data heavy models should be avoided. The proportionality principle will entail use case scoping of the activity and the AI model required. Specialised models are prone to lesser hallucinations as they are easier to fine-tune due to limited use of data. Further, implementing models with restricted designs will also be required to prevent scope creep. Scope creep is when an AI that has been deployed for specific task but is used for some other tasks which it has not been designed for due to its versatility.
Transparency in all legal documents: An overt declaration of the use of AI by lawyers in their documentation and in court documents (by courts) will go a long way in reducing submission of documentation containing hallucinated outputs. Such an overt declaration puts the adversary (and all parties) on alert and drive them to scrutinise better. In part, this transparency requirement already finds place in Supreme Court (SC) White Paper disclosure requirement within the guideline. However, its applicability should also be extended to all pleadings filed before the courts. A declaration should be submitted by lawyers disclosing the use of AI and the purposes for which it has been used like drafting plaint, petition, research for citations for arguments, etc.
Restricted use of AI for administrative work: The use of AI must be strictly restricted to administrative work of court. Any activity that involves decision-making even on the part of the court staff should not involve engaging AI in the process (decision to number a petition or application). This will ensure that all judicial or activities that require decision-making will stay shielded away from AI reducing the possibility of miscarriage of justice. The permissible uses12 in the draft guidelines does list the use cases for implementing AI.
Proper training for court staff on AI: The guideline does mandate training to the court staff that use AI systems. Training should be a repetitive exercise and just not be one time activity to ensure that the staff is reminded time and again of the risks in using AI. A technical support person should also remain in site always to guide the initiates before they are comfortable with the use of new technology. When the staff is getting trained, AI system should be treated on par with a new software or an application and assumed that it will take a while for them to get familiarised and comfortable with the system. While the court staff is getting trained, similar initiative should be launched by Bar Councils for the lawyers in the form of workshops or online training sessions to educate them on the ways of usage and risks of over reliance on AI systems. Such a two-pronged training approach will address upstream and downstream actors will enable to stay alert and identify hallucinations by humans. To keep the staff alert, gamifying13 the identification of mistakes within court system will be an innovative way to approach this issue.
Human-in-loop: The draft regulation imposes a human-in-loop requirement. This enables cross-verification of every significant action that AI system performs. This is a significant step in the process of deploying AI in courts. The guideline also suggests the Bar Council of India to come up with guidelines for the “Responsible Use of AI for Lawyers”. This mechanism will be effective only if the training is good enough to equip the staff with required skills.
Imposition of heavy costs on lawyers — Repeated submissions of hallucinated judgments by lawyers or multiple occasions of non-disclosure of use of AI in their documented submitted before courts should attract imposition of heavy costs. Initiating disciplinary action will not be an effective way to address this issue because there may be cases of inadvertent mistakes. As the usage of AI is rampant, disciplinary action would also increase the burden on Bar Councils and the court. Establishing a legal basis for the act of initiating disciplinary proceeding will also require some kind of amendment in Advocate Act, 1961 or Bar Council of India Rules, 1975. Easier way to deal would be to leverage the existing laws and impose heavy fines. However, heavy costs are a good deterrent as lawyers regularly slapped with costs will lose their clients trust.
Internal disciplinary action against Judges — Similar to the possibility of lawyers presenting hallucinated judgments/citations in their documentation, the Judges who rely on AI as for legal research support can also end up with such judgments. Such use cases of AI will fall outside the scope of prohibited usage of AI but will still have similar inherent risks. Repeated instances of relying on such judgments reflects on the Judge’s inability to act independently due to over reliance on such technology. Unless some internal disciplinary mechanism to address such instances are in place, the Judges cannot be made accountable for mistakes committed from their side.
Conclusion
Even with the state-of-the-art technology, it is not clear if hallucinations can ever be avoided completely and it is also apparent that AI is definitely going to play a fairly significant part in legal field. Till technology evolves to address the efficiency issues, it is necessary for the judicial fraternity as a whole to adopt alternative measures to keep a check on the hallucinations. Failing to do so may have an opposite effect and increase the burden on the judicial system.
*Privacy Legal Counsel, Philips, Amsterdam, The Netherlands.
**Partner, M/s Nomos Solutions, Solicitor (England & Wales) and Advocate, practising at Madras High Court.
2. Rule 20(1)(e) — Prohibited uses of AI.
3. Rule 3(1)(k) — “anonymisation” means the process of irreversibly altering or removing personal information from a dataset such that the data subject cannot be identified, directly or indirectly, including by reference to any name, identification number, location data, or any other factor specific to the identity of that person.
4. Anonymisation/Pseudonymisation, European Data Protection Board, available at <https://www.edpb.europa.eu/topics/ai-and-technology/anonymisationpseudonymisation_en>.
5. Rule 3(1)(za) “harm”, in relation to AI Incidents, includes any kind of physical or financial damage, or damage to the reputation or rights of any individual, institution, or infrastructure.
6. Presumption in favour of responsible AI adoption.
7. William Fish, Perception, Hallucination, and Illusion (Oxford University Press, USA, 2009).
8. Meng Chen, Weihua Yu and Ke Liu, “A Meta-Analysis of Third-Person Perception Related to Distorted Information: Synthesizing the Effect, Antecedents, and Consequences” (2023) 60(5) Information Processing & Management 103425, ISSN 0306-4573, available at <https://doi.org/10.1016/j.ipm.2023.103425>.
9. Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala and Edwin Zhang, “Why Language Models Hallucinate” (4-9-2025) available at <https://arxiv.org/pdf/2509.04664>.
10. Parth Pulkundwar, Vivek Dhanawade, Rohit Yadav, Prof. Minal Sonkar, Prof. Medha Asurlekar and Prof. Sarita Rathod, “A Concise Review of Hallucinations in LLMs and their Mitigation”, available at <https://arxiv.org/pdf/2512.02527v1>; Aisha Alansari and Hamzah Luqman, “A Comprehensive Survey of Hallucination in Large Language Models: Causes, Detection, and Mitigation”, available at <https://arxiv.org/pdf/2510.06265v1>.
11. Parth Pulkundwar, Vivek Dhanawade, Rohit Yadav, Prof. Minal Sonkar, Prof. Medha Asurlekar and Prof. Sarita Rathod, “A Concise Review of Hallucinations in LLMs and their Mitigation”, available at <https://arxiv.org/pdf/2512.02527v1>
12. Rule 19.
13. Turning the identification of hallucinations and mistakes into a game with some minor reward attached to it.
