Structure-aware chunking for legal RAG

Before a language model answers, RAG systems find suitable text fragments in the corpus, and in legal documents this retrieval turns out to be the bottleneck: ordinary splitting into fixed-length fragments cuts sections in the middle, separates a clause from the section it belongs to, and loses references like “subject to section 167”. The authors test how much retrieval quality depends on the splitting method. There are five datasets: the Indian Code of Criminal Procedure (CrPC, 1973), the Code of Civil Procedure (CPC, 1908) and the GDPR, with 501 questions each; 776 questions of the LegalBench-RAG benchmark on contracts; 3,730 questions of NitiBench-CCL on 36 Thai financial laws. The questions for the Indian codes and the GDPR were written by the authors themselves with an LLM and checked manually; the last two datasets were labeled independently of them.

StrucChunk parses a document as a hierarchy: part, chapter, section, clause. A section remains a single fragment if it is no longer than one and a half times the limit (the limit is 512 tokens); otherwise it is split at sentence boundaries. Each fragment is prefixed with a path like [CrPC 1973 > Chapter XII > Section 167], and if a section refers to others, a summary of up to three of them, 200 characters each, is appended. Retrieval combines BM25 keyword ranking and dense BGE-M3 embeddings by Reciprocal Rank Fusion. The baseline is recursive splitting at 512 tokens with the same retrieval models; in the Thai dataset, the ready-made NitiBench fragments of 553 characters and by section. A separate 2×2 experiment separates the contribution of splitting from that of the retrieval method.

On the Indian CrPC, Recall@5 rises from 0.557 to 0.804, and the share of questions where the right section ranks first from 0.214 to 0.627; on the CPC from 0.158 to 0.259, with the absolute level remaining low. Splitting accounts for 88% and 73% of the total gain, the retrieval method for the rest. On LegalBench-RAG, character-level recall at eight results is 48.18% versus 23.59% for the baseline with the same model and 26.30% in the benchmark’s original paper; precision of the first result is lower (4.51% versus 6.64%) because a section fragment is longer than the labeled answer. On the GDPR the best result comes from StrucChunk without BM25 (Recall@5 0.908): keyword retrieval pulls in recitals that refer to the relevant article. The Thai dataset shows the source of the effect: moving from 553-character fragments to splitting by section raises HR@5 from 0.594 to 0.861, with StrucChunk at 0.856, that is, the structural path and the summary of references add 2.3% only on questions with references.

The authors measure only retrieval; answer quality was not tested, and according to NitiBench data splitting by section barely changes the final answer F1 (0.739 versus 0.722). The questions for three datasets were written and labeled by one person, and in the section on label verification the agreement rate with a second lawyer is not filled in; the text reads “X/100”. References are recognized by regular expressions and implicit ones are missed; in the Indian CPC some sections are lost due to uneven PDF layout. For systems that search for provisions in codes the conclusion is practical: splitting by section while preserving its place in the structure of the code gives more than changing the retrieval model. On court decisions and other texts without numbered structure the authors expect a smaller effect but did not test it.