← writing
15 October 2024

PyMuPDF and pgvector: what the chunking decides

In a retrieval-augmented pipeline, attention goes to the model. Yet quality depends on two earlier steps: how a PDF is split, and what the extraction has already lost.

The demonstration is built in an afternoon, and the disappointment comes the following week: the system misses the question, and the model is suspected. In the cases I have seen, the error was almost always upstream. A model cannot answer from a passage it was not given, nor make sense of a passage cut through the middle of a table.

01What the extraction loses

A PDF does not describe a structure; it describes what to draw and where. Extraction returns characters and positions, and everything that was structure (a table, a footnote, a repeated header) becomes running text. Before any splitting, you therefore need to examine the extraction output on the most difficult documents in the corpus, not on the cleanest one.

python
import pymupdf
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter

def read(path: str) -> list[Document]:
    document = pymupdf.open(path)
    pages = []
    for number, page in enumerate(document, start=1):
        # "blocks" rather than "text": this returns a reconstructed reading order
        # instead of a stream where columns interleave.
        blocks = page.get_text("blocks")
        content = "

".join(b[4].strip() for b in sorted(blocks, key=lambda b: (b[1], b[0])))
        # The page number is not decorative: it is what will later let the source
        # be shown, and an answer without a source cannot be checked.
        pages.append(Document(page_content=content, metadata={"source": path, "page": number}))
    return pages

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=150,
    # Order matters: cut at paragraphs first, and only as a last resort in the
    # middle of a word.
    separators=["

", "
", ". ", " ", ""],
)
02Overlap and chunk size

Two consecutive chunks that do not overlap cut a sentence in two, and then neither contains the answer. An overlap of ten to fifteen per cent fixes this at the cost of a slightly larger index. The main trade-off is size. A long chunk gives the model context but an imprecise vector; a short one gives a precise vector but too little context. No value is correct in absolute terms, and it has to be measured on about thirty questions whose answers are known.

03A database you already have

The reflex is to add a vector database to the architecture. For a few tens of thousands of chunks, an extension of the relational database already in use is enough, and it brings what a separate service lacks: vectors and metadata in the same transaction, the same backup and the same permissions. There is one service fewer to administer, and that saving is felt mostly at handover.

python
from langchain_postgres import PGVector

store = PGVector(
    embeddings=embedding_model,
    collection_name="documents",
    connection="postgresql+psycopg://…",
    use_jsonb=True,
)
store.add_documents(splitter.split_documents(pages))

# The metadata filter in the same query as the distance: this is what avoids
# pulling back a thousand chunks in order to keep five.
results = store.similarity_search(
    "what are the termination conditions",
    k=5,
    filter={"source": {"$in": permitted_documents}},
)
04Reranking, hybrid search and access rights

I do not cover reranking or hybrid search combining keywords and vectors. Both improve precision, but they are only justified once measurement is in place: adding them before you have a reference set of questions means piling up settings without knowing which one helped. Nor do I cover per-document access rights, which are the main difficulty of such a system in a company. The filter above assumes a list that has already been computed, and computing that list is a project in its own right.