PyMuPDF and pgvector: what the chunking decides

The demonstration is built in an afternoon and the disappointment arrives the week after: the system answers beside the question, and the model is suspected. In the cases I have seen, the fault was almost always upstream. A model cannot answer from a passage it was not given, and it cannot make sense of a passage cut through the middle of a table.

A PDF is not a structured document: it is a description of what to paint and where. Extraction returns characters and positions, and everything that was structure (a table, a footnote, a repeated header) becomes running text. The first job, before any chunking, is to look at what the extraction produced on the corpus’s ugliest documents, not on its cleanest.

import pymupdf
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter

def read(path: str) -> list[Document]:
    document = pymupdf.open(path)
    pages = []
    for number, page in enumerate(document, start=1):
        # "blocks" rather than "text": this returns a reconstructed reading order
        # instead of a stream where columns interleave.
        blocks = page.get_text("blocks")
        content = "

".join(b[4].strip() for b in sorted(blocks, key=lambda b: (b[1], b[0])))
        # The page number is not decorative: it is what will later let the source
        # be shown, and an answer without a source cannot be checked.
        pages.append(Document(page_content=content, metadata={"source": path, "page": number}))
    return pages

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=150,
    # Order matters: cut at paragraphs first, and only as a last resort in the
    # middle of a word.
    separators=["

", "
", ". ", " ", ""],
)

Two consecutive chunks that do not overlap cut a sentence in two, and then neither carries the answer. Ten to fifteen per cent of overlap settles that case at the price of a slightly larger index. The real trade-off is elsewhere: a long chunk gives the model context and a blurred vector, a short one gives a sharp vector and not enough context. There is no correct value, there is a value to be measured on thirty questions whose answers you know.

The reflex is to add a vector database to the architecture. For a few tens of thousands of chunks, an extension on the relational database already in service is enough, and it brings what a separate service does not: vectors and metadata in the same transaction, the same backup, the same permissions. It is one service fewer to administer, and it is the kind of saving that shows at handover rather than at delivery.

from langchain_postgres import PGVector

store = PGVector(
    embeddings=embedding_model,
    collection_name="documents",
    connection="postgresql+psycopg://…",
    use_jsonb=True,
)
store.add_documents(splitter.split_documents(pages))

# The metadata filter in the same query as the distance: this is what avoids
# pulling back a thousand chunks in order to keep five.
results = store.similarity_search(
    "what are the termination conditions",
    k=5,
    filter={"source": {"$in": permitted_documents}},
)

I say nothing about reranking, nor about hybrid search mixing keywords and vectors. Both improve precision, and both are justified only once you measure: adding them before you have a reference set of questions is stacking settings without knowing which one helped. And I say nothing about per-document access rights, which are the real difficulty of such a system in a company: the filter above assumes an already-computed list, and computing that list is a project in itself.

• PyMuPDF · extraction de texte
• pgvector
• langchain-postgres