PyMuPDF and pgvector: what the chunking decides
In a retrieval-augmented pipeline, attention goes to the model. Yet quality depends on two earlier steps: how a PDF is split, and what the extraction has already lost.
The demonstration is built in an afternoon, and the disappointment comes the following week: the system misses the question, and the model is suspected. In the cases I have seen, the error was almost always upstream. A model cannot answer from a passage it was not given, nor make sense of a passage cut through the middle of a table.
A PDF does not describe a structure; it describes what to draw and where. Extraction returns characters and positions, and everything that was structure (a table, a footnote, a repeated header) becomes running text. Before any splitting, you therefore need to examine the extraction output on the most difficult documents in the corpus, not on the cleanest one.
import pymupdf
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
def read(path: str) -> list[Document]:
document = pymupdf.open(path)
pages = []
for number, page in enumerate(document, start=1):
# "blocks" rather than "text": this returns a reconstructed reading order
# instead of a stream where columns interleave.
blocks = page.get_text("blocks")
content = "
".join(b[4].strip() for b in sorted(blocks, key=lambda b: (b[1], b[0])))
# The page number is not decorative: it is what will later let the source
# be shown, and an answer without a source cannot be checked.
pages.append(Document(page_content=content, metadata={"source": path, "page": number}))
return pages
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=150,
# Order matters: cut at paragraphs first, and only as a last resort in the
# middle of a word.
separators=["
", "
", ". ", " ", ""],
)Two consecutive chunks that do not overlap cut a sentence in two, and then neither contains the answer. An overlap of ten to fifteen per cent fixes this at the cost of a slightly larger index. The main trade-off is size. A long chunk gives the model context but an imprecise vector; a short one gives a precise vector but too little context. No value is correct in absolute terms, and it has to be measured on about thirty questions whose answers are known.
The reflex is to add a vector database to the architecture. For a few tens of thousands of chunks, an extension of the relational database already in use is enough, and it brings what a separate service lacks: vectors and metadata in the same transaction, the same backup and the same permissions. There is one service fewer to administer, and that saving is felt mostly at handover.
from langchain_postgres import PGVector
store = PGVector(
embeddings=embedding_model,
collection_name="documents",
connection="postgresql+psycopg://…",
use_jsonb=True,
)
store.add_documents(splitter.split_documents(pages))
# The metadata filter in the same query as the distance: this is what avoids
# pulling back a thousand chunks in order to keep five.
results = store.similarity_search(
"what are the termination conditions",
k=5,
filter={"source": {"$in": permitted_documents}},
)I do not cover reranking or hybrid search combining keywords and vectors. Both improve precision, but they are only justified once measurement is in place: adding them before you have a reference set of questions means piling up settings without knowing which one helped. Nor do I cover per-document access rights, which are the main difficulty of such a system in a company. The filter above assumes a list that has already been computed, and computing that list is a project in its own right.