← writing
22 November 2020

Public figures do not make proof

French open data is plentiful and free, which is what makes it misleading. What data journalism teaches an engineer, and the step almost nobody takes.

Data journalism and data engineering handle the same files, but their results are judged by different people. In a company, a table is approved by whoever asked for it. In a newspaper, it is challenged by the reader who disagrees. That second check is more demanding, which is why an engineer has something to learn from it.

01Where the data is

In France most of it is open and documented: the interministerial portal, the land registry, the address base, the geoportal, and datasets published directly by operators such as the railways or the electricity distribution network. The difficulty lies in joining files more than in accessing them. Two public files almost never share a granularity or a cut-off date, and that is the step where errors appear that are no longer detected afterwards.

02What you scrape for want of better

When the figure is not published, you have to read it where it is displayed. This is a last resort: extracting from a page depends on its current layout and stops working at the first change. It is justified for a one-off investigation, never for a lasting system. Confusing the two leads to delivering a pipeline that will break without warning.

03A warehouse that fits in a file

The tooling I kept from that period is the kind that stays small: a dataset published as a database you can query without a server, or a notebook run in the browser. In both cases the article and the data are published together, and the reader can redo the calculation. A corporate data pipeline almost never allows this, and it would benefit from doing so.

04The figure, and its colour

In a research paper, a figure illustrates a result established elsewhere. In a news article, the figure is the result. The choice of scale, axis origin and palette then becomes an editorial choice, and a continuous palette applied to categories misleads the reader without their noticing. The tools that produce these figures are good, but the responsibility for those choices stays with the author.

05The 2020 links

My 2020 notes were half a directory of links, and I am not republishing them as they were, since a directory is what ages fastest. The order of operations still holds: find, join, verify, publish. The step I most often see skipped in companies is the third, and it is the one that allows a table to be presented as evidence.