Ten years before the trade had a name
From soil mechanics to data warehouses, a reading log kept since 2011 covers a decade in which the discipline was named only after the fact. What the log shows, and what it leaves out.
Since 2011 I have kept a reading log: what I read, in which month, and what it was for at the time. It was never written to be read end to end, and that is why it is useful now, since it was not rearranged afterwards. Read in one sitting, it shows something I could not have said at the time. For the first half of the decade, the trade I would end up practising had no settled name.
The first entries are about calculation more than computing. A soil mechanics handbook sits next to a C++ treatise and one-line scripts for editing text. I came from mechanical engineering, where the computer was used for calculation without being a subject in itself. What I noted were mostly reference sheets, the dense pages you print and keep within reach. That engineering-school habit took ten years to disappear.
In 2012, three articles in the business press announced the same thing: data was becoming a raw material, and someone would have to handle it. The log notes explicitly that they changed my direction. In hindsight their accuracy was partial, and their effect mattered more: they gave a name to work some people were already doing. Other disciplines got their names the same way.
“Thus, I thought dynamic programming was a good name. It was something not even a Congressman could object to. So I used it as an umbrella for my activities.”
Bellman is talking about 1950 and operations research, and the description applies to data sixty years later. The name did not describe the work; it made it possible to fund it. Knowing this does not excuse anyone from practising the trade, but it avoids believing the word names something new.
- Harvard Business Review · Data Scientist: The Sexiest Job of the 21st Century (2012)
- Harvard Business Review · Big Data: The Management Revolution (2012)
From 2013 to 2015 the log is taken up with problems that have largely disappeared from discussion: producing office documents from markup, handling rich text, drawing charts without code run by the browser, taking over inherited code written in languages that already had no successors. An online calculator appears as the first notable experience of heavy computation run somewhere other than my own machine. Without my noticing at the time, that is where the cloud part of this story begins.
From 2016 the subject changes. Version control becomes a way of working with others, not only a tool. Then come functional languages and distributed computing, with memory to watch and jobs to submit. The difficulty moves: writing the computation becomes secondary, and most of the work is making it run on machines you cannot see.
Between 2017 and 2019 the log fills with statistics and machine learning: lecture courses, reference works, notebooks. That is when data science takes up the whole conversation. What I was noting alongside it (serving a model, running it in production, maintaining the pipeline that feeds those experiments) had no name yet and interested nobody. The data engineering trade formed at that point, out of the problems data science left aside.
One change runs through the whole decade without ever being announced. In 2011 my main source was the vendor’s documentation. By the end it is the public repository: you read the code before the manual and the open issues before the landing page, and you judge a tool by how active its discussions are rather than by its front page. GitHub hosted code, and it also changed where people learn. With one limit: what you read there is mostly what is visible, that is, what is popular.
It records only what I read, and reading does not confer competence. It says nothing about the months spent on a problem without noting anything, nor about the tools dropped because they did not work. Read in sequence, it therefore suggests a coherent trajectory, when it mostly records what was available when I was looking. I publish it as a dated document with no value as career advice, all the more since half of what it cites no longer exists.