1 The Data-Generating Process
In data science, an overarching goal is to develop an understanding of some phenomenon or process in the real world. This phenomenon could involve understanding human behavior, predicting natural processes like storms or floods, or characterizing the performance of complex machines like automobiles or airplanes. In all of these cases, we aim to collect data and use that data to tell us something about what is happening with these real world phenomena.
We think of the data going through a process before it arrives on our desks or computers. That process, depicted below, involves the data-generating process and the data collection process.
\[ \text{Data-generating process} \longrightarrow \text{Data collection process} \longrightarrow \text{Data} \]
Before we actually analyze or look at the data we receive, it is valuable to think about the process by which the data have come to use. Our understanding of both the data-generating process and the data collection process can help us to interpret the results we produce from the data and evaluate any uncertainty about the evidence that we generate.
1.1 The Data-Generating Process
The data that we collect and then analyze is generated by what we call the data-generating process. This generic term is meant to represent the underlying phenomenon that we are trying to study. The data-generating process ultimately is the thing we want to learn more about. Very often we need to make some assumptions about the data-generating process in order to draw inferences from the data. Sometimes we use explicit models to represent a part of the data-generating process and then use the data to estimate parameters in that model. The hope is that we can learn something about the data-generating process and use that knowledge to guide future actions or investigations.
The exact nature of the data-generating process can vary widely depending on the application we are considering and the question we are trying to answer. If we are studying air pollution concentrations in a city, we might imagine that there are sources of pollution scattered across the city each spewing out pollution into the air. In this case we might want to know what exactly are the levels of pollution in different neighborhoods around the city. If we are studying cancer in humans, the data-generating process might involve the complex sequence of biological processes that lead to the development of cancer in people. There, we might want to study what it is about these biological processes that lead some people to develop cancer and some people to not develop cancer. Regardless of the exact situation, it is important to conceptualize the data-generating process and consider what features of that process that we want to learn more about.
The biggest challenge with the data-generating process is that we typically cannot observe the process directly. Rather, we have to design a separate process to allow us to take measurements that tell us something about the data-generating process. This separate process is the data collection process.
1.2 The Data Collection Process
One important question to ask when analyzing any kind of data is “How did the data come to me?”. Sitting between the data-generating process and the actual data that we see on our computers is the data collection process. If we think of the data-generating process as something that exists in the natural world, the data collection process describes how humans interface with nature and take measurements. These measurements ultimately become the data that we analyze, as illustrated in the examples below.
While the specifics of the data-generating process are generally unknown to us (because we do not directly observe it), the data collection process is something that we may control or at least have some information about. Understanding how the data were collected can be important for interpreting data analysis results, especially if those results are not what we expected to see.
It is important to see the data collection process as a system that interfaces with the real world or nature and produces the measurements and data that data scientists will analyze. This process can be a complex multi-stage process where the data are possibly transformed at each stage. The better the understanding of that system that we have, the stronger our conclusions will be. In the event that results are questioned or challenged, having a deep understanding of the data collection process will be helpful for defending those results or conclusions.
1.2.1 Example: Coin Flipping
Suppose a friend plans on flipping a coin 10 times. After each flip he show you the side of the coin that faces up and you record the value (i.e. “heads” or “tails”). After 10 flips you have a collection of 10 data points, each indicating heads or tails. In this simple example the data-generating process was the process of the person flipping the coin. The data collection process was the act of reading the coin after the flip to determine if it was a heads or tails. Ultimately, the data that we would analyze are the ten heads and tails values that we recorded after reading the coin.
1.3 Making Assumptions
In order to make sense of the data we observe, we often need to make assumptions about the data-generating and data collection processes. These assumptions can originate from a number of places, including the analyst’s real-world experience or their understanding of the previous research on this topic. All of an analyst’s past experience with a given problem will likely play into the development of assumptions about the data-generating process. With these assumptions, we can make predictions about what the data will look like once we observe them.
A key task for an analyst is to carefully consider the evidentiary basis for their assumptions, if it exists. Sometimes assumptions are developed from personal opinions or experience and do not have any clear documentation. Other times, there may be published studies or materials that one can point to that serve as a basis for making certain assumptions about the data-generating process. Regardless of the specific basis for an assumption, having a detailed record of how the assumption was formed can provide a meaningful way to assess the validity of that assumption, either before seeing the data or after seeing the results.
In the coin flipping example above, what assumptions did we make about the data-generating process and the data collection process? For example, we might have assumed that the coin flip would be random and that the coin was equally likely to be heads or tails. We might also have assumed that the coin was a typical coin with both a heads and tails side. We will talk about assumptions and their implications in much greater detail in Chapter 12. For now, it is useful to consider what assumptions we make, either implicitly or explicitly, about the processes involved in bringing the data to us.
1.4 More Examples
In this section we provide a few more examples of data-generating processes and data collection process. In particular, we try to highlight how a given data-generating process might allow for different data collection processes. Depending on what question we want to answer, we might design different ways to collect the data we want to analyze.
1.4.1 Example: Coin Flipping (phone version)
Now suppose a friend plans on flipping a coin 10 times and after each flip he shows a second friend the side of the coin that faces up. This second friend then calls you on your phone and tells you the value (i.e. “heads” or “tails”), which you record. As before, after 10 flips you have a collection of 10 data points, each indicating heads or tails.
On its face, this example seems very similar to the original coin flipping example. In fact, the data-generating process is the same (the first friend flipping the coin). However, we have modified the data collection process by adding the second friend who reads the coin and then phones you with the result. While this data collection process might seem equivalent to the first example above, it is not. Whether or not the two data collection processes produce the same dataset depends critically on the reliability of the second friend. What if this friend decided to take a nap before calling you with the result? If the second friend forgets the value before calling you on the phone, you might end up with unreliable data. Because you haven’t seen the actual coin value, there is no way for you to know whether the second friend is accurate or not.
Just for fun, we can take this example to an extreme and insert 20 friends in between the one flipping the coin and you. Here we have the classic game of “telephone” where each person has to call the next and tell them the result of the coin flip. How confident would you be that every single friend heard correctly what the coin flip result was and accurately reported it to the next person (and ultimately to you)?
The second data collection process is a more complex system than the first one because it has more stages and therefore more places where we as data analysts have more uncertainty about what exactly is going on. When analyzing data, it can be important to know as much as possible how the data were collected and transmitted to you as the analyst. Sometimes processes that appear equivalent are not actually the same.
1.4.2 Example: A New Medical Treatment
The flipping of coins might seem a bit silly or abstract, but the mechanism underlying the data generated by “coin flips” reflects processes in the real world that we often care about. In his book Statistical Evidence: A Likelihood Paradigm, statistician Richard Royall describes a clinical study involving the treatment of critically ill infants (Royall 1997). The study was designed to evaluate the use of a new treatment called extra-corporeal membrane oxygenation (ECMO) as a treatment for infants with respiratory problems. When new treatments are developed, typically an initial small study is conducted to ensure the safety of the approach. In this case, a study was designed to use the ECMO approach on a few infants to ensure that it was no worse than the existing treatments.
Although the setting could not be more different, the data-generating mechanism in the study was identical to the coin flip. In this case, the outcome was the survival of the infant: A “heads” meant the infant survived and a “tails” meant that it did not. Needless to say, the stakes for such a study were extremely high. At the time, it was not known what the success rate for ECMO would be as a treatment for this problem. However, it was known that the existing treatments only had a success rate of about 20%. Therefore, there was substantial interest in developing a treatment that had a higher success rate.
The initial study recruited 11 infants. If the ECMO treatment were no better or worse than the existing treatment, we would expect only about 2–3 infants to survive (about 20%). Some of the researchers were optimistic that the method would be more successful, with perhaps an 80% success rate. Those researchers therefore expected 8–9 infants to survive. When the study was conducted, all 11 infants survived, providing strong evidence that the ECMO treatment was much better than the existing approach.
In this example the data-generating process is the complex mechanism by which the proposed ECMO treatment interacts with the respiratory disease process and the human body. The data collection process involved the researcher observing whether the infant survived or not. There was not necessarily anything complicated about the data collection process that might have us question the data in this particular study. However, the data-generating process likely involved many unknown factors about which the researchers were hoping to learn.
1.4.3 Example: Outdoor Air Pollution
Outdoor air pollution is a problem throughout the world and decades of research has shown that exposure to outdoor air pollution is harmful to human health. There are a variety of different sources of air pollution depending on nearby population centers, climate, and the level of industrial development. As a result, there is often significant interest in measuring air pollution levels, particularly if there are national standards that require pollution to be below certain levels.
Suppose we want to study air pollution in a large city. In that case, the thing we want to study is the concentration of air pollution all across the city. We can imagine a complex “surface” blanketing the city with peaks where air pollution is high and valleys where air pollution is low. We cannot observe this surface directly and therefore have to design some sort of data collection strategy.
Consider two different ways we could measure the levels of air pollution outdoors:
One strategy is to place a single monitor in the middle of a city and collect pollution measurements each day for a year, resulting in 365 measurement values. The monitor is highly accurate and is considered the gold standard for pollution measurement.
A second strategy is to use 50 small low-cost sensors placed all around the city, each of which will collect daily measurements. Because the low-cost sensors are not very reliable, they can only collect good data for about 2 months, after which they can no longer be trusted.
In this scenario the data-generating process involves the set of pollution sources that are present in a given city and the pattern with which they generate the “surface” of air pollution on a daily basis. The two data collection processes described above clearly result in different data. The first process has a single monitor that produces 365 reliable measurements from a single location in the city. The second process results in about 60 days of measurements (2 months) from 50 different locations across the city, for a total of about 3,000 measurements.
Which strategy is better? It’s difficult to say without knowing exactly what question is going to be asked. The first strategy collects more data across the year, which may be important if you are going to study seasonal variation in pollution levels. The second strategy collects data over a much shorter time period, but gives more information about different locations. This might be important if you are interested in studying the variation of pollution across different neighborhoods in the city.
1.4.4 Example: Voter Polling
During election seasons it is common to poll eligible voters to make predictions about which candidate will win the election. Fundamentally, the information that pollsters are seeking is who each person plans to vote for. With that data, they can calculate the proportion of the electorate who will vote for a candidate and whether that proportion is greater than 50%. The data-generating process is the process that leads an individual to decide to vote for a given candidate. How we obtain that information can come in a variety of ways.
One approach is a telephone survey, where eligible voters are called and asked if the election were held today, who would they vote for. This approach requires having a list of phone numbers of people who live in the area holding the election (city, state, or country) and it needs people to answer the phone when the pollster calls (and not hang up once they find out who is calling). One advantage of this approach is that it is economical and therefore allows for reaching a large number of people, potentially reducing the uncertainty regarding the final result.
A second approach would be to go door to door and ask people in person who they plan to vote for. This might be feasible for a city-wide election or a smaller district where a group of people could realistically reach a representative sample of people who live there. This approach requires people to be home at the time the pollster arrives and to answer the door. It is more time-consuming to go door to door in a community so the approach may not be able to contact that many people. However, people might be inclined to talk with someone who is standing right in front of them as opposed to someone who calls on the phone.
As a data scientist, you have to consider how the data were collected and whether that might have any impact on how you approach the analysis. In this example, we can ask whether one approach might be more reliable or accurate than the other. If the less reliable approach is the one that was used, we have to consider what problem this approach might introduce and whether there are things we can do in the analysis to mitigate those problems.
1.4.5 Example: Objective vs. Self-Report Measures in a Survey
There are numerous surveys in the United States and around the world that try to determine how people spend their money on a regular basis. For example, the U.S. Bureau of Economic Analysis conducts a monthly Personal Consumption Expenditures survey, which asks people what they spent their money on and how much they spent. This survey is used, in part, to calculate a measure of inflation in the U.S. economy. If people are spending more money for the same thing, then we have inflation. If people are spending less money, then we have deflation.
The ultimate goal of this survey is to determine how much money people are spending. You could say that the data-generating process is the process by which people decide where and when to spend their money. But how do we find out what those numbers are? Let us consider two different data collection approaches.
One data collection approach is to simply ask them how much money they spend and on what. This is known taking a self-report measure. We are relying on the survey participant to remember what they bought possibly one month ago (and to be honest about it). This approach is very simple and can be done over the phone or an online survey, but people’s memories can be shaky sometimes. You might ask yourself whether you remember the exact dollar amount you spent on a cup of coffee one month ago.
Another data collection approach could involve asking people to submit their credit card statements for the past month so that we could look at the transactions reported there. This gives us what is sometimes referred to as an objective measure because it doesn’t rely on the participant remembering the dollar amounts and giving us the numbers themselves. Rather, we can rely on the credit card company to accurately account for every dollar that was spent using the card. This approach relieves the participant from having to remember every dollar, but we do lose some information here too. For example, we won’t know about any cash purchases, or we may have to exclude people entirely if they don’t use or have a credit card. In addition, credit card statements often do not have detailed descriptions of what was purchased. Finally, people may be reluctant to give away their credit card statements to a stranger conducting a survey, which could result in a poor survey response.
This example provides two very different approaches to collecting data on what people spend their money on. Self-report measures are simple but can be unreliable if people do not remember well what they purchased. Objective measures do not require that we rely on people to have strong memories, but may exclude other information based on the specific measure that we choose (i.e. credit card statements). When designing a data collection strategy, it is important to balance these considerations in the context of the overall goals of the survey. When analyzing data from a survey like this one, it is important to understand exactly what the data collection strategy was and to determine whether any weaknesses of the strategy affect the data analysis.
1.4.6 Example: Barton Springs Salamander Data
The City of Austin, Texas periodically collects data on the Barton Springs Salamander, which is a threatened species that is endemic to the area. The City has made various efforts to restore the salamander’s habitat to maintain the population in the City’s natural springs. The data on the salamanders is available on the Austin City data portal (https://data.austintexas.gov).
At a basic level, the City of Austin wants to study the population of Barton Springs salamanders and see if it is either increasing or decreasing. Therefore, we can think of the data-generating process as the complex biological and ecological systems that govern the size of the salamander population. In particular, they want to know if that process is affected (positively) by various restoration efforts undertaken by the City.
The data collection process is quite sophisticated and is described in detail in Bendik et al. (2026). Briefly, the study team visited Eliza Spring1, a small natural spring located in south Austin, approximately every 3 months, where 4 to 6 people searched the pool and an associated stream for salamanders. When a salamander was found, it was caught, photographed and measured, and the released back into the spring. The photographs allowed the team to track the salamanders over time using their skin patterns. In order to estimate the size of the population, they used a commonly used technique known as closed-population capture-recapture.
With respect to the primary question of gauging the size of the salamander population, the data collection procedure seems reasonable. The careful sampling of the area is likely to identify a good number (but not all) of the salamanders during each visit. The basic idea with the capture-recapture method is that if the team is constantly seeing the same set of salamanders at each visit, then the overall population must be relatively small. However, if they are constantly finding new salamanders that they haven’t seen before, then the population must be relatively big.
1.5 Summary
The data-generating process is the abstraction that we use to describe the process that produces the data we analyze. In many areas of study we can envision a data-generating process that produces the numbers, text, or other forms of data. Often, we are interested in learning more about the data-generating process and understanding how it works. In those cases, our goal is to use the data to build that understanding and to help us to decide what to do next.
The data collection process is the approach that we take to interrogate the data-generating process and to obtain the actual data. For any given data-generating process, there may be many ways in which we can collect data or take measurements. Determining which is the ideal data collection process will depend on the problem and the question being asked.
The data collection process is a system that is constructed to take measurements from the real world. This system can become very complicated depending on the complexity of the phenomenon we are trying to measure. It is important for the data scientist to have an understanding of this system in case subsequent data analyses raise questions that cannot be answered from the data alone. In those cases, the data scientist may need to inspect the data collection process in order to determine if any errors or problems have been introduced.
1.6 Exercises
In a study of bird movement patterns, a researcher goes out into the woods every day and marks down the location of a specific bird that had been tagged with a radio transmitter. Over the course of a month, the researcher produces a series of daily maps containing the locations of that species of bird in the forest. For this example, what is the data-generating process? What is the data collection process?
In the example given above, give an example of a different data collection process that could be used to study bird movement patterns.
Consider the voter polling example given in Section ??? and the two different data collection strategies described (telephone survey vs. door-to-door collection). Describe what you see as the strengths and weaknesses of each approach. Which strategy do you think will provide a more accurate estimate of the proportion of people willing to vote for a given candidate? Why do you make this conclusion?
Consider the Personal Consumption Expenditure survey example in Section Section 1.4.5 where we are trying to learn about how much money people spend on various things in a month. Suppose the researchers decide that they will ask survey participants to (voluntarily) submit their credit card statements so that the researchers can examine their expenditures. Suppose also that they discover that people who use credit cards tend to have higher incomes on average than those who do not use credit cards. What implications might this fact have for our interpretation of the credit card data as a measure of how much money people are spending?
Location of Eliza Spring: https://maps.app.goo.gl/qa3Eh4PAwqfPpdvT7↩︎