Have you ever wondered what’s behind the weather forecast TV reports? Or who is managing government satellites? Perhaps you’ve wandered across a sensor — where does that data go?
In the Netherlands, that data is collected and managed by the KNMI, or the Koninklijk Nederlands Meteorologisch Instituut — a.k.a., the government’s meteorological branch. The KNMI forecasts weather, researches climate change, and monitors seismic activity.
Their reach goes far beyond the Netherlands. In 2017, the KNMI-built TROPOMI was launched into the atmosphere aboard the Sentinel-5 Precursor satellite. Recently, the TROPOMI measured and mapped the decline in nitrogen dioxide concentration over China from December to March due to COVID-19.
But how is this data collected? How does it get from the TROPOMI and other field sensors to the KNMI, and what do they do with it once it’s received?
TNW recently spoke with Saskia Wagenaar and Roy Engelen, two developers who work with data collection and distribution for the KNMI, to find out.
From sensors to the weather room
The KNMI is made up of several departments. “There are departments where research is conducted, and departments who actually go out in the field and replace the sensors and test them. And then you have the meteorologists which actually are there to give out the warnings to the Netherlands,” Roy Engelen, a cloud engineer, tells us.
If we were to paint a simple picture of how data funnels from point A to B within the KNMI, the field sensors would be the starting point. Engelen works as a cloud engineer in the IT migration team, which is responsible for the whole information chain, starting from the sensors all the way to the weather room: “My department is the helper that keeps the whole chain working on the operational and information side, so more IT-related.”
The sensors do what humans have done for hundreds of years — they observe the weather. Except now, they automatically collect data such as temperature, rainfall, and air pressure.
“So you have 48 sensor fields — 34 on land and 14 in the sea — in the Netherlands,” Engelen tells TNW. “There’s also people locally who have a small station in their garden or farmers who have sensors in their fields… all that data is then collected at the KNMI.”
Most of these people are individual volunteers — around 300 in the country, Engelen estimates. But there is a “decrease in the number of volunteers that are willing to help,” because it’s a daily effort. This is an issue for the KNMI, Engelen tells us, because local volunteers are incredibly important: “Those 300 extra data points are really essential to get a clear picture, because the forecast model needs to be high resolution.” More data means a more accurate weather prediction.
Once the data is collected by field sensors, it is run through the KNMI’s models. “All of that data output generates maps, videos, etc.”
Raw data from radars and satellites are also collected, Engelen says, “and those are used as input for the forecast models, and real-time maps are generated. These maps then go to the meteorologists in the weather room (and famous Buienradar) who give warnings and information about the weather. The raw data is archived so it can be used by researchers who work on climate change and making better weather prediction models.”
This data is collected continuously — or, at least, should be. Occasionally, a sensor will go down in the field.
“For example, a laser which sends a beam into the sky to measure how cloudy it is… and birds made a nest on it, for example,” Engelen tells TNW. “Also with rain measurements, a long time ago the fields (with sensors) were not closed, so anyone could cross them. And drunk people would urinate in them, for example, and suddenly there was a lot of rainfall measured there.”
But, Engelen assures, the sensors are continuously being improved and worked upon: “Things like that happened all in the past.”
What happens with the data?
Saskia Wagenaar is a software engineer working on the KNMI data platform (KPD), or the platform which gives access to all the KNMI’s open data, and licensed data via APIs. Wagenaar’s team uses the cloud infrastructure that Engelen’s sets up.
“My responsibility is really collecting the data and making it available,” Wagenaar tells TNW. “What’s nice about my role as a distributor of data is that not only scientists at the KNMI can use it for research, but everybody around the world has access as well. Because we’re a governmental institution, we have to make our data available as open data, free of charge.”
The people who most often access this open data are academics, government researchers, but also commercial companies, such as Schiphol Airport and KLM Airlines, which use the data to plan flights around the weather. “It’s really a mix of both academic use and commercial use,” Wagenaar tells TNW.
Managing such a vast trove of open-access data is not without its challenges. Besides for the cost of human labor and data storage, there is also the challenge of making sure it’s provided in a useful way: “Of course, the sheer amount of data is a very big challenge. How do you make it available to users so they can use it correctly? That’s what my team is working on. We gather data from many different domains, like radar measurements, wind measurements, or satellite data… you have to handle all these kinds of data differently.”
“So our idea is to start moving towards tailored APIs which give access to the data of those different domains. We are really listening to our users, asking them how they actually want to use the data. Based on their input we can make sure that the APIs will be useful… That’s really interesting.”
The KNMI’s research has importance beyond the Netherlands. Wagenaar has worked on projects with the goal of combining data across European countries, which all collect and make it available in different ways. This, Wagenaar says, makes it “really difficult to get a quick situational awareness when big events happen.”
Natural disasters and climate change
A “big event” includes natural disasters, such as when the Icelandic volcano, Eyjafjallajökull, erupted in 2010. Wagenaar tells TNW that a central European data trove would increase “warning capability and situational awareness,” which could be crucial: “So it’s really in the European collaborations, making sure that both ground-based measurement but also measurements from satellites are made together in one central point so that everybody in Europe can access the data and use it to get some situational awareness if an event like that will happen again.”
While most of the data collected in the field is used for weather forecasts, the archive of data is also essential for studying climate change, Wagenaar says: “After archiving, (the data) becomes important for climate change research. To see climate change trends, it’s important the sensors always stay in a similar place, and that the trends are not broken.”
One recent event that caused a change in trends, as mentioned before, is COVID-19: “I think that the most interesting thing is we actually see, for certain places around the world, that the air quality is improving… we can see (via the satellite) there are fewer particles in the air where there’s a lockdown happening.”
It’s always sunny
Government dev work doesn’t seem like the most independent or creative — especially when it concerns the weather, something that is outside of human control. But Engelen says his job as a cloud engineer in the IT migration team comes with a lot of freedom. In fact, it’s how he came to work for the KNMI in the first place.
“You have a lot of freedom and input — you’re free to give your input to the whole IT process,” Engelen says.
Wagenaar was attracted to working with the KNMI data platform because of the opportunity to work closely with researchers, and new technology: “I’ve been here for three years now, and I think it’s a very dynamic environment. When I started to work at the KNMI everything was done on the premises with on-premise services. In the time I’ve been here we’re now moving towards the cloud. And we’ll be using newer technology, which is a very interesting journey.”
“It allows us to learn and develop ourselves a lot, as well as the way we work,” Wagenaar tells us.
Source: TheNextWeb