MAELSTROM will develop Europe’s supercomputing architecture of the future. It will optimise high-performance computing (HPC) system designs for machine learning (ML) applications for weather and climate (W&C) predictions and thereby enable the W&C community to make efficient use of new ML capabilities on exascale supercomputers. The co-design cycle will bring together system designers, ML experts, and W&C domain scientists to create compute system designs, a software framework, and large-scale ML applications customised for W&C science. The developed ML tools and software environment will be usable across the workflow of W&C sciences and improve weather and climate predictions in Europe.
At its core, the MAELSTROM software environment will use Mantik as the mediator between the users of ML tools and HPC architectures. The software will abstract the usage of prevalent ML tools, data loading, and deployment on HPC facilities and, consequently, enhance the workflow of W&C scientists.
Earth system modeling is virtually impossible without dedicated data analysis. Typically, data are big and, due to the complexity of the system, adequate tools for the analysis lie in the domain of ML or atrificial intelligence (AI). However, earth system specialists have other expertise than developing and deploying state-of-the art programming code which is needed to efficiently use modern software frameworks and computing resources. In addition, cloud and HPC infrastructures are frequently needed to run analyses with data beyond Tera- or even Petascale volume and corresponding requirements on available RAM, GPU and CPU sizes.
The KI:STE project will extend the concepts of the Mantik platform such that handling of data and algorithms is facilitated for earth system analyses while abstracting technical challenges such as scheduling and monitoring of training jobs and platform-specific configurations away from the user. The principles for design are collaboration and reproducibility of algorithms from the first data load to the deployment of a model to a cluster infrastructure. In addition to the executive part where code is developed and deployed, the KI:STE project develops a learning platform where dedicated topics in relation to earth system science are systematically and pedagogically presented.
Mantik is part of the Nvidia Inception startup accelerator program.
For the KI:STE project Ambrosys builds the KI:STE AI platform to ease AI usage for scientists and ensure the reproducibility of AI experiments. Based the Mantik platform, the team implements a cloud-based plug-and-play framework for AI. All common frameworks (e.g. tensorflow, sklearn etc.) are abstracted to be used out-of-the-box without requiring framework-specific user knowledge. The KI:STE AI-platform aims at a native support of execution of workloads in a local, cloud, or HPC environment.
4cast is dedicated to generate power production forecasts for wind and solar power plants at the highest level of precision. Especially wind power is notoriously hard to predict. Regional small-scale effects like vegetation or neighboring plants affect the power production on a large scale. This makes forecasts based solely on weather data unreliable as their geographical resolution is too coarse.
4cast uses historical production data of wind and solar plants and complements them with weather data from a multitude of sources to feed state-of-the-art ML algorithms (symbolic regression, deep learning) that infer the regional peculiarities of any site. As a result, the algorithms develop optimal models which suit the specific circumstances of any wind or solar plant. Combining the models with weather forecasts yields power forecasts of unprecedented precision. Here, the company is able to provide high-precision intra-day as well as short- and long-term predictions.
For MAELSTROM, 4cast will develop the software environment of the project and make use of Mantik to fulfill the particular requirements of the W&C community.