Alpha Close - Space Apps Challenge

Alpha Close| A Flood of Ideas

A Flood of Ideas

Your challenge is to develop a new methodology or algorithm that leverages Earth observation and critical infrastructure datasets to estimate damages to infrastructure caused by flooding. Make a measurable impact on the resilience of nations by helping the Earth observations community contribute to the United Nations’ primary effort to reduce disaster risk!

Flood Level Prediction with Temporal Graph Autoencoder

Summary

Using JAXA rainfall and environmental time series dataset, we designed a system to predict the flood level with a temporal graph autoencoder.

How We Addressed This Challenge

We planned to develop

Data feed pipeline from available data sources (done for JAXA)
Data normalisation pipeline (half-done)
Data visulisation pipeline (mvp done)
Prediction model (baseline done)

Importance

It will collate the available data sources
It will help understand the existing datasets with the visualisation
It will generate flood level predictions which can be used for flood prevention or damage control

What does it do?

Mostly covered in our planned development - floor level prediction

How does it work?

Data source collation - summarize data location, format, region in a shared document
Data extraction (for JAXA) - Python script for converting .nc files into .csv files with a common interface
Data visualisation - Reading .csv files and visualising with plotly
Prediction model - baseline: regression, planned: LSTM with graph neural network based autoencoder at each time step

What do we hope to achieve?

Overall reach better understanding and find common grounds in the available datasets
Through visualisation and prediction

How We Developed This Project

Inspiration

One of the most serious and urgent challenges faced by the world population
Really interesting challenge since we see this as an opportunity to consider the following aspects
1) Overall it's both temporal and spatial so our model must be coping with both
2) From a ML perspective, this is suggesting a spatial model such as a graph neural net as the input layer, and then a temporal model such as a LSTM at the backbone to connect the time steps
3) From a scale perspective, water system is a global problem, which means the flood happening in Thames River might be indirectly correlated to ice melting in the Arctic
4) From a data perspective, the features to be considered in such problem are countless - from rainfall to policy decisions

Approach

Agile (with Asana)

Tech Stack

Revision control: Git(Hub)
Language: Python (plotly, sklearn, Pytorch)
Platform: Apache Airflow, Jupyter Notebooks

Problems

Data availability - despite parallel-tasking enabled by our agile methodology, a big proportion of our time was spent on finding the usable datasets and decoding the formatted data
Time - due to the remote nature of this hackathon, it is harder to find concentrated time for working on the project

Achievements

Data collection - we managed to retrieve useful datasets from JAXA with given FTP credentials and decoded the .nc files into a .csv file
Prototyping - we managed to visualize a sample dataset and built a baseline model with the limited dataset

How We Used Space Agency Data in This Project

We mainly used JAXA rainfall and flood level dataset to build our visualisation pipeline and prediction model

Project Demo

slides - https://docs.google.com/presentation/d/1rMsdsFeUuVE7S7o4vnMMWISkcRpW32MU8bU2dmK5jus/edit?usp=sharing

all resources - https://drive.google.com/drive/folders/1q93otqn0OnaW5FHU3ZkuJhmc-9womOfM?usp=sharing

Data & Resources