SenseTheSpace| Hey! What Are You Looking At?

Hey! What Are You Looking At?

The High Energy Astrophysics Science Archive Research Center (HEASARC) archives space agencies' data from missions studying electromagnetic radiation from extremely energetic cosmic phenomena (e.g., gravitational wave detections, gamma ray bursts, and supernovae). The Canadian Astronomy Data Center (CADC) is another repository containing missions studying comets, asteroids, and exoplanets among other things. Your challenge is to create a visualization tool that can help people interested in these phenomena to access the data quickly and easily.

SenseTheSpace (Collect, Analyse and Visualise Space Data - detect useful patterns with AWS services)

Summary

Website: https://sensethespaceres.co/index.htmlSenseTheSpace is about sensing data coming from outer space arriving as a continuous stream of information to The Mother Earth via satellites or probes in combination with ground-based systems of observation. Outer space observations provide information patterns that can be identified as either predefined/pre-observed or new objects, events that are important for the study carried out by researchers. Distinguishing between identified and unidentified objects, phenomena or events in the celestial space within a desired amount of time (before hand) can identify threats from objects in our solar system or help Space Missions for launch/travel.

How I Addressed This Challenge

Website:https://sensethespaceres.co/index.html


Goal and Demand:

Considering the exponentially increasing data payloads from a spectrum of observation systems, this mere distinction of objects and events in outer space shown in a visualisation and/or notification solution/service is not so easy without utilising the power of AI, Cloud & Quantum Computing. To architect and design an approach to effectively use these technologies from AWS cloud is the primary aim of SenseTheSpace team. It has become increasingly difficult to have multiple Astrophysicists or Space Data Researchers that can manually or even semi-automatically perform data collection, data analysis and observations for events, super-events and phenomena in the outer space and provide timely notifications for multiple reasons. As every industry is moving towards the data explosion and aims to solve this using Data, AI and Quantum Computing technologies, SenseTheSpace targeted to solving the challenge by developing some disintegrated demos with real Space Data from NASA datasets, and then proposing a unified approach to develop a system for observational effectiveness. A constant urge to utilise multiple upcoming tech stacks in the Data, AI and Quantum Computing industries, and an inherent love towards Astronomy and Space Sciences motivated me to devise the solution for this challenge.


Mission:

SenseTheSpace focusses on demonstrating appropriate use of a spectrum of technologies in Data and AI so that this can be utilised for developing Minimum Viable Products that are scalable for large scale data analysis and information extraction and pattern identification, with respect to this particular space! We aim at a unified system of pattern identification, pattern mining and pattern matching to detect objects, events and phenomena in the celestial space.


Vision: 

SenseTheSpace aims to utilise multiple technology stacks in the area of Data, AI and Quantum Computing to provide the low latency pattern recognition and notification solutions in a simple and useful visualisation dashboard, that can be leveraged later in terms of the number and complexity of features in the visualisation.


Roadmap:

SenseTheSpace aims to provide a concrete roadmap of at least 1 year that can facilitate new threads of innovation, research and application of the advanced tech space in data and AI for solving this particular challenge over a longer run.

How I Developed This Project

PPT Slides:https://drive.google.com/file/d/1OXX4WshgV5HCpfvel-Eem8mn6UG5zRXP/view?usp=sharing

I subdivide the work into following sections for simplicity:


  1. Demonstration of Collection, Analysis and Visualisation of Data from Far Extragalactic and Intergalactic Space: This was completed on local machine by running python code for detecting a super_event - Detection of a Super Massive Black Hole in 2015. Observations using LIGO Hanford and LIGO Livingston data were demonstrated with inferences on data collection, analysis and visualisation.
  2. Demonstration of Collection, Analysis and Visualisation of Data from Solar System Space: This was done for an exoplanet detection using Hubble Space Telescope data. Jupyter notebooks were installed on the local machine to showcase use of MAST for HST observations as well as TESS observations. Plot of folded light curve is shown.
  3. Design of mechanism to convert Jupyter Notebook Python code into containerised micro services on AWS cloud: Typically a Jupyter Notebook performs more or less unitary functionality - like collecting a particular dataset, analysing it and inferencing the finding in some visualisation. Converting this unitary functionality even if it is specific or generic task performed on astronomical and space data, and then wrapping it around a containerised micro service on AWS cloud can be beneficial. If this service demands intensive compute, then we can come up with associated architecture of utilising the respective data compute service on AWS cloud.
  4. Explain segregation of functional code for Pattern Identification, Mining and Matching: Consider the example of performing filtering techniques for time series data obtained from LIGO and VIRGO observatories. If we are able to simplify the python code as a service that performs this filtering on massive datasets, it is possible to achieve performance by deploying this as a scalable containerised micro service on AWS cloud. Same is with distinguishing a cosmic ray object from a start while studying the cosmic ray python code.
  5. How AWS cloud services will facilitate a long term MVP... Progressing
  6. Integration of Real-Time Data Feeds using AWS Ground Station... Progressing
  7. Integrating ML workflows using AWS Sagemaker... Progressing
  8. Identifying Compute Workloads with exponential complexity to be distributed to Amazon Bracket... Progressing
  9. Envisioning an Integrated Cloud+Satellite based Hybrid system of visualisation, analysis and notification... Progressing
How I Used Space Agency Data in This Project

Code and Notebooks Used:


  1. notebooks-master (DrizzlePac, JWST, MAST, NICMOS): https://drive.google.com/file/d/14B6jjgeEkUDbVcEGsMTWGFgBqR15YMBc/view?usp=sharing
  2. Aladin Lite (uses ligo.skymap): https://drive.google.com/file/d/1ZVHutOUEXwdaFtRfCvMAOGvrRlgHvUyW/view?usp=sharing
  3. ligo.skymap: https://drive.google.com/file/d/1b6GRutvbSQLhhurgqJ3znEvMIrA_G1bz/view?usp=sharing
  4. hst_cosmic_rays_master: https://drive.google.com/file/d/1fZqkwM_FbHXd_utRYZi6ohzMjRCqW61B/view?usp=sharing
  5. LIGO Pattern Visualisation (python program): https://drive.google.com/file/d/1YAq_eMhNt4MHjoyJntvH3AsiyF5FExCU/view?usp=sharing
  6. Filter Time Series - Detect Gravity Waves (Super Massive Blackhole super event 2015): https://drive.google.com/file/d/1y3dCG4LhcvM7brr7fwBBhlDMv6sbnC5o/view?usp=sharing
  7. Q-Transform of Time Series (Detection of supermassive black hole - also used for music signals): https://drive.google.com/file/d/1y3dCG4LhcvM7brr7fwBBhlDMv6sbnC5o/view?usp=sharing
  8. qiskit-community-tutorials-master: https://drive.google.com/file/d/1bLHoRBI68M8Iv321JVVF6KUnLcddhSlL/view?usp=sharing
  9. DWave Solver Usage White Paper: https://drive.google.com/file/d/18tr2--5dQ6V7HTyKytoUMR5lWJZ1sJCz/view?usp=sharing

Datasets Used:


  1. LIGO:
  2. HST:
  3. TESS

Some datasets tried but not fully included in the demo: Cosmic Ray Data, X-Ray Data. Provided given more time these could be integrated into the solution.

Tools Used:


  1. LIGO Libraries
  2. Aladin Lite uses ligo.skymap
  3. XAMIN WebApp
  4. DrizzlePac
  5. MAST
  6. GraceDB

AWS Services Proposed:


  1. AWS S3 - for storing disparate datasets on a reliable, always available storage on cloud
  2. AWS Lambda - for serverless deployment of unitary functionality
  3. AWS EC2 - for deploying preprocessing of datasets as and when they arrive on S3
  4. AWS Data Exchange - Seamless Data Integration for Batch and Streaming Data Integration from disparate data sources
  5. AWS Sagemaker - for automating ML/DL/AI or Analytical Workflows
  6. AWS Ground Station - for integrating real time data streams from satellite data
  7. Amazon Bracket - for distributing massive, heavy compute workloads that are parallelizable and reducible to an optimisation problem (mostly exponential time complexity search or pattern matching or path identification problems with exponential time complexity that gets reduced to polynomial time on quantum computing infrastructure). Explored graph partitioning on AWS Bracket Notebook Instance. Kindly refer the latest PPT slides for details.
  8. AWS Pinpoint - for developing quick web or mobile applications
  9. AWS Simple Email Service - for providing email notifications for super events observed
Project Demo

1. Detection of Supermassive BlackHole in 2015: Category: SuperEvent (Extragalactic or Intergalactic Events)

LIGO Hanford and LIGO Livingston Data of Detecting Super Massive Black Hole (super event 2015)

Q-Transform ofTime Series Data Detecting Super Event in 2015 (Supermassive Black Hole)

Time Series Filtering for Detecting Supermassive Black Hole (super event in 2015)

2. Detection of Exoplanet by observing Hubble Space Telescope Data (Solar System Events)3. Sky Matching (HST Telescope Data Analysis):Sharing link due to limited space here: https://drive.google.com/file/d/1f2e2ZDAQI53Usb0DKPs0nvc23KptUxOh/view?usp=sharing

Project Code
Data & Resources
  1. LIGO Data
  2. HST Data
  3. TESS Data
Tags
#artificial_intelligence, #scalable_machine_learning, #space_data, #super_events
Judging
This project was submitted for consideration during the Space Apps Judging process.