Kryptonians| Hey! What Are You Looking At?
Hey! What Are You Looking At?
Visionary, a 3D sky map and multilayered-mission catalog visualization tool in Tableau.
Summary
Visionare is the name for an idealized visualization tool and interface, potentially useful for scientists and researchers of NASA and other space agencies, in the sense that in its scope, it encompasses dynamic 3D sky mapping of sources from HEASARC archives, including diverse missions (NuStar, Swift, XMM-Newton) and their respectively catalogs (Observation Type tables). It was also thought to provide a multilayer information about sources, meaning that a close look up on a source would correlate it how it was seen and when, by different missions. In this way, it is expected to allow users to follow up celestial events developed over time with different observatories instruments.
How I Addressed This Challenge
The idealized tool Visionare was sketched up to be a visualization tool to aid specialists in the source-hunting tasks. That means that it would make the life of people going through missions' data, to find the same source among different datasets, so that one is able to compare different observations of the same target. In this way, the tool comes as a facilitator and a first step for finding celestial objects through different missions/catalogs.
In order to achieve that, a dynamic 3D sky map was thought to be the visual interface a user would rely upon to find his/her interests. In this sphere, the dots rendered on it surface would represent the actual source using the right ascension and declination measures. The user would also be able to see: time of start of exposure, time of end of exposure, name, instrument name of the observatory (if provided), observation ID.
A multilayer visualization would be accomplished by a set of filters, that would display different catalogs. The navegation itself through the sky map is to happen through mouse dragging on the sphere or button controlled.
The hope was to achieve a useful solution for specialists at NASA who may feel overwhelmed by massive amount of data and with a hard time trying to find sources on different catalogs.
In order correlate the differente catalogs/datasets, and with the advice of the experts from the chat, the 'name' field for each source would be used to intersect datasets and form the multilayered visualization.
Further futures could be implemented if time permitted, like providing a constructed Sky View URL for seeing the source appearance. An animated visualization based on time would be also added if possible. Finally, colors would also be added to the visualization as an extra feature.
How I Developed This Project
Since I am the only member of this team and its creator, it is pretty straightfoward and personal the motivation behind this endeavour. I always liked astronomy since I was a little kid and I read books about it, both for the open public and more advanced ones, too. My field of occupation is Computer Science, a formal degree obtained years ago. Since I got out of college, I had the opportunity to work in a Synchrotron accelerator which boosted my passion for the Natural Sciences, specifically Physics. Nonetheless, I engaged advance level studies in Data Science and Big Data recently and I am also learning about Visualization and Story Telling in this program. It came to my knowledge the hackathon event by the same university I am studying. I never participated in a Hackathon before, and because all of my background and interests I thought it to be a good experience, which was indeed. Finally, I chose the Observe category and found this challenge. It was very similar to a research project proposal I once did, which would use neural networks in order to classify large astronomical datasets in one of the two categories: star or galaxy. Since I elaborated this proposal, I was aware of the existence of Virtual Observatories and the scientific effort of providing ease access to unexplored huge datasets.
Unfortunately, I faced two problems in the development of the project. One of them was the data pipeline I created was broken for some bug that is happening with Tableau Prep (the version I used is listed below). I searched the web for the occurrence of the problem, and indeed, there are other people experimenting this too. The software gets in a eternal loop for "validating flux and generating schema" at some point of one's data transformations. What I was doing was to load and organize/clean columns associated with name, right ascension, declination, start time for exposure, end time for exposure, observation ID, instrument name from the following datasets:
- NuStar Mission nuaftl catalog/dataset;
- NuStar Mission numaster catalog/dataset;
- Swift Mission swbatsfxt catalog/dataset;
- Swift Mission swiftmastr catalog/dataset;
- Swift Mission swifttdrss catalog/dataset;
- Swift Mission swiftuvlog catalog/dataset;
- XMM-Newton Mission xmmmaster catalog/dataset;
- XMM-Newton Mission xmmomsuob catalog/dataset.
Tipically, at the point of having all of these datasets inserted in an ETL flux in Tableau Prep, the transformation of cleaning entries with right ascesion and/or declination fields being null would get me stucked at the cited problem. I hoped that Tableau would be stable and reliable, but in the end it blocked the development of the solution right at the pre-stages of the project, which is disappointing.
The second problem which occured before the one above, got me precious hours overnight of October 3rd-4th. I was trying to find a sphere model, converting it to an acceptable format in Tableau Desktop (from .stl to .tde) and loading the converted file. The first model I found was had a non-readable file (ascii), so I could not convert it at all. I then found one that was convertible, but conversion seemed to whipe out some chuncks of data. Besides, getting access to a converting tool took me time, also.The references I used did not provide any sphere model. Furthermore, the calculated rotation measures from the workbook³ were all null values. I followed the steps in the tutorials cited, but I was facing a total different scenario of inadequacy. During all of this time, I was chatting with experts/ambassadors/moderators and I formally thank their help which showed me possible workarounds many times. In the afternoon of October 4th I decided to simplify the project in order to make it viable, so I decided with the advice from NASA experts, to project right ascension and declination in a 2D image of the sky. That's when I went over to Tableau Prep and got stucked. This lead me to the conclusion I wouldn't get any further with more attempts. Substituting Tableau Prep was not an option, I did not have time. Dead end. I understood it was time to stop and accept the state of the situation and start writing the Project for submission.
The original idea was to publish the 3D sky map in Tableau Public server, so anyone could use the tool.
Below I list down all the resources used:
Software
- Tableau Desktop 2020.3.1 (64 bits);
- Tableau Prep 2020.3.3 (64 bits);
- Alteryx Designer 2020.3 (64 bits) and its package [4] Convert STL to TDE.
Hardware
- Intel Core i7 10750H 12 cores
- 8 GB RAM
- 256 GB SSD
- Nvidia GeForce GTX 1650 4GB GPU
Addiotionally I wrote a documentation right at the hackathon start that I provide below:
PROJECT DOCUMENTATION v1.0
NOTATION AND CONVENTIONS USED FOR THIS PROJECT
- Source = Celestial object
- Database = NASA's HEASARC database
- Catalog = A sky survey in the database indexed to a mission
- Appearance (of a source)= gathered light radiation recorded in a catalog in a pictorial format, no matter the wavelength
- Event = Extremely energetic phenomena involving a source
- Dataset = Isolated event's data in a catalog
- Visualization = Dynamic 3D celestial map visualization
- History (of a source) = Event development as a function of time
- Moment (of a history) = A specific moment in history
SCOPE
- Intended Audience/potential users: Professional Astronomers, Physicists, Researchers, Amateur Astronomers;
- Sources: HEASARC's database archives;
- NASA's Missions: NuStar, Swift, XMM-Newton;
- Examples of events: Gravitational waves, Supernovae explosions, Gamma-ray bursts.
GOAL
Create a VISUALIZATION TOOL that can help people interested in EXTREMELY ENERGETIC COSMIC PHENOMENA to access the data quickly and easily.
FUNCTIONAL REQUIREMENTS (for the visualization tool):
- Selection of different wavelengths for displaying different appearances of the same object(s);
- Selection of different missions for providing mission-wise selection of events;
- Be able to see the history of the celestial source before/after the event happens:
- What the source looked/look like;
- Behavior on a time series (hours/days).
- Assuming that 'look(ed) like' is the appearance of the source, and, the 'behavior' as the appearance through history.
OBJECTIVES
- Use HEASARC database;
- Scrutinize missions in the database to be included in the visualization;
- Create a 3D dynamic celestial map (sphere-shaped);
- Provide ways for the user to navegate through/select the missions/wavelengths/sources;
- Enhance and modify the visualization to show the selected source's appearance up-close;
- Show the history of the source both as a continuous and in a discrete time-frame.
TASKS
- As a user I'd like to see the map of the night sky by looking a 3D sphere simulation;
- As a user I'd like to rotate the field of view and the sky itself to see other portions of data available by using some type of rotation control;
- As a user I'd like to see gamma-ray and x-ray wavelengths of sources by selecting some kind of filter;
- As a user I'd like to see data from missions NuStar, Swift and XMM-Newton by selecting some kind of filter;
- As a user I'd like to find a source from its coordinates by inserting its location in numeric fields;
- As a user I'd like to see a selected source in a specific moment by inserting the moment in a numeric field;
- As a user I'd like to see the history of a selected source by setting a time interval.
METHODOLOGY
- Download the databases from the online archives;
- Analyse the database schemas;
- Select a view of the relevant data (columns);
- Load and transform the selected view data in Tableau;
- Select and index missions'records among the database;
- Create a 3D simulation of the night sky, building a visualization in Tableau based upon Flerlage's¹;
- Insert filters in the visualization for selecting wavelengths and missions being displayed in the visualization;
- Create pop-ups when selecting a source for showing its history in a slide-show fashion.
How I Used Space Agency Data in This Project
I used mainly HEASARC Browse (<https://heasarc.gsfc.nasa.gov/db-perl/W3Browse/w3catindex.pl>) web interface for looking for datasets and finding missions/catalogs. Issueing queries there I was able to generate text files that were then loaded in the ETL tool.
The catalogs/dataset used were (all Observation table types):
- NuStar Mission nuaftl;
- NuStar Mission numaster;
- Swift Mission swbatsfxt;
- Swift Mission swiftmastr;
- Swift Mission swifttdrss;
- Swift Mission swiftuvlogt;
- XMM-Newton Mission xmmmaster;
- XMM-Newton Mission xmmomsuob.
The reason for using the Observation table types is that, after consulting experts in the chat, it provides all the information a user would be interested to see in this tool and to identify the same source in different catalogs. In other words, answering what and when was imaged, with the use of the name and coordinates, for what, and exposure time interval, for when.
Project Demo
Unfornately, there is no solution to be shown due to reasons explained above.
Data & Resources
1.Flerlage, Ken. "Creating a 3D Star Map in Tableau". January 20, 2018. Source:
<www.flerlagetwins.com/2018/01/creating-3d-star-map-in-tableau_55.html>. Visited on
October 3rd, 2020.
2.A'Hearn, Anya. "The 3D Tableau Full Monty". November 30, 2016. Source:
<https://www.datablick.com/blog/2016/11/22/the-3d-tableau-full-monty>. Visited on October
4th, 2020.
3.Beran, Bora. "Going 3D with Tableau. December 18, 2015. Source:<https://public.tableau.com
/views/3DChartsinTablea
/Simple3DScatterplot?:embed=y&:display_count=no&:showVizHome=no>. Visited on October
4th, 2020.
4. Riggs, Philip D. Package for converting STL files to TDE. Source: <https://gallery.alteryx.com/#!app/Convert-STL-to-TDE/583e52c8f499c70468980488>. Visited on October 4th, 2020.
5. Sphere model from Wikimedia. Source: <https://commons.wikimedia.org/wiki/File:3d_model_of_Sphere.stl#file>. Visited on October 4th, 2020.
6. Burkardt, John. Sphere model from Florida State University: <https://people.sc.fsu.edu/~jburkardt/data/stla/stla.html>. Visited on October 4th, 2020.