Data Wrangling and Advanced Projects
Feb. 7th, 2026 10:08 am![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
tcpipTen years ago, when the Spartan HPC system was launched at the University, it was small, innovative, and experimental, its very name a laconic reference to the funding provided (i.e., not much). But the tricky combination of traditional HPC flexibly supplemented by cloud VMs for single-node jobs worked, and over the years, it would become one of the world's most powerful supercomputers, supporting thousands of researchers with the computational power required in engineering, astronomy, mathematics, economics, climate science, and especially the various medical and life sciences. One of the decisions we made at the time was that Spartan would be open and collaborative; researchers could come from anywhere in the world, as long as the project's principal investigator was from the University of Melbourne. It was an openness that has allowed hundreds of researchers to access the supercomputer resources.
There has been, however, a change in policy and not one to my liking. Now each project requires a university supervisor, and each external collaborator requires a University email address, with the supervisor making a separate application for each individual. This is an overly bureaucratic procedure, in my opinion, and if there's anything IT workers hate doing, it's wrangling systems to meet unnecessary bureaucratic requirements. It's wasted work and time that provides no change in outcome; mathematicians would describe it as "inelegant", engineers would call it "suboptimal", economists would call it "damaged goods", you get the idea. My unfortunate role this week has been to get a list of active non-University researcher accounts and craft individual emails to each of them and their university supervisor, informing them of their need to apply for new email addresses. Due to nuances that I won't go into (such as one user many projects) it was not a matter of just making a single SQL database extraction, but rather required several steps of data wrangling.
The procedure was a bit of an annoyance, an interesting technical challenge, but the real moment of joy was achieved by going through the various projects: ecosystem population connections in tropical oceans, molecular modelling of novel antivirals against SARS-CoV-2 proteins, cosmic birefringence from the South Pole telescope, subterranean dark matter studies - and so it goes on. It is the range, diversity, and importance of these projects that inspire me, a quest for objective knowledge without partisanship, in a world where universal norms are betrayed by the influences of power and wealth, and aesthetic expressions are either trite or manipulative rather than sincere expressions of the imagination. As I tell researchers in my introductory class, you are the people who will make the discoveries and inventions that hopefully will make the world a better place. I'm just going to show you how to harness the resources of a supercomputer to make this easier for you. So even when I'm deeply engaged in a project I find grossly and even offensively unnecessary, there are still some parts that bring joy and hope.
There has been, however, a change in policy and not one to my liking. Now each project requires a university supervisor, and each external collaborator requires a University email address, with the supervisor making a separate application for each individual. This is an overly bureaucratic procedure, in my opinion, and if there's anything IT workers hate doing, it's wrangling systems to meet unnecessary bureaucratic requirements. It's wasted work and time that provides no change in outcome; mathematicians would describe it as "inelegant", engineers would call it "suboptimal", economists would call it "damaged goods", you get the idea. My unfortunate role this week has been to get a list of active non-University researcher accounts and craft individual emails to each of them and their university supervisor, informing them of their need to apply for new email addresses. Due to nuances that I won't go into (such as one user many projects) it was not a matter of just making a single SQL database extraction, but rather required several steps of data wrangling.
The procedure was a bit of an annoyance, an interesting technical challenge, but the real moment of joy was achieved by going through the various projects: ecosystem population connections in tropical oceans, molecular modelling of novel antivirals against SARS-CoV-2 proteins, cosmic birefringence from the South Pole telescope, subterranean dark matter studies - and so it goes on. It is the range, diversity, and importance of these projects that inspire me, a quest for objective knowledge without partisanship, in a world where universal norms are betrayed by the influences of power and wealth, and aesthetic expressions are either trite or manipulative rather than sincere expressions of the imagination. As I tell researchers in my introductory class, you are the people who will make the discoveries and inventions that hopefully will make the world a better place. I'm just going to show you how to harness the resources of a supercomputer to make this easier for you. So even when I'm deeply engaged in a project I find grossly and even offensively unnecessary, there are still some parts that bring joy and hope.
