Prior to adopting this new solution, when clusters were torn down they would lose code, work, and most importantly - time. The new solution separates concerns and gives them the ability to now have clusters running independently of the notebooks.
Existing clusters took over 45 minutes to bootstrap. With the new solution, the clusters launch in about 7 to 8 minutes. Goldfinch is now much more willing to change cluster configurations, and launch clusters tailored to the compute job because they can be more agile with this new solution in place.
With Privo owning the responsibility of maintaining the Hail AMI, Goldfinch can focus their time and effort on creating tailored clusters to their data scientists, and not have to worry about building and maintaining the complexities of Hail.
