The Promise of the Cloud and the IRIS DMC

Date: 4/25/2019

Time: 05:00 PM

Room: Elliott Bay

The IRIS Data Management Center (DMC) has operated a repository of time series data for 3 decades using its own computational and storage infrastructure. The rise of cloud computing comes with many promises, such as nearly unlimited compute and storage capacity, load-driven dynamic scaling, turn-key data replication and more. We have started exploring which of these cloud resources could potentially be leveraged by the DMC in the GeoSciCloud project, which is supported by the National Science Foundation’s (NSF) EarthCube program. One of the project goals is to evaluate how some of our core systems operate within a cloud environment. For this project, we selected Amazon’s AWS and XSEDE’s Jetstream and Wrangler systems. Web services were deployed in these environments using Docker and Kubernetes running across a cluster of virtual machines. This architecture proved to be powerful, allowing for relatively simple scaling of service capacity and an abstraction of service instances. Looking ahead, the platform created by Docker + Kubernetes is an enticing model for a large data processing framework. Initial results, common to both cloud systems, indicate two clear advantages for DMC users. First, the ability to scale data access capacity allow for servicing more concurrent requests than the DMC is currently capable of handling. Second, providing the data in close proximity to powerful compute infrastructure used by researchers eliminates the significant bottleneck of transferring data over the internet. A number of downsides were also identified including monetary cost, uncertainty regarding long term operation, cost in adapting to cloud resources, and reduced insight into systems. The DMC is currently exploring the potential to partner with peer data centers that are members of the Council of Data Facilities and an infrastructure operator to develop shared infrastructure that would provide the advantages of cloud computing and mitigate most of the disadvantages.

Presenting Author: Timothy Ahern

Authors