Background
Moving large amounts of data poses a significant challenge. In most cases, networks optimized for business operations are neither designed nor capable of meeting the data movement requirements of data-intensive research. When scientists attempt to run data intensive applications over these so-called “general-purpose” or enterprise networks, the result is often poor performance. In many cases, this poor performance significantly impacts the scientific mission, leading to challenges such as not receiving data on time or resorting to “desperate” measures, such as physically shipping disks.
There has been a significant increase in available network capacity and a greater need to be able to transfer large amounts of data efficiently. With the implementation of the SANReN 100Gbps backbone network capacity, 100Gbps Data Transfer Nodes (DTNs) have been implemented nationally, with Globus (globus.org) data transfer software installed.
The best international data transfer results seen were between Johannesburg/Cape Town DTNs and Colorado National Centre for Atmospheric Research’s (Globally Accessible Data Environment (NCAR GLADE). Data transfer speeds reached between 4.78GB/s-5.48GB/s, which resulted in 1TB of data being moved in only 3 minutes!
Data Transfer Pilot
This SANReN PERT have been around for a number of years. A SANReN Data Transfer Pilot (DTP) project was initiated in 2017, on SANReN’s 10Gbps network infrastructure, with the intent of evaluating the data transfer movement needs of the research and education community and to facilitate the development of the Data Transfer Service (DTS).
Although the DTP was beneficially to many, the data transfer speeds achieved were limited by the hardware selected at the time. The SANReN LDT team have since improved the infrastructure, provide exceptional service, and go above and beyond to assist researchers with their data movement needs. The SANReN DTS infrastructure is now available on 100Gbps infrastructure, leading to improvements in efficiencies for researchers and scientists and enhancing its offering to CSIR stakeholders.
The SANReN LDT team have been novel in their approach of the implementation of a combined perfSONAR/DTN to support large data transfers for researchers, being one of the firsts to implement this combination successfully. Each function has a dedicated 100G backbone connection, capable of utilizing the full 200Gb/s of combined network bandwidth when needed.
Although started with data transfer scope only, the SANReN PERT have embraced the NICIS integrated services commitment to ensuring that South African cyberinfrastructure is world-class and that the data transfer movement infrastructure will be able to work seamlessly with the other pillars of NICIS, ie. Data Intensive Research of South Africa (DIRISA), who are responsible for managing data storage, and the Centre for High Performance Computing. The team now assists clients wholistically on advising clients via the extended Performance Enhancement Response Team (PERT) team.
The iRODs/Globus connector was tested successfully on a SANReN DTN and is a solution to integrate IRODs (Data Intensive Research Institute of South Africa (DIIRISA’s) chosen storage solution) and Globus data transfer software (DTN data movement solution) to enhance the NICIS’s data transfer and data management solutions.
Through the support of the SANReN PERT, their infrastructure and advice are enabling data generated from single pieces of specialized instrumentation to have global use and impact.