ESCAPE DIOS: Development and application of an automatic workflow for the replication of scientific data

Note: adapted from original article written by Agustin Bruzzese, Jordi Delgado, Pau Tallada, Gonzalo Merino, from Port d’Informació Científica (PIC) and High Energy Physics Institute (IFAE, Institut de Física d’Altes Energies). IFAE is a partner in ESCAPE project.

Prototype of MAGIC data at PIC: Development and application of an automatic workflow for the replication of scientific data

Thus, we would like to develop a suitable workflow to handle large data sets produced by the gamma ray telescope, and continuously stream these files to the datalake for permanent storage and access, while keeping free space in the data source center. For this, Rucio is an open-source software framework that provides scientific collaborations with the functionality to organize, manage, and access their data at scale. Also, Rucio can trigger the automatic deletion of files once they have been successfully replicated to its destination.

By means of Rucio, the following frame replication workflow has been developed (Fig. 1 see below). Due to the current restrictions established by the experimentalists in charge of the distribution data from MAGIC do not allow free access to the real data, from 5 up to 10 mock files are generated locally at the root base of the non-deterministic RSE (PIC-INJECT). As a result, it can be seen in Fig. 2.1 the increase from 5 to 10 every 30 min (blue line), corresponding to the creation of files in the RSE PIC-INJECT. Once these new files are generated, the script that initializes the automatic replication workflow activates. In the first place, those files will be registered in the Rucio catalog (through the command "add_replicas"). Consequently, in Fig. 2.2 you can see peaks corresponding to the number of records added to the Rucio catalog. Then, the script will proceed to organize and create the registered files according to the MAGIC namespace in the RUCIO collections (dataset and containers), and the replication rules are created pointing to the deterministic RSE (PIC-DCACHE)(Fig. 2.3). All the files of our current data samples will be organized in 8 main rules according to their namespace of MAGIC. For this reason, in Fig. 2.4 we see 8 rules organized as containers (blue).

To achieve the automatic deletion of files in PIC-INJECT, the creation of temporary rules was implemented, which were called carrier rules(Fig. 1). The intention is that once the data is successfully replicated to the PIC-DCACHE RSE, the files in the PIC-INJECT RSE are automatically deleted, thus conserving space in the origin RSE (Fig. 2.4). Herein, the creation of two datasets is seen in Fig. 2.4 (green), one which corresponds to a dataset that points to the PIC-INJECT, and another that points to PIC-DCACHE (Fig. 1). Shortly after, the line decreases when the content has been correctly replicated in the destination RSE, which means that the dataset has been erased. Consistently, in Fig. 2.1, it is also seen a drop in the total count number of files in PIC-INJECT, suggesting that the files that were replicated have been deleted. Hence, it is possible to consistently add new files to the main rules, while eliminating the files from the source.

After 72 h of continuous running of this workflow, the replication and deletion of ~1000 files of 10 mb with the MAGIC namespace injected into the data lake were recorded Fig. 2.5. Specifically, an increase in the number of files is seen in PIC-DCACHE each 30min, which corresponds to the successful replications of files appearing at PIC-INJECT. The results show consistency in the upload fluctuations of a file and the elimination of a rule, so it can be suggested that the current setup of the ESCAPE DIOS instance is optimal. Thus, our script implements a functional fully automated workflow that could be used for current experiments in the datalake.