Lappeenranta University of Technology Department of Information Technology 01065210 Parallel Computing AstroGrid Teemu Toivola, Tite 5, 0109337 Tero Viitala, Tite 5, 0080692 1 Abreviations 3 2 Introduction 4 3 Virtual Observatory Prototype 5 3.1 Introduction 5 3.2 Project management 5 3.2.1 Three year project goals 5 3.2.2 One year goals 6 4 Astrogrid's technology needs 7 4.1.1 Science Requirements 8 5 Phase B Plan 9 5.1 Milestones 9 5.2 Preliminary architecture 9 5.2.1 Resource Registry 10 5.2.2 Community 10 5.2.3 Workflow 10 5.2.4 Portal 11 5.2.5 DataAccess 11 5.2.6 Tools 11 5.2.7 Security 11 5.2.8 Messaging 12 5.2.9 GridSpace 12 5.3 Phase B implementation 12 5.3.1 Iteration 2 (June 2003) 12 5.3.2 Iteration 4 (December 2003) 13 5.3.3 Iteration 6 (June 2004) 13 5.3.4 Iteration 8 (December 2004) 13 6 Challenge: metadata in grid datamining 15 6.1 Different types of metadata 15 6.2 Metadata service 15 6.3 Replicas 16 7 References 17 Abreviations CAS Community Authorization Server CoPS Community Provilige Server, Manages creation, removal, authentications of user and communities. OGSA Open Grid Service Architecture OGSI Open Grid Service Infrastructure VO Virtual Observatory XML eXtended Markup Language Introduction The aim of the AstroGrid project is to produce a working datagrid for major UK astronomy related databases, with associated datamining facilities, by the beginning of 2005. The project is part of the worldwide drive towards the concept of a Virtual Observatory, and can be seen as the UK contribution to this vision. However, in various ways it is both wider and more focussed than other initiatives. It is wider in that it covers astronomy, solar physics, and space plasma physics, and covers all wavelengths from radio to X-ray. The project is also part of a coherent UK e-science programme, with links to projects in particle physics, bio-informatics, and basic grid technology development. AstroGrid is a three-year project: it was officially started in September 2001. Development is split into two phases. Phase A was essentially an intensive one-year R&D study. The Phase A Report was completed in October 2002. Phase B began in January 2003. /1/ AstroGrid is one of three major Virtual Observatory projects worldwide, the others being the European Astrophysical Virtual Observatory (AVO) and the US National Virtual Observatory (NVO). These three projects collaborate quite closely via an International Virtual Observatory Alliance, which is hoped to grow to encompass other VO projects worldwide. Virtual Observatory Prototype Introduction The idea of the Virtual Observatory arose gradually over a period of time in which astronomical data archive sites continually improved their facilities and user-interfaces, but in a piece-meal and uncoordinated way. Many have realised that a coherent and planned approach, with standardised interfaces, would permit a radical advance. As part of AstroGrid's initial programme, a short survey of the most advanced existing web sites and software packages was carried to determine the facilities they provided and how they worked. Nearly all of these rely on technical solutions which pre-date Data Grids and the Web Services paradigm, so it wasn’t expected there to be much scope for direct technology transfer, but the facilities reflect the perceived needs of the astronomical community, and it was hoped to learn a lot from them about what AstroGrid needs to provide, and note the strengths and weaknesses of the current solutions. These investigations were carried out as a joint exercise between the AstroGrid grid technology and database technology teams. They represent a snapshot of practice in the early part of 2002, and that many facilities have changed since then. Project management Within the three-year project, the project has been divided into two phases. Phase A is essentially an intensive one-year R&D study, which is developed in as concrete manner as possible. Phase B delivers the agreed AstroGrid facility, and has preliminary goals. Three year project goals The final deliverables of the project will not be agreed until completion of the Phase B plan. However, the goals include a working datagrid for key databases, requiring an associated distributed storage systems and high-throughput datamining machines, implementation of middleware to enable distributed processing and data management and ingestion of 14 key target databases. /2/ A uniform archive query and data-mining interface has been planned for the above. It would include simple public query interface by standard web page, an advanced query and datamining interface using supplied software environment and an intelligent on-line help system (AstroGrid Assistant). Goals also include the ability to browse simultaneously multiple datasets and tools for integrated on-line analysis of data containing images and spectra and the ability for user to upload code to run own algorithms. One year goals The key deliverables for phase A are as follows: Phase B plan Science Requirements Document including set of example Use Cases) Functional Requirements Document Functionality Market Survey Report Agreed division of labour with international partners Federation of SOHO and Yohkoh databases Federation of SuperCOSMOS, SDSS, and INT-WFC databases Federation of XMM and Chandra data products Federation of CLUSTER and EISCAT databases Federation of selected MERLIN and VLA database subsets Preliminary visualisation tool Preliminary database management system Astrogrid's technology needs AstroGrid is now seen as basically a service grid, but with a datagrid incorporated to move data between services and to store intermediate results. At present, there are no plans to include a compute grid. Although some substantial processors (e.g. Beowulf clusters) are available to AstroGrid, these will initially be the sites for pre-installed application-services, not resources to which code is uploaded by end-users. All access to bulk datasets in archives is through data-selection services that isolate data-extracts small enough to move about the datagrid. In the rare cases where a job needs to access all of a major dataset, then the processing services for that job are constrained to run at the data-centre where the data-set lives. Hence, large-scale datamining operations are likely to be at major data-centres, not at arbitrary places on the grid where there is spare CPU power or storage. /3/ Dataprocessing services are needed that wrap up existing, general software in astronomy, such as the Interactive Data Language, the Astronomical Image Processing System and the Starlink Software collection. These services should be available at many points on the Grid. Where possible, they should be available at the places where data extracts are made, such that the extracts do not have to cross the network. Most of AstroGrid's work will be the analysis of data in archives that have already been processed in standard data-reduction "pipelines". AstroGrid's mission is, in part, to promote the use of these standard reductions in preference to reprocessing of the raw data by individual researchers. However, there will be cases where pre-reduced data are not archived, notably in solar and radio astronomy, or where re-reduction with improved techniques is desirable. Specialist dataprocessing services are needed that wrap up the data-reduction pipelines for the major surveys and facilities. Results from services must remain in the datagrid for later use. A user typically wishes to feed the results back into another calculation or to share the results with collaborators or to publish the results directly from the Grid. A general system for storing and cataloguing results is needed. "GridSpace" is the current working title for that service. Science Requirements The planned usage of AstroGrid has naturally driven the technology and architecture of AstroGrid. Early in the project 10 use cases were selected and the final AstroGrid should address the problems of these use cases. Expected future usages are discovery of High Redshift Quasars, locating galaxy clusters at a range of redshifts, Brown Dwarf selections, deep Field Surveys, Low Surface Brightness galaxy discovery, Supernova galaxy environments, Solar Stellar Flare Comparison, Solar Coronal Waves, STP Solar Event Coincidence and Magnetic storm onsets. All these require combining existing data and performing data mining within data. /5/ Phase B Plan Milestones Unlike in the first phase, the phase B is implemented using iterative approach. This method would produce a functional VO in the early stage of project implementation. Some new technologies are used and the experience got from them is taken account for the following iterations. Phase B iterations are phrased in terms of system capability. The goal was to have a working VO-like system available from at least the second iteration. Preliminary architecture AstroGrid is seen mostly as a service grid, to which data-grid has incorporated to move data between services and to store intermediate results. AstroGrid is complex system that can be roughly divided into 9 components. The components are implemented iteratively in phase in such a way that as early as possible the system is functional. Most of the components can be developed individually even though they work tightly together in the end stage. The crucial points of the architecture are as follows: Portal, Security, Messaging, Workflow, DataAccess, GridSpace, Tools and Community. /4/ Resource Registry This is the heart of the Virtual Observatory. All the resources and their descriptions are stored to the Resource Registry. Registry was made so that it contains as much information as possible of the available resources so maximum number of queries could be fulfilled without asking the resource. This sort of implementation is called a fine-grained registry. Resources can be discovered in multiple ways according to the metadata. In the future, the registry will be able to discover any resources relevant to a user's enquiry using inference engine. Community The Community component has the functionality of Globus’ Community Authorization Service and number of extra features. Community component provides the ability to specify and certify the groups which a person or resource may belong to on the grid. Each community will have the ability to define groups and members and assign rights to each. There are group of resources and groups of users. Resource providers define resource-groups and grant access on resource groups to appropriate user groups. Community is charge of user authorisation and authentication procedures for the resources. In the future, it will also define the look and feel of the Portal component. Community Privilege Server (CoPs) implements these features. Important reason for having communities and community right is the fairly big authorization task involved. There are around 10 000 users, 100 000 000 resources and tens of permission. Traditional approach would leave managing the access rights for the Data Centers, which in this case would paralyze the system. Workflow This service will be mainly used by the user interface programs. It is accessed via the web-based and PC-based ones. It controls job’s life cycle. It is used to create, edit, schedule, run and monitor jobs. A job template can be created which can be then run one or more times, possible changing the input variables. It supports running sequential jobs. Portal Portal is the web-access point to the system. The Portal is planned to be astronomer friendly meaning that deep knowledge of the system isn’t required for efficiently employing the system. Portal component communicates mostly with the workflow. Jobs are created in the Portal and then sent to the workflow. There is also PC-client access to the system. The interface is similar. The Portal provides access to a range of notification services. A user can elect to be informed about the progress of a job. Notification could be for example fired when final or intermediate results are available. DataAccess DataAccess is the second heart of the AstroGrid. Its purpose is to provide access to different media and hide the various interfaces from the system point of view. The service provides details of how a subset of data is to be selected and retrieved and will process queries to the dataset and return results. Result could be also a pointer if the result dataset is too big to be moved. The used access method for queries is AQL. It stands for Astronomical Query Language, which is better equipped to represent astronomical data and data relations than query languages designed for conventional data. Tools Tools are divided into two categories. There are predefined sets of algorithms for data mining purposes and data analysis tools. Mostly Tools provide the interfaces to for the third party to create specific data mining tools. All tools operate in a server environment. Security Though security is described as a component in the picture, it is an interface, which is implemented by other components. It is implemented in GridSpace, DataAccess, Workflow, Community and Portal. It provides common security routines for checking if a user has access rights for the components and outside resources that are being inquired. Messaging Messaging is interface, which Portal and Workflow components employ in their communication. GridSpace GridSpace allows the user to own space on the Virtual Observatory. The space can be distributed across several computers to enhance the performance of the queries and usage of disk space. From user point of view the GridSpace is one entity. All GridSpace objects have access rights, which can be set in a UNIX -fashion. The objects in GridSpace can be automatically and transparently transferred closer to data processing to enhance the data processing. This data transfer is implemented with the fastest available protocol, which in most cases is GridFTP. If GridFTP is not available then regular FTP is used. Phase B implementation The AstroGrid project is tightly scheduled though sticking to the schedules has been less successful. The milestones are set every three months. In every milestone, iterations should be completed. In the following iterations are viewed using half-year scales to make the list more practical. /3/ Iteration 2 (June 2003) Astronomers are able to log into portal, recall preferences and past activities, set preferences and search for registered resources. Dataset Access is on the level that astronomer will be able to select two catalogs and run simple join operation across them. Astronomer is able to reserve space at specific data centre and have results of query returned their using MySpace. Community administrator is able to create community on demonstration CoPs –server. Members can be registered and removed. Groups can be created and modified. Both group and member data access rights can be set. Data centre administrator will be able to create permissions for access to a resource for community, group and member. Iteration 4 (December 2003) Analysis tools can be used to run server-based analysis against datasets. CoPS Server is able to govern all access to data resources by community, group and member rights combined with data permissions Job Control can be used to manually construct set of sequential tasks, monitor progress of tasks and user can query job control for task progress and completion. MySpace is able to handle user can request space on number of distributed servers. The tasks will make use of scratch space on servers, moving results into MySpace area. User can make seamless queries on the distributed GridSpace. Data Mining is operational. It can perform complex queries, data can be loaded in optimal form and custom algorithms can be run against loaded data. Registry/Workflow demo is operational. Drag-and-drop workflow editor can be used to construct job. Workflow editor can query registry for details of resources. Iteration 6 (June 2004) Astronomer will be able to use Data Mining facilities for moving data resources to warehouse and running complex queries. Workflow editor will be integrated into portal. Jobs can be stored and rerun with different parameters. Additionally workflow editor will be able to detect if joined tasks are incompatible and possible suggest data conversion. Jobs can be visualised via server. Visualisation is still not integrated into workflow and portal. OGSA Integration will be on stage where system will be implemented on grid servers and all authorisation and permissions will be done via grid certificates. Iteration 8 (December 2004) User will be able to upload own algorithms to the data Tools. User can request run of complex algorithm on data in a data warehouse. Data warehouse is powerful computer system, which has good network connections, lot of disk space for additional datasets and local copies of the more popular astronomical datasets and fairly comprehensive set of standard astronomical data processing packages. Queries are optimized before being run. Job lengths can be estimated. Resource Registry has ontology-based metadata and inference engine to assist with queries. Visualisation of data will be integrated into workflow and Portal. User will be able to represent the result of his queries in visual form, which is based on the result data. OGSA Integration will be fully OGSA compliant. Challenge: metadata in grid datamining It has become common practice to associate with scientific datasets metadata that describes the contents and structure of that data. Metadata is used to optimize data transfer and as a way of transferring results between hosts. Different types of metadata Metadata can be dividing three different types according to its usage and content. The first type is application metadata. The metadata may describe the content represented by file, the circumstances under which data was obtained, and/or other information. It can be viewed as defining the logical structure or semantics that should apply to the uninterrupted bytes that make up a file instance or a set of file instances. The second type is replica metadata. It is used to manage replication of data objects since it includes information for mapping file instances to particular storage system locations. With this information, a replica manager can decide the best sources for any data it requires. The last type is usually known as system configuration metadata. It describes the fabric of the datagrid itself; network connectivity, details about storage systems, like capacity and usage policy. Metadata service Metadata service is a grid component that is concerned with the management of metadata. It provides a uniform means for naming, publishing, and accessing different types of metadata. Each type has its own characteristics in frequency, mechanism of update and its logical relationship to other grid components and data items. However, there is still a single interface for accessing all types. Applications identify files of interest by posing queries that specify the characteristics of the desired data to a metadata service that includes a metadata repository or catalog. The metadata repository associates such characteristics with logical files and entities with globally unique names that may have one or more physical instances. When metadata service has identified logical files with the desired attributes, replica manager uses replica metadata to locate the physical file instance to be accessed. The difficulty of specifying a general structure for all metadata is apparent when one considers the variety of approaches used to describe application metadata. Some applications build a metadata repository from a specified list of file instances based on data stored in self-describing formats. High-energy physics applications are successfully using a specialized indexing structure. The Digital Library community is developing sets of metadata for different fields. Other user communities are pursuing the use of XML to represent application metadata. The situation is further complicated when one considers the additional requirements imposed by large-scale data grid environments. Besides providing a means of integrating the different approaches to metadata storage and representation, the service must operate efficiently in a distributed environment. In order to be efficient, the metadata service must be structured as a hierarchical and distributed system that allows to achieve scalability, avoids any single point of failure and facilitates local control over data. However, distribution complicates efficient retrieval. This can be solved if data organization exploits the hierarchical nature of the metadata service. Replicas Replicas are usually the real data that metadata is used to refer. The replica manager is a service that manages distributing file. Its role is to create/delete copies of file instances, or replicas, within specified storage systems. A replica is a user-asserted correspondence between two physical files that is created because new storage location offers better performance or availability. It might be deleted because of storage need and it is assumed to be read-only. Where replicas are to be selected is based on access time, Grid information services can provide information about network performance, and perhaps the ability to reserve network bandwidth, while the metadata repository can provide information about the size of the file. Based on this, the selector can rank all of the existing replicas to determine which one will yield the fastest data access time. Alternatively, the selector can consult the same information sources to determine whether there is a storage system that would result in better performance if a replica was created on it. References /1/ AstroGrid: home http://www.astrogrid.org/ /2/ AstroGrid: project http://www.astrogrid.org/project /3/ RbGridTechnologyReport http://wiki.astrogrid.org/bin/view/Astrogrid/RbGridTechnologyReport /4/ AstroGrid: Architecture Overview http://wiki.astrogrid.org/bin/view/Astrogrid/RbArchitectureOverview /5/ AstroGrid: The Science Analysis Summary: defining key requirements for AstroGrid http://wiki.astrogrid.org/bin/view/Astrogrid/RbScienceRequirementsSummary