As part of an effort to continually improve CCSM software process, the following sections describe a proposal of steps to implement that, if done in a coordinated way, will improve CCSM software quality and control. To a large degree, these policies are general and should be applied to all CCSM components as well as to the CCSM model as a whole. It is recognized that individual components have different needs and concerns, and that slight variations in policy may be required for each component. In addition, there may be distinctions required between scientific development and software engineering development. However, there shouldn't be any distinctions between internal and external developers.
There is recognition that CCSM is fundamentally a research project and that individuality is an important part of the way people work. There also must be recognition that CCSM is a project that has a mandate to release a stable, high-quality software product to the community. This plan attempts to balance the needs of the individual developer with the needs of the overall community. Hopefully the developer will not feel significant changes in the way they work, but the release product quality will be higher. Many of the new processes implement additional controls between what is considered development code versus what is considered production code.
The recommendations in this document strive to improve the software quality of CCSM and make people in the project more productive, which will help achieve the ultimate CCSM goal, to do better science.
As the current time, an atmosphere CRB has been formed. This group has been meeting at irregular intervals mainly to address CCSM/CAM release issues. Some time has been spent outlining a CRB process, but this is still not well defined. It is recommended that the atmosphere CRB meet regularly; first, to outline a CRB process and role, and second, to carry out this process. It is recommended that the CRB process include:
These items are very important to assess the impact of changes, understand the potential for hidden bugs and unintended interactions, and coordinate and prioritize the work.
At the same time, a formal CRB should be formed to address CCSM issues. In particular, there is currently lack of coordination with regard to cross component interactions. The CCSM scientists and programmers meetings have served as an arena to discuss cross component interactions, but there is currently no responsibility on any person or group to coordinate these activities.
It is recommended that CRBs be created for all CCSM component model groups. However, before doing so, the benefit and cost of forming additional CRBs for the ocean, ice, land, and coupler components should be considered.
Over the past several months, a CVS policy and access request procedure has been put in place, largely formulated out of this document dated Oct 25, 2001. As this process moves forward, the policy and request procedure should be updated when necessary to address individual concerns. There are still open issues related to security in the CVS framework and tools should be built or local modifications to CVS made to ensure CVS is being used properly and that individuals are following policy. The CSEG group is currently investigating the use of CVS import/export as a way to instantiate multiple copies of the CCSM repository as a means of providing enhanced security. This work should continue. The CSEG group is also looking into improving the CVS access request and status capabilities mainly by implementing a web based form and database tool. This work should continue.
CCSM should continue to use CVS as a tool for source code control. CVS is free and CCSM developers have become experienced and comfortable with it. Evaluating, choosing and spinning up a new source control tool would require a committment of at least one person full time for several months for tasks like evaluation, testing, and training. The risks and cost are significant in terms of potential resources required, potential financial cost, and/or potential loss of productivity during the transition period. In the background, CCSM should look into the availability of other tools in conjuction with outside collaborators. There should be an ongoing community effort to evalute new (free and commericial) software tools, so if a new tool becomes available and recommended, CCSM could migrate to it.
An individual has been tasked with developing, running, and maintaining testing software for CCSM on a full-time basis, largely as a result of this document dated Oct 25, 2001. The CSEG CCSM test engineer is developing test procedures and tools to validate both release and development versions of CCSM. Ultimately, the goal is to have a suite of tests that are used by the community to accomplish testing tasks, such as weekly tests of released software on all validated machines, daily ``build and smoke'' tests of software under development, and regression testing of source code changes. These testing capabilities must be made available to the community as soon as possible in order to increase the quality and decrease the risk associated with many developers working on the models simultaneously.
Status accounting is the ability to understand what changes have been implemented, what changes are proposed, what defects have been discovered, who is working on these, and the status of each. Over the last 6 months, a ccsm-bugs group has been added to the CGD wreq system and is being used to track CCSM bugs. While this is not ideal, it is a significant improvement over previous actions.
In the future, a better software tool to document the CCSM status, bug tracking, and testing status needs to be found. There are several free, publicly available tools for status accounting and tracking, including GNATS and Jitterbug. Someone should be given responsibility and time to explore and implement one of these tools for use in CCSM. The accounting tool could be an integral part of the change review boards and vice versa.
Documents are important for good communication as well as project tracking. Documents include things like task lists, meeting notes and web pages. There are several documents that need to be developed or improved. Some, like the CVS policy statement, planning documents, and a status accounting document have been discussed above and implemented. The CCSM developers guide, which also needs to be kept current and more complete, serves as the guide for all CCSM development. An individual should be put in charge of coordinating documents, including the developers guide, to assure they remain up to date. It is also recommended that a review of the CCSM public and local web pages be undertaken in an attempt to clarify and improve document visibility.
Because of the size and importance of CCSM, CCSM development needs to become better focused on stability and quality than it is now. Better planning is required not only for CCSM science and software engineering, but for infrastructure support as well. CCSM needs to improve short-term, long-term, and strategic planning, and a process should be put in place to address this task. There is little short- or long-term planning now, and what's done is not formally recorded. CCSM software engineering strategic planning does occur now to some degree. There is a ``CCSM Software Engineering Plan 2000-2005'' that was prepared largely by the software engineering working group that addresses many strategic planning issues. This document is currently under review and is being revised. Part of the planning process on all time scales should include improved cost/risk/benefit analysis to set priorities to make best use of all resources.
Some of the planning issues will be solved if change review boards are created. However, each component and the CCSM as a whole should regularly outline some long-term goals, what individual models are going to accomplish in time; when code is going to be frozen; how much time is going to be set aside for testing, porting, validation, tuning, and documentation; and when new code is going to be released. The goals need to be realistic and the planning documents should include schedules, milestone targets, and priorities. In addition, longer-term planning should be used to prioritize issues like model performance, base code rewrites, infrastructure support, and new physics.
It is recommended that a formal planning process take place before any work is done on the next major CCSM release (CCSM2.1?), outlining goals and schedules.
In order to improve our ability to plan changes and estimate schedules, regular reviews of progress should occur. Does CCSM meet short-term goals on schedule? Are long term-goals met? How accurate are the scheduling estimates, and are there particular issues that come up regularly that cause delays? Reviews should become a standard part of CCSM software engineering process.
Many of the recommendations in this document require a stronger management structure. There needs to be agreement within the community that these policies will be used and enforced. The CCSM scientific coordinator and the SSC need to be in agreement with these policies, and they need to make sure processes and policy are understood. There also needs to be a oversight process where individuals can be disciplined if they violate policy. The goal is to better control quality and improve productivity, while still allowing individuals to carry out independent, creative scientific and software development.
Over the last year, there have been a number of useful training activities including formal book discussions and local training by an external company. These activities have focused on improving the group's understanding of software engineering process. It is now time to review what has been learned and use some of that new knowledge. In particular, the action items outlined in the Construx training need to be revisited.
There is currently a project to improve the CCSM GUI and scripts that will attempt to take advantage of these newly learned skills. Currently, the project is in the requirements collection phase. The project will move through an architecture and design phase before beginning implementation. It is recommended that new projects also adopt some formal software engineering processes in order to improve quality and productivity.
Training opportunities should continue to be made available to CCSM software engineers in areas related to procedure, process, project management, and technical skills.
There are a number of other issues that are worth bringing up as a part of this discusssion.
A code review policy should be implemented. All component code changes should be reviewed before merging onto the main trunk and/or before release to CCSM. Another final review is probably required of CCSM code that is to be formally released to the public. The CRB process will handle some of the reviews, but maybe there needs to be more formal review upstream of the CRB.
Model output control and access needs to be better defined. A policy should be put in place with regards to data storage, naming conventions, data locality, data compression, and access. A separate data management group should be formed to deal with this issue immediately, and it should include members from the CCSM community, especially the Climate Change and Paleoclimate working groups. Along the same lines, standard analysis plots need to be better defined and available, and questions regarding analysis and plotting tools needs to be addressed. Some progress has been made on these issues in the last six months, particularly with regard to naming conventions and standardization of model analysis plots, but there is still a deficiency in procedures related to model output and analysis.
The role of a computer scientist in the overall strategic planning of CCSM software engineering should be investigated.