Minutes of the CCSM SSC Meeting

18 and 19 November 2003

NCAR, Boulder, CO


Attendees: Bill Collins (Chair), Chris Bretherton, Maurice Blackmon, Phil Merilees, Ping Chang, Scott Doney, Cecilia Bitz, Bill Large, Steve Reid, Jay Fein, Jim Hack, Gordon Bonan, Danny McKenna, Jim Carton, and Lydia Shiver


Guests: Frank Bryan, Dai Tsumune, Phil Rasch, Mike Alexander, Peter Thornton, Marat Khairoutdinov, Marika Holland, Natalie Mahowald, Byron Boville, Roland Garcia, Tony Craig, Peter Backlund, and via phone Bette Otto-Bliesner and Rick Smith


1. Welcome. Bill Collins welcomed Gordon Bonan, Ben Santer, and Jim Carton as new SSC members. He also welcomed Jeff Amthor as the new DOE representative, and thanked Ping Chang for his several years of service on the SSC and on the CCSM project.


2. Update on Earth Simulator and CCSM Collaborations. Collins reported that Tim Killeen was very positive about the CCSM working group's efforts to develop and use the next version of CCSM after hearing presentations from several of the Working Group Co-chairs at the CCSM Advisory Board meeting.


The issue of distribution of model data to the scientific community was discussed. It was decided that the CCSM Data Policy at http://www.cesm.ucar.edu/experiments/datapolicy.html would guide distribution. The policy states that CCSM output data will be available to members of any CCSM Working Group no later than 6 months following the conclusion of the simulation, and it will be released into the public domain without restriction as soon as a scientific paper on the results has been submitted by the simulation originator or 1 year after the date of creation, whichever comes sooner.


Collins will circulate a list of the data fields that will be saved and stored to the SSC and Working Group Co-chairs so that all fields needed are included. Collins stated that the IPCC runs done in Japan and U.S. will be run the same and viewed the same. The cost of saving and storing IPCC ensemble runs was another issue discussed.


Frank Bryan gave the SSC an update on the CCSM running on the ES by describing the MEXT Kyosei Program and the Kyosei Proposal submitted by CRIEPI/NCAR/LANL/Kyushu University consortium that funds the CCSM/CRIEPI collaboration and provides access to the ES. Bryan reported that work is well under way to port the CCSM component models to the ES, along with needed scripts and diagnostics. Once the porting is finalized, each component will be validated, and the coupled system will include exact restart testing, round-off evaluation, debug testing, 5-day runs with instantaneous history files, multiyear climate runs used as a comparison with baseline, and performance documentation.


The Kyosei project component of the CCSM IPCC runs to be carried out on the ES will emphasize long runs and will be complementary to the CCSM runs done in the U.S.


Bryan discussed the high-resolution modeling approach and what should be considered in choosing experiments to run on the ES. He said they should fit with the CCSM Science Plan and Kyosei Proposal objectives; exploit the initial software engineering and scientific investments that have been made; focus on a few projects leading to a unifying end point, have only modest manpower demands in Japan, and include scientific participation of CRIEPI scientists.


3. Science Team Meeting. Many guests (see list above) joined the SSC meeting to discuss the ES Science Team, its guidelines, and what science should be done other than IPCC scenarios. It was reported that the vectorization of CCSM is going well, and a fully coupled CCSM will start running on the ES in January 2004. The SSC will manage the ES Science Team and their experiments on the ES, and SCD will be a member of the Science Team because software engineering and data resources will be issues.


Boville discussed his plans for running WACCM (Whole Atmosphere Community Climate Model) as doing long control runs with present-day climate and long series of runs with forcings of the last 50 years to try to explain variability and ascertain if a WACCM-type model makes a difference. ACD is a collaborator on WACCM, and Rasch said that the AMWG is interested in seeing the WACCM results. Collins stated he wants WACCM to feed back for use into CCSM, and he requested an experiment outline with specific details from Boville. Collins also asked Boville for quantitative estimates of the amount of data that would need to be stored to help allocate funds for data media and storage. Boville stated that WACCM data would be publicly available.


Bryan reported that he is running a stand-alone ocean model on the ES now, and he wants to run the coupled model so that the ocean can make its own intrinsic variability and measure energy represented in eddies, and then the scientists can analyze and understand eddies and mixed layer at the mesoscale in the ocean to determine the effect of temperatures of eddies. The Ocean Climate Process Team (CPT) and the Ocean Model Working Group are also interested in using the ES for future experiments.


Rick Smith, LANL, has POP2.0 ready with a tripole grid and partial bottom cells, so that a coupled ocean-ice simulation can be done and then we could move on to a fully coupled simulation.


Hack stated that he is interested in using the ES to look at parameters as a function of resolution and the role of scale interactions on transient and mean state, and he also stated that daily diagnostics of the ES simulations would help to analyze the atmosphere component. The atmosphere and ocean components will need to be coordinated to do high-resolution simulations at the same time.


Other uses of the ES that were suggested are using superparamaterizations; running a 3D cloud resolving model; running superparamaterizations on top of the coupled system, but resources would be needed to vectorize the superparameters; and carbon cycle simulations. Bryan and Doney volunteered to discuss biogeochemistry experiments with CRIEPI scientists, since they are also interested in that field, and Bryan agreed to check with CRIEPI about their interest in doing biogeochemistry experiments on the ES. T. Dai, CRIEPI, stated that they are planning to buy a new supercomputer in 2005 that could be used to run biogeochemistry experiments.


Large stated that CCSM needs the criteria of the ES major review coming up in 2005, so we can do well at the review and probably obtain additional computing resources.


Collins will collect estimates of resources needed (software engineering, data, computational requirements, science) for future experiments on the ES and then circulate the information for discussion, including to Craig and Bettge. The Science Team will identify what scientific priorities should be set for the ES and send them to the SSC. It as also noted that the CCSM Working Groups need to be kept informed of what is being done on the ES and what the future plans are.


The BGCWG needs to define their computing resource needs, and Doney stated that he and Fung have committed to do the intermediate steps between CCSM1 and CCSM3 to bring the BGC component up to date with the other components.


Collins will request information from Bryan and Blackmon about what level of detail and information will be needed for the next MEXT proposal.


Collins will discuss responsibilities of monitoring day-to-day ES IPCC runs with the CCWG because someone must make sure that the ES and U.S. runs are the same.


4. Status of IPCC Version of CCSM. Collins reported that we are close to freezing the model, and a meeting on 1 December 2003 has been scheduled to make a decision so that the SSC can endorse a model configuration soon thereafter. The goals for IPCC are that a T85 Eulerian or equivalent finite-volume dycore be used to improve resolution for regional impact studies and to improve resolution for fidelity in coastal and stratus regions. The goal for CCSM is to have a single physics package for multiple dynamics and resolutions. The Eulerian dycore will be used for the standard IPCC scenario applications, and the finite-volume dycore will be used for future development and experimentation. For IPCC, CCSM will use T85 1x1, and CCSM will also support a T42 2x2.5 and T31 versions.


The SSC advised Collins to continue to run the CCSM3 version of the model with and without diurnal cycle until the 1 December meeting, and the metrics that will be used to decide which model will be used for IPCC are mean state, variability, and moving toward the most realistic physics. It was requested that difference plots from both models be available on 1 December to help with the decision.


The IPCC production schedule is at http://www.cgd.ucar.edu/ccr/ipcc/. All the model component's physics, datasets for the control runs for T31, T42, and T85, and technical documentation will be released to the community via NCAR and PCMDI in winter 2004.


5. Discussion of Future Plans for CCSM. Collins proposed calling the next version of the model CCSM3, and the SSC agreed with the proposal. The components will be CAM3, CLM3, CSIM4, and POP1.x. He also reported that a 0.5 FTE associate scientist was added to support the BGCWG, and two software engineers were added to work on CAM and portability of the model. We are working toward hiring a Paleo Working Group liaison, and DOE will fund a CCWG liaison for 2 years.


Further discussion was held about the contents of the public release of CCSM3. The SSC suggested that the release include a 500-year control run at T85 and 1000-year control runs for T42 and finite volume for present-day climate, datasets, pointers to control runs on the NCAR Mass Store, web-based documentation, and source code for vector and scalar configurations. In June or July 2004 or when the IPCC code repository deadline occurs, the IPCC runs and datasets will be released; this will give the CCWG time to analyze the runs and publish their results. It was suggested that Collins request a schedule of release dates for scenario runs from the CCWG.


The target platforms for CCSM3 are IBM, SGI, HP/CPQ, Cray X1, NEC SX6, and Linux (carefully described). Support of software and infrastructure and retention of output datasets were discussed. It was stated there is no NSF data retention policy, but some NSF programs require that data be sent to the national archive. The SSC decided that the policy for the lifetime of the software engineering and infrastructure (output datasets) of CCSM models must be 5 years including the coupled system and scripts. It was suggested that we request statistics from SCD on what is being used from earlier versions of the model, and tape storage is an option.


Collins stated there would be no major releases of CCSM in the next 4 to 5 years, but there probably would be intermediate releases.


Collins suggested the BGCWG produce a Technote to document the equations of their model using CCSM1 and note that CCSM1 is not supported.


6. Carton Presentation. Jim Carton discussed his ocean reanalysis activities and data assimilation project called SODA-POP (slab ocean model with data assimilation).


7. Update on Earth Simulator MOU. Backlund discussed the remaining issues of the MOU. The SSC endorsed the use of CCSM on the ES.


8. Another Special Issue of Journal of Climate? The SSC felt that a special issue was needed for CCSM3, and the goal of the issue would be to write an integrated description of the coupled system, what changed in each component model and the effects of the changes on the coupled system, and the impact of the additional computer nodes. Collins will contact Randall to confirm that Journal of Climate is still interested in a special issue for CCSM, and if so, what the deadlines need to be. Collins will draft a "critical papers list" and request authors to write the papers. This list will help the SSC know who will need access to the control runs and 1%/year CO2 simulations right away. Other papers than the ones identified as critical can be submitted as long as they meet the deadline imposed. The SSC will decide on a deadline date of 1 August or 1 September 2004.


9. Update on CCSM Potential Funding. Fein stated that the FY04 budget is still being deliberated. Fein does have a modest amount of funds for SGERs, visitors (moderate to long-term), and postdocs. He also stated he would try to build increases into CCSM's budget every year starting in FY05.


The SSC asked for clarification about the SGERs. It was agreed that Fein would add additional criteria on the web (http://www.cesm.ucar.edu/management/SGER_grants.html) that states the proposal must be linked to its relevance to CCSM and a final report must be submitted except if the proposal is from an NCAR scientist. All NCAR scientist's report their scientific research results in the Annual Scientific Report. The SSC asked that the results of the SGERs to date be discussed at the summer SSC meeting. SGERs are supposed to have fast turnaround, so Fein suggested a deadline for recommendations to be two weeks. The SSC requested that Shiver include a deadline date and send a reminder two days before the deadline for each SGER with "REMINDER" in the subject line. Another suggestion regarding SGERs was that the $25,000 amount be increased to keep up with inflation rather than more SGERs being funded.


10. CCSM Implementation Plan. Merilees discussed the draft CCSM Implementation Plan that was included in the SSC's notebook, and he requested that the SSC members provide feedback to him on the plan, and especially on how to fund and run a CCSM Visitor Program. It was strongly suggested that the Implementation Plan needed to be rewritten based upon present budget expectations.


11. Regional Climate Modeling and CCSM. Merilees reported that at the Joint Meeting of the CCSM Working Group Co-chairs on 29 August 2003 that the Working Group Co-chairs said they were interested in pursuing regional climate modeling (RCM) activities in CCSM. Jim McWilliams and Joe Tribbia joined the meeting to discuss the RCM project, stating that MMM's Weather Research and Forecast (WRF) model and a regional ocean model were being coupled, and that the CCSM radiation and land surface physics would be incorporated into WRF. The RCM project will carry out downscaling experiments using WRF and running CCSM at T85 in the background. A workshop on RCM/CCSM activities is also being planned in August 2004.


McWilliams and Tribbia said they wanted an endorsement from the SSC to continue collaborations with the CCSM and the RCM project, consultation help from the CCSM software engineering team, consultation help with the CCSM physics, and GAUs from CSL to support some experiments over the next 6 months. They estimated they would need 3000 GAUs/month for the next 6 months. All datasets from this project will be available to the CCSM community. Fein stated that CCSP will direct its interest into the regional climate area, but CCSM needs to be careful not to spread its resources too thin. Collins will talk to the Working Group Co-chairs and Merilees about consultation with the RCM project and GAUs request. The SSC endorsed the RCM project as a two- way coupling between global and regional models, but they also stated that the highest priorities right now for CCSM are IPCC, ES, and Journal of Climate. McWilliams stated that there would be a future need for software engineering support to complete the two way coupling.


12. Update on CCSM and ESMF. Craig updated the SSC on the implementation of CCSM using the Earth System Modeling Framework (ESMF). Max Suarez, NASA, joined the meeting at this time.


ESMF's goal is to increase software reuse, interoperability, ease of use, and performance portability in climate, weather, and data assimilation applications. ESMF is a superstructure that provides a framework for writing a "main" driver; coupling framework that provides a framework for writing couplers; coupling infrastructure that provides communication and regridding/interpolation/mapping; useful datatypes such as grids, fields, bundles, and arrays; and utilities such as time manager, LogErr, performance profiling, and input/output. ESMF enables modeling applications to be scalable, portable (consistent interface across computer architectures), exchangeable (among NASA/NOAA/DOE/NSF/university modeling centers), and flexibly coupled (hub and spokes/pairwise, sequential/concurrent). Platforms that are currently supported are IBM RS6000, SGI Origin, HP/CPQ S45, and Linux Intel, and testing is being done on Sun, several Linux platforms, and MAC. Future platforms that will be supported are NEC ES, NEC SX6, and Cray X1.


In November 2004 all the interoperability experiments will be complete, and all testbed applications, such as CCSM, must be compliant. For CCSM to be in ESMF compliance, a coupled CCSM demonstration using ESMF coupling at the atm/lnd/ocn/ice level of a non-production version of CCSM and using at least 3 ESMF utilities must be shown. Adoption of ESMF into the CCSM production system will not take place before early 2005 pending a formal evaluation by CCSM.


The new CCSM coupler, CPL6, has demonstrated good performance, the ability to change coupling fields easily, and coupling layer abstracted away from the individual components. The coupler's limitations are that a single executable has not yet been demonstrated and there is a lack of flexibility for changing/adding components and swapping packages, such as physics, dynamics, modules, etc.


ESMF provides solutions for future CCSM coupler goals, such as a single executable with mixed sequential and concurrent integration of models for flexibility and performance portability; a single CCSM model framework that supports "CAM stand-alone" model needs; better management of task/thread allocation and load balance; ability to add new components or change the number of components easily, nesting components, and adding more components such as the slab ocean model, dynamical cores, etc.; ability to modify frequency and number of coupling fields easily; on-the-fly generation of mapping weights; and performance portable coupling infrastructure and new robust utilities.

Craig reported that the CSEG wants ESMF to be successful because it will provide critical software capabilities that match well with CCSM goals/needs so that CCSM resources can be directed toward other capabilities, the development for the next version of the coupler will be led by NCAR and will use the best tools available to do better science, and only the appropriate and useful portion of ESMF will be adopted. Some resources will be needed in the ESMF effort, and Craig stated that it would be a good investment for CCSM.

13. Update on 9th Annual CCSM Workshop. Collins described the Santa Fe, New Mexico venue for the annual workshop, and he also reported that 3 of the 4 plenary speakers are confirmed. They are David Archer, Inez Fung, and Colin Prentice.

14. Scientist Candidates and Hiring Priorities. Collins reported that he and Merilees are involved in looking at all applicants to be able to identify candidates that are interested in CCSM-related research. He reported they found 3 applicants that fit into the CCSM Strategic Business Plan. Further information on this topic will be sent as the process evolves.


15. Status of SSC and Role of Software Engineering. Collins recommended to the SSC that one of the co-chairs of the SEWG be added as a member of the SSC because of the need for expert software engineering advice that is needed at the SSC level. The SSC endorsed adding Tony Craig to the SSC, and Collins will pursue this issue with Anthes, Killeen, and Fein. The SSC also requested that a balance be kept between NCAR and non-NCAR scientists on the SSC. Collins also recommended that the Code Review Boards be changed to Design Review Boards for each component model.


Collins again thanked Ping Chang for serving on the SSC and encouraged him to stay involved in the project. He also thanked Jim Carton for agreeing to serve and attending this meeting as an observer so that he could be aware of SSC issues.