Documenting research software in engineering science

We want to investigate how research software is documented in a field where scientists usually don’t have a computer science background. Due to the highly disciplinary nature of research software, we focused on our discipline, Engineering Science. We conducted a multi-case study with a case-based approach¹¹. To see how the documentation is implemented, one specific research software was chosen: Neweul-M²¹², a research software that has been developed over years in an institute by engineers without formal software development training. Neweul-M² continues to be actively developed and is often used to address specific research questions. We cross-case synthesis with two other research software’s documentation habits, to compare the gained insights. We selected these best practice examples because they received funding from the DFG (German Research Foundation) sustainable software funding call to improve their documentation¹³. In contrast to Neweul-M², both best practice examples are open source. DuMu^x14 is a research software from engineering sciences, which is also programmed by Research Software Developers with an engineering background. The other example preCICE¹⁵ is a research software developed from more experienced Research Software Developers working in an informatics institute; their users are mainly engineers. The central rival hypotheses we considered are the lack of time to document and the lack of training of researchers in software engineering^2,6.

Two main research questions structure the investigation.

Research questions

1. RQ1

   What are the recommendations for documenting research software? Which rules and best practices exist? Do the given recommendations cover the defined categories?

2. RQ2

   What is the practice of documenting research software? How is research software documented in the daily life of researchers? Which workflows are implemented? What are the obstacles to document research software?

Data collection

For collecting data, we choose different sources of evidence:

• Documentation: An evaluation of literature was conducted to assess what recommendations are given (RQ1). Furthermore, the three research software documentation were evaluated (RQ2).

• Participant observation: Both authors are familiar with Neweul-M², one author from a new Research Software Developer perspective, and the other from years of experience. One author is part of the project from the sustainable software funding call and has thus witnessed the discussions about the possibilities for improvement and shortcomings of the documentation of DuMu^x (RQ2).

• Direct observations: The concepts, thoughts, and insights were further discussed with the old and later the new Research Software Engineer from Neweul-M² and with DuMu^x and preCICE Researcher Software Engineers (RQ2).

• Interviews: The two best practice examples were evaluated with semi-structured interviews (RQ1, RQ2).

Data analysis

Our first idea was to evaluate the research software using given recommendations from the literature. As we soon noticed, the recommendations for research software do not give a complete picture of what documentation should actually contain. Therefore, we switched to an inductive strategy and formed categories that we consider necessary from everyday work with research software, supplemented with categories from literature and internet resources like blogs and wikis. Moreover, we decided to include the best practice examples to answer the research questions. We defined four documentation categories for research software, intending to picture possible documentation forms. Based on the defined categories, we evaluated the recommendations given for, Neweul-M², DuMu^x, and preCICE. In the following, we introduce the categories, followed by the recommendations and conclude with the analyzed research software examples.

Categories

Domain Research software can belong to different domains¹⁶: private, shared and open. Usually, research software is developed in the private domain with one main Research Software Developer. The shared domain varies from a few users at an institute to many users all over the world, nevertheless the research software is unavailable to the broader public. Published research software in the open domain is accessible for everyone. Where open can have two different meanings: only the source code is available open source or the software is developed openly. The domains may change over time and require more documentation, as more people need to understand the research software.

Role As we noticed, it is essential who documents for whom; we differentiate between three roles: Research Software Engineer (RSE), Research Software Developer (RSD) and user. One person can have multiple roles, multiple people can share the role and the role of a person can change. As the roles in classical software engineering are conceptualized¹⁷, we defined the roles from our perspective—which is biased from our education as engineers and work in an interdisciplinary research cluster. When we speak about engineers, we think of the classic engineering fields such as mechanical engineering, civil engineering or chemical engineering. We explicitly neglect software engineers—due to their formal education in software development and maintenance, which is mostly missing in the other fields.

* Research Software Engineers are responsible (i) for the infrastructure and maintenance of the software, (ii) they give the rules of how research software should be written, (iii) but are often not part of the active feature development. The problems of funding, education and missing credit of Research Software Engineers are discussed in the RSE movement¹⁸.

* Users are (i) research software users who want to do either computer-aided engineering or computer-based experiments without writing code.

* Research Software Developers are (i) the link between Research Software Engineers and users, (ii) they develop new features, mainly to answer–with the research software–specific new research questions (iii) in engineering without education in software development and are often less experienced than Research Software Engineers. They typically need a specific answer for their research question, for which they need to implement a specific new or missing feature in existing research software. A typical example in Neweul-M2: A RSD implemented the calculation of reaction forces. This new feature can be reused for other research questions from other researchers and, therefore, need to be documented. RSDs are an essential part of the documentation process; they mainly know their developed features but are usually not deeply involved in the maintenance and documentation process.

Purpose The purposes describe the content of the documentation: why, what and how. The documentation of the problem should describe why the research software or a feature is written–similar to describing the research question, i.e. the RSE Manuscript/Dev docs row from Table 3. The feature’s documentation should describe what is needed to be done to solve the research question. How the feature is implemented should be documented in a technical documentation, i.e. the Help/Handbook/User Docs row from Table 3.

Type The type describes the characteristics of the documentation¹⁹: problem, product and technology. The three above introduced categories can be expressed in different types of documentation: The documentation of the problem can involve how the problem is implemented and why a solution was preferred. The product documentation contains the list of all features provided by the software and how they work together. The technical documentation should help the RSD and RSE to understand the code, how the research software is engineered and how to build over the existing source code. It should contain different schemas about the used model, the logical, and physical architecture. The types are intended to be umbrella terms for different forms of documentation types. For example, code comments are a form of technical documentation or tutorials as a type of product description.

We assume that each of these categories requires a different type of documentation. In each domain, researchers can document purposes in their role as RSE or RSD for different roles. The combination leads to several types of content, which then appear in a variety of forms. For example, a RSD can describe why they solved what and how in a problem description such as an article. Or a RSE describes how to solve a problem for the user in a product description as a how-to-guide.

Different aspects of documentation.

Recommendations

Aspects of good research software, and its documentation, has also been addressed in various recommendations. In order to find recommendations for documenting (research) software, we conducted a literature review in Web of Science using the terms “research software” and “documentation” or “reusability”. Most articles refer to the whole process of developing research software, and not only to documentation. Often just one small paragraph is dedicated to documentation. The selection of the articles was limited to those that include rules or best practices for documenting research software in at least one paragraph. Ten articles with interdisciplinary and different disciplinary focus were found. As described above, the evaluation of the recommendations did not provide a complete picture of what the documentation in our opinion should contain. Therefore, we also analysed the recommendations according to the categories we defined (Fig. 1).

Analysis of research software

Neweul-M² Neweul-M² is a software package that allows the dynamic analysis of mechanical systems in calculating multibody systems with symbolical equations²⁰. The first version of Neweul was written in FORTRAN with an own symbolic formula manipulator engine in the mid 1970s and was rewritten in 2003 using MATLAB. The new version is called Neweul-M². In Kurz et al.¹², the history and changes are documented until the year 2010 (for further information see Table 1). Neweul-M² is used from:

The source code is developed and administrated by PhD students within the developing institute, they aim for a degree in mechanical engineering and usually don’t have a formal software development education. One experienced RSD is the RSE, a new colleague is briefed as RSE from the previous one.

For the external people and students, a content-obscure (P-code) version of Neweul-M² is provided. One part of the documentation is in an integrated help within MATLAB. The help includes a product description, tutorials and examples and a function reference, automatically generated from the code. For PhD students from the developing institute, the full source code is accessible. The source code is managed via a Git repository hosted at an institutional GitLab instance. Bug fixes and support are the responsibility of the RSE. Another part of the documentation is done in a local wiki with information on how to get started and how to document with coding guidelines, tests and checklists. Decisions and discussions are documented via GitLab. For the RSE, an additional document gives information on how everything is organized. PhD students, who use Neweul-M² for their research, develop new features in Neweul-M² that they need for their research. They document these features mainly in publications.

DuMu^x The research code for the free and open-source simulator is written in C++ and is based on DUNE (Distributed and Unified Numerics Environment). DuMu^x stands for “DUNE for Multi-{Phase, Component, Scale, Physics, …} flow and transport in porous media²¹. The main intention is to provide a sustainable and consistent framework for implementing and applying of porous media model concepts and constitutive relations (for further information see Table 1). All documentation is linked on the Website. The documentation consists of a collection of documented code examples within the institute’s publicly accessible GitLab instance, a manual in PDF format, code documentation within Doxygen, a reference to the most important publications and a wiki that is still under construction. The software is written by PhD candidates in civil engineering with a predominantly engineering background, who have taught themselves to program.

preCICE The research software preCICE²² is an open-source coupling library for partitioned multi-physics simulations, including fluid-structure interaction and conjugate heat transfer simulations. The research is about methods how two systems can be coupled (for further information see Table 1). In preCICE the research question is not solved with the software, rather the research results are provided to users of the software. For this reason, there are only RSEs and users. The software is written by PhD candidates with different backgrounds, who are aiming for a degree in computer science. The documentation is bundled in a website. By using GitHub pages, pull requests can be made to all the documentation. The website is divided in quick start, docs, tutorials, community and blog. The section docs start with a user documentation with fundamentals, installation, configuration, tooling and provided adapters. The API is described in the category “couple your code”. Followed by a developer documentation, with a link to the source code documentation in Doxygen, and a description of coding conventions, tooling, workflow and testing. Even a description, how the documentation is build, exists. For the users–in addition to the conventional documentation–a community page gives insights on workshops, other contributors and publications. Furthermore, there is a blog where there is also the possibility to ask questions.

Validity analysis

We have deliberately chosen projects in which we can gain a more in-depth insight. These projects are in the engineering field to establish comparability of the training background of RSEs, RSDs and users. The interview partners are based on personal relationships and recommendations, which could lead to a specific research bias due to the small sample size and the personal connection. More extensive studies based on the research hypothesis of this study should be conducted in the future, e.g. at research software conferences. In order to validate our conclusions, we presented and discussed our results and methods with the RSEs and RSDs of the three software projects. We also presented a poster at the internal SimTech conference to receive feedback on our method and conclusions from other researchers. The feedback received confirmed our approach. The poster and other material is published in the case study database²³. The selection of the case studies initially limits the generalizability of the results. However, the feedback received confirmed that our approach could be transferable to other research software projects. For example, PhD students at the conference confirmed that our approach is similar to their experiences with research software. The generalizability of the findings obtained in this study will be tested in another larger interview study with more extensive surveys in the future.