Find answers to common questions that may arise during the effort to prepare and submit an AD or AE appendix to the SC Conference. If your question is not addressed here, please contact us: [email protected].
AD and AE Appendix Requirements
Impact of AD and AE Appendices
ACM Artifacts Available and Artifacts Evaluated Badges
Q. Are the artifact description and artifact evaluation appendices required in order to submit to SC18?
A. No. These appendices are not required. If you do not submit any appendix, it will not disqualify your submission. However, if two submissions are otherwise comparable in quality, the existence and quality of appendices may be a factor in ranking one over the other.
Q. Is the artifact description appendix required in order to be considered for Best Paper or Best Student Paper at SC18?
A. Yes. If you want your paper to be considered for the Best Paper or Best Student Paper awards, your submission must include an AD appendix.
**Q. Is the artifact evaluation (formerly CRA) appendix required in order to be considered for the Best Paper or Best Student Paper at SC18?
A. No. The AE appendix is completely optional.
Q. Do I need to make my software open source in order to complete the AD appendix?
A. No. You are not required to make any changes to your computing environment in order to complete the appendix. The AD appendix is meant to provide information about the computing environment you used to produce your results, thereby reducing barriers for future replication of your results. However, in order to be eligible for the ACM Artifacts Available badge (see below), your software must be downloadable by anyone without restriction.
Q. Who will review my appendices?
A. The AD and/or AE appendices will be submitted at the same time as your paper and will be reviewed as part of the standard review process by the same individuals who review the rest of your paper.
Q. Will the information in these appendices reveal an author’s identity, thus interfering with the double-blind review process?
A. It is possible that information provided in an appendix could reveal some information about an author’s identity, but this is most likely no more revealing than self-citations in the bibliography. Listing the software you used does not necessarily mean you developed the software. As with self-references in your text, you should refer to software from a third-person perspective. If you think any particular piece of information requested in an appendix could unambiguously identify you, you may state: “Information will be provided if and when the paper is accepted.” Finally, if you have any unaddressed concerns about completing appendices for SC18, please contact us: [email protected].
Q. Does the artifact description appendix really impact scientific reproducibility?
A. The AD appendix is simply a description of the computing environment used to produce the results presented. By itself, this appendix does not directly improve scientific reproducibility. However, if this artifact is done well, it can be used by scientists (including the authors at a later date) to more easily replicate and build upon these results. Therefore, the AD appendix can reduce barriers and costs of replicating published results. It is an important first step toward full scientific reproducibility.
Q. Does the artifact evaluation appendix really impact scientific reproducibility?
A. A thorough artifact evaluation effort is an effective way to increase the trustworthiness of computational results from “boutique” platforms. Leadership computing platforms, novel testbeds, and experimental computing environments are of keen interest to the supercomputing community. At the same time, these platforms are often volatile and possess a higher risk of unanticipated behavior.
Furthermore, access to these systems is typically limited, making it nearly impossible for most reviewers to independently compute author results. Finally, the volatility of these platforms often makes it hard for the authors themselves to recompute their own results in the future, since changes in the environment (compilers, libraries, components, etc.) impact computational results, and there is no way to revert to previous system states.
For all of these reasons, the introduction of upstream setup testing, peri-execution time testing and post-execution analysis can improve confidence that computational results from these special platforms are correct.
Q. The Artifacts Available badge description on the ACM website states: “Personal web pages are not acceptable for this purpose.” I do not have free access to any repositories aside from my personal website. Does SC provide a permanent repository?
A. No. There are numerous source hosting platforms that provide free or inexpensive hosting. You should use one of these.
Q. Do I get any recognition for submitting an AD and/or AE appendix?
A. ACM Artifacts Available badge: An accepted paper with a fully completed artifact description appendix is eligible for the ACM Artifacts Available badge if the artifacts listed in the appendix are downloadable by anyone without restriction.
ACM Artifacts Evaluated -- Functional badge: An accepted paper with fully completed artifact description and artifact evaluation appendices is further eligible for the ACM Artifacts Evaluated -- Functional badge if its artifacts are downloadable and positively evaluated by reviewers.
ACM Replicated Computational Results badge: An accepted paper that contains an AD appendix will be considered for inclusion in the SC Student Cluster Competition (SCC), where student teams will attempt to replicate the results presented in the paper. Papers will be selected based on their suitability for use in the SCC. Papers selected for the SCC will receive the ACM Results Replicated badge.
Please see the ACM Artifact Review and Badging page for details.