There is a large push to utilize evidence-based practices in medical education. At the same time, credentialing bodies are evaluating the use of simulation technologies to assess the competency and safety of its practitioners. At the 2008 Academic Emergency Medicine Consensus Conference on "The Science of Simulation in Healthcare," our breakout session critically evaluated several issues important to the use of simulation in emergency physician (EP) assessment. In this article, we discuss five topics felt to be most critical to simulation-based assessment (SBA). We then offer more specific research questions that would help to define and implement a SBA program in emergency medicine (EM).