Skip to main content
SpringerLink
Log in

Validity evidence for the Fundamentals of Laparoscopic Surgery (FLS) program as an assessment tool: a systematic review

  • Published:
Surgical Endoscopy Aims and scope Submit manuscript

Abstract

Background

The Fundamentals of Laparoscopic Surgery (FLS) program uses five simulation stations (peg transfer, precision cutting, loop ligation, and suturing with extracorporeal and intracorporeal knot tying) to teach and assess laparoscopic surgery skills. We sought to summarize evidence regarding the validity of scores from the FLS assessment.

Methods

We systematically searched for studies evaluating the FLS as an assessment tool (last search update February 26, 2013). We classified validity evidence using the currently standard validity framework (content, response process, internal structure, relations with other variables, and consequences).

Results

From a pool of 11,628 studies, we identified 23 studies reporting validity evidence for FLS scores. Studies involved residents (n = 19), practicing physicians (n = 17), and medical students (n = 8), in specialties of general (n = 17), gynecologic (n = 4), urologic (n = 1), and veterinary (n = 1) surgery. Evidence was most common in the form of relations with other variables (n = 22, most often expert–novice differences). Only three studies reported internal structure evidence (inter-rater or inter-station reliability), two studies reported content evidence (i.e., derivation of assessment elements), and three studies reported consequences evidence (definition of pass/fail thresholds). Evidence nearly always supported the validity of FLS total scores. However, the loop ligation task lacks discriminatory ability.

Conclusion

Validity evidence confirms expected relations with other variables and acceptable inter-rater reliability, but other validity evidence is sparse. Given the high-stakes use of this assessment (required for board eligibility), we suggest that more validity evidence is required, especially to support its content (selection of tasks and scoring rubric) and the consequences (favorable and unfavorable impact) of assessment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  1. Derossis AM, Fried GM, Abrahamowicz M, Sigman HH, Barkun JS, Meakins JL (1998) Development of a model for training and evaluation of laparoscopic skills. Am J Surg 175:482–487

    Article  CAS  PubMed  Google Scholar 

  2. Fundamentals of laparoscopic surgery. http://www.flsprogram.org/

  3. Fried GM, Feldman LS, Vassiliou MC, Fraser SA, Stanbridge D, Ghitulescu G, Andrew CG (2004) Proving the value of simulation in laparoscopic surgery. Ann Surg 240:518–525

    Article  PubMed Central  PubMed  Google Scholar 

  4. ABS to Require ACLS, ATLS and FLS for General Surgery Certification. http://www.absurgery.org/default.jsp?news_newreqs

  5. Cook DA, Beckman TJ (2006) Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med 119(166):e7–e16. doi:10.1016/j.amjmed.2005.10.036

    PubMed  Google Scholar 

  6. Downing SM (2003) Validity: on meaningful interpretation of assessment data. Med Educ 37:830–837

    Article  PubMed  Google Scholar 

  7. Vassiliou MC, Dunkin BJ, Marks JM, Fried GM (2010) FLS and FES: comprehensive models of training and assessment. Surg Clin N Am 90:535–558. doi:10.1016/j.suc.2010.02.012

    Article  PubMed  Google Scholar 

  8. Messick S (1989) Validity. In: Linn RL (ed) Educational Measurement, 3rd edn. American Council on Education and Macmillan, New York, pp 13–103

    Google Scholar 

  9. American Educational Research Association—American Psychological Association & National Council on Measurement in Education (1999) Standards for educational and psychological testing. Washington, DC

  10. American Educational Research Association—American Psychological Association & National Council on Measurement in Education (2014) Standards for educational and psychological testing. Washington, DC

  11. Moher D, Liberati A, Tetzlaff J, Altman DG (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med 151(264–9):W64

    Google Scholar 

  12. Cook DA, Brydges R, Zendejas B, Hamstra SJ, Hatala R (2013) Technology-enhanced simulation to assess health professionals: a systematic review of validity evidence, research methods, and reporting quality. Acad Med 88:872–883. doi:10.1097/ACM.0b013e31828ffdcf

    Article  PubMed  Google Scholar 

  13. Cook DA, Zendejas B, Hamstra SJ, Hatala R, Brydges R (2014) What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment. Adv Health Sci Educ Theory Pract 19:233–250. doi:10.1007/s10459-013-9458-4

    Article  PubMed  Google Scholar 

  14. Cook DA, Hatala R, Brydges R, Zendejas B, Szostek JH, Wang AT, Erwin PJ, Hamstra SJ (2011) Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA 306:978–988

    CAS  PubMed  Google Scholar 

  15. Brydges R, Hatala R, Zendejas B, Erwin P, Cook D (2015) Linking Simulation-Based Educational Assessments and Patient-Related Outcomes: A Systematic Review and Meta-Analysis. Acad Med. 90(2):246–256. doi:10.1097/ACM.0000000000000549

    Article  PubMed  Google Scholar 

  16. Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33:159–174

    Article  CAS  PubMed  Google Scholar 

  17. Reed DA, Cook DA, Beckman TJ, Levine RB, Kern DE, Wright SM (2007) Association between funding and quality of published medical education research. JAMA 298:1002–1009. doi:10.1001/jama.298.9.1002

    Article  CAS  PubMed  Google Scholar 

  18. Fransson BA, Ragle CA (2010) Assessment of laparoscopic skills before and after simulation training with a canine abdominal model. J Am Vet Med Assoc 236:1079–1084. doi:10.2460/javma.236.10.1079

    Article  PubMed  Google Scholar 

  19. Fraser SA, Klassen DR, Feldman LS, Ghitulescu GA, Stanbridge D, Fried GM (2003) Evaluating laparoscopic skills: setting the pass/fail score for the MISTELS system. Surg Endosc 17:964–967. doi:10.1007/s00464-002-8828-4

    Article  CAS  PubMed  Google Scholar 

  20. Vassiliou MC, Ghitulescu GA, Feldman LS, Stanbridge D, Leffondré K, Sigman HH, Fried GM (2006) The MISTELS program to measure technical skill in laparoscopic surgery: evidence for reliability. Surg Endosc 20:744–747. doi:10.1007/s00464-005-3008-y

    Article  CAS  PubMed  Google Scholar 

  21. Ritter EM, Kindelan TW, Michael C, Pimentel EA, Bowyer MW (2007) Concurrent validity of augmented reality metrics applied to the fundamentals of laparoscopic surgery (FLS). Surg Endosc 21:1441–1445. doi:10.1007/s00464-007-9261-5

    Article  CAS  PubMed  Google Scholar 

  22. Sansregret A, Fried GM, Hasson H, Klassen D, Lagacé M, Gagnon R, Pooler S, Charlin B (2009) Choosing the right physical laparoscopic simulator? Comparison of LTS2000-ISM60 with MISTELS: validation, correlation, and user satisfaction. Am J Surg 197:258–265. doi:10.1016/j.amjsurg.2008.02.008

    Article  PubMed  Google Scholar 

  23. Cook DA (2014) Much ado about differences: Why expert-novice comparisons add little to the validity argument. Adv Health Sci Educ Theory Pract. [Epub ahead of print]

  24. Fried GM, Derossis AM, Bothwell J, Sigman HH (1999) Comparison of laparoscopic performance in vivo with performance measured in a laparoscopic simulator. Surg Endosc 13:1077–1081 discussion 1082

    Article  CAS  PubMed  Google Scholar 

  25. Kolkman W, Put MAJ, Wolterbeek R, Trimbos JBMZ, Jansen FW (2007) Laparoscopic skills simulator: construct validity and establishment of performance standards for residency training. Gynecol Surg 5:109–114. doi:10.1007/s10397-007-0345-y

    Article  Google Scholar 

  26. Yurko YY, Scerbo MW, Prabhu AS, Acker CE, Stefanidis D (2010) Higher mental workload is associated with poorer laparoscopic performance as measured by the NASA-TLX tool. Simul Healthc 5:267–271. doi:10.1097/SIH.0b013e3181e3f329

    Article  PubMed  Google Scholar 

  27. Stefanidis D, Scerbo MW, Korndorffer JR, Scott DJ (2007) Redefining simulator proficiency using automaticity theory. Am J Surg 193:502–506. doi:10.1016/j.amjsurg.2006.11.010

    Article  PubMed  Google Scholar 

  28. Feldman LS, Hagarty SE, Ghitulescu G, Stanbridge D, Fried GM (2004) Relationship between objective assessment of technical skills and subjective in-training evaluations in surgical residents. J Am Coll Surg 198:105–110. doi:10.1016/j.jamcollsurg.2003.08.020

    Article  PubMed  Google Scholar 

  29. Vassiliou MC, Feldman LS, Andrew CG, Bergman S, Leffondré K, Stanbridge D, Fried GM (2005) A global assessment tool for evaluation of intraoperative laparoscopic skills. Am J Surg 190:107–113. doi:10.1016/j.amjsurg.2005.04.004

    Article  PubMed  Google Scholar 

  30. McCluney AL, Vassiliou MC, Kaneva PA, Cao J, Stanbridge DD, Feldman LS, Fried GM (2007) FLS simulator performance predicts intraoperative laparoscopic skill. Surg Endosc 21:1991–1995. doi:10.1007/s00464-007-9451-1

    Article  CAS  PubMed  Google Scholar 

  31. Zheng B, Hur H-C, Johnson S, Swanström LL (2010) Validity of using fundamentals of laparoscopic surgery (FLS) program to assess laparoscopic competence for gynecologists. Surg Endosc 24:152–160. doi:10.1007/s00464-009-0539-7

    Article  PubMed  Google Scholar 

  32. Swanstrom LL, Fried GM, Hoffman KI, Soper NJ (2006) Beta test results of a new system assessing competence in laparoscopic surgery. J Am Coll Surg 202:62–69. doi:10.1016/j.jamcollsurg.2005.09.024

    Article  PubMed  Google Scholar 

  33. Sankaranarayanan G, Lin H, Arikatla VS, Mulcare M, Zhang L, Derevianko A, Lim R, Fobert D, Cao C, Schwaitzberg SD, Jones DB, De S (2010) Preliminary face and construct validation study of a virtual basic laparoscopic skill trainer. J Laparoendosc Adv Surg Tech A 20:153–157. doi:10.1089/lap.2009.0030

    Article  PubMed Central  PubMed  Google Scholar 

  34. Xeroulis G, Dubrowski A, Leslie K (2009) Simulation in laparoscopic surgery: a concurrent validity study for FLS. Surg Endosc 23:161–165. doi:10.1007/s00464-008-0120-9

    Article  PubMed  Google Scholar 

  35. Jayaraman S, Trejos AL, Naish MD, Lyle A, Patel RV, Schlachta CM (2011) Toward construct validity for a novel sensorized instrument-based minimally invasive surgery simulation system. Surg Endosc 25:1439–1445. doi:10.1007/s00464-010-1411-5

    Article  CAS  PubMed  Google Scholar 

  36. Stefanidis D, Hope WW, Scott DJ (2011) Robotic suturing on the FLS model possesses construct validity, is less physically demanding, and is favored by more surgeons compared with laparoscopy. Surg Endosc 25:2141–2146. doi:10.1007/s00464-010-1512-1

    Article  PubMed  Google Scholar 

  37. Azzie G, Gerstle JT, Nasr A, Lasko D, Green J, Henao O, Farcas M, Okrainec A (2011) Development and validation of a pediatric laparoscopic surgery simulator. J Pediatr Surg 46:897–903. doi:10.1016/j.jpedsurg.2011.02.026

    Article  PubMed  Google Scholar 

  38. Dauster B, Steinberg AP, Vassiliou MC, Bergman S, Stanbridge DD, Feldman LS, Fried GM (2005) Validity of the MISTELS simulator for laparoscopy training in urology. J Endourol 19:541–545. doi:10.1089/end.2005.19.541

    Article  PubMed  Google Scholar 

  39. Cook DA, Beckman TJ, Mandrekar JN, Pankratz VS (2010) Internal structure of mini-CEX scores for internal medicine residents: factor analysis and generalizability. Adv Health Sci Educ 15:633–645. doi:10.1007/s10459-010-9224-9

    Article  Google Scholar 

  40. Bloch R, Norman G (2012) Generalizability theory for the perplexed: a practical introduction and guide: AMEE guide no. 68. Med Teach 34:960–992. doi:10.3109/0142159X.2012.703791

    Article  PubMed  Google Scholar 

  41. Lijmer JG, Mol BW, Heisterkamp S, Bonsel GJ, Prins MH, van der Meulen JH, Bossuyt PM (1999) Empirical evidence of design-related bias in studies of diagnostic tests. JAMA 282:1061–1066

    Article  CAS  PubMed  Google Scholar 

  42. Nasca TJ, Philibert I, Brigham T, Flynn TC (2012) The next GME accreditation system—rationale and benefits. N Engl J Med 366:1051–1056. doi:10.1056/NEJMsr1200117

    Article  CAS  PubMed  Google Scholar 

  43. Cogbill TH, Ashley SW, Borman KR, Buyske J, Cofer JB, Deladisma AM (2014) The General Surgery Milestone Project. http://www.acgme.org/acgmeweb/tabid/150/ProgramandInstitutionalAccreditation/SurgicalSpecialties/Surgery.aspx. Accessed 27 May 2015

  44. Fundamentals of endoscopic surgery. http://www.fesprogram.org/. Accessed 5 May 2015

  45. Fundamental Use of Surgical Energy. http://www.fuseprogram.org/. Accessed 5 May 2015

  46. Fundamentals of robotic surgery. http://frsurgery.org/. Accessed 5 May 2015

  47. Ahmed K, Miskovic D, Darzi A, Athanasiou T, Hanna GB (2011) Observational tools for assessment of procedural skills: a systematic review. Am J Surg 202(469–480):e6. doi:10.1016/j.amjsurg.2010.10.020

    PubMed  Google Scholar 

  48. Avgerinos DV, Goodell KH, Waxberg S, Cao CGL, Schwaitzberg SD (2005) Comparison of the sensitivity of physical and virtual laparoscopic surgical training simulators to the users level of experience. Surg Endosc 19:1211–1215. doi:10.1007/s00464-004-8256-8

    Article  Google Scholar 

  49. Fichera A, Prachand V, Kives S, Levine R, Hasson H (2005) Physical Reality Simulation for Training of Laparoscopists in the 21st Century. A Multispecialty, Multi-institutional Study. JSLS 9:125–129

    PubMed Central  PubMed  Google Scholar 

Download references

Disclosure

Drs. Zendejas, Ruparel, and Cook have no conflicts of interest or financial ties to disclose.

Author information

Authors and Affiliations

  1. Department of Surgery, Mayo Clinic College of Medicine, Mayo 12-W, 200 First Street SW, Rochester, MN, 55905, USA

    Benjamin Zendejas & Raaj K. Ruparel

  2. Division of General Internal Medicine, Mayo Clinic College of Medicine, Rochester, MN, USA

    David A. Cook

  3. Multidisciplinary Simulation Center, Mayo Clinic College of Medicine, Rochester, MN, USA

    David A. Cook

Authors
  1. Benjamin Zendejas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Raaj K. Ruparel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. David A. Cook
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Benjamin Zendejas.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zendejas, B., Ruparel, R.K. & Cook, D.A. Validity evidence for the Fundamentals of Laparoscopic Surgery (FLS) program as an assessment tool: a systematic review. Surg Endosc 30, 512–520 (2016). https://doi.org/10.1007/s00464-015-4233-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00464-015-4233-7

Keywords

Search

Navigation