Assessment Workshop, Plenary 2.pptx
Download as PPTX, PDF0 likes222 views
Professor Paul Maharg presented on assessment in legal education at a workshop of the Law Society of Ireland. The presentation included: 1. A taxonomy of task analysis for assessments in simulations, ranging from discrete tasks to assessments involving an entire case file and performative skills. 2. Results from a study on using simulated clients for interviewing assessments, which found it to be more reliable and valid than traditional assessment methods. 3. Ways that problem-based learning can be designed to ensure both breadth and depth of learning, and forms of assessment that align with problem-based learning methods like exams involving analysis of new or previously seen case studies.
Assessment Workshop, Plenary 2.pptx
- 1. Professor Paul Maharg paulmaharg.com/slides https://www.zotero.org/groups/4722732/lsi_assessment_workshop_27.6.22/library Law Society of Ireland Workshop on Assessment 27.6.22 Plenary 2 Zeugma legal education::technology:: writing::professional education
- 2. preview of workshop 2 Times Session 1000-1040 1. Introductions (10 mins) 2. Plenary 1: Research, ideas, applications (RIA): Framing research & concepts (20) 3. Brief discussion (10) 1040-1115 Breakout groups: Group discussions (20), plenary discussion (15) 1115-1130 Break 1130-1200 Recap on breakout issues. Plenary 2: Examples of the architecture of assessment 1200-1230 Breakout groups: Group discussions 1230-1300 Plenary discussion of ways forward & possible projects Zeugma legal education::technology:: writing::professional education
- 3. in this plenary – some framing research & concepts Zeugma legal education::technology:: writing::professional education ideas application research Some key questions: 1. Is the research out there? Is it relevant to my subject? 2. Are my ideas feasible re time, finance, effort? 3. Is the application practical? 4. Will the application improve professional learning? 5. Is it sustainable? 6. How will it fit programme learning? 7. How do I think students will remember my assessments?
- 4. 1. Task analysis 2. Assessment with Sim Clients 3. Assessment in professional PBL
- 5. 1. Task analysis of assessments in simulations
- 6. task analysis of assessments in simulations Here’s a taxonomy of task analysis, from simple to complex: 1. Discrete tasks (eg drafting, letter-writing, research) (Estate Planning) 2. Whole file + performative skill (Personal Injury) 3. Discrete tasks + whole file (Conveyancing) 4. Discrete tasks + file + performative skill (Litigation) Zeugma legal education::technology:: writing::professional education
- 7. • Set context (or not: let student figure that out – the clearing in the forest…) • Set task (but in how much detail? Supported with templates? Guidelines? Commented examples?) • Design feedforward (but don’t do the task for students) • Deadline a task (bearing all contextual factors in mind) • Task completed (and sent to staff in role) • Feedback on task (by staff in role) • Debrief (either in role or out of role) 1. discrete tasks Zeugma legal education::technology:: writing::professional education
- 8. • Holistic assessment of document chain • Bodies of evidence generally, but can embed critical points of assessment, eg report to client, speech plan, etc • Preparation for performative skill, including overlap with other skills – eg relation of legal research to professional negotiation. 2. whole file + performative skill Zeugma legal education::technology:: writing::professional education
- 9. • Specific tasks are the foreground, eg draft the completion certificate… • … but students must also complete entire file process. No completion, no competence. • Tasks may shadow tutorial work or precede tutorial work or neither • Quaere: How many attempts at each task? 3. tasks + whole file Zeugma legal education::technology:: writing::professional education
- 10. • Most complex, most authentic and most demanding • Potentially 1-3 plus more – eg performative skill can be assessed in role. 4. tasks + whole file + performative skill Zeugma legal education::technology:: writing::professional education
- 11. • Example: PI project: – PI mentor: passes information in real time; takes all fictional roles including PI senior partner (instructs, praises, warns), e-comm only: student responses are assessed – Surgery mentor: gives detailed feedforward on task, f2f, out of role: responses not assessed – Discussion forum: gives detailed feedforward on task, e-comm, out of role: responses not assessed – Practice Manager: gives coaching on firm experiences, in role: support & coaching not assessed, but the result is… use of interleaved learning support & assessment Zeugma legal education::technology:: writing::professional education
- 12. 2. Assessment with SCs
- 13. aims of our Simulated Client Initiative back in 2005 • Could we correlate interviewing assessments to the degree of client satisfaction & confidence arising from them? • ie answer the following questions… 1. Was our current system of teaching and assessing interviewing skills sufficiently reliable and valid? 2. Could the Simulated Patient method be translated successfully from medical education to the legal domain? 3. Was the method of Simulated Client training and assessment more reliable, valid and cost-effective than the current system? Zeugma legal education::technology:: writing::professional education
- 14. results from Strathclyde University pilot Questions Results 1 Was our current system of teaching and assessing interviewing skills sufficiently 1. reliable? 2. valid? 1. No 2. No 2 Could the Simulated Patient method be translated successfully to the legal domain? Yes 3 Was the method of Simulated Client training and assessment more 1. reliable, 2. valid 3. cost-effective than the current system? 1. Yes 2. Yes 3. Yes Zeugma legal education::technology:: writing::professional education
- 15. what changed…? • We made what the client thought important in the most salient way for the student: an assessment where most of the grade was given by the client • We did not conclude that all aspects of client interviewing could be assessed by SCs – We focused the assessment on aspects we believed could be accurately evaluated by non-lawyers – We focused the assessment on initial interview (which was been extended in various ways at other centres to other contexts) • This changed the way we enable students, trainees and lawyers to learn interviewing & client-facing ethical behaviour
- 16. current status of SCI University of Strathclyde Law School (Glasgow, Scotland) WS (Writers to the Signet) Society (Edinburgh, Scotland) University of New Hampshire Law School (Concord, NH, USA) The Australian National University College of Law (Canberra, Australia) Northumbria University Law School (Newcastle, England) Kwansei Gakuin University Law School (Osaka, Japan) Solicitors Regulation Authority - Qualifying Lawyer Transfer Scheme (QLTS) (London, England) Law Society of Ireland - Continuing Professional Development of Solicitors (Dublin, Ireland) Hong Kong University Faculty of Law (Hong Kong) National Centre for Skills in Social Care (London, England) The Chinese University of Hong Kong Faculty of Law (Hong Kong) Flinders Law School (Adelaide, South Australia) Nottingham Trent University Law School, (Nottingham, England) Osgoode Hall Law School + OPD (Toronto, ON, Canada) Canadian Centre for Professional Legal Education PREP programme for articling students (AB, MB, SK + NS) U of Windsor Faculty of Law (Windsor, ON, Canada)
- 17. independent studies… Gerkman, A., Harman, E., Bond, L., Sullivan, .M. (2015). Ahead of the Curve: Turning Law Students into Lawyers. A Study of the Daniel 17 Professor Paul Maharg | CC BY-NC-ND 2.5 CANADA • In focus groups, members of the profession and alumni said they believed that students who graduated from the program were a step ahead of new law school graduates; • When evaluated based on standardized client interviews, students in the program outperformed lawyers who had been admitted to practice within the last two years; and • The only significant predictor of standardized client interview performance was whether or not the interviewer participated in the
- 18. SCs: people as co-producers, co- designers The SC approach challenges: 1. Curriculum methods 2. Ethics of the client encounter 3. The cognitive poverty of conventional law school assessment 4. Law school as a monolithic construct 5. Law school categories of employment 6. The curricular isolation of clinic within many (not all) law schools 7. Hollowed-out skills rhetoric 8. Conventional forms of legal education regulation by regulatory bodies 9. The role of regulator: to be an encourager of innovation & radical reform…? 10.Disciplinary boundaries – what about a SC Unit that’s interdisciplinary? 11.Local jurisdictional practices: how might such a project work globally?
- 19. 3. Assessment in professional PBL ‘the key to expert problem-solving lies in how knowledge is organized, not the quantitative knowledge acquired’ (Lung 2008)
- 21. breadth of learning: what we focus on We: • Ensured we had breadth of learning as prescribed by regulatory and other codes (Priestley, AQF, CALD, etc) • Designed the new context of subject clusters in order to improve learning and assessment across the entire program • Used PBL as a heuristic to link courses, eg private and public law courses; pervasive ethics; linkages of substantive legal rules with sociolegal research and policy issues arising from the problem ‘trigger’ and student outcomes in first PBL sessions and review sessions Zeugma legal education::technology:: writing::professional education
- 22. depth of learning: what we focus on • deep learning through active learning. Students committed to being engaged from day 1 – PBL obliges them to do this. • Experiential process, including awareness of learning & knowledge, colleagues’ learning, spiral learning, self- management, ethics. • We focused students on dealing with sophistication and complexity and updating knowledge, as well as learning legal principles, leading cases, statutory knowledge, problem handling, etc. Zeugma legal education::technology:: writing::professional education
- 23. more depth vs more breadth, or both? On knowledge acquisition, Schmidt et al (2009) noted what many others observed: that PBL students better integrate their knowledge, which resulted in more accurate reasoning; that in the clinical case recall (a measure of expertise) and processing speed (a sign of better understanding) they were superior to the conventionally- educated cohorts (227). In skills acquisition, PBL students demonstrated much better interpersonal skills, and knowledge about skills (a variable closely related to skilled performance – 236). Student and expert perceptions of the quality of PBL education were higher than the results for the conventionally-educated cohorts, with students commenting positively in particular on their practices in independent study and critical thinking. In passing, Schmidt et al also noted that PBL schools graduate students faster and in larger numbers and retain students better (237). (Maharg 2015, 12-13) Zeugma legal education::technology:: writing::professional education
- 24. forms of PBL assessment • For specialist skills, OSCEs are well-suited • For knowledge in examinations: – Use the PBL learning method, eg provide: • A case worked on during the year, with different questions • An unseen case and require students to analyse it using the PBL method • A student response to a seen case study, and ask students to analyse its strengths and weaknesses Zeugma legal education::technology:: writing::professional education
- 25. forms of assessment in PBL • OSCEs for skills • Forms of exams: – Align with the learning method. So: • Use PBL cases seen in workshops, but with different questions • Use new PBL cases and require learners to analyse it according to PBL methods • Present a case + a (fictional) student response, and ask them to assess its strengths and weaknesses Zeugma legal education::technology:: writing::professional education