From Idea to IRB: Implementing Educational Research in Your Program

The bottleneck in educational research is not bad ideas. It is sloppy execution and IRB paralysis.

You do not need a PhD in education to run solid studies in your program. You need a repeatable process. And the discipline to follow it.

Below is exactly how I walk busy clinician-educators from “we should study this” to “IRB-approved project enrolling next month.” Step by step, no theory fluff.

---

Step 1: Turn a Vague Idea into a Testable Question

Most “ideas” I hear in faculty lounges are not researchable questions. They are complaints.

• “Our residents hate M&M.”
• “No one reads the feedback we give.”
• “The new simulation course feels better.”

None of that is research-ready.

You need to force the idea through a basic structure. I like a stripped-down PICO for education:

• P (Population): Which learners, exactly?
• I (Intervention): What are you doing differently?
• C (Comparison): Compared to what?
• O (Outcome): How will you know it worked?

Example transformation:

• Vague: “Our morning report is not effective.”
• Structured: “Among internal medicine residents (P), does a case-based, faculty-facilitated morning report with pre-assigned roles (I), compared to our current unstructured, presenter-driven format (C), improve clinical reasoning scores on a standardized case-based assessment at 3 months (O)?”

Now you have something you can actually test.

Do this today:

1. Write your idea as one sentence.
2. Under it, write:
   • Population:
   • Intervention:
   • Comparison:
   • Outcome:
3. If any line is blank or fuzzy, you are not ready for IRB. Fix that first.

---

Step 2: Map Your Question to a Feasible Study Design

Do not jump to “randomized controlled trial.” You are not running a drug trial. You are running a program with limited time, messy schedules, and a coordinator who works 0.3 FTE.

Pick the simplest design that can answer your question honestly. For 80–90% of educational projects, that is one of the following:

Design Type	When To Use	Core Data Structure
Pre–post (single group)	Testing immediate effect of change	Before vs after scores
Cohort comparison	New vs prior curriculum	Group A vs Group B
Randomized groups	Two viable options, limited learners	Randomize to A or B
Cross-sectional survey	Attitudes, exposure description	One-time survey

1. Pre–post (single group)
You change something; you measure learners before and after.

• Pros: Easy, minimal ethics concerns, fast.
• Cons: Weak on causality; other things may change over time.

Suitable for:

• New workshop on feedback skills.
• New consultation curriculum.
• Added simulation session.

2. Cohort comparison (old vs new)
You compare the “before” cohort (last year’s residents) to the “after” cohort (this year’s) exposed to a new curriculum.

• Pros: Realistic in programs that iterate yearly.
• Cons: Cohorts differ for reasons unrelated to curriculum.

3. Randomized groups
You randomize learners to two versions of something (e.g., standard online module vs interactive module).

• Pros: Stronger evidence.
• Cons: Logistically harder; equity and fairness issues.

Only do this if:

• Both options are ethically acceptable.
• You can deliver both options consistently.
• You have enough learners (at least ~20–30 per arm for most basic outcomes).

4. Cross-sectional survey
Describes knowledge, attitudes, or self-reported behavior at one time point.

• Pros: Simple; good for early-stage exploration.
• Cons: Self-report bias, no causal claims.

If your question is really, “What is happening now?” not “What is the effect of X?”, then a survey may be enough.

Your decision rule:
If this is your first real project and you are clinically busy, pick either:

• Single-group pre–post; or
• Cross-sectional survey.

Anything more complex will die in your inbox.

---

Step 3: Choose Outcomes That Actually Matter (and That You Can Measure)

This is where most educational research gets weak. People pick outcomes based on what is convenient, not what is meaningful.

Use a stripped-down version of Kirkpatrick’s model:

• Level 1: Reaction – satisfaction, perceived usefulness
• Level 2: Learning – knowledge, skills, attitudes
• Level 3: Behavior – what they actually do in practice
• Level 4: Results/Impact – patient or system outcomes

Typical Outcome Levels in Educational Research

Category	Value
Reaction	80
Learning	60
Behavior	25
Results	10

(Made-up distribution, but that pattern is real. Most projects never leave Level 1–2.)

Here is the fix:

1. Aim for at least Level 2 for any intervention.
   If you change curriculum and only ask “Did you like it?”, that is a missed opportunity.

2. Pick the highest level you can measure without blowing up feasibility.

   • Behavior? Maybe you can track chart audits, observed encounters, or procedure logs.
   • Results? Maybe you can track exam pass rates or funnel metrics (e.g., time-to-disposition in ED simulations).

3. Use existing instruments when possible.
   Stop inventing surveys for everything. Look for:

   • Validated knowledge tests in your area.
   • Published checklists for procedures (e.g., central line insertion).
   • Standardized feedback tools (e.g., mini-CEX, OSTE checklists).

If you must create your own measure, do it like a grown-up:

• Write more items than you need.
• Pilot with 5–10 learners.
• Check for clarity and completion time.
• Kill confusing items.

---

Step 4: Build a Minimal, Coherent Methods Plan

Before you ever open the IRB portal, you need three things locked:

• Who you are including (and excluding)
• Exactly how you are collecting data
• How you will protect learners and their grades

I use a barebones, 1–2 page “methods sketch” with these headings:

1. Setting and participants

   • Program, level of learners, approximate N.
   • Inclusion criteria: “All PGY-1 residents in internal medicine, academic year 2024–2025.”
   • Exclusion criteria: “Residents who opt out of data use.”

2. Intervention
   Precise description of the educational change:

   • Duration (e.g., four 1-hour sessions).
   • Format (didactic, small group, simulation).
   • Who delivers it (faculty, fellows, peers).
   • Key content elements (so others could replicate).

3. Study design
   State it plainly: “single-group pre–post design with mixed methods” or “cross-sectional survey.”

4. Data collection procedures
   You must be painfully explicit, step by step:

   • What instruments (tests, surveys, checklists).
   • When they are administered (before/after/3 months).
   • How (online, paper, during mandatory session).
   • Who has access to identifiable data and when it gets de-identified.

5. Data analysis plan
   You do not need complex statistics. But you do need a clear, appropriate plan:

   • Pre–post numeric scores: paired t-test or Wilcoxon.
   • Two independent groups: t-test or Mann–Whitney.
   • Categorical data: chi-square.
   • Likert-type items: descriptive stats, maybe nonparametric tests.

If that sounds overwhelming, keep it simpler: “We will compare pre and post mean scores with paired tests and report effect sizes and 95% confidence intervals.” Good enough.

---

Step 5: Decide if You Need IRB – and What Type

A lot of educational projects get stuck here because faculty are guessing.

Here is the blunt truth: if you are planning to present or publish the results as research, you almost certainly need IRB involvement. Whether that ends up being “exempt” or “not human subjects research” is an IRB decision, not yours.

Typical educational projects fall into 3 buckets:

1. Quality improvement / routine program evaluation

   • Main goal is internal improvement.
   • No plans to generalize beyond local setting.
   • Often still submitted for a “not human subjects research” determination.

2. Exempt educational research

   • Conducted in established educational settings.
   • Uses normal educational practices.
   • Or involves anonymous or minimal-risk surveys.

3. Expedited or full review

   • More than minimal risk.
   • Sensitive data (e.g., mental health disclosure).
   • Power differentials with potential coercion.

IRB Pathway for Educational Projects

Step	Description
Step 1	Project Idea
Step 2	Q I or Program Eval
Step 3	Optional IRB consult
Step 4	Submit for Exempt Review
Step 5	Submit for Expedited or Full Review
Step 6	IRB Determination
Step 7	Will you publish or present?
Step 8	Normal educational practice and minimal risk

Your move:
Do not play amateur IRB. Email or call them early with a 1–2 paragraph summary. Use words like:

• “Minimal risk”
• “Established educational setting”
• “Voluntary participation”
• “No impact on grades or evaluation”

Ask explicitly: “Does this appear to qualify as exempt educational research, and what application type should I use?”

They would rather advise you early than fix a mess later.

---

Step 6: Build an IRB-Safe Plan to Protect Learners

Educational research is messy because of power dynamics. Learners depend on you for grades, letters, and jobs. That makes them a vulnerable population.

Your IRB will care about four things:

1. Voluntariness

   • Participation in the research portion must be voluntary.
   • Opting out must not affect grades, evaluation, or standing.

2. Confidentiality

   • Limit who sees identifiable data.
   • De-identify or code data as early as possible.
   • Store linking files separately, encrypted or in secure institutional drives.

3. Timing

   • Avoid surveying right after high-stakes evaluations by the same faculty, if possible.
   • Collect data during protected educational time, not off-hours demands.

4. Separation of roles
   Ideally:

   • The person grading learners is not the one holding the raw survey data with names.
   • A coordinator or non-evaluating faculty member manages consent and data.

Example of a clean structure:

• Residents complete a required quiz as part of the course.
• Grades are recorded in the LMS by faculty with names attached.
• For research, an administrator exports scores, removes names, assigns codes, and hands de-identified data to the research team.
• Consent for research use of quiz scores is obtained separately; those who decline are excluded from the dataset.

Write this structure into your IRB application explicitly. Do not make the IRB guess.

---

Step 7: Assemble Your Documents Before Opening the IRB Portal

This is where people waste weeks, clicking around an online system without the content ready.

Before you touch the portal, have these files drafted:

• Short protocol (3–6 pages) – based on your methods sketch.
• Survey instruments – in final or near-final form.
• Test or checklist – any assessments you will use.
• Recruitment email or script – clear, concise, IRB-friendly.
• Consent form or information sheet – depending on level of review.

Consent language that does not annoy IRBs

Your consent needs to hit the basics:

• Purpose: “to study the impact of X curriculum on Y outcome.”
• Procedures: what participants do, how long it takes.
• Risks: usually minimal for typical educational research (e.g., discomfort, time).
• Benefits: to learners, future cohorts, or education in general.
• Voluntariness and no impact on evaluations.
• Contacts for questions and concerns.

Stay away from promising direct benefit if there is none. Do not use marketing language.

A simple trick: Get a colleague’s old approved consent for a similar educational project from your institution, and adapt the structure.

---

Step 8: Submit to the IRB Like a Professional, Not a Grad Student

When you finally log into the IRB portal:

1. Pick the correct application type (often “Exempt” with educational research category).
2. Answer briefly and directly. Long, rambling responses get more questions.
3. Copy-paste from your protocol. Do not invent new descriptions that might conflict.
4. Flag the power dynamic issue and your mitigation upfront. It shows awareness.

Typical phrasing that works:

• “The principal investigator is a course director. To minimize coercion, participation in research components is voluntary, and a non-evaluating coordinator will manage recruitment and data. The course grade will not include research participation.”

Expect some back-and-forth. That is normal. Do not take it personally. Fix what they ask for, not five other things.

---

Step 9: Operationalize – Turn the Protocol into a Real Workflow

An IRB approval letter does not implement your project. You need an actual operational plan.

Use a ridiculously simple execution checklist:

1. Who sends what, when?

   • Calendar invites for sessions.
   • Survey links scheduled (email, LMS announcements).
   • Reminders for non-responders.

2. Where are the instruments housed?

   • Survey platform (REDCap, Qualtrics, SurveyMonkey).
   • Assessment tools embedded in LMS or simulation software.

3. Who owns the data pipeline?

   • Who exports the data?
   • Who de-identifies it?
   • Where is it stored?

4. Stop rules and protocol deviations

   • What happens if something changes mid-year? (e.g., pandemic, schedule change)
   • Define what counts as a minor vs major deviation and who you notify.

I have seen too many good IRB-approved projects die because no one actually owned the timeline. Assign roles explicitly.

---

Step 10: Plan the Analysis and Write-Up Before the Data Arrives

Backward-design your output. Decide now what your eventual abstract or paper will look like.

Sketch a one-page “fake” results section:

• Primary outcome: What do you hope to see?
• Secondary outcomes: What else might shift?
• Tables and figures you will probably include.

Example Pre-Post Knowledge Scores

Category	Value
Pre	62
Post	78

Maybe:

• Table 1: Learner characteristics.
• Table 2: Pre vs post scores with means, SDs, p-values, effect sizes.
• Figure 1: Change in confidence ratings by domain.

Why do this now?

Because you will discover:

• Which variables you must collect (but have not planned yet).
• Which comparisons are impossible without additional data (e.g., baseline characteristics).
• Whether you need identifiers temporarily to match pre and post data.

Fix those gaps before data collection starts, not after.

---

Common Pitfalls – And How to Avoid Them

I have watched hundreds of educational projects make the same avoidable errors:

1. Tiny, underpowered samples with complex statistics

   • If you have 12 residents, you do not need regression modeling.
   • Describe, compare with simple tests, and be honest about limitations.

2. No control or comparison group when one was available

   • If you can compare to prior cohorts, do it. Even with confounders, it is better than nothing.

3. Conflating program evaluation and research

   • Program eval: required, for improvement, often anonymous, may not require IRB.
   • Research: systematic, designed to produce generalizable knowledge. IRB territory.
   • You can run both, but keep them conceptually distinct.

4. Ignoring qualitative data

   • A small number of focus groups or open-ended survey questions can salvage a study with weak quantitative shifts by explaining the “why.”

5. Forgetting dissemination timing

   • Work backward from your target meeting submission deadline.
   • You need IRB approval and enough data collection completed several months before.

---

A Minimal, Realistic Timeline

For a typical, modest educational research project at a residency program:

Idea to IRB to Data Collection Timeline

Task	Details
Planning: Refine Question	a1, 2026-01-01, 2w
Planning: Methods Sketch	a2, after a1, 2w
Planning: Draft Instruments	a3, after a2, 3w
IRB: IRB Application Prep	b1, after a3, 2w
IRB: IRB Review and Revision	b2, 2026-02-15, 4w
Execution: Launch Intervention	c1, 2026-03-15, 4w
Execution: Data Collection	c2, after c1, 8w
Analysis: Clean and Analyze Data	d1, 2026-06-01, 4w
Analysis: Draft Abstract/Paper	d2, after d1, 4w

You can compress this, but only if:

• You already have instruments.
• Your IRB turnaround is fast.
• You keep the design simple.

---

If You Are Starting Tomorrow: A Concrete 30-Day Plan

Here is how I would move you from idea to IRB in one month.

Week 1: Clarify and Design

• Lock your PICO-style question.
• Pick the simplest viable design (ideally pre–post).
• Decide primary and secondary outcomes.

Week 2: Build Instruments and Methods Sketch

• Identify or build your survey/assessment.
• Draft a 1–2 page methods sketch with population, procedures, and analysis.
• Email IRB with a brief description and ask for category guidance.

Week 3: Draft IRB Materials

• Expand sketch into short protocol.
• Draft consent and recruitment text.
• Finalize instruments with colleagues’ feedback.

Week 4: Submit

• Enter IRB portal and complete application using your prepared text.
• Respond quickly to any queries or modifications requested.

If you stay disciplined, you will have IRB approval roughly 4–8 weeks later, depending on your institution. Then the real work—implementation and data collection—begins.

---

Key Takeaways

1. Most educational research dies from over-complex design and under-prepared IRB submissions. Keep it simple and plan before you click.
2. A clean, minimal protocol with clear protection of learners will get through IRB much faster than a bloated, vague one.
3. Backward-design from your eventual abstract: define your outcomes, data structure, and analysis now, then build the project around that.

---

FAQ

1. Do I really need IRB approval for routine course evaluations if I might publish them later?
If you have any intention of publishing or presenting beyond internal quality improvement, assume you need IRB involvement. Many institutions will categorize routine evaluations as program evaluation or “not human subjects research,” but that determination should be official. Submit a brief protocol or a determination request; do not try to retroactively declare something as research after you have the data.

2. What if my sample size is very small – is it still worth doing research?
Yes, if you are realistic. With small cohorts (10–30 learners), you are unlikely to produce definitive quantitative findings, but you can still generate valuable pilot data, rich qualitative insights, and detailed descriptions of educational innovations. Focus on clear effect sizes, confidence intervals, and strong narrative interpretation rather than chasing p-values. Many respected medical education journals publish small but well-executed studies, especially from single programs.

3. How do I protect residents from feeling coerced into participating?
Separate roles and make the protection explicit. Ideally, someone who does not evaluate the residents handles consent and data collection. Use language like, “Participation is voluntary and will not affect your grades, evaluations, or standing in the program.” Allow opt-out without penalty and without explanation. If participation happens during scheduled teaching time, make clear which parts are required for education and which are optional for research.

4. What if my IRB insists this is ‘quality improvement’ and not research, but I want to publish it?
Ask for a written determination letter or email stating the project is “not human subjects research” or is categorized as QI. Many journals now accept that documentation as sufficient. In your methods, describe the project as QI conducted in an educational setting, with IRB determination noted. If you later substantially change the project to be more systematic or generalizable, go back to the IRB with an updated description rather than assuming the original determination still applies.