Jenny Motioned to Approve the Notes from Last Week and Mark Seconded

Jenny motioned to approve the notes from last week and Mark seconded.

Jenny introduced her draft for guidelines for developing new AACR questions. She asked to identify a group of people who would like to regularly meet over the summer to work on the first two points on the outline: 1) Identify content and 2) Outline content in such a way that is clear what the difficulties are. The outlining step is important because it may not be clear at first where the difficulties for students are. The third point focuses on how to modify multiple-choice questions: leave out jargon; make sure the question isn’t answerable with one word, etc. The rest of the outline, steps 4 – 10, focuses on how to get feedback on questions and get them ready for piloting.

Jenny then went through two questions that had been modified previously. Jeremy had worked on the origin of alleles question and found that there wasn’t much of a difference in how students answered it for animals vs. bacteria. Also, the incorrect ideas students gave talked about how alleles changed, not how they arose, so they weren’t even answering the question that was asked. This made it difficult to analyze if students had misconceptions about origins of alleles because all the students discussed were alleles changing over time. Jeremy changed the question to explicity ask about how a new wing pattern arose in moths and then how that wing pattern could change over time. Moths

John approved of Jenny’s outline, although he offered that we should call it something different than “question development cycle” because the QDC group already exists and they focus primarily on lexical analysis. Perhaps “question construction” would be a better name for this working group. Jenny agreed and is happy to call it question construction.

Mark suggested that point 9 from the outline (develop scoring rubric) could be facilitated by some of the lexical analysis. According to Jenny, steps 8 – 10 should involve the QDC group since they are knowledgeable about the software and computer analysis. Jenny wanted this document to focus more on the role that the biologists would be playing (steps 1 – 7). Mark asked what is the most effective way to get through the process. Perhaps it’s ok to just develop a question and then ask it in a classroom and analyze their answers without doing preliminary student interviews? Jenny disagreed; she thinks it is worth it to interview students because we can get information from them that we might otherwise miss. Even talking to just a couple of students will help. Mark agrees with that and thinks this is a reasonable approach to question development.

John offered that this proposal is actually part of the question development cycle; maybe this process is a sub-category and not a separate process entirely.

Mark liked the moths question and asked if students had been interviewed. Jenny said a previous version was interviewed on and he found from those responses students tended to conflate origin of alleles and change over time. This led to his new question stem. Jenny has hired an undergraduate to conduct interviews with students over the summer on this question and any others.

Michelle asked if they had thought about using a different species, since moths are an example that shows up in most biology textbooks. Jenny said that yes, he had thought about it and interviewed some students using questions from his activity (i.e., what causes a cheetah to start running fast). They did make an effort to make the question more general and not cue the recall of moths changing from white to black in response to the environment.

Ross said it’s important to mention the level that students are at when considering the construction of new questions.If Ross gave this question to his intro bio students, he is concerned that the question would give them the answer. ‘Frequency change’ is a documented misconception. By including frequency, he is concerned that the student misconceptions wouldn’t be elicited, but rather it would cue them to focus their response and use reasoning they might not otherwise use. Since ‘frequency’ is a technical term, Ross and John suggest using different language so we don’t cue them to answer in a certain way but rather see how they answer on their own. Jenny said that absolutely, when we have our team working on new questions, one of the ideas to keep in mind is what we know students either don’t know or how they might tend to answer. We won’t be able to flesh it out for every question/topic but the question development teams should keep in mind the ideas students will express and/or the models they will use to answer.

Ross said that in the design principles, we should pay attention to framing and cueing of scientific ideas. We know that certain language can cue students towards using a specific model. As a design principle, we want to avoid language that elicits specific ideas so that the answers that students give are more authentic. Jenny clarified: when it is studied, avoid language that cues students and focus on neutral language? Ross said yes, we don’t want to get pre-packaged memorized answers so using neutral language will help.

Ross brought up one of Megan’s papers from a couple of years ago about ‘fitness’ and how that term has lexical ambiguity. He also sees this in his own class – most of his students think that fitness refers to overall health. It’s important to avoid language that is ambiguous to students, otherwise we might not know if their answers are naïve due to the language or lack of scientific understanding.

Jenny agreed and suggested modifying this question to remove buzzwords or jargon words. She will re-write based on that feedback to get a better question. Ross also cautioned that lexical ambiguity and jargon buzzwords are different, and we should avoid both.

Mark brought up research in physics education where students tend to ask, “do you want the physics answer or do you want to know what I think?” Do students even have the scientific idea? How do we prompt for it? Can they apply scientific knowledge outside the classroom? We have to decide what we want to test.

Jenny repliedthat, in some circumstances, we need to cue students by saying “ how would a scientist or biologist explain this,” which keeps the students in the scientific realm and doesn’t use either jargon or lexically ambiguous terms. We would like to not artificially cue them to regurgitate a previously heard explanation but instead hear their ideas. Mark agrees and says this is one place where interviews can be useful. Asking how a scientist may answer a question can cause other issues, and student interviews can be very helpful for that.

Luanna and Andrea are interested in ecology questions.

John asked what makes these questions different from ACORNS? Jenny clarified that this question was developed based on the need to improve the AACR origin of alleles question. It’s a question for the larger group of whether or not we want to put all evolution questions over onto the EvoGrader interface.

Ross suggested that as we develop questions we hypothesize a progression of understanding. How would low level, mid-level, and expert students answer these questions? Good questions should have a rich gradient of responses. It’s hard and we might not be able to do this in all cases, but it’s important to keep in mind when writing questions. At this point Jenny left the meeting.

Jill offered to create a Google doc where people could sign up for topic sub-groups for question development. Mark then wondered how much of the work in the question development cycle is generic and benefits from the larger group, and how much work should be done in smaller groups? Jill thought that subgroups might be a more efficient for developing questions.

John thinks there is too much expertise in the group and that perhaps we aren’t ready to break up into smaller groups yet. Mark brought up the point that perhaps everybody meeting in the FLC-PD group is even interested in question development so it might not be best to devote time in this meeting to question development.

Michelle wants to keep talking about QDC during summer meetings, even if a faculty member isn’t interested in question development, per se. She feels that we could take turns during the summer between talking about question development and professional development. Also, Mindy Summers is a postdoc who works with Michelle, and she would like to invite Mindy to participate in meeting where we talk about ecology and evolution questions.

John suggested rethinking our working groups to make sure they are most efficient. We want everyone to be informed, but we don’t want to confuse goals. We want everybody to be informed but we want to serve the goals efficiently. This meeting is supposed to be the FLC-PD meeting, but we have started talking about question construction and that takes the conversation away from what is happening in the FLCs. Michelle said that there is a lot of data analysis that is being conducted on the FLCs by her, Karen, Jill, Paula, etc. It’s nice to have these weekly meetings where we can engage

John thinks we need to split off the question construction process, especially into groups where people feel comfortable with technical knowledge. However, refining the question development cycle document that Jenny created is something that should include the entire group. The FLC-PD meeting

Jill doesn’t feel particularly confused, especially since she sees the question development as related to supporting faculty in professional development. She is also concerned that we don’t have enough to discuss PD-wise for weekly meetings. She really appreciates feedback from all the PIs every week. John asked for feedback from others, and Tessa offered that separating out the aspects of the project would be helpful so the appropriate people could attend the appropriate meeting. Ross thinks specialized meetings are nice but that it’s nice to have everyone together to get caught up. Perhaps we should change the name of this meeting?

Mark asked how we envision these meetings. Do individuals interested in a sub-topic meet, and then meet with a larger group, and then meet with the FLC-PD group? Ross suggested that as a group we should flesh out the question origination cycle and work on the draft before we move on to the next step. Clearly we need some anchor points to make sure we don’t drift off. We need to clarify the goals so that we can stay on track.

John asked if Michelle, Jill, Paula, and Karen have to meet separately to discuss the FLC-PD aspect of the project. Michelle and Jill clarified that they do meet to do data analysis and discuss papers

Jill said that as far as the FLC-PD aspect of the project goes, the FLCs only meet once per month. There doesn’t seem to be a need for that discussion every week and it’s fine to devote this meeting to other topics once and awhile. She doesn’t want to have to add too many more meetings to everyone’s schedule. Perhaps we could offer the agenda ahead of time and we could decide if we attend or not? Michelle laughed and worried that she would be the only one to show up!

John said that this project is growing rapidly and demands a lot of time and attention. If we feel the need for another branch, or another meeting, so be it. Jill suggested that in the fall it might be very useful to separate the question development and the FLC-PD meetings. It’s also important, like Michelle said, to have the faculty that lead FLC meetings involved in the early part of the cycle because it keeps everyone informed. The faculty have also brought up questions/concerns in Jill’s interviews. In the summer there aren’t regular FLC meetings so we have the time to talk about multiple topics. However, in the fall we have regular FLC meetings resuming, and likely more activity/curriculum development groups, so there might be a real need for separate meetings.

Mark agrees that over the summer particularly we don’t have ongoing activities so it is reasonable to keep using this time for multiple purposes. We can also add new meetings if necessary once we have a feel for how often we need to get together. Mark said a Google doc or a poll would be helpful to gauge interest. Michelle asked if these topics were coming from the faculty or us? Jill offered to make a document consisting of faculty requests for questions/topics once she is done with this round of interviews. Michelle suggested putting that on the agenda for the next meeting.

Karen likes starting from the point of what faculty want.

Ross brought up the trouble his FLC faculty have with technology. If we have a broader goal of getting more faculty to use this technology, or for faculty to use the portal themselves to upload data, we may face some challenges. He has spent around 10 hours this semester assisting his faculty with using AACR and the learning management software. He thinks it is an important research question. Michelle said it’s not just about using Blackboard, but also the concept of a pre-/post-test. There are basic aspects that fall off the faculty member’s radar and they need a lot of assistance. Ross said it’s critical to know how big of a problem this is because it can cause issues downstream. Jill agreed that it’s an important question, and in her experience at least, there are likely many faculty that may struggle with the technological aspect. Mark said that he has had this discussion with other people in the QDC group. One issue is that some faculty don’t regularly use their online course management system, so it’s not just about the AACR portal but providing support for those faculty who need help with other aspects of their learning management system. He also brought up the point that it’s very difficult to get faculty trained to use these tools.

Michelle wanted to thank everybody for their hard work collecting COPUS data. Way to go everybody!