11:00Am 12:30Pm Breakout Session 1: Early in the Grant

11:00am – 12:30pm Breakout Session 1: “Early in the Grant”

Topics during this breakout session cover early “nuts and bolts” of implementing the S-STEM grant.

·NSF Annual and Final Reports – Salons A, B, C

NSF Program Officers will explain differences between annual and final reports, and the dos and don’ts of what to report in both.

Salon C, Monday October 15, 2012

Chair: Christine R. Leverenz

Scribe: Cathy Middlecamp

NSF officer: Joe Grabowski, DUE

We have an outline to follow today, given to all session chairs

Strengths and visions

Christine: What do you see as strengths that can be envisioned from writing annual reports?

Responses from participants:

-NSF is looking perhaps for best practices

-All the data we collected actually strengthened our grant

-Finding what doesn’t work is helpful because then you can make adjustments

Joe: when you wrote your research grant, you got the money a year later. NSF understands that you learn as things advance. You are not expected to strictly stick from what you articulated some years back. Tell us about how you adapted, what didn’t work and why. Things that you got to work, why are they working now?

Note that this is a private document between you and your program officer. The rest of the world doesn’t see it.

Re: vision, how would you take what you did and share it with the rest of the community?

Joe: In January 2013, the reporting system is going to change to research.gov. It follows the same general format with prompts. The problem is, there is one reporting system currently, and it was designed for the research directorate. The new form will really prompt you to put things in the prompted text boxes instead of appending documents.

The strategy – the data mining systems can harness the text boxes. It cannot harvest pdf files.

Question: two entirely different reporting systems. Could you explain?

Annual and final report goes through FastLane (and will go through research.gov next year). The standard annual and final report required for ALL NSF reports. No personal and identifiable information goes here.

The S-STEM reporting website reports the data back to Congress.

Question: I got scolded for putting names in the report. Why, if this is a private document?

Joe: Freedom of Information Act. Somebody could request the document.

Question: I’d like to know the difference between the final report and the annual report. And what level of detail?

Joe: The final report summarizes the entire project. The annual report should only speak to a 12-month period. Delete anything that is populated previously from another report and is not related to the current 12 months.

As a Program Officer, I read and approved 14- annual and final reports. When I get 125 pages, this is too much. Useless. What I need to know is what you tried and did not work? Did work? What were the ultimate successes? The elevator talk. That’s what I need. I do not every piece of data you ever collected. A 10-page summary of the high points, for example, what got institutionalized, what got slashed in your campus budget.

Question: Could you give me guidelines on an annual report length?

Joe: The big picture. It is like a peer-reviewed publication. An abstract. Summary charts and figures. 10 pages. Maybe 20 at the outset. I don’t need to see your survey instruments. Rather, I’d like to learn what you learned from them and how you morphed the project. A summary.

Question: How do annual reports help us set up for the next S-STEM grant submission, especially given that this is a one on one document. This makes me wonder that the time we put into them really doesn’t help us much.

Joe: Think about what the time and effort is worth in the end. But if you put the synopsis together for the final report, this sets you up for the critical part of the proposal, help you build upon what you learned from the first. What is beneficial? Again, it is a private document. But it should set you up for the next one.

Question: Could you talk about the difference in content between the annual and final report.

Joe: Half of the 30 Program Officers in DUE are rotators; the other half is permanent Officers. If you have a 5-year S-STEM grant, you might get somebody changing every year. This person does not have the time to read your proposal and compare it line-by-line. Your program officer is likely to be changing.

Keep this in mind as you write your report.

Look back at any comments from Program Officers.

The gold standard is a peer-reviewed publication. This is more valuable than an annual report.

Angst about getting these in on time.

Unapproved annual or final reports ONLY hold up funding going out the door for you or for your co-PIs.

Annual reports are a pass-fail.

Question: A moment ago, you mentioned that an annual report should be kept to the reporting period. If we are dealing with cohort building, don’t include personal specifics. General term, even if person reading it

Joe: Yes. Good synopsis. All previous reports are available. You can write, as we reported in our last report.

Question: I’m interested in a description of products. Mine or my students?

Joe: All of the above.

Question: could you suggest to me my obligations?

Joe: I’ve never seen any obligations written down anywhere. Remember EHR is newer to NSF. Peer-reviewed publications are still the gold standard, because the whole world can see them. Presentations are more limited.

Comment: Don’t lose focus as to what this grant is about. You have the opportunity to figure out what works in keeping students engaged in a STEM major. Intrusive advising? Cohort building? There are tons of things you can report on.

Question: Outcomes. How do you measure the success? Number of students who graduated? What do you mean about practices being institutionalized?

Joe: Institutionalization is things like “intrusive advising.” Put into play during the grant and continued when the grant funds go away. We don’t expect you to institutionalize the funds. Put in a first year experience course that is unique, worked, and the university saw the value in continuing it. You have changed some part of the culture because you achieved success.

All NSF funds stop at some point.

NSF is completely electronic right now.

Highlights. If you have something, no matter how small, that worked, pull that out. This is really cool. We need to get highlights out to the directorate, to the broader NSF, to Congress. We need to document success. Pull them out!

Question: All these really cool things. It would be great to see what is working. Does NSF have plans to create something in which we could share what has worked?

Joe: Great question. Most of the other programs (STEP) have a central worksite. S-STEM does not. I have been unable to convince them that this is a valuable entity. You could write a proposal to NSF to fund this.

Question: I still don’t understand the privacy issue you mentioned.

Joe: No names, PII, no social security numbers, no GPAs, no personally identifiable information. We are NOT looking for the names of all your students. But presentations are matter of public record. The annual report is a synopsis of the program. For a presentation, you can give the names at the meeting.

Question: Also the POR report, public outcomes report. What is it?

Joe: I am aware of it, but NSF has nothing to do with it except that we require it. It is a report for the public and it is a fairly new requirement. You will have to do this when the grant runs out. As program officers, we don’t have to do anything with this. You will get an email from NSF when it is due (or so I think).

Anybody filled out a POR? (silence in the room)

Question: If I have an annual report due in February, will I get the new system? Sometime in January, all folks will transition, even if you are in the middle of an awarded project.

Question: No cost extensions.

Joe: Great question about NCEs. Are extremely common (not frowned upon at all). Projects can get up to three NCEs. The first can get a grantee approved NCE. You want it? You got it (if you do it within 30 days of the ending date). The second is NSF-approved by the program officer, essentially a gimme. I’ve never seen any NCE turned down. Well over half with S-STEM. Because sometimes the money does not arrive for you to put the money into play.

Question: Submitting for a second S-STEM award. What are the odds? Not a renewal. Perhaps the PI changes.

Joe: You’ve raised a very gray area. They are NOT renewals. They are not to be used to be used to continue funding of the first cohort students. In terms of statistics, no data reported to me that has funding rates of first-time grantees vs. established investigators. Becoming increasingly discussed for old vs. new. There is no bottom line. We are being challenged to pay attention to the solicitation: if there is another s-STEM on your campus, you need to talk about what was learned from the other one as we make our funding decision. Additional S-STEM grants are certainly possible.

Question: The two NSF-approved NCSs. What information would you like to see? We will have some extra money.

Joe: A couple of comments. When you fill out the FastLane form, do not say we have money left over and we want to spend it. Rather, say why it would help us to be successful if we spend it. If you start another cohort, what will happen when the money runs out? This could be a concern. Make a cogent argument about what works for your students.

Question: Transfer students … (part of question missed)

Joe: It is acceptable in NF to best meet your local circumstances. As long as it is not changing participant support costs. You can do 10 in year one and 12 in year two and here is why.

Question: Multiple S-STEMs. A different cohort – different year. What are the parameters?

Joe: The proposals are reviewed independently. How the new one is going to be different. If you have been paying attention, S-STEM is going up every year. The funding amount is staying the same. NSF is now requiring indirect costs to be included on S-STEM costs. But we are allowing the indirect costs to go over the $600,000 limit rather than having the PIs lose the funding for undergraduates. Right now one proposal per dean. It may change to one proposal to institution. Check this before you submit next year.

Question: The allowable indirect cost is only for the new proposals?

Joe: Correct.

Question: What about community colleges partnering with a university:

Joe: The current solicitation is one proposal per dean. In the future, it may change to one proposal per institution.

Question: If the university has 3 campuses, separate by administration, is each one eligible?

Joe: My understanding is that geographically separate campuses can submit. This will continue to happen. Talk to a Program Officer for your specific case.

Question: How does the NCE affect your final report?

Joe: A NCE triggers an annual report. You have one more year in your project.

Question: Peer-reviewed publications. If the students covered by S-STEM are publishing, do we acknowledge NSF on theirs?

Joe: Never hurts. Look at the awarding. You can find it in your award letter.

Question: Annual report in research.gov. Will it automatically populate?

Joe: Fingers crossed. We hope so. But the new site is different with different prompts. If I had a bet, you would be able to see last year’s annual report but you will have to cut and paste.

Question: Final reports … say more?

Joe: Give us a synopsis of your report. What worked, what didn’t work, what was different? The failure is not to tell us that something did not work.

Question: Comparison data with non-S-STEM students?

Joe: Would be fantastic. If you had students applying who did not get the scholarships.

Question: What about external assessment?

Joe: Great question about assessment and evaluation in general. The general guideline is that the person should be external to the leadership team, not necessarily external to the department or to the campus. S-STEM does not have a lot of money to support an external evaluator. Most have somebody on their own campus.

Question: The reporting website, I got the email earlier that you need to put in your information. Has anybody else seen it?

Joe: All I can do is to look at the reporting website.

Audience: There is a session on this later today.

Question: How many pages in the final report?

Joe: Not 125 pages. I want to read something short, sweet, and concise. How institutionalization has happened. Like a peer-reviewed publication.

Question: Making a comparison with students supported by S-STEM and those that were not. Easy to do but perhaps not scientific. We would then need to match students.

Joe: It is the gold standard of evaluation to have two groups.

Comment from assessment person: We can get cohorts by working with people across campus. We are able to parse this out. It took us a couple of years to get everybody on board.

Question: How necessary is an outside evaluator for S-STEM?

Joe: very valuable. The reviewers are looking for this. Then again, NSF is not providing much funding for this and NSF understands this very well.

[Applause for Joe!!!]