(Since we are unsure how readable this response will be as cut and paste text, we have also uploaded the same text as our PDF cover letter for this revision)

Wednesday, August 27, 2014

To whom it may concern:

On behalf of the Scalable Game Design writing team, we appreciate the reviewers’ and editor’s constructive comments and suggestions. As you will find, the paper has been revised significantly and is a much better paper in many ways. Below is an outline of the main concerns and changes made by the writing team:

~~~

As you can take from the attached reviews, the most crucial point is how your paper fits into these special issues. Originally, we had intended to collect case studies about the situation and perspectives of Computer Science Education IN schools in certain countries or states. Your paper describes a very impressing and successful project that is more or less located outside of or additional to regular school courses. Also, the number of schools even in smaller countries or states is at least about 400. Hence, the number of schools that have participated in your project (46 in the last 3 years, as we could take from your paper) is about 10% of this intended impact scale. Yet we acknowledge that your project might be a substitute in cases where the institutional education fails to deliver sufficient computer science education. This is why we are willing to include your paper. But nevertheless, its peripheral character regarding the focus of our issues should be reflected by its size.

The primary goal of Scalable Game Design is to integrate opportunities to program in regular school courses. We fault ourselves in giving the reviewers the impression that this project is located outside of regular school courses or in after school programs. Scalable Game Design is almost exclusively in schools and is part of existing classes.

We have improved the communication of “getting computer programming into schools” in this version in multiple ways: in the discussion of how the SGD strategy emerged from prior work in schools, in the discussion of the strategy and the theory of change, and in the explicit focus on professional development of IT, math, science, and other teachers.

With respect to the scope of the project in relation to other articles, from just the cumulative count of student uploads to the arcade we have a conservative estimate of schools 113 schools that are implementing Scalable Game Design, through initiatives that started in 2008. These schools are located in 21 of the 50 United States (i.e., Alaska, California, Colorado, Florida, Georgia, Indiana, Maryland, Minnesota, Montana, Missouri, Mississippi, New York, North Carolina, Ohio, Oregon, Pennsylvania, Rhode Island, South Dakota, Tennessee, Texas, and Wyoming). The main study includes over 110 schools but in terms of impact the number is much higher as the majority of AgentSheets users are not participating in any study at all. We can try to put an estimate on this but number of all schools using AgentSheets is quite high. Also, during the week of the Hour of Code alone we had about 250,000 participants.

The rapid uptake, over a period of six years, of embedding computer programming in regular school activities in spite of the broken pipeline is what has been remarkable. With all of this said, we are also careful to present our case in a manner that is a bit more objective (as suggested by reviewers).

In some parts, your paper looks still more like an advertisement than like a scientific discussion. We would welcome some critical remarks about your project (e.g. “lessons learned”). Please try to keep the presentation as objective and unemotional as possible.

Throughout the entire manuscript we reduced the direct references to AgentSheets and AgentCubes. It is now less gratuitous and more directly relevant to the argument and description of the SGD implementation.

As an example, in section 2.3 we revised a paragraph that referenced the AS and AC software multiple times to read as follows: Early on it became clear to us that students did not sign up for computer science courses because of their negative perceptions of the subject. Asked in the context of a typical computing course, one middle-school student summarized her perception of programming as “hard and boring,” which does not suggest a workable tradeoff but instead a lose-lose proposition. The “hard” part is a cognitive challenge that requires a visual programming approach such as drag and drop programming. The “boring” part is an affective [Picard et al. 2004] challenge that relates back to motivation. Why should students really want to program? In our research we found student ownership [Repenning 2013] to be the key to motivation. When students are enabled to create artifacts that are personally meaningful to them, they are much more likely to be motivated. Once they have created shapes such as people, animals and cars and assembled them into a world, they are highly motivated to bring them to life through the process of programming.

Please adjust the layout of your paper to the correct style (Small Standard Format 2012), see

Style, cites and references have been updated to the requested format

In particular, we would ask you to add the missing elements on the front page (ACM classification etc.) and to adjust the citation style of your paper to [ACM Number, Year].

Done

As already suggested by the reviews of your first submission, please remove multiple references for the same topic, e.g. [1,2,3]. Please restrict the references to the necessary minimum, in most cases to one single reference (e.g. the paper that has presented a certain concept for the first time or the most recent paper about it).

Done

In many cases, you have just given an indication of source without any explanation of its relevance. We would ask you to check the text around your references in order to make sure that the necessary information about the relevance of the referenced for the respective topic is provided (e.g. “XY was comprehensibly described by [Smith, 2003]).

As directed, we have added further elaboration throughout the manuscript, where appropriate, of how particular references support arguments and claims.

Please give some information about the temporal extent of your courses (in total number of hours)

Additional information regarding professional development courses was included in Section 3.2.1.

“This training is conducted in three blocks: 25 hrs (3 days) beginner training focused on basic game design, pedagogy and computational thinking; 16 hours (2 days) intermediate training focused on STEM simulations and; 16 hours (2 days) advanced training focused on 3-D designs of games and simulations.

Check the placement of commas

Done

Please update your introduction regarding recent activities that are relevant for your paper, e.g. regarding code.org or the 10k initiative.

We have added paragraphs in the introduction about the CS10K initiative and code.org

You refer to the “Scalable Game Design curriculum” several times, e.g. in 3rd paragraph of page 3. Although the term curriculum might be interpreted in a very general way (e.g. as “set of courses”), it has a much more specific meaning in the context of international K12 education. In particular, a curriculum contains at least a set of topics and/or learning goals. Hence, we would ask you to provide such information about the SGD curriculum in your paper (e.g. in a table of learning content in the appendix) or to drop the term “curriculum” in the context of SGD.

We have improved the consistency of the use of the term curriculum in the paper. As requested, we have also added an appendix section that articulates the Scalable Game Design curriculum vis-a-vis computational thinking patterns.

In lines 6-7 on page 12 you describe a survey among 10.000 students. Please explain this research in detail, for example:

Which question was asked to these students? (david adds)

Which responses were coded for the “characteristics”. (all student responses from the post survey)

The prompt used is now included in the text, and we clarified the way in which the responses were selected.

As already suggested in the reviews of the last submission, please explain the meaning of the first sentences (printed in italics) of sections 3.1.1. – 3.1.4. How should the reader understand these sentences (e.g. as introductions or as self-set goals)?

This is explained now with the following framing at the beginning of section 3.1: “Goals were created for each of these four components as detailed below. The goals are presented in a format that communicates our theory of change [Ployhart and Vandenberg 2010]. Each introductory sentence, in italics, is written a format: <approach> so that <outcome>”

Added reference:

Ployhart, R. E., & Vandenberg, R. J. (2010). Longitudinal research: The theory, design, and analysis of change. Journal of Management, 36(1), 94-120.

Please adjust and improve the size and quality of the text in all the figures. Sometimes it is not readable at all.

Done

On pages 27f you discuss the CT skill scores of students. Please explain in detail how this score is calculated and provide information about the variance over the samples. Otherwise it is not possible to assess the significance of all the discussed differences.

The information regarding the computation of CT skill scores is now included in Section 4.2. We included the equations used and the following text:

Each Skill Score is the normalized CTPA vector length. That is, a value of 0 indicates that the values of all the Computational Thinking Patterns are 0 (no evidence of any skill). A value of 1.0 indicates maximal values of all the Computational Thinking Patterns (100% evidence of all skills).

I don’t understand the message of section 4.5 beyond saying “we did programming with primary school students”. Hence, I would suggest deleting this section.

We agree and have deleted this section.

In my opinion, the list of references contains too many references on your own papers compared to the rest of the references. Please try to reduce the number of self-references.

Done

Generally, please follow the advices of the reviewers as closely as possible. Please don’t forget to update figures, facts or other statements due to the current situation.

We agree with most of the reviewers’ comments and suggestions, and have noted how we have responded to these remarks in the sections that follow.

Referee #1

The claim that the SGD-initiative is based on its own curriculum seems overstretched. This claim needs to be either drastically reduced or better presented.

We have clarified the use of the term curriculum, when used, and have included an appendix that outlines the SGD curriculum.

References need to be fully spelled out.

As directed, we have added further elaboration throughout the manuscript, where appropriate, of how particular references support arguments and claims.

Details re edits

P2 / 42 “that” is duplicated. - corrected

P6 / 32 “broaden” => “broader” -corrected

P6 / 50 “interesting” => “interested” -corrected

P7 / 40 “... transferred?” “... transferred.” -corrected

P10 / 55 “… make students” => “… making students” - corrected

P13 / 20 I am not sure whether “Seven Big Ideas” are widely known. What is well known are the “Seven GREAT ideas” identified by Peter Denning. Kindly clarify and quote appropriately. Statement has been corrected.

P14 / 30 “ … of almost exclusively of …” Drop one of the “of”s. - corrected

Fig. 3 and others: Some readers will use black/white prints. In these cases, green and orange are not identifyable in the Kiviat (Spider-web) graphs.Graphics improved

P17 / 46 “that” is duplicated. Perhaps “that this student …” - corrected

P21 / subsection 3.2.2: This definitely suggests a starting point for working with a class and sketches how one might progress. But this is certainly not a curriculum. –clarified throughout

P22 / subsection 4.1: Critics might argue that the level of abstraction of these patterns allows transfer. But they are stereotypes that do not necessarily foster creativity as such. You might want to preempt critique of this sort by addressing the issue of transfer, generalizability, or use as elementary building block.With several revisions in this and preceding sections, we feel the argument for transfer has been moderated more appropriately for this paper.

P24 / 6,7 Something seems missing in this sentence. Kindly rework it.corrected

P26 / 48 “This makes CTPA _a_ unique …”corrected

P27 / Fig 8 Kindly enlarge it. Text is hard to read.Improved

References Quite a number of them is incomplete. Check and rework.References are now complete, reconciled with citations, and corrected to proper TOCE format.

Referee #2

Can you please elaborate more the following statement (in the manuscript not the abstract):

"However, K-12 students in the United States are facing a pipeline for computer science education that is broken." In my eyes this is extremely important, being not from US I want to understand exactly where this pipeline is broken, why and potential solutions. If the manuscript will address those things (making it clear even to people like me, not being experienced with decentralized ed. systems like the US one), you will have made a major contribution in the area.

New text and references have now been added to the first two sections to enhance the various descriptions of factors and initiatives in place in CSE in the United States. The revisions in these sections are too comprehensive to itemize here.

The authors are focusing to the gender and race inequality issues in CS and in the second section they propose their model. However, it is not clear to me the current situation, the problems of the current situation are clear. But I need to know how K-12 CSE is currently being performed, in order to be able to grasp where all these problems (gender etc.) coming from. The authors need to define the current situation of K-12 CSE, even epidemically (since it is a part of decentralized ed. system) before proposing their model, this will allow the international readers to see the whole picture instead of only the tree and better understand the proposed model, its applications and benefits.

Starting with the second paragraph on (“At present…) we have improved the articulation of the current situation of K-12 CSE, and this carries through the first two sections. We are confident that with this version of the manuscript readers will have a better understanding of the situation in the U.S. We have also included several recent references from CSTA and others that support the same statements and claims.

The qualitative results are being presented in a "messy" way, the authors put all the quotes from the interviews (which is OK) but the only categorization was, positive, negative, neutral. I expected from the authors to "make sense those data" in a better way. For example, the authors could have coded those responses in order to reflect the current situation of CSE in K-12 and give the specific benefits of their strategy. Or highlight possible future directions, in terms of the content, the practices and the frequency of CSE. Even now the authors can discuss in depth the qualitative findings, no need for coding them again, but at least discussing upon them.

We agreed that the inclusion of this section required additional information regarding methodology and approach. Given the focus of the paper, its current length, and the tangential nature of this data, we felt it was best to delete this section.

What those figures in 30-31 pages are? there is no connection with the text, no captions etc. please put them in the context or delete them.

We agree and have deleted this section.

At the conclusions, what was the case before/without SDG, where SDG helped K-12 CSE? what other CS ed. systems can learn from your case ?

We have revised the conclusion section accordingly (noted in italics):

This article has described Scalable Game Design as a strategy to introduce students to computer science through game and simulation design. To successfully address pipeline issues, CS projects must create sustainable programs, which challenge and motivate students to engage in computational thinking. Furthermore these projects must be accessible by teachers and must support training in a way that enables teachers to grow into more complex engagement with the material. Finally, it must broaden participation for women and minorities, and reach students at a variety of grades and abilities.The data presented suggest that this strategy is not only highly motivational for students but, when combined with the inquiry-based teaching approaches...

And in the next paragraph:

The SGD approach has demonstrated that teachers and students are eager to engage in computer science education when the proper resources and supports are available. Unlike most professional development opportunities where only 10-15% of teachers actually implement changes as found by [Richardson 1998], SGD support has resulted in more than 80% of the teachers implementing SGD past the initial year, and more than 15% of teachers who return for additional training to strengthen and expand their skills. Even without a national curriculum for CSE in the U.S., teachers recognize the value of computer programming and opportunities for students to design games and STEM simulations.

Referee: # 3

I did feel that, in the early discussion, an important element that was missing was the value gaming could make to many areas of the curriculum. That is, to expect scalable game design to be placed within a compulsory unit of study (p3) would require an argument as to the value of computational thinking to other subjects within the curriculum. This also becomes an important argument to develop and would strengthen the paper from the beginning. As was pointed out on page 4, line 43/3,’ schools require persuasive value propositions to instigate change’. There is also some disjoint aspects within the paper and, in particular, the discussion of computational thinking and this occurs on p 8 line 34/5 where the authors states "Systemic computational thinking is highly attractive to schools because it already aligns well with existing math and science standards".