OASIS Specification Template s6

A More Agile Public Review Process

Revision 1

9 November 2016

Guideline URIs:

Editor(s):

TAB members

Abstract:

This document provides a new approach to public reviews.

Status:

This document is not yet approved.

Interested parties should send comments on this specification to the TAB.

Table of Contents

1 Issues with current PR process 3

2 Proposed PR Process 4

2.1 At a Glance 4

2.2 How it works 6

2.3 What it looks like from the TC and Editors viewpoint 7

2.4 What it looks like from the Comment Submitter viewpoint 7

3 Comparisons on Two Scenarios 8

3.1 Scenario #1: Most Favorable to the new Process 8

3.2 Scenario #2: Least Favorable to the new Process 8

ntifier] Y]

1 Issues with current PR process

The process is rather rigid & involves significant administration overhead, especially when a subsequent Public Review (2nd or beyond) is needed due to comments filed.

· The bridging from PR1 to PR2 is fraught with risks of delay and administrative overhead both on TC side and staff side. (The TC needs to approve the new draft - needs at least 7-day internal review either as part of electronic ballot – then ballot. TC admin needs to get an on-line request that may be flawed and resubmitted. Overall, it may take between 15 days at best and 1 month, before a 2nd PR can be initiated after the end of PR #1.)

· TCs are under pressure to deliver and are tempted to downgrade the comment severity – e.g. rate them non-material or editorial – in order to avoid starting a subsequent PR.

· The decision to approve a drat for review, or to approve a comment resolution, requires a motion. This level of formality introduces friction and delays in the process (need to freeze the draft for a review period, need to get quorum.)

· OASIS reviewers lose track of their comments because of the long delay between filing and resolution. They need to pay attention when the next PR will take place – if any – and may realize too late that their comments have not been addressed (they have no visibility into the resolution process.)

· All of the above, added to the serialization of public reviews, contribute to a long process that needs administrative overhead all along, especially when a new PR is needed.

2 Proposed PR Process

2.1 At a Glance

We propose a more nimble specification review process, closer to open-source practices: the “rolling updates” review.

In this new review process, there is a single review period that can be dynamically extended depending on comments received. Comment resolution goes on in parallel with the review. There are incentives to synchronize both as much as possible – although not required.

Some major differences with today’s process are:

· No 2nd PR is needed.

· No serialization of tasks is needed such as (review) à (resolution+approval)à(new review).

· Comments can be addressed as soon as received and a new draft rolled-out for review while the PR is still ongoing. So the PR period is a very active period for the TC – not a resting period as with today’s process. The TC has interest in advertising the review.

· A string of new drafts may then be produced over the review period (e.g. one every week - 2 or 3 total). This is possible because the approval on TC side is based on consensus and informal (see later) – the editors are the deciders.

· During review, the latest draft supersedes the others, but reviewers can still file comments on previous drafts.

· The review ends automatically when 7 days elapse after the latest draft is rolled-out and no new comment has been submitted since then. There is a minimal duration for the review – 3 weeks is recommended here. There is also a maximal duration beyond which the TC can ignore comments. At that time the TC may formally approve (vote) the final draft to avoid a 7-day lapse.

Handling the risks of conflicting or inadequate comments, parallel with open-source:

The concurrency between (a) review & comment filing, and (b) comment resolution and editing new review drafts, may increase such a risk. As submitters may not closely watch the sequence of drafts being rolled-out during the PR, the comments they file may be on an earlier draft than the latest available. This means there is a risk of their comment being obsolete or inadequate for the latest draft. We consider this a minor risk, and its low occurrence makes it possible for the TC editor to detect and handle. This risk is similar of receiving conflicting comments from different submitters, that the TC editor has to resolve anyway.

It is akin to the risk of conflicting updates in version control of open-source software, when merging branches (modern version control does not “lock” the source code between developers). Yet the benefits of concurrency in developers’ work has been deemed to outweigh the risk of having to resolve “manually” possible conflicts. We speculate that the same trade-off applies to specification editing. While reviewers are encouraged to track the draft releases and to adjust their comments if needed, It is in the interest of TC editors to spot conflicting or obsolete comments early on and to notify the submitters.

The most prominent features are:

· A relaxed level of formality for approvals on TC side. The approval relies more on consensus (as opposed to formal approval or motion). Consensus building relies on the comment process itself: a strong disagreement can be expressed by TC members by filing critical comments, which in turn brings more visibility to the issue. Ignoring these will introduce more delays per the new process, as the unhappy submitter can re-submit several times. There is however a way for the TC to override such obstructions by a formal vote (but voting is a last resort).

· A shorter and more dynamic comment-to-resolution cycle. The TC has interest in addressing filed comments and in rolling-out a new draft very shortly. The process encourages TC editors and comment submitters to collaborate immediately and to converge toward a satisfactory comment resolution. The reward for TC is a shorter review period and a faster specification release. The reward for submitters is to not have to wait and possibly forget for a next PR to deal with their comments.

A swim-lane diagram for both processes (current and new) is rendered below:

Current process: iterating review periods including comment resolution: total = 30day + N*( request for new PR + 15day )

New (proposed): “rolling updates” reviews including comment resolution: 21 days, dynamically extensible to a total of ~5 weeks, depending on comments flow. (But TC may decide to go longer.)

2.2 How it works

(a) A single review period, of 3 weeks but dynamically extensible based on the comments flow (no TC admin request needed for a subsequent PR…)

(b) During the review period, as comments come in, the TC is encouraged not to wait to work on these (TC could wait but it is in their interest not to). As soon as TC resolves a batch of early comments, TC approves a new PRD rev and posts as the “latest in review”, diffed with previous rev and along with comments resolved so far. All this while the review period is still ongoing.

(c) The reviewers (membership) can file comments at any time during the review period. It is OK to file comments on a PRD rev that is not the latest (in that case, the comment may be obsolete or superseded)

(d) The review period ends when 7 days have elapsed after the latest PRD rev is posted with all the comments disposed of and no new comment was filed. The TC may ignore any comment that comes in after the 30th day of the review. (although they are encouraged to dispose of them in some way to prevent a negative response from the submitter, should a voting period entail - e.g. for OASIS standard).

2.3 What it looks like from the TC and Editors viewpoint

The decision on the TC side to roll out a new draft does not require a ballot… the editors consult informally for the disposition of each comment (or during a TC meeting) with the TC membership, then do the fix. Entirely based on consensus: if a TC member is in strong disagreement, s/he can always file a new comment that will need be addressed and slow down the process. In case of stalemate, in absence of consensus, the TC can set-up a ballot to approve the latest draft.

It is in the interest of the editors to follow-up quickly with the comment submitters, because of the potential gain in the PR duration.

Also the new process is not disruptive: a TC can still use it in the old way: TC can rest and wait for the end of the PR period, fix the comments just after, approve the new draft the way they like (e.g. motion), and request a new PR.

2.4 What it looks like from the Comment Submitter viewpoint

The comments submitter must expect to be notified by editor as soon as a new draft resolving the comments is posted. S/he is expected to review, and in case the draft does not address the concerns, to file another comment(s) within 7 days.

3 Comparisons on Two Scenarios

3.1 Scenario #1: Most Favorable to the new Process

Scenario: a substantive (material) comment is submitted after 1 week of review of a new specification. No other comment filed after.

Current process: the 30-day review still needs to complete. The TC is resolving the comment. A 2nd PR (15-day) must be requested, even if the TC has resolved the comment and approved the new draft a week after the comment was filed. The comment submitter is satisfied with second draft. Total: 2 months.

“rolling updates” process: The TC resolves the comment immediately then approves and posts a new PRD rev a week later (day 15 of review). The comment submitter is satisfied. 7 days elapse with no more comments. Total: 3 weeks.

3.2 Scenario #2: Least Favorable to the new Process

Scenario: several comments filed over 3 weeks. The last substantive (material) comment is submitted at day 21 of the review of a new specification.

Current process: the 30-day review still needs to complete. The TC is resolving the comments, then a 2nd PR (15-day) must be requested. The comment submitters are satisfied. Total: 2 months.

“rolling updates” process: The TC decide to resolve the comments as they come in over the 2 first weeks. TC approves and posts a new PRD rev2 at day 21. But new comments came in during 3rd week, plus the rev2 got a new comment (say on day 25). TC works on these, approves & post rev3 a few days later (say, day 30 of review). The comment submitter is satisfied. 7 days elapse with no more comments. Total: 1 month and 1 week.

ntifier] Y]