Administrative Handin a Work in Progress

Administrative Handin – A work in progress

This document is the first draft of the administrative user’s guide to the Handin system and is targeted towards TA’s and Instructors who are managing courses that use Handin as a Submission/Testing/Grading platform. Each section of this document corresponds to one of the panels in the Administrator Console, which is seen after logging into the system. Each sub-section corresponds to a link within that panel and the appropriate form fields on the pages that they link to are described. A more task-oriented document to complement this reference will soon be provided.

- Ashish Venugopal, Mike Maxim

Assignment Management

Overview

Assignments in Handin refer to the combination of the submission restrictions as well as the initial base code that was provided for the assignment. Before students can turn in their solutions for a given project, an Assignment must be created via Handin to hold their submissions as well as manage the testing and grading for the assignment.

Create Assignment

Description / Usually includes the course specific name for the project as well as its working title in the class. Usage: “HW5-Building Google”
Release/Due Date / Selecting the release and due date impacts decisions made regarding lateness of submissions.
Handin Root / Choose a informative folder name to serve as the root for submissions as well as base, test code. This folder name should at minimum help identify the project number and semester. Usage: hw2fall02-arv
Base Code / Allow the TA to upload the base code that is combined with student submissions to build and run the assignment. The base code will overwrite student code files of the same name. Since only one file can be uploaded, if there are multiple base files, they must be archived into one file, ie jar
Unpack Command / Unpacks the base code if it is an archive. Typical usage :
“jar –xvf *.jar”
Compile Command / Compiles the base code with the student submissions in preparation for testing. Typical usage: “javac *.java”

Creating the Assignment will make the assignment available for submissions after the specified Release date. It will not become unavailable after the Due Date since late submissions are allowed.

Manage Assignments

Shows a listing of current assignments, with the option to modify or remove them from the system. Removing an assignment does not remove the directory structure that was created for the assignment. This must be done manually by Handin administrators. When modifying the Handin Root, you will notice that a path has been pre-pended to the directory that was entered at creation time. Only modify the file directory in order to maintain the directory structure of the system.

Add Test

Each assignment can have a set of associated tests, which are simply combinations of additional driver code that are run in combination with student submissions. Adding a test to an assignment allows these tests to be run against a reference implementation for comparisons. A test is essentially a driver file that hooks into a student submission and performs a series of operations. The output of this computation is saved and evaluated against reference submissions for grading purposed. Pre-Tests are a special type of Test that is run at submission time, and shows the student his submissions output for that test immediately. Students can compare their output against the provided reference output to provided immediate initial feedback for the assignment.

Description / Informative name for test that should include at minimum the userID of the creator, a short description of their function. Usage: “arv-DivisionByZero”
Type / Pre-Tests are run immediately at submission time to give students instant feedback as to the basic functionality of their submission. Pre-Tests are typically computationally IN-expensive since they are run concurrently with the submission process. Normal tests are typically run a few days after the Due Date are more rigorous.
Difficulty / General indicator of difficulty of the test, can be used for performance analysis.
Points / Total number of points award for successful completion of this test.
Max Running Time / Number of second that the test will be allowed to run. This is primarily used to prevent the system from continuing to execute code that is in an infinite loop.
Test Base Code / Base code (typically driver files) used to hook into the student submission to run the test. IMPORTANT: For Pre-Tests, the uploaded archive must include a file name “precorrect.txt”. This file should contain the correct text output of the pre-test with the reference submission to allow students to compare their output with the correct output.
Unpack Command / Similar to Assignment Unpack
Compile Command / Similar to Assignment Compile
Execute Command / Runs the test. Typically this is a call to the uploaded driver program to begin the test. Usage: “java –Xms400m DoIt.java”

List Tests

Shows tests available for this assignment and tests can be modified or removed from this page.

Correct Submission Locations

An administrative utility that deals with the implications of changing assignment handin root directories. This function is currently not supported.

Grading Management

Overview

Grading in Handin involves comparing the output of a student submission’s results on a test with a reference implementations output on the same test. Once the appropriate tests have been created for the assignment, the TA in charge of the assignment must submit, using the same mechanism that students use, a reference implementation that is known to perform perfectly on each of the assignment’s tests. This encourages sample solutions to be ready in advance as well as provides a level of feedback regarding the rigorousness of the tests and the sample solution. This general approach to the grading problem allows student submissions to be compared against each other, allowing innovative competition based grading, as is currently done in 15-411 (Compilers). In 15-211, only TA’s are allowed to submit tests for assignments, but any user’s submission can be used as the reference.

Comparing the reference solution output to the student submission output is typically a simple task, but can become complex depending on the nature of the assignment. For this reason, we have built in a general notion of a Comparator. A Comparator is a Java class that takes two strings, corresponding to the two output sequences, and compares them based on some arbitrary criteria, the basic example being exact match. The Comparator returns a score form 0.0 – 1.0 corresponding to the closeness of the outputs, which is then multiplied against number of points assigned to the running Test, resulting in a final score for that Test.

Create A Grading Job

Grading is typically done in batch mode, where a set of users, typically one section, is selected for grading against a reference implementation. After the assignment is selected, a set of users, a reference implementation, the set of tests and a comparator are coupled to create a grading job. The grading job is then sent to the DemonServer (a dedicated testing machine) to run these Tests. The DemonServer provides administrative tools to check the status of jobs and to modify their execution state. The details of job creation are described below.

Job Name / An informative job name that is used to identify the jobs on the DemonServer. Usage “arv-hw1-regrade”
User Set / Select one or more users or groups to be tested.
Test Set / Select one or more tests from those available for this assignment.
Reference Set / Select one reference users submission to grade the user set against. This is typically the TA’s reference solution for this assignment.
Comparator Set / Select one comparator to compare the output of the reference solution to the user solutions. Typically comparators include “Basic String Match” - checks to see if the outputs are identical, “Count Lines” – counts the number of lines in the output.
Grading Option / The job’s two essential components, the running of the tests and the grading of the outputs, can be run individually and in combination with the appropriate selection. Typically the scenario when only the grading is performed occurs when a change has been made in the Comparator.

Hitting the TakeAction button, send the job to the DemonServer, and a receipt of the job is displayed. The job name that is included in this receipt is important because it may not necessarily be the name that you requested when you created the job due to potential naming conflicts. It is important to keep this job name for your records to track the job’s progress on the DemonServer.

View Job Status

This link provides a way to manage jobs running on the server, specifically to view their status, get their creation details, and to stop/pause/resume them. Selecting one of these options and hitting “Take Action” will make the appropriate requests to the DemonServer. Jobs are run sequentially on the server on a first-come-first-serve basis and the job creator will receive e-mail notification of job completion (feature not completely implemented, coming in a few weeks).

View Grades for Assignment

This link allows you to select a set of users or groups to view their output, results and grades for the selected assignment. Their output and code can be viewed on a per user basis or as a complete group. The code viewing functionality is primarily to determine style points.

Go To Student Handin

As described above, grading is performed against a reference solution that the TA in charge of the assignment must submit to the system. This link allows a TA to submit a reference solution, a step that is critical to perform before the test/grading process begins.

Group/Section Management

TA’s are encouraged to create a Class Section group to manage their students. This can be performed via the Add Group link and it is recommended to name section groups with their appropriate section number ie “Section E”. Please keep track of your students as they switch sections, by using the Add, Remove functionalities.