MSC—Multi-State Collaborative to Advance Learning Outcomes Assessment
quick guide to the Online upload process
Updated 1/8/2015 to include Assignment Submission and Common Upload Issues
Introduction
The MSC is a partnership between state systems and institutions in nine states designed to provide meaningful evidence about how well students are achieving important learning outcomes. The initiative foregrounds a distinctly different form of assessment than the traditional standardized test. To complete this work each institution has submitted a sampling plan for how they will identify and collect student artifacts to submit and score. Each institution has agreed to submit at least 75 artifacts for each learning outcome they intend to assess. (Only three learning outcomes, written communication, quantitative literacy and critical thinking, are included in this pilot.) Once these artifacts are collected someone at the state or institutional level will be responsible for uploading them into the VALUE Database where they will be housed and scored. This Quick Guide provides an overview of how to upload these documents into this system.
Confidentiality
The MSC VALUE Database has been designed to ensure high levels of security and confidentiality for all information that it houses. Identifying course information (course number, name, and faculty) will not be collected so that artifacts cannot be linked to a specific course or faculty member. Although artifacts will be linked to the institution and the student in the system, these linkages will be kept confidential[1] to all other state and institution leads. Each uploader will be able to view only those assignments and artifacts that they have uploaded. Representatives from SHEEO and AAC&U, the organizations supporting this project, will NEVER have access to student identifying information submitted as part of the student ID field (although they will have access to other demographic fields). These organizations will have access to institutional information ONLY for the purpose of administration and report creation. No data identifying institutions will be reported at the national or state level without the institution’s consent. For the purposes of scoring, designated scorers will be able to view de-identified artifacts through the Artifact Management and Scoring system; artifacts will not contain any information about the student who produced the artifact or the institution from which the artifact was collected. Scorers will not score artifacts from their own institutions.
Overview
In order to upload artifacts, each artifact must be linked to an assignment with an assignment cover sheet. The information from this cover sheet will be used to upload the appropriate assignment information along with assignment instructions and supplementary documents (if appropriate). Once information on the assignment is completed, relevant student artifacts and the accompanying demographic file containing student demographic information (CSV) should be uploaded. The following sections provide an in-depth description of how to complete each of these steps.
Collect cover sheet from faculty members and submit assignment information
The assignment cover sheet is a Word document that faculty are asked to complete with assignment-related information. The first step to completing the cover sheet is putting an assignment code into the system. Any numbers or letters can be used for the assignment code.
The identifying fields on the cover sheet, including the faculty member’s name, department, the course name, and course number and section (screenshot 1) cannot be submitted into the system. These fields are collected only for the institution representative to keep track of assignment source information. Thus, neither a state nor the Multi-State Collaborative as a whole will be able to link student artifacts to faculty members. However, this information should be used to put both a course level and a course discipline into the system. One hundred and two hundred level courses (or their equivalent) should be considered lower division while three hundred and four hundred level courses (or their equivalent) should be categorized as upper division. Course discipline is optional but will provide valuable insight into the diversity of courses included in this sample and courses should be grouped into disciplines using the ten NSSE major categories (see Appendix I). Please note that Fall 2014 is the default term and year setting in the system.
The cover sheet also asks faculty to identify the learning outcome for which an assignment should be scored: written communication, quantitative literacy, or critical thinking (only one may be selected). Faculty must also indicate which criteria of the rubric student artifacts should NOT be assessed on. The default setting is that each criterion will be scored, therefore it is necessary to change the option from “Yes” to “No” in the system.
Screenshot 1: Assignment Information
Assignment instructions and supplemental information
In addition to the assignment cover sheet, faculty will submit the assignment instructions (i.e., description or guidelines for the assignments from their syllabi) as a separate Word document or PDF. Instructions will not be visible to the scorer but will be used by project staff to confirm that the assignment is appropriate for scoring on the selected learning outcome.
Faculty may also submit supplementary assignment information or information that is helpful for scorers to be able to score the assignment. Specifically, for quantitative reasoning assignments an answer key or schematic of required calculations will be helpful for scorers since the rubrics are designed to be used by faculty from any discipline. For written communication, please include information about intended audience and purpose of the assignment. This information will provide the basis on which scorers can evaluate the context and purpose of the writing assignment.
These documents may be uploaded either when the assignment is initially created or when the student artifacts are uploaded (if this is done as a separate step). Assignment and supplementary information should be uploaded as Word documents or PDF (also see supplementary information templates). Screenshot 2 provides an example of the page that will be used to submit this information.
Although failure to submit these documents will not stop the artifacts from being included in the sample, we strongly recommend submitting them, particularly the assignment instructions. Submitting both the assignment instructions and the supplementary material will significantly help evaluate the artifacts’ relationship to the appropriate rubric and ensure the highest quality of assessment.
Screenshot 2: Upload assignment instructions and supplementary information
Preparing the student demographic files
When collecting the artifacts it is important to also ensure that each artifact can be linked to the demographic information of the student who produced the artifact (although all student identifying information should be removed from the artifact). Before uploading the student artifacts for an assignment, prepare a CSV that contains the demographic information of the student associated with each artifact and random identification code or number for the student (see “StudentID” field). This file will serve as a reference document linking each artifact to a student’s random identification number. This CSV MUST be named LearningArtifact.csv and should contain the following fields for each student (you can use the “LearningArtifact” template to construct this file):
Field / Definition / Acceptable ValuesStudentId / Any non-duplicated ID including the student’s actual university ID, an existing ID such as an email address, or even a random ID generated specifically for this project. / A set of string alpha-numeric characters or numbers of any length is acceptable
ArtifactFileName / The file name of the individual artifact associated with this student that will be uploaded into the system / A set of alphanumeric values of any length is acceptable but the file must be Word, PowerPoint, Excel, PDF or txt
PELLEligibility / An indicator for whether the student has been eligible for a Pell grant at any point during their career as a student / Yes
No
Gender / The student’s gender / Female
Male
Other
RacialEthnicGroup / The student’s race or ethnicity based on the categories used in IPEDS / American Indian or Alaska Native
Asian
Black or African American
Native Hawaiian or Other Pacific Islander
Hispanic Latino
White
Two or more races
Non-resident alien
YearOfBirth / The year only in which a student was born
DO NOT include month and day. / YYYY
CreditsEarned / The number of credits a student earned from successful completion of courses at the institution or transferred into the institution from any other regionally accredited two- or four-year institution / A numeric value
StudentProgramMajor / The six-digit classification of instructional program (CIP code) associated with the student’s major / Available at: http://nces.ed.gov/ipeds/cipcode/browse.aspx?y=55
DegreeLevel / An indicator of the degree level in which the student is enrolled / Certificate
Associate’s
Bachelor’s
At many institutions the institutional leads will send the course rosters that contain student ID numbers to the IR office and have them provide a file with the demographic information collected. Each assignment will be loaded separately and a separate demographic CSV file must be loaded with each set of artifacts uploaded (see webinar for details).
Collecting artifacts and uploading them into the system
For each student artifact collected use a common convention to name the document. This name must also be put into the “ArtifactFileName” field of the demographic file so it can be matched as part of the upload process. The system will automatically generate a unique identifier, based on the institution and assignment, for each artifact making it is possible to sequentially number the student artifacts without the concern that other institutions will use the same codes. Any student or institution identifying information in an artifact may need to be manually removed before the artifact is submitted.
Once the artifacts have been collected and named and the demographic file has been constructed, please follow the recommended steps below for preparing the artifacts for upload:
1. Create a folder for the assignment (on your desktop or a local drive)
2. Add the student artifact files for that assignment to the folder
3. Add the demographic file (LearningArtifact.csv) to the folder
4. Right-click on the group of files within the folder and create a .zip file (screenshot 3)
5. Upload to the system by dragging it into the screen (screenshot 4)
Screen shot 3: organize files and create a .zip file
Screen shot 4: Upload Student Artifacts .zip file
Validation rules and upload results
The system will show the status and provide feedback about any problems associated with the upload of each assignment (screenshot 5). This screen will also provide confirmation of which learning outcomes and dimensions an assignment should be assessed on as well as confirm that artifacts have been successfully uploaded.
Screen shot5: Validation
An upload may create an error for any of the following reasons:
· There is no LearningArtifact.csv in the .zip file.
· The LearningArtifact.csv file does not have the appropriate fields (see template)
· There is a missing value in the StudentID or ArtifactFileName field of the LearningArtifact.csv file
· There is a duplicate StudentID or ArtifactFileName in the LearningArtifact.csv file
· An artifact in the file does not appear in the ArtifactFileName field of the LearningArtifact.csv file.
· An artifact listed in the ArtifactFileName field of the LearningArtifact.csv file is not in the .zip file
· The artifact is not one of the accepted format types (Word, PowerPoint, Excel, PDF or txt).
Assignment Submission (Added 1/8/2015)
Once the assignment has been created, all the attachments have been added and all the artifacts have successfully uploaded you will need to submit the artifact for scoring (screenshot 6). This signifies that the assignment and its associated artifacts are complete and ready to be scored. Keep in mind that once an assignment has been submitted you will not be able to adjust the information associated with the assignment or add additional artifacts.
Requesting log-in credentials and assistance
To access the system, each institutional lead will need to request log-in credentials from their state’s MSC point person for the appropriate individual(s). Each state point person will provide SHEEO with the contact information for all individuals that should be allowed to upload files for each institution. The service provider, Taskstream, will then provide credentials directly to each user so that they can access the Artifact Management and Scoring system. Any updates or changes to system access for an individual must go through the state point person who must submit change requests to SHEEO.
Questions specific to each state’s individual state’s process should also be directed to the point person for each state.
Questions related to logging in and using the VALUE Database should be directed to Taskstream customer support at: 1.800.311.5656 or
Questions specific to the LEAP VALUE Rubrics and Learning Outcomes should be directed to AAC&U at: Terrel Rhodes, Vice President of the Office of Quality, Curriculum and Assessment, AAC&U, OR Ashley Finley, Senior Director of Assessment and Research, AAC&U, .
For all other questions or concerns please contact SHEEO at: or 303-541-1623
MSC Quick Guide for the Online Upload Process_version_01082015 Page 1 of 9
Common Upload issues
(Added 1/8/2015)
1. If an error message appears when you log in (“Encrypted value cannot be null”), switch to another browser (Chrome is recommended; Internet Explorer is not recommended).
2. Do not include the assignment instructions, cover sheet, and supplemental information in the zip file.
3. If the upload of artifacts was not successful, (1) check the spelling of the column headers in the CSV file; (2) make sure that the column headers perfectly match the way the column headers appear in the guide—for example, do not put spaces between the words in the column headers (it is “ArtifactFileName” and not “Artifact File Name”).
4. In the CSV file, the file names in the ArtifactFileName column needs an extension (i.e., .doc or .pdf). For example, the file name should be “520-01.doc” and not “520-01”).