One-to-One+Evaluation

FORMATIVE EVALUATION ACTIVITIES


 * One-to-One Evaluation - SME **

1. **Participant: Subject Matter Expert.**
 * 1) **Troy Stevens - Shippensburg Area Schools Technology Coordinator **- As the Technology Coordinator of Shippensburg Area School District, Mr. Stevens, is the resident expert in all areas of technology and training that occurs in the District. His credentials include a Masters of Learning Technology from Bloomsburg University and he is currently seeking his Doctorate in the same. Mr. Stevens has a vast working knowledge of Instructional Design and is acutely aware of the technology needs for the District. He and agrees that a solution is needed to reduce on-site calls.

2. **Materials: SME will be provided with the following**:  3. **Revisions: Submitted to the SME for comments/revisions in the** ** following ** **areas**:
 * 1) Instructional Analysis
 * 2) Performance Objectives
 * 3) Instructional and Embedded Assessments
 * 1) Objective statement
 * 2) Instructional analysis
 * 3) Appropriate sequential steps - for both classroom and hands-on portion of the training
 * 4) Appropriateness of the instructional materials in vocabulary, interest value, sequence, chunk size, and learner participation activities
 * 5) Clarity and appropriateness of test items and assessments situations
 * 6) Placement of this piece of instruction relative to prior and subsequent instruction


 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 16px; line-height: 23px;">One-to-One Evaluation - Participation by Learners **

<span style="font-family: 'Times New Roman',Times,serif; font-size: 120%;">1. **Participants:**
 * 1) <span style="font-family: 'Times New Roman',Times,serif;"> Teacher - confident in technology - high level of computer skill - will perform a few troubleshooting techniques (check cables to see if tight, etc.) before contacting IT Dept
 * 2) <span style="font-family: 'Times New Roman',Times,serif; font-size: 16px; line-height: 23px;">Secretary - very confident in technology - high level of computer skill - however high tendency to contact IT Dept for all troubleshooting issues.
 * 3) <span style="font-family: 'Times New Roman',Times,serif;"> Cafeteria Manager - not very confident with technology - adequate level of computers skills but only within areas that she is comfortable - will call IT Dept for all technology related issues

<span style="font-family: 'Times New Roman',Times,serif; font-size: 120%;">2. **Arrangement and Instructions to Participants (Learners):**

<span style="font-family: 'Times New Roman',Times,serif; font-size: 120%;">Make arrangements to meet individually with target learners to discuss the potential training and to get their reactions on the material as proposed. Have them evaluate all test, sequencial steps and materials of the proposed workshop. Record their reactions and note all comments for future revisions. Apply revisions from one target learner to the next target learner to see if the changes make an impact on the flow of the workshop as proposed.

<span style="font-family: 'Times New Roman',Times,serif; font-size: 120%;">3. **Introduction of Instruction to the Target Learner:**

<span style="font-family: 'Times New Roman',Times,serif;"> Explain to the learner that a new workshop is being proposed for the District and that you would like them to review all the instructional materials for their reactions, comments and possible revisions. Explain to them how important their comments will be to the overall design. Remind them that most mistakes in the learning process are usually due to poor design. Encourage the learner to relax and feel free to talk about the material and changes at anytime during the evaluation. Inform them that they will be taking and evaluating the pre- assessment test and the post-assessment test. In addition, they will be reviewing all other training materials to include PPT presentation, TRP manual and workbook.

<span style="font-family: 'Times New Roman',Times,serif; font-size: 120%;">3. **Materials to be review:**
 * 1) <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%;">Pre-assessment Test
 * 2) <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%;">Powerpoint presentation
 * 3) <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%;">TRP Manual
 * 4) <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%;">TRP Workbook
 * 5) <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%;">Post-assessment Test

<span style="font-family: 'Times New Roman',Times,serif; font-size: 120%;">4. **Data Collection :**
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 16px; line-height: 23px;">Clarity of Instruction:
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 16px; line-height: 23px;">Sit with learner while they take the pre-assessment test. Observe as they read through the test, watch for any changes or notes they make to the questions. Take note of the occurrences when additional clarification of a question is required. Ask the learner if they can suggest a better sequencing of information, longer or shorter segments. Ask them mark with a "circle" or "x" any vocabulary or directions that are unclear. Record their ideas along with yours for later revisions to the test.
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%;">Evaluate the TRP manual and corresponding workbook. Are the directions in the manual clear, are the sequences of instruction easy to follow, rate the graphics, figures and illustrations for clarity. Ask them to make any revisons on the pages or within the margins. Do you think the manual would be a good troubleshooting tool? Is the workbook easy to follow with the TRP as a reference? Do you think the workbook is a good training tool? Again, ask them to make any revisons/comments to all materials.
 * <span style="font-family: 'Times New Roman',Times,serif;"> Ask learners to evaluate the slide presentation (in paper form). Do the slides convey the material so it is easy to understand and follow. Are the slides congruent to the training materials? Are the graphics a true depiction of the hardware.
 * <span style="font-family: 'Times New Roman',Times,serif;"> Explain to the the learner about the proposed second phase of the training. The "hands-on" portion. Do they think it would be beneficial to the training? If so why, if not why not? Note all answers.
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%;">Ask if they have ever attended a similar training and if so what were the procedures. Would they add anything to the training that was overlooked.
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%;">Again, observe the learner as they take the Post-assessment test. Give the same instructons as the pre-assessment test. Have them mimic all actions as with the pre-assesssment "circle" or put an "x" on unclear directions and/or vocabulary. Rate the questions for clarity. Again, record all revisions as indicated. After reviewing all the training materials, was the post-assessment test congruent with what you reviewed/learned?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 16px; line-height: 23px;">Impact on Learner:
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%; line-height: 27px;">Question the learner on the information as presented and what the overall feelings is about the potential workshop. Did she/he feel the training was relevant to the work environment. Did the workshopprovide the skills needed to perform the tasks as assigned, check for connectivity? Did you find the training difficult to follow or understand. If so, please explain. Were the training examples and materials easy to follow? Did having the hardware available for demonstration purpose make the process easier to comprehend? Do you feel that you can perform the actions, check for connectivity, from the materials and training that you <span style="font-family: 'Times New Roman',Times,serif; font-size: 16px; line-height: 27px;">evaluated today? Do you think the workshop, as described today, would make you more confident with technology and able to perform problem solving techniques to resolve connectivity issues in your work environment without IT intervention? If no, can you give examples of what changes are needed in the instruction to help you better understand/learn the problem solving skills as indicated.
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%;">Feasibility:
 * <span style="font-family: 'Times New Roman',Times,serif;"> Evaluate from the collected information how feasible the training actually is dependent on the target learners. Is their maturity level appropriate for the amount of information and time allotted for the training. Determine from the target learners, if they will be physically able to perform the task as required. Will the training make them confident enough to actually perform the problem solving techniques to solve connectivity problems in their own environments? Estimate the time, resources, and training materials. From all the data collected evaluate if the workshop is really feasible.

<span style="font-family: 'Times New Roman',Times,serif; font-size: 120%;">5. **Assessment & Questionnaires** : <span style="font-family: 'Times New Roman',Times,serif; font-size: 16px; line-height: 23px;">Assessment - Review the all training materials with the learner, ask them about their comments and any notes that they made while evaluating the materials. Ask them about the questions they had difficulty with, how/why would they change the question for better clarity. Do the same for each piece of the training materials as reviewed. Use this information to remove any faulty items before each target one-on-one and before the small group evaluation.
 * <span style="font-family: 'Times New Roman',Times,serif;"> Questionnaire :
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0.5em; padding-bottom: 0px; padding-left: 3em; padding-right: 0px; padding-top: 0px;">How do you feel about the workshop as proposed?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0.5em; padding-bottom: 0px; padding-left: 3em; padding-right: 0px; padding-top: 0px;">Did the instruction hold your attention?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0.5em; padding-bottom: 0px; padding-left: 3em; padding-right: 0px; padding-top: 0px;">Was the instruction too long or too short?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0.5em; padding-bottom: 0px; padding-left: 3em; padding-right: 0px; padding-top: 0px;">Was the instruction difficult or too easy?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0.5em; padding-bottom: 0px; padding-left: 3em; padding-right: 0px; padding-top: 0px;">What parts of the instruction were unclear, and why?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0.5em; padding-bottom: 0px; padding-left: 3em; padding-right: 0px; padding-top: 0px;">Was the PowerPoint presentation graphically appealing, informative, work well in conjunction with the TRP Manual
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0.5em; padding-bottom: 0px; padding-left: 3em; padding-right: 0px; padding-top: 0px;">Did you find the TRP Manual easy to follow?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0.5em; padding-bottom: 0px; padding-left: 3em; padding-right: 0px; padding-top: 0px;">Were the instructions and the sequential steps as outlined in the TRP Manual easy to follow and mimic in a true life classroom scenario
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0.5em; padding-bottom: 0px; padding-left: 3em; padding-right: 0px; padding-top: 0px;">What did you like most from the training?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0.5em; padding-bottom: 0px; padding-left: 3em; padding-right: 0px; padding-top: 0px;">What did you dislike from the training and why?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0.5em; padding-bottom: 0px; padding-left: 3em; padding-right: 0px; padding-top: 0px;">Do you think you will use the problem solving techniques learned today?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0.5em; padding-bottom: 0px; padding-left: 3em; padding-right: 0px; padding-top: 0px;">Do you think having the materials available to see and use in the instruction made the problem solving strategies easier to comprehend?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0.5em; padding-bottom: 0px; padding-left: 3em; padding-right: 0px; padding-top: 0px;">Did you enjoy the "hands-on" potion of the training?
 * <span style="font-family: 'Times New Roman',Times,serif; font-size: 120%; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0.5em; padding-bottom: 0px; padding-left: 3em; padding-right: 0px; padding-top: 0px;">Would you be interested in attending other TRP workshops? If not, please explain.

<span style="font-family: 'Times New Roman',Times,serif; font-size: 120%;">6. **Outcomes:**
 * <span style="font-family: 'Times New Roman',Times,serif;"> Evaluate all the data collection to determine if the proposed changes will makes the instruction contain more appropriate in reference to vocabulary, language complexity, examples and illustrations for the participating learner. Will it yield a more reasonable learner attitudes and achievement? Is it the workshop feasible to the target audience, available resources and materials?
 * <span style="font-family: 'Times New Roman',Times,serif;"> The One-on-One was a very positive experience. The data and attitudes of the target learners showed a very positive results with minimal changes to the workshop as a whole. There are a few changes to the questions that were noted and the revisions will be implemented before the Small-Group Evaluations are conducted. Visual changes will also be made to the PPT. It was noted, that the color choices (red with black lettering) was not visually appealing to the learners. Most indicated that white font on a darker background might be a better visual choice ( Al). The learners were in support of the TRP manual and thought the workbook was easy to follow and a good training aid in conjunction with the manual. They also felt that both the manual and the workbook will be a valuable resource as a quick reference guide after the completion of the training. Most felt that the workshop would provide them with the skills needed to problem solve their connectivity issues without IT intervention. The results were also positive in the offering of additional TRP workshops.
 * <span style="font-family: 'Times New Roman',Times,serif;"> SME results are still pending in receipt of the forwarded materials and questionnaire.