E-6A Aviation Maintenance Training Curriculum Evaluation: A Case Study

As proposed earlier, the proper organization of declarative knowledge into schemata or mental models is the first step on the road to expertise. In maintenance training, building appropriate mental models is an essential precursor to building skills. As the factor analysis indicates, students and instructors are very concerned about how well the current curriculum teaches facts, concepts and principles that relate to component location, function and system operation.

Survey results indicated that both instructors (M=3.94, SD=.85) and students (M=3.70, SD=.81) were confident that, after completing instruction, students would be able to locate a specific component on the aircraft and explain its function. As Figure 2 illustrates, the results of a Mann-Whitney U-Test indicated that there was no significant difference (U=591, p<.116). However, survey results do appear to indicate that instructors are slightly more confident in students� ability to perform this memory task than the students are themselves.

Another survey question that dealt with declarative knowledge asked instructors and students to comment on how well they understood how the systems they were studying interacted with other systems on the aircraft. Figure 3 illustrates those results. Here, there were significant differences reported using the Mann-Whitney U-Test (U=459, p<.002). On this issue, instructors (M=4.43, SD=.51) were more confident than students (M=3.96, SD=.77) were in their ability to understand system interaction.

Given that the responses by both students and instructors to these issues were, on the whole, positive, how important are these results? On one hand, the low self-efficacy (Gist, 1989) displayed by students may be a natural consequence of their interaction with unfamiliar facts, concepts and principles associated with aircraft components and systems. On the other hand, these results may indicate that current instructional strategies and delivery technologies may be interfering with building learner confidence. This issue is investigated further in the subsequent sections of this study.

While declarative knowledge deals with how mental models are constructed, procedural knowledge is used to deploy strategies or procedures based on those mental models. Naturally, in this situation, the novice learner is prone to mistakes. That is why opportunities for practice are so essential to building skills. The factor analysis revealed student concerns with the quality and quantity of practice, with job knowledge and troubleshooting among their concerns. Instructors shared this concern, listing the same issues.

In maintenance training, the schoolhouse expends considerable effort making sure that students get the practice they need to build skills. Both instructors and students think that the labs are a crucial component of training. As one student told me, "The classroom is just setting you up for the lab. Honestly, I could care less about the class. The lab is where everything keys in." The instructors told me that they go to great lengths to ensure that students get adequate time in the lab. Yet when queried about this, students indicated that they only somewhat agree with this statement.

Figure 4 shows significant differences reported (U=366, p<.001) between students (M=3.43, SD=.83) and instructors (M=4.19, SD=.60) on this issue. Once again, perhaps it is just the beginner�s anxiety regarding their understanding of novel procedures. However, this issue may also indicate concerns about the quality of the lab experience as well as concerns about the quantity of lab time.

In the case of both the panel trainers and the open-frame trainers, only a limited number of students can interact with the trainer at once. This means that some students must wait for a turn, either observing the exercise or studying at an alternate location. Instructors are sensitive to this problem and attempt to schedule as much time with the trainers as the class schedule permits.

However, instructors remain concerned about this issue. Figure 5 indicates significant differences. Instructors (M=3.58, SD=.81) more readily agreed (U=491, p<.010) with the statement that the students (M=3.09, SD=.83) needed more hands-on practice.

There are also quality issues relating to the limitations on the number and types of gripes (faults) the instructor can program into a problem-solving scenario during lab sessions. This is particularly true when using the panel trainers to teach troubleshooting. Although instructors say the panel trainers can be effective teaching tools, some planning and effort needs to be expended before the lab to ensure that the students can practice solving the gripe in a somewhat realistic manner. The problem most often centers on the MOMI (Manual of Operation and Maintenance Instructions) which document panel trainer system operation. The fundamental problem is that the trainer MOMI often calls for procedures that the technical publications do not. According to instructors, this problem is compounded by the fact that the panel trainers cannot replicate many of the gripes found in the technical publications. In discussing the panel trainers, one instructor told me. "They�re obsolete because you can�t change them. You can�t make them conform to the maintenance manual and that�s how we fix airplanes."

The factor analysis also reveals that both students and instructors are concerned about how much effort students are willing to invest time in studying. The surveys indicate that both students and instructors agree that it is important to study frequently. A Mann-Whitney U-Test found no significant differences between groups. Since none of the training tracks require any work (homework or otherwise) outside of class time, the most students might have to do on their own would be to review their class notes. According to instructors, students need only show up, do the minimum acceptable level of work and they should be able to get through their training track with no problems. Still, students have difficulty. "I didn�t think this stuff was hard to learn, but a lot of these guys are having problems grasping it. And it�s not that difficult to learn, especially when you have (the manual) right open in front of you, telling what it is. And when you take a test, there�s books with every answer in there and these guys still fail tests," said one instructor, referring to this performance deficit among, what he described, as a sizeable minority of students. Another instructor said, "I think the Navy school system has come along like civilian schools. We�re pushing people through. We�re not looking at people, okay? Like, why is he failing? If he does fail a section, we�ll kind of push that aside and move him on to the next phase. We get a lot of students like that."

Is this instructional philosophy in the best interests of the schoolhouse or the E-6 community at large? Most instructors don�t think so. "We should not be afraid to drop someone or fail them from the program and then send them to another rate, mess cooking or whatever. But we get students coming through here that have failed numerous tests. When they go to an ARB (Academic Review Board) and (subsequently) fail again, they should get the boot. But they end up staying somehow or another and going on through to the squadron and they don�t amount to anything at the squadron."

During one site visit, I had an opportunity to observe an ARB that was convened to counsel an airman who had just failed his third attempt at the final exam in the reel operator course. During the session, the ARB committee asked him why he performed so poorly on the test. The student said that he had a medical condition that was distracting him from studying and, on his last try to pass the test, the order of the questions on the test had been changed from the two previous versions. He thought this was both confusing and unfair. The student went on to say that he really didn�t want to become a reel operator anyway and asked to be transferred to another rate. After some discussion amongst the committee members, the student was dismissed from the program for non-academic (motivational problems) reasons.

How often are students dismissed from the program for academic reasons? In FY97, the student throughput for all training tracks at the schoolhouse was 213. Of those 213, six were dismissed. Of the six that were dismissed, only two were released for poor academic performance.

The results of the classroom observations indicated that, although military decorum is enforced, instructor attitudes towards students in the classroom lend themselves to the creation of a moderately informal atmosphere. The instructors are, by and large, enthusiastic in their delivery of instruction. "Everybody�s relaxed. Everybody is calm. Time just flies. You almost enjoy yourself," said one student.

Instructors generally react well to student needs, answering questions or explaining points of difficulty when brought up by students. There is some effort to use questions to guide learning. However, instructor questioning tends to be routine with few attempts at using questioning as a strategy to generate discussion among the class.

The instructional approach centers on lecture. Occasionally, the instructor may call upon a student to point out a component on a chart. Students mostly remain silent, passive onlookers. Student participation appears to be limited to students taking turns reading enabling objectives aloud. During lecture, the instructors make frequent use of review and use examples (i.e., sea stories) from their own experience to elaborate on the concepts and principles being presented. Instructors also make sure that all the class understands the material before moving on.

The technology used in the classroom is limited to charts (for example, photo enlargements of components in racks for the communications course) or overhead transparencies. According to the survey, students and instructors generally thought that classroom media were clear and easy to understand. But there are some design issues that may hinder effectiveness. For example, during one observation, I noticed that the instructor taught the entire lesson from a single overhead. The overhead itself illustrated a subsystem and was surrounded by call-outs (labels). Without the instructor�s interpretation, I would have found the overhead quite perplexing. One student told me that, "I noticed there were a lot of overheads and schematics they (the instructors) didn�t use. They were just too confusing or incorrect."

As was the case with classroom instruction, the labs also provided a moderately informal atmosphere for learning. The instructors displayed a fair amount of flexibility in dealing with questions brought up by students, often taking extra lab time to ensure that questions were answered to the student�s satisfaction. There was also some effort to use questions to guide discussion where appropriate. Unlike the classroom, students were much more involved in the learning process. There were multiple opportunities for students to practice preflight checks, maintenance checks, and troubleshoot gripes.

Technology issues affecting instruction in the lab were discussed in detail in an earlier section. Suffice it to say that there are design issues relating to the panel and open-frame trainers that also affect the quality of instruction. These issues are discussed in a later section.

According to the survey, both the students and instructors were confident that, by the time they finish training, students will have the necessary knowledge and skills to perform well as entry-level aviation maintenance technicians.

A factor analysis based on descriptive statistics derived from the supervisor and technician surveys was conducted in order to elucidate areas of concern for each group. The results of this analysis are summarized in Table 5.

Table 5: Technician/Supervisor critical issues.

As noted in the Process Evaluation, procedural knowledge refers to the student�s ability to properly deploy the cognitive strategies (skills) needed to successfully interact with the job environment. Assuming a solid foundation in system concepts and principles, the key to preparing entry level maintenance technicians for the challenges they will face on the job rests on the quantity and quality of their skills training. Obviously, the locus for this type of training is the schoolhouse lab.

While students (M=3.43, SD=.83) and instructors M=4.19, SD=.60) believed that students get enough lab time for practice, application of a Kruskal-Wallis H-Test shows significant difference (X² (3) =72.63, p<.001) in attitudes towards this question when compared to technicians (M=2.71, SD=1.20) and supervisors (M=1.88, SD=.74) responses. As Figure 9 shows, both more experienced technicians and supervisors believed students need significantly more lab time to practice. Moreover, the survey indicates that both groups believe that entry-level technician skills are deficient in several areas. As one supervisor said, "The initial qualified people think. �Oh cool, I know all this stuff.� And then they really get slammed when they get to the shop and realize they know just enough to be dangerous."

A Kruskal-Wallis H-Test of the mean responses on survey items concerned with procedural knowledge yielded statistically significant differences when looking at all four groups. In particular, the data indicated that shop supervisors and more experienced technicians believe that students are ill prepared to meet job requirements.

One of the critical job requirements is the ability to understand technical publications. Classroom observations and extensive discussions with instructors and students indicate that the schoolhouse puts great emphasis on understanding how to use the technical publications on the job. Some supervisors think that the schoolhouse is making progress in this area but not in others. One supervisor said, "They go through the initial and they might know the pubs and that�s about it. So we�re back to how it was when I first came in. You�re out there teaching them how to do everything. How to read schematics, how to order a part, the whole nine yards." Unfortunately, it appears that the aforementioned supervisor appears to be in the minority. While students (M=3.94, SD=.73) and Instructors (M=4.03, SD=.66) were confident in students� ability, Figure 10 indicates that most experienced technicians (M=3.29, SD=.87) somewhat agreed and supervisors (M=2.81, SD=1.04) did not believe that entry-level technicians have a basic understanding of how to use the technical publications (X² (3) =37.9, p<.001).

Survey results also showed significant differences (X² (3) =84.14, p<.001) among respondents in regards to entry-level technicians� ability to properly use test equipment, as Figure 11 illustrates. While students (M=3.7, SD=.78) and instructors (M=3.81, SD=.79) were confident that students had this skill, supervisors (M=2.12, SD=.82) and more experienced technicians (M=2.35, SD=.81) disagreed with this assessment.

Another job skill is to be able to document gripes and order parts using the software designed for that purpose, NALCOMIS (Naval Aviation Logistics Command Management Information System). The schoolhouse does not have responsibility for including instruction in NALCOMIS in the curriculum, although it still uses the paper-based VIDS/MAF (the precursor to NALCOMIS) in some of its practical tests. The squadrons report that it is difficult to schedule classes in NALCOMIS so instruction for technicians (entry-level and otherwise) becomes something that is handled on the job as the opportunity arises. Since there is no systematic instruction in the system, users are often left to their own devices.

The survey asked instructors (M=2.79, SD=.87), supervisors (M=4.44, SD=.81) and more experienced technicians (M=4.27, SD=.99) whether entry-level technicians should know NALCOMIS. As one might expect, a Kruskal-Wallace H-Test yielded significant differences (X² (2) =40.6, p<.001), as Figure 12 illustrates.

All of these skills, using the technical publications, test equipment, and using NALCOMIS to track gripes and order parts, are viewed by supervisors as being critical to the job. As one supervisor said, "It would be nice if (entry level technicians) knew how to use test sets. It would be nice if they knew the pubs. If they could order parts and use NALCOLMIS that would make my job a lot easier." All of these skills also contribute to troubleshooting a gripe. While it would be unreasonable to expect that the schoolhouse should turn out expert technical problem solvers, all groups said it was reasonable to assume that entry-level technicians should have a basic understanding of the process.

What is also interesting is how significant the differences in perceptions between groups were in regards to the breadth of training transfer (X² (3) =48.36, p<.001). While the students (M=3.96, SD=.88) and instructors (M=3.9, SD=.87) that were surveyed were confident that, by the time they finish training, students have a good understanding what is expected of them on the job, more experienced technicians (M=2.76, SD=1.0) and supervisors (M=2.72, SD=.97) disagreed with this assessment. They indicated that entry-level technicians had little understanding of the job requirements, as Figure 13 shows.