EduGuru

Examining the 12 Elements for Data Systems (America Competes Act)

The Feds are going to require all states taking stimulus money for P-12 education to have 12 “elements” in their statewide education information systems.  The Eduguru is going to examine each of the 12.  In case you’ve been in a cave for the past few years, the first 10 elements are the components from the Data Quality Campaign recast in the America Competes Act.  Nothing bad there (see The Process for Ensuring Data Quality, ESP’s Optimal Resource Book at www.espsg.com/resources.php for a review).

 The Eduguru rates each element on two scales from 0 (lowest) to 10 (highest).

• How important is it for the Feds to track this? (So what?)
• What’s the likelihood the states are going to pull this one off? 

Then the Eduguru makes a summary declaration.

• What’s the bottom line on this one?

Number 1:  A Unique Statewide Student Identifier

• How important is it for the Feds to track this? 1
• What’s the likelihood the states are going to pull this one off? 10
• What’s the bottom line on this one? Save everyone’s time reporting this one.

The unique student identifier must be assigned statewide and “not permit the student to be individually identified by users of the system.”  What?  Isn’t that the purpose of a unique identifier?  In this case the point is to allow a student’s records to be matched up across years and across files without a viewer of reports being able to figure out that student’s personal identity.  That can be done these days. 

In fact count this one as done in all states.  Maybe not done completely, but there’s no argument about this one being reality.  The challenge will be whether or not the safeguards will protect every student’s identity.  The betting line?  There will be problems through carelessness.  Get ready for them.

The Eduguru speaks:  Glad this is number one, so we can begin with a victory.  Each one gets harder as we go.

Number 2:  Student-Level Enrollment, Demographic, and Program-Participation Information

• How important is it for the Feds to track this? 10
• What’s the likelihood the states are going to pull this one off? 7
• What’s the bottom line on this one? They can do it. They should do it.

Ahh, content for the data system.  The key here is defining the data elements in each area.  Yes, the metadata standards are paramount.  Even the terms enrollment, demographic, and program have different meanings across our schools—and even across schools within districts at times. 

All across the nation, people will start slapping into data warehouses all the data they already have about when students enroll, what ethnicity they are, and what programs they are in—but wait—that’s the problem.  The data they already have is inconsistent, incomplete, inaccurate, and potentially incomprehensible to the National Education Data Model (NEDM) with which everything may get aligned.  

The Eduguru speaks:  There’s important work to be done up front to clean up our definitions, codes, and formats before we concatenate a cacophony of connotations into an incomparable little data warehouse of horrors.

Number 3:  Student-level information about the points at which students exit, transfer in, transfer out, drop out, or complete P-16 education programs

• How important is it for the Feds to track this? 9
• What’s the likelihood the states are going to pull this one off? 9
• What’s the bottom line on this one? Simple accounting.  Get it done.

Pardon me.  How simple is it for us to count when students come and go?  Our student information systems do this every day.  The significant data are those about why, what factors contributed to the exits, what would have changed the student’s decision.  There will be significant effort invested in moving the same old data once again from schools to states to the Feds to count students, when we need to be taking new, fresh looks at our students.

The Eduguru speaks:  Let’s get beyond this one.  The important, actionable data will tell us why kids come and go.

Number 4:  Capacity to communicate with higher education data systems

• How important is it for the Feds to track this? 8
• What’s the likelihood the states are going to pull this one off? 3
• What’s the bottom line on this one? Look to the higher education systems, especially the private institutions who keep their shades drawn and the community colleges with tremendous mobility to make this difficult.

Let’s assume this means a two-way communications.  (Again, this element suffers from ambiguity.)  Requires someone on both ends.  Wrong word—capacity.  Should have been willingness.  Private schools may not play this game.  Community colleges may not see the point and be overmatched with the volatility in their student mobility.  What do they gain from exchanging data bout students that attend their college for such brief periods, take such random courses, and don’t complete degrees?

This could have been the most vague of the 12 elements had it not been for #12 taking that dishonor.

This overlaps significantly with #9, which refers to transcript information.  #9 should be communicated—electronically—using common standards.  We should be focusing on what is to be communicated and for what purpose.  This “element” is dysfunctional as any kind of mandate to which a state could plan or design their information system.  Is capacity irrespective of ever actually exchanging any meaningful data?  No.

The Eduguru speaks:  Let’s define the meaning of capacity or we risk its being defined as what we have already.

Number 5:  State data audit system assessing data quality, validity, and reliability

• How important is it for the Feds to track this? 10
• What’s the likelihood the states are going to pull this one off? 2
• What’s the bottom line on this one? There are no standards and no will to enforce any.

What is the standard for data quality?  Validity?  Reliability?  Fact is validity and reliability have definitions in the statistics textbooks.  What complicates the matter is that one must set the criterion for each like setting the odds for a bet.  Do we want tough 99% or loose 67%?  Oh, and we need to make the decision as to who sets the criterion. 

Remember when NCLB let small groups off the hook if their failing percent proficient was not “reliable”?  Some states set the criterion high, some low.  Some states declared groups as meeting AYP if their failing percent was unreliable—therefore not reportable.  Yes, they really did—I mean, they really do.  Deciding who sets the criterion is very important.

The reality in states that monitor data quality now is that there are two factors that matter.

• Submitting data (reports) on time
• Submitting complete data (filling all required the blanks/fields)

Hopefully, this “element” expects more, like…

• Alignment with standard definitions, codes
• Conformance with formats and transmission standards
• Accuracy reflecting the data within local transactional systems
• Validity reflecting the intent of the data element being recorded and reported

The Eduguru speaks:  Quality is in the eye of the beholder.  If the state will be holding the data, then the state will be defining the quality.

Number 6:  Yearly test records of individual students with respect to assessments under section 1111(b) of the Elementary and Secondary Education Act of 1965 (20 U.S.C. 6311(b));

• How important is it for the Feds to track this? 10
• What’s the likelihood the states are going to pull this one off? 9
• What’s the bottom line on this one? There are no tech, practical, or fiscal reasons not to comply.

Let’s assume this includes linking yearly assessment to measure growth for individual students.

We may still argue about what to test, when to test, and how hard the questions should be, but we can score those test and match them up from one year to the next.  The technical, practical, and fiscal challenges of this “element” are being handled now.  Unless the Feds make demands for more standardization of what, when, and how hard, then the states can do this. 

The Eduguru speaks:  Don’t we all wish that our cumulative test record meant that the scores were added together across the years?  Eventually we’d all have pretty decent scores.

Number 7:  Information on students not tested by grade and subject

• How important is it for the Feds to track this? 10
• What’s the likelihood the states are going to pull this one off? 6
• What’s the bottom line on this one? There are no tech, practical, or fiscal reasons not to comply; however, there are tremendous disincentives for keeping accurate records.

This hallmark of Bush’s NCLB was bashed initially, then admired for ensuring that every child counted.  Schools could no longer suggest some students stay home on test day.  (They really had done that.)  The low rating for the likelihood the states are going to pull this one off is not that they can’t report this statistic.  They can report.  The pessimism is that they will ever faithfully commit to testing every student and reporting accurately every one not tested.

The Eduguru speaks:  All skate!  Even if someone is going to fall down. 

Number 8:  A teacher identifier system with the ability to match teachers to students

• How important is it for the Feds to track this? 10 for secondary schools, 5 for elementary schools
• What’s the likelihood the states are going to pull this one off? 9 for secondary school, 5 for elementary schools
• What’s the bottom line on this one? Who really teaches each elementary student each subject area?

Surprised this isn’t already done routinely?  Surprised to know a state would have a law preventing such linking?  Why do this anyway?  Simply put, this is for teacher accountability.  Also for research and evaluation.  This has been done for decades in secondary schools with distinct courses and class periods.  For elementary schools—a culture change and/or a recordkeeping change is required most places. 

Would it be appropriate to suggest that maybe the culture of elementary schools still argues for a grade-level or even schoolwide evaluation in many schools? 

The Eduguru speaks:  Assigning the identifiers—trivial.  Using them to link teachers and students for accountability—priceless. 

Number 9:  Student-level transcript information, including information on courses completed and grades earned

• How important is it for the Feds to track this? 10
• What’s the likelihood the states are going to pull this one off? 6
• What’s the bottom line on this one? There are no tech reasons not to comply.  States vary in their capacity.

Does this level of detail need to reside at the state?  The time may just be here that the answer is “yes.”  Student mobility, research, the comprehensiveness of mandated reporting, state-level scholarship programs, college admissions rules, etc. may be compelling reasons.  Schools can even use the state for back-up or permanent record storage for archived student records. 

States vary considerably in their capacity to comply with this.  The rating of 6 for likelihood could rise over time with the proper use of stimulus dollars. 

The Eduguru speaks:  No more parchment, no more gold stickers, no more registrar’s embossed seals.  XML digital files with electronic signatures are the only acceptable academic currency for exchange going forward.

Number 10:  Student-level college readiness test scores

• How important is it for the Feds to track this? 5
• What’s the likelihood the states are going to pull this one off? 3
• What’s the bottom line on this one? Are we going to mandate universal participation at a single administration date?

Who says the college readiness tests are worthy of our use for evaluating school effectiveness?   Aren’t these tests being criticized for not predicting success in college?  Aren’t admission officers using alternative criteria to make their selections.  A reading of Outliers shows us that these tests may be useful as cut scores—up or down decision makers, but not good enough to differentiate students along a meaningful scale.  That would mean they don’t predict how well someone will perform in college after they get past the hurdle of getting through the door.  After all, the competition beyond the door is not the same across those colleges.

Are we yet able to compare the ACT and SAT with their contrasting philosophies, content, scoring, and participants?  Not all students are planning to go to college, so they are not going to be serious about one of these tests.  These tests are expensive. 

The Eduguru speaks:  There just is not a compelling justification for this expensive, unreliable, difficult to interpret “element.” 

Number 11:  Information regarding the extent to which students transition successfully from secondary school to postsecondary education, including whether students enroll in remedial coursework

• How important is it for the Feds to track this? 9
• What’s the likelihood the states are going to pull this one off? 3
• What’s the bottom line on this one? First define a few terms, then start gathering some data.

This element needs to be better defined, maybe divided into more than one criterion.  Transitioning successfully appears to refer to being academically prepared and not requiring remediation.  Does it also refer to being mature enough to handle college life, to manage the responsibility, to earn credits whether or not the student has the skills and knowledge already?

Then we’ll need to agree on what a remedial course is contrasted with an introductory course.  We’ll want to level the playing field across colleges for when remedial courses are required.  Oh yes, let’s distinquish whether a student failed other courses because the student was ill prepared, took too difficult a course, didn’t study, partied too much, etc.

We’ve been at this for far too little time to have worked out the kinks.  The postsecondary institutions are clearly behind elementary and secondary education in the development of their information systems.  (Remember we are talking about the ability of those information systems to exchange data across institutions and with the state and Feds.) 

The Eduguru speaks:  Standards must be developed first. 

Number 12:  Other information determined necessary to address alignment and adequate preparation for success in postsecondary education

• How important is it for the Feds to track this? 0
• What’s the likelihood the states are going to pull this one off? 0
• What’s the bottom line on this one? This statement itself is clear evidence of how far higher education is behind elementary and secondary education.

How is this different from #11? 

The Eduguru speaks:  Any court would throw this “element” out as being too vague to be enforceable.  My fear is that it is so vague that the Feds will be able to use it to mean whatever they want it to mean.  Watch out for this one.

 

Diagnosis for Teachers.

With the changes in the Executive Branch, NCLB will change, but how?  The Eduguru is continuing a series of blogs on the problems opponents have had with NCLB and what they might or should do about them now.

The seventh is focus:  Diagnosis for teachers.

The annual statewide assessment for accountability is not for teachers.  That’s correct, depending upon this high-level, annual assessment to provide teachers useful diagnostic data is silly.  The focus of these assessments is accountability.  Teachers need a separate set of assessments for diagnosis.  See Why Eva Baker Doesn’t Seem to Understand Accountability, ESP Solutions Group Optimal Reference Guide.

NCLB naively requires timely reporting to teachers for diagnosis.  That’s fine, but ask a teacher how useful the state assessment results are to them.  Interesting, yes.  Important, yes.  Meeting their needs for diagnosis, no.  Psychometrically, what makes for a great accountability test does not make for a decent diagnostic assessment.

Congress should authorize separate diagnostic assessments if they want teachers to benefit from the statewide assessment programs.

The EduGuru speaks:  Seen the satellite photos of your house and neighborhood?  Great for judging where you are relative to community services and stores.  Not very useful for determining what renovations need to be made.

Timely Reports for Parents.

With the changes in the Executive Branch, NCLB will change, but how?  The Eduguru is continuing a series of blogs on the problems opponents have had with NCLB and what they might or should do about them now.

The sixth is practical:  Timely reports for parents.

The aspect of NCLB that may be the farthest from being implemented well is the timely reporting of AYP results for parents to make school choice decisions.  The culprit is the long cycle time for scoring assessments.

Pushing the testing dates earlier in the school year diminishes the validity of the results for parents.

This requirement is one that should not be changed.  The states and assessment companies need to rethink their processes to reduce the cycle time for scoring and reporting.  Granted, all states pile up with their testing and scoring needs during the same time period.  How can Texas get millions of tests scored and reported back to schools within a few weeks and other states require much longer?  Negotiation skills and money.

The states with the best cycle time are those that manage their processes on the front end the best.  Pre-coding accurately and shortly before testing.  Using the assessment contract as a vehicle for scoring, not for collecting demographic and other student data that should already be in the state’s data store.  Ensuring schools provide compliant answer documents that will not require trouble shooting by the scoring service.

The Eduguru speaks:  There’s plenty of money in the system for timely scoring and reporting.  This is a case where processes need to be improved to reduce cycle time.

Grade Levels Assessed.

With the changes in the Executive Branch, NCLB will change, but how?  The Eduguru is continuing a series of blogs on the problems opponents have had with NCLB and what they might or should do about them now.

The fifth is expensive:  Grade Levels Assessed.

What if we add more grade levels to the accountability system?  Some grade levels (e.g., PK-2, 11-12) are not formally assessed for inclusion in AYP.  Some sort of measurement is required, especially for schools with only these unassessed grade levels.  Adding appropriate assessments statewide would be expensive.  Continuing to leave them out makes for a spotty accountability system.  This is especially true where student performance is being linked to teachers for pay-for-performance or annual evaluation.

Alternative measures are available, but typically are even more expensive, unreliable, and difficult to quantify (e.g., teacher ratings of students, individual reading inventories, performance measures graded by the teacher).  Eva Baker, past AERA President, has recommended a new generation of performance measures that will take years to develop, validate, and standardize.  States could merely be tasked with the obligation to figure this all out.

Growth measure advocates require multiple years of preferably adjacent grade level testing for their calculations.  If growth can’t be measured until the end of grade 4 or after grade 10, then the two spans that get the most attention in the research are missing.

The Eduguru speaks:  The NAEP-sayers must admit their omnipotent assessment falls short here also.  Will on-line assessments arrive in time to solve this dilemma?

Proficiency Across States.

With the changes in the Executive Branch, NCLB will change, but how?  The Eduguru is continuing a series of blogs on the problems opponents have had with NCLB and what they might or should do about them now.

The fourth is politimetrics:  Proficiency across states.

The contrasts in definition of proficiency across the states has been one of the most fascinating aspects of NCLB.  States rights advocates applaud allowing each state to adopt its own standard.  Of course, NAEP is the only practical and available basis for judging relative difficulty of proficiency cut points.

Proficiency will be defined more normatively—across states—possibly demanding an alignment with NAEP.  This is not as difficult to pass as you might expect considering that many of the states that have the lowest standards for proficiency and the lowest NAEP scores voted Republican.

Debates have raged for decades as districts and states adopted their own tests called minimum skills, basic skills, essential skills, grade-level skills, exit-level skills, and many other terms to designate that what’s measured is somewhere between the lowest level acceptable or a high standard for excellence.  NCLB created proficient as the goal, expecting it to be a high standard but not requiring it to be as high as NAEP.

Politimetrics, the combining of psychometrics and politics to make decisions, will determine the definition of proficiency and how it’s aligned across states.  Look for this to be one of the liveliest debates.  One current strategy of some, following NCLB but having local or state accountability ratings as well, will get more difficult to defend.  As NCLB softens up in other areas, this proficiency issue may get more controversial.  No state can publicly adopt a local proficiency standard that is below the Federal standard.  Having one that’s different in how it’s calculated or what it includes is fine, but expecting less of our proficient children is not going to work.

The Eduguru speaks:  What if NAEP became the standard, but NAEP’s Basic Level became the criterion?  Then states can be encouraged to adopt standards higher than basic—if they want.

100% in 2014.

With the changes in the Executive Branch, NCLB will change, but how?  The Eduguru is continuing a series of blogs on the problems opponents have had with NCLB and what they might or should do about them now.

The third is an easy one:  100% in 2014.

No one believed this goal would be reached, but it did set a long-term target that influenced annual objectives.  Besides, how do you argue against wanting every student to succeed?  Which ones are we willing to leave behind?

Look for Congress, the states, or the Education Secretary to make these changes.

1. Proficiency will be defined more normatively—across states—possibly demanding an alignment with NAEP.  This is not as difficult to pass as you might expect considering that many of the states that have the lowest standards for proficiency and the lowest NAEP scores voted Republican.

2. Growth will become the new controlling factor.  The indication that standards are falling would be that growth is no longer defined as being on pace for proficiency by the end of n years.  If any measurable, reliable growth becomes acceptable, then the barn door will be opened.

3. An IEP for special education or English language learners may trump test scores as the measure of growth.

4. Full academic year will be replaced by a longer term to reduce the number of low performers in the mix and to allow for them to recover using growth.  Look for a two-year term to be used to determine which students are included in AYP.

5. The Education Secretary may adopt a national standard for small subgroups to level that issue.  Look for the subgroup size to be 30 or above, or to be based upon confidence intervals that might require even larger groups.  (This is for AYP, not for reporting descriptive statistics for annual public report cards.)

6. Will subgroups survive as a disaggregation criterion?  Look for alternatives such as majority/minority.  Keep in mind that the new President fits into multiple race categories.  Maybe race goes away and is replaced by income.  Now that would be consistent with campaign talk.

7. Reporting 100%?  Congress may want to clean up the FERPA conundrum of not allowing reporting of 100% categories, unless new guidance is deemed sufficient.

8. Requiring 100%?  Very difficult to keep.  Very difficult to abandon.  100% can be kept as a distant target (some states already do this)—one that continues to move out into the distance as we approach it.  Like a mirage that disappears when you approach and reappears farther down the road.  Look for a 5 to 10 year horizon for 100% instead of a specific year—if 100% survives at all.

The Eduguru speaks:  I’m 100% certain there will be multiple changes, the combined impact of which will not be known for years.  Unless…Congress could authorize some impact studies for rule changes—after all, the states have longitudinal data now with which to test the proposed changes.

Including the Disabled in Accountability.

With the changes in the Executive Branch, NCLB will change, but how?  The Eduguru is continuing a series of blogs on the problems opponents have had with NCLB and what they might or should do about them now. 

The second is a touchy one:  Inclusion of the Disabled.

NCLB required that the disabled students be included in the accountability process.  Much delight was expressed by the parents of those students who before had been excluded from accountability processes and now have the extreme attention of educators who see many of their schools being designated low performing only because the special education student subgroup did not meet adequate yearly progress (AYP).

Relaxed guidance allowed more students to take alternative assessments.  All quota systems with a national percent cap are difficult to implement at the local level.  Local schools don’t all have a perfectly random sample of special education students matching the national numbers. 

The NCLB requirement that all students be tested on grade level impacts the special education students directly.  A grade level test may be frustrating with a preponderance of items that are too difficult for students performing below grade level. 

The challenge is how to include all students in the accountability system, but do it fairly. 

The EduGuru speaks:  The parents of special education students will rule this issue with a sympathetic new administration.  Look for special education to no longer be a subgroup (assuming the whole subgroup notion survives) with the same rules as others.  Look for Congress to give the Secretary and/or states more leeway, but ensure special education students are a part of the formula for accountability.

Change Impacts NCLB.

The Nation’s Children Love Barak, that’s not the NCLB we are going to talk about changing.

With the changes in the Executive Branch, NCLB will change, but how?  The Eduguru is starting a series of blogs on the problems opponents have had with NCLB and what they might or should do about them now. 

The first is the easy one:  Underfunded.

Congress will need to find that extra funding someplace among the bailouts.  The dollars don’t appear to be available from a windfall profits tax on big oil with gasoline below $2 a gallon again.  The reality is that Congress typically funds their own projects at a level lower than is authorized in the enabling legislation, so NCLB is not that unusual. 

Understanding what is meant by NCLB being underfunded is tough.  Just saying more money for schools in need of improvement is too vague.  Title I has been funding them for decades—and the amount of unspent Title I dollars each year raises the question of the ability of those schools to wisely use even more money.  With the complaints about the What Works Clearinghouse, where will the new administration find worthy programs on which to spend millions of dollars? 

The dollars appropriated for parents to find alternative tutoring and services for their children have been largely unspent—partly because schools discourage parents from seeking external help, and partly because the districts and states haven’t identified and publicized the alternatives as well as they could have. 

Let’s assume Congress provides more dollars for the Elementary and Secondary Education Act, aka Improving America’s Schools Act, aka the No Child Left Behind Act.  How will they be allocated to states, and what restrictions will accompany them?

The EduGuru speaks:  Education wasn’t much of an issue during the election.  Looks like GM/Ford/Chrysler, mortgage lenders, and another round of stimulus checks will get the first money.  Low-performing schools don’t cost us money—directly.  They don’t depress the economy—directly.  They don’t reduce our savings—directly.  So, Congress will likely not get to them directly—as they say in the South.

Everyone who passed the test, get on the bus. Sorry, Charlie, only the best tunas get to go to Sea World.

The Texas Education Commissioner had to remind everyone that publicly rewarding students who pass the state test violates FERPA.  (Associated Press, October 14, 2008)  Ummm…what about those National Merit finalists, the magna cum laude graduates, the honor roll students?  All those students not wearing the gold tassel or the gold sash at graduation failed—to a degree.  They don’t get to go on the bus trip to the elite universities.

Should the high school yearbook include each student’s height and weight along with clubs and honors?  

How did we get so differentiated in our proclamations of success and failure between academics and sports?  Is there something personal or embarrassing about being slow in the classroom more so than slow on the field?  We publish athletes’ height and weight, their speed, their won-lost records.  By the way, the local school board must still designate sports vital statistics as public to follow FERPA.

The EduGuru speaks: School is about the curriculum, so the smart kids rule over the others. Even FERPA doesn't trump the naming of the valedictorian. Sports are extracurricular, that is "less" curricular, not "even more" curricular.

We’re taking our school voucher to Korea!

U.S. students rank below 14 other nations? True. However, do you really believe that? Even in the presidential debate, the moderator quoted the miserable ranking of U.S. students among all nations of the world. We spend more and achieve less.

Is the comparison fair?  Our public schools include everyone—almost.  We test them all—thanks, NCLB.  We even graduate 80% of them—including the special education students with modifications.  Our English language learners have to be tested in English—do other countries test only in their official language?  Our schools teach their own state’s standards—not some international panel’s opinion of what’s important globally.  Our school day is more reasonable—shorter.  We rely inordinately upon homework—actually less than other countries.  We educate the illegal immigrants—shouldn’t their scores be assigned to their home countries?  We take the summer off so kids can help their parents back on the farm—or travel to those other countries.  We blame ourselves for global warming and teach green—well, they blame us too.  Our course catalogues list more elective courses than solids—does anyone still use that term?  The Pope should take responsibility for the Catholic schools.  Charter schools aren’t doing well enough.  College engineering schools should give preference to Americans.  Did you see the last National Spelling Bee?—our students are from those other countries.

If we had universal vouchers, would parents send their kids to other countries? We can make too many excuses, but if you believe in face validity, those rankings just don’t add up to us.  Are our U.S. members on that panel really representing us well? 

The EduGuru speaks:  How in the world can they determine what someone in Beijing, Beruit, and Baytown should know and measure it fairly?  Is too much faith being placed in those tests and the standardization of them?  The U.S. doesn’t have to be first, but are those other countries really that good?