The Feds are going to
require all states taking stimulus money for P-12 education to have 12 “elements”
in their statewide education information systems. The Eduguru is going to examine each of the
12. In case you’ve been in a cave for
the past few years, the first 10 elements are the components from the Data
Quality Campaign recast in the America Competes Act. Nothing bad there (see The Process for
Ensuring Data Quality, ESP’s Optimal Resource Book at www.espsg.com/resources.php
for a review).
- What’s the likelihood the states are going to pull this one off?
Then the Eduguru makes a
summary declaration.
- What’s
the bottom line on this one?
- How important is it for the
Feds to track this? 1
- What’s the likelihood the states
are going to pull this one off? 10
- What’s the bottom line on
this one? Save everyone’s time reporting
this one.
The unique student
identifier must be assigned statewide and “not permit the student to be
individually identified by users of the system.” What?
Isn’t that the purpose of a unique identifier? In this case the point is to allow a
student’s records to be matched up across years and across files without a
viewer of reports being able to figure out that student’s personal
identity. That can be done these
days.
In fact count this one as
done in all states. Maybe not done
completely, but there’s no argument about this one being reality. The challenge will be whether or not the
safeguards will protect every student’s identity. The betting line? There will be problems through
carelessness. Get ready for them.
The Eduguru speaks: Glad this is number one, so we can begin with
a victory. Each one gets harder as we
go.
Number 2: Student-Level Enrollment, Demographic, and
Program-Participation Information
- How important is it for the
Feds to track this? 10
- What’s the likelihood the
states are going to pull this one off? 7
- What’s the bottom line on
this one? They can do it. They should do
it.
Ahh, content for the data
system. The key here is defining the
data elements in each area. Yes, the
metadata standards are paramount. Even
the terms enrollment, demographic, and program have different meanings across
our schools—and even across schools within districts at times.
All across the nation,
people will start slapping into data warehouses all the data they already have
about when students enroll, what ethnicity they are, and what programs they are
in—but wait—that’s the problem. The data
they already have is inconsistent, incomplete, inaccurate, and potentially
incomprehensible to the National Education Data Model (NEDM) with which
everything may get aligned.
The Eduguru speaks: There’s important work to be done up front to
clean up our definitions, codes, and formats before we concatenate a cacophony
of connotations into an incomparable little data warehouse of horrors.
Number 3: Student-level information about the points at
which students exit, transfer in, transfer out, drop out, or complete P-16
education programs
- How important is it for the
Feds to track this? 9
- What’s the likelihood the
states are going to pull this one off? 9
- What’s the bottom line on
this one? Simple accounting. Get it done.
Pardon me. How simple is it for us to count when
students come and go? Our student
information systems do this every day.
The significant data are those about why, what factors contributed to the
exits, what would have changed the student’s decision. There will be significant effort invested in
moving the same old data once again from schools to states to the Feds to count
students, when we need to be taking new, fresh looks at our students.
The Eduguru speaks: Let’s get beyond this one. The important, actionable data will tell us
why kids come and go.
Number 4: Capacity to communicate with higher education
data systems
- How important is it for the
Feds to track this? 8
- What’s the likelihood the
states are going to pull this one off? 3
- What’s the bottom line on
this one? Look to the higher education
systems, especially the private institutions who keep their shades drawn and
the community colleges with tremendous mobility to make this difficult.
Let’s assume this means a
two-way communications. (Again, this
element suffers from ambiguity.)
Requires someone on both ends. Wrong
word—capacity. Should have been
willingness. Private schools may not
play this game. Community colleges may not
see the point and be overmatched with the volatility in their student
mobility. What do they gain from
exchanging data bout students that attend their college for such brief periods,
take such random courses, and don’t complete degrees?
This could have been the
most vague of the 12 elements had it not been for #12 taking that dishonor.
This overlaps significantly
with #9, which refers to transcript information. #9 should be
communicated—electronically—using common standards. We should be focusing on what is to be
communicated and for what purpose.
This “element” is dysfunctional as any kind of mandate to which a state
could plan or design their information system.
Is capacity irrespective of ever actually exchanging any meaningful data? No.
The Eduguru speaks: Let’s define the meaning of capacity or we
risk its being defined as what we have already.
Number 5: State data audit system assessing data
quality, validity, and reliability
- How important is it for the
Feds to track this? 10
- What’s the likelihood the
states are going to pull this one off? 2
- What’s the bottom line on
this one? There are no standards and no
will to enforce any.
What is the standard for
data quality? Validity? Reliability?
Fact is validity and reliability have definitions in the statistics
textbooks. What complicates the matter
is that one must set the criterion for each like setting the odds for a bet. Do we want tough 99% or loose 67%? Oh, and we need to make the decision as to
who sets the criterion.
Remember when NCLB let small
groups off the hook if their failing percent proficient was not
“reliable”? Some states set the
criterion high, some low. Some states
declared groups as meeting AYP if their failing percent was
unreliable—therefore not reportable.
Yes, they really did—I mean, they really do. Deciding who sets the criterion is very
important.
The reality in states that
monitor data quality now is that there are two factors that matter.
- Submitting data (reports) on time
- Submitting complete data (filling all required
the blanks/fields)
Hopefully, this “element”
expects more, like…
- Alignment with standard definitions, codes
- Conformance with formats and transmission standards
- Accuracy reflecting the data within local transactional systems
- Validity reflecting the intent of the data element being recorded and reported
Number 6: Yearly test records of individual students
with respect to assessments under section 1111(b) of the Elementary and
Secondary Education Act of 1965 (20 U.S.C. 6311(b));
- How important is it for the
Feds to track this? 10
- What’s the likelihood the
states are going to pull this one off? 9
- What’s the bottom line on
this one? There are no tech, practical,
or fiscal reasons not to comply.
Let’s assume this includes
linking yearly assessment to measure growth for individual students.
We may still argue about
what to test, when to test, and how hard the questions should be, but we can
score those test and match them up from one year to the next. The technical, practical, and fiscal challenges
of this “element” are being handled now.
Unless the Feds make demands for more standardization of what, when, and
how hard, then the states can do this.
The Eduguru speaks: Don’t we all wish that our cumulative test
record meant that the scores were added together across the years? Eventually we’d all have pretty decent
scores.
Number 7: Information on students not tested by grade
and subject
- How important is it for the
Feds to track this? 10
- What’s the likelihood the
states are going to pull this one off? 6
- What’s the bottom line on
this one? There are no tech, practical,
or fiscal reasons not to comply; however, there are tremendous disincentives
for keeping accurate records.
This hallmark of Bush’s NCLB
was bashed initially, then admired for ensuring that every child counted. Schools could no longer suggest some students
stay home on test day. (They really had
done that.) The low rating for the
likelihood the states are going to pull this one off is not that they can’t
report this statistic. They can
report. The pessimism is that they will
ever faithfully commit to testing every student and reporting accurately every
one not tested.
The Eduguru speaks: All skate!
Even if someone is going to fall down.
Number 8: A teacher identifier system with the ability
to match teachers to students
- How important is it for the
Feds to track this? 10 for secondary
schools, 5 for elementary schools
- What’s the likelihood the
states are going to pull this one off? 9
for secondary school, 5 for elementary schools
- What’s the bottom line on
this one? Who really teaches each
elementary student each subject area?
Surprised this isn’t already
done routinely? Surprised to know a
state would have a law preventing such linking?
Why do this anyway? Simply put,
this is for teacher accountability. Also
for research and evaluation. This has
been done for decades in secondary schools with distinct courses and class
periods. For elementary schools—a
culture change and/or a recordkeeping change is required most places.
Would it be appropriate to
suggest that maybe the culture of elementary schools still argues for a
grade-level or even schoolwide evaluation in many schools?
The Eduguru speaks: Assigning the identifiers—trivial. Using them to link teachers and students for
accountability—priceless.
Number 9: Student-level transcript information,
including information on courses completed and grades earned
- How important is it for the
Feds to track this? 10
- What’s the likelihood the
states are going to pull this one off? 6
- What’s the bottom line on
this one? There are no tech reasons not
to comply. States vary in their
capacity.
Does this level of detail
need to reside at the state? The time
may just be here that the answer is “yes.”
Student mobility, research, the comprehensiveness of mandated reporting,
state-level scholarship programs, college admissions rules, etc. may be
compelling reasons. Schools can even use
the state for back-up or permanent record storage for archived student
records.
States vary considerably in
their capacity to comply with this. The
rating of 6 for likelihood could
rise over time with the proper use of stimulus dollars.
The Eduguru speaks: No more parchment, no more gold stickers, no
more registrar’s embossed seals. XML
digital files with electronic signatures are the only acceptable academic
currency for exchange going forward.
Number 10: Student-level college readiness test scores
- How important is it for the
Feds to track this? 5
- What’s the likelihood the
states are going to pull this one off? 3
- What’s the bottom line on
this one? Are we going to mandate
universal participation at a single administration date?
Who says the college
readiness tests are worthy of our use for evaluating school effectiveness? Aren’t
these tests being criticized for not predicting success in college? Aren’t admission officers using alternative
criteria to make their selections. A
reading of Outliers shows us that
these tests may be useful as cut scores—up or down decision makers, but not
good enough to differentiate students along a meaningful scale. That would mean they don’t predict how well
someone will perform in college after they get past the hurdle of getting
through the door. After all, the
competition beyond the door is not the same across those colleges.
Are we yet able to compare
the ACT and SAT with their contrasting philosophies, content, scoring, and
participants? Not all students are
planning to go to college, so they are not going to be serious about one of
these tests. These tests are expensive.
The Eduguru speaks: There just is not a compelling justification
for this expensive, unreliable, difficult to interpret “element.”
Number 11: Information regarding the extent to which
students transition successfully from secondary school to postsecondary
education, including whether students enroll in remedial coursework
- How important is it for the
Feds to track this? 9
- What’s the likelihood the
states are going to pull this one off? 3
- What’s the bottom line on
this one? First define a few terms, then
start gathering some data.
This element needs to be better
defined, maybe divided into more than one criterion. Transitioning successfully appears to refer
to being academically prepared and not requiring remediation. Does it also refer to being mature enough to
handle college life, to manage the responsibility, to earn credits whether or
not the student has the skills and knowledge already?
Then we’ll need to agree on
what a remedial course is contrasted with an introductory course. We’ll want to level the playing field across
colleges for when remedial courses are required. Oh yes, let’s distinquish whether a student
failed other courses because the student was ill prepared, took too difficult a
course, didn’t study, partied too much, etc.
We’ve been at this for far
too little time to have worked out the kinks.
The postsecondary institutions are clearly behind elementary and
secondary education in the development of their information systems. (Remember we are talking about the ability of
those information systems to exchange data across institutions and with the
state and Feds.)
The Eduguru speaks: Standards must be developed first.
Number 12: Other information determined necessary to
address alignment and adequate preparation for success in postsecondary
education
- How important is it for the
Feds to track this? 0
- What’s the likelihood the
states are going to pull this one off? 0
- What’s the bottom line on
this one? This statement itself is clear
evidence of how far higher education is behind elementary and secondary
education.
How is this different from
#11?
The Eduguru speaks: Any court would throw this “element” out as
being too vague to be enforceable. My
fear is that it is so vague that the Feds will be able to use it to mean
whatever they want it to mean. Watch out
for this one.
Recent Comments