PUBLICATIONS
- RESEARCH DESIGN AND STATS
Reliability and Validity for Life Care Planners
Why are we, as clinical practitioners, interested in research
aimed at validating the process of life care planning? There
are multiple answers. Through research we can improve the
process,raise standards,help
to answer ethical questions and resolve ethical dilemmas.
Perhaps most importantly we protect the patient’s access
to life enhancing care through this very useful case management
tool.
Ensuring the future of life care planning is essential to
protecting the catastrophically injured patient’s access
to quality of life enhancing care through visionary case
management practices. In light of the recent challenges presented
by the Daubert rulings, the future of life care planning
as a forensic tool is dependent upon validating the Life
Care Planning process in the eyes of the court (Countiss & Deutsch,
2002).
We can do this through definitive research attesting to
consistent methodology employed by life care planning professionals
that is reliable, valid and relevant to the individual patient’s
case. Because we intend to objectively validate the utility
of life care planning as a tool in case management, let’s
first consider the facets of reliability that are prerequisite
to validation of life care planning.
Top
Issues Related To Reliability
Demonstrating the reliability of life care planning as a
case management tool is at the heart of validating life care
planning as a specialty area of practice. Reliability is
comprised of the dependability and consistency of the life
care planning process to yield similar results under similar
conditions. In other words, life care plans (LCPs) are reproducible.
If life care planning is a reliable tool in case management
and the provision of patient care, then the results of a
given LCP can be consistently replicated. We can convincingly
demonstrate reliability of life care planning by appropriately
designed research studies. First we need to discriminate
between two aspects of reliability: intra-planner
and inter-planner reliability (Bellini & Rumrill,
1999, Chap. 3).
Top
Intra-Planner Reliability
Intra-planner (a.k.a., Intra -rater, or Intra -observer)
reliability provides internal consistency to the process
much like “test-retest”. Intra-planner reliability
attests to the consistent application of an individual life
care planner's processes and the reliability of the results
of that process. Given similar circumstances, the process
of developing the LCP recommendations and cost estimates
are the same. Certainly, similar forms and procedures would
be used for collecting the information needed for similar
cases. But intra-planner reliability goes beyond that type
consistency.
Because the LCP is a document that makes recommendations
for case management and estimates the costs of those recommendations,
it will produce similar recommendations and cost estimates
given patients with similar disabilities and life circumstances.
Differences between the individual patients, their families,
and geographic locations would be appropriately noted as
modifying factors in the comparisons.
For example: The basis for establishing the skill level
involved in the provision of care for a C-5 tetraplegic should
remain consistent regardless of geographic location and irrespective
of subsequent development of cost data. What is being validated
is the basis for establishing need for care, level of care,
and availability of care. Costs are incidental to these issues.
It does not vary depending upon the geographic locale or
any other predisposing factor for bias.
The significance of intra-planner reliability is that a
given life care planner will produce the same life care plan
whether it is produced in Indiana or California, whether
it is produced as an Independent Medical Examination (IME),
or as a forensic tool of the defense or plaintiff, or even
whether funding is available to implement the recommendations.
Top
Inter-Planner Reliability
Inter-planner (a.k.a., Inter -rater, or Inter -observer)
reliability provides external consistency to the process.
Inter-planner reliability indicates that life care planning
is a standardized process, consistently applied by life care
planners across the country in a similar manner. Given similar
patient disabilities and circumstances, life care planners,
in general, present similar recommendations and cost estimates.
Wide discrepancies do not occur between plans generated by
different life care planners for the same patient.
Let’s consider an example demonstrating these concepts:
What if Dr. Smith who always works for the plaintiff, always
gets his cost estimates for custom modification of a vehicle
to accommodate a wheelchair from Jaguar, Porsche, and Mercedes?
Dr. Smith could show a tight range of costs, and his results
would be consistent and dependable over time, for each and
every LCP he develops.
On the other hand, his colleague, Dr. Jones, only accepts
defense work. Dr. Jones consistently gets his cost estimates
from Bubba's Junkyard, Billy Bob's Pre-Owned Palace, and
Honest Eddie's. [Note: Honest Eddie's motto is "We'll
beat any deal or give it to you for free," so Dr.
Jones always checks with him last.]
Dr. Jones’s results are also consistent and dependable
over time, for each and every LCP he develops. However, reliability
between the LCPs provided by Dr. Smith and Dr. Jones does
not exist. The results vary depending upon which life care
planner writes the LCP!
Validity is Case Specific
Dependability and consistency in applying the life care
planning process will reliably yield similar results under
similar conditions. While general reliability is necessary
to establish validity, it is not entirely sufficient. Validity
is case specific. A standard, "valid" life care
plan for all people who are infected with HIV does not exist.
Each LCP must be valid for the individual for whom it was
developed. If life care planning is a valid process, then
a good LCP will accomplish its mission: to decrease the frequency
and severity of medical complications for a particular patient,
avoiding case management by crisis intervention, and improving
the patient’s overall quality of life. Establishing
the validity of the life care planning process shows that
the basic tenets of life care planning are sound.
Top
Issues Related To Validity
Validity has four major aspects: face validity, content
validity, criterion-related validity and construct validity.
These four aspects can be used as lenses through which to
view validity. For validity to be established, evidence of
each of the four aspects should be demonstrated (Bellini & Rumrill,
1999, Chap. 3).
Face Validity
Face validity in life care planning refers to whether the
LCP "looks like" it appropriately details the disability-related
needs of a given individual. Although face validity is not
evidence of whether or not the LCP accurately presents an
individual's needs, it is still important to life care planners.
Because the LCP is a tool for educating people about disability-related
needs, if it does not appear to represent those needs accurately,
then family members, judges, juries, insurance adjusters,
etc. may not accept it as a useful instrument. The establishment
of face validity speaks to the lay audience.
Content Validity
Content validity relates to whether the elements included
in the LCP actually address all the disability-related needs
of an individual with a particular disability and set of
circumstances for enhancing their life across their life
span. The specific LCP should address all of a particular
patient’s needs without providing for extraneous treatments.
For example, an annual urological exam would be an important
part of the LCP for a person with a spinal cord injury, but
not for an individual who has had her leg amputated. Furthermore,
to find a meaningful and useful life, the patient’s
needs could reasonably be expected to extend beyond medical
care. These needs may include vocational education and retraining.
One approach to examining content validity is to have a
group of recognized experts come to consensus about which
items are most appropriate. Relating your recommendations
to published treatment protocols and standards of care could
help you to demonstrate the content validity of elements
in your life care plan. An excellent reference is Outcomes
Following Traumatic Spinal Cord Injury: Clinical Practice
Guidelines for Health-Care Professionals issued by the Consortium
for Spinal Cord Medicine (July 1999).
Criterion-Related Validity
Criterion-related validity is the gold standard of validity.
Evidence of criterion-related validity is presented when
a relationship exists between the LCP’s recommendations
and estimated costs and some outside measure, or criterion,
relevant to those recommendations and estimated costs. There
are three types of criterion-related validity: concurrent
validity, predictive validity, and convergent and divergent
validity.
Concurrent Validity
Concurrent validity refers to the relationship between elements
of the LCP and objective findings available at the time the
plan was developed, ( concurrent “at the same time”).
If the client fits the demographic profile and circumstances
of individuals studied in Aging With Spinal Cord Injuries
(Whiteneck, Charlifue, Gerhart, et al., 1993), it should
be possible to demonstrate similarity between the LCP’s
recommendations and the recommendations made for the individuals
in that published study. As medical research and education
advance, the standards of practice for life care planning
must necessarily evolve to keep pace. Demonstrating concurrent
validity shows that the specific LCP meets the current standards
of practice and is not obsolete.
Predictive Validity
Predictive validity is of greatest interest to those utilizing
the LCP for reserve setting, budgeting, or in a forensic
setting. Demonstrated predictive validity answers these questions:
1) Do the recommendations and cost estimates accurately
predict the services that will be needed by the individual
for whom the plan was written, and at what cost?
2) If the LCP’s recommendations are implemented in
full, and the LCP predicts they will help reduce the incidence,
frequency, severity, and duration of complications, will
the research study demonstrate a difference from the occurrence
of those complications in patients with similar injuries
but without LCPs?
3) If projections of life expectancy are included in the
LCP, are those projections accurate?
Predictive validity examines the quality and quantity of
follow-up on patients with whom LCPs have been completed.
The goal is to re-examine these patients and update the plans
later in time to establish predictive validity.
Convergent and Divergent Validity
Convergent and divergent validity are specific types of
criterion-related validity. They could be addressing either
concurrent or predictive variables. Essentially, if you have
evidence that two things that should be similar to each other,
are similar to each other (converge) you have evidence of
convergent validity. Likewise, if two things that should
be different from each other, actually are different from
each other (diverge), you have evidence of divergent validity.
Evidence of convergent validity is demonstratedwhenthe
recommendations in a LCP for an individual who has lower
extremity paralysis are similar, in some ways, to those in
a plan for an individual who has lower extremity amputation.
On the other hand,therecommendations
for an individual who has congestive heart failure would
be expectedto differfrom
those of an individual who has a hearing impairment. If the
life care plans for these two individuals are not different,
then the divergent validity of the two plans is not demonstrated.
Construct Validity
Construct validity is the essential validity to establishing
that the basic tenets of life care planning are sound and
are evolving contemporaneously with related health care fields.
Construct validity might best be understood as theoretically
related validity. Evidence of construct validity is presented
when a theory or hypothesis predicts a particular finding,
and the results of an analytical study correspond to that
prediction.
Although life care planning is not a theory per se, many
of the underlying tenets of life care planning could be considered
theoretical propositions. For example, we operate under the
assumption that the development and implementation of a good
LCP will decrease the incidence of medical complications,
and increase an individual's quality of life. To test that
assumption, we could compare the rate of complications for
people who have implemented a LCP with those who did not
have a LCP. We could also interview individuals with implemented
LCPs and individuals without such plans, and ask them questions
about their quality of life.
Consideration of construct validity leads us to conclude
that validation must be an on-going process. There are many
different aspects of validity most of which change with time.
The large number of variables involved in LCP complicates
the process of researching reliability and validity relative
to life care planning . The only solution is to reduce the
number of variables in any given research project and increase
the number of projects being conducted.
No single study will conclusively validate life care planning
once and for all. Every research study provides another piece
of evidence establishing life care planning as a valid case
management tool. We need to begin collecting elements that
contribute to establishing the validity life care planning’s
place in the management of the catastrophically injured patient’s
case.
Top |
