Chapter 9. Outcomes Evaluation of Pharmacy Operations

Dr. Chui is currently an assistant professor at the University of Wisconsin in Madison, Wisconsin. She earned a B.A. in biological sciences from the University of California—Davis, a Pharm.D. from Creighton University, and an M.S. and Ph.D. in pharmacy administration from Purdue University. Her research centers on applying human factors approaches and utilizing a systems perspective to characterizing community pharmacy practice and its practitioners to understand processes that impact medication safety and quality of care.

After completing this chapter, readers will be able to

1. Discuss the different facets of the ECHO model and their application to evaluating existing and proposed pharmacy goods, services, and operations.

2. Describe differences among the five types of economic analyses as tools for managerial decision-making.

3. List the advantages and disadvantages of the various methods of measuring humanistic outcomes, and describe the role that humanistic outcomes play in evaluating goods, services, and operations.

4. Identify the objectives of a service and determine the types of outcomes a pharmacist would need to measure in a pharmacy-related service.

5. Identify the advantages and disadvantages of the various methods used to collect data when evaluating services.

6. List and describe the uses of various outcomes evaluations and assessments.

Cynthia Marshall is a pharmacist who works at a closed-staff-model health maintenance organization (HMO) pharmacy. Pharmacists at this clinic have access to patient charts that can be used to supplement information given by the patient to the pharmacy staff and what is written on the prescription. Cynthia enjoys working at this pharmacy because she feels that the added information is helpful when she is assessing and counseling her patients. She also believes that the HMO she works for values high-quality health care and is seeking to improve its ratings on some of its quality indicators.

Cynthia sees a number of heart failure patients every month. These patients see their physicians (including primary care physician, cardiac specialist, and frequently one or two other specialists) annually. Many times, these physicians do not coordinate care or know about the other medications that her heart failure patients are taking. She notices that these patients can improve or deteriorate quickly, have a difficult time remembering the complicated medication regimen they are on, and that their health conditions frequently change from month to month. They occasionally complain that their symptoms get so bad that they have to go to the emergency room. She thinks that these patients could have better outcomes if they were better educated about their conditions and monitored more closely.

Cynthia approaches the cardiologists in her clinic with an idea that she could start a heart failure clinic to provide additional education and monitoring for these patients. While the physicians would oversee the clinic, she could conduct monthly educational seminars about heart failure for patients, order and assess laboratory tests, order medications and adjust dosages under physician-approved protocols, and communicate with patients' other physicians about concurrent medications to prevent medication errors.

The HMO is also supportive, as their administrators recognize that Cynthia's clinic may contribute to higher scores on their quality indicators ratings related to heart failure. Given the HMO's patient load, Cynthia figures that she would have to spend 50 percent of her time with the heart failure clinic. Because the pharmacy is always busy, the HMO would have to hire an additional part-time pharmacist to take care of her dispensing duties.

1. Why is it important to measure baseline data for individual patients and/or a cohort of patients before initiating a new service?

2. How can outcomes be used to prove the value of your service?

3. How do pharmacists use outcomes measures to assess the success of their services?

4. What type of economic analysis would be appropriate to evaluate the service described in the scenario? How should the costs and outcomes be measured?

5. Why is it important to choose a perspective for the analysis prior to starting a project? How do perspectives differ?

Pharmacists make many contributions that improve the lives of their patients on a daily basis. These contributions range from catching a drug–drug interaction, counseling a patient about his or her medication, or providing immunizations. To keep up with a transitioning profession and evershrinking margins, pharmacies must persistently evaluate its services and the resources that they use to provide those services (e.g., automated dispensing system, bar code scanning) to determine if they contribute to patient satisfaction, allow pharmacists to provide optimal care, and are cost-effective. In addition to evaluating the services and programs that pharmacists are already typically offering, pharmacists frequently conceptualize opportunities to provide a new service, such as the heart failure clinic proposed by Cynthia Marshall in the scenario.

While many pharmacists would like to offer services that would improve the quality of care to their patients, they can have upfront costs that cannot be ignored. How can pharmacists provide these services and show that they both help their patients and actually decrease the overall cost of care? Many studies have been conducted that show that pharmacists can make a difference both in cost and in the quality of their patients' lives (Perez et al., 2009; Boyd and Briggs, 2009). How can Cynthia Marshall do the same in her situation?

How does one begin documenting the value of a pharmacy program or service? The first step is to define the problem being addressed. Stating the problem upfront seems rather obvious, but it is not always done. Many pharmacists waste time and energy trying to answer poorly defined questions. For example, one can start with a very broadly defined problem such as heart failure, which arises from a combination of factors and results in numerous health outcomes. Evaluating all the possible causes and effects of heart failure may be an overwhelming task, so the problem should be narrowed. One could evaluate the effect of medication adherence in patients with heart failure or even one particular outcome of heart failure (e.g., blood pressure or quality of life). The more narrow the problem, the easier it will be to address. Once having identified one or more worthy problems, the pharmacist will need to prioritize which one should be the focus of the service. These criteria may be useful in determining which service or program will provide the most impact and demonstrate success.

• Impact: Is the service properly aimed to help the greatest number of high-risk heart failure patients?
• Improvability: To what extent will the service improve the most appropriate aspects of care in a heart failure patient?

After the problem has been stated, the next step is to define the objectives. A very broad problem will result in broadly defined objectives. If heart failure is broadly identified as the problem, an objective could be to reduce the number of patients with heart failure. However, no specific method of reducing heart failure is addressed, and heart failure itself has many potential outcomes. On the other hand, if the problem is addressed more narrowly, a more specific objective could be studied. For example, one may look to reduce the number of heart failure exacerbations related to nonadherence with medications. In addition, other specific objectives may be included, such as increasing the quality of life of patients with heart failure, reducing the number of medications used by patients with heart failure, or making patients more aware of behavioral and dietary modifications related to heart failure.

All objectives, regardless of whether they are broad or narrow, must be measurable. While this also may seem obvious, many objectives (especially if they are broadly defined) can be difficult, if not impossible, to measure. The broad objective of reducing heart failure may be difficult to measure because so many factors must be considered when arriving at a diagnosis of heart failure. On the other hand, specific objectives such as reducing blood pressure or improving quality of life in patients with heart failure are much easier to quantify and measure.

The third step is to establish a framework to evaluate the question. When establishing the framework, pharmacists need to know some basic facts about the programs and diseases they are studying. Pharmacists undertaking these projects first should prioritize their objectives by determining what interventions and outcomes most need to be evaluated. For example, Cynthia Marshall may wish to determine the effect of her monitoring program on cost of care (an economic outcome), blood pressure or adherence to heart failure medication (clinical outcomes), and quality of life (a humanistic outcome).

These steps are not dissimilar from evaluating more common or “traditional” services and resources such as an automated dispensing system, or a refill reminder or patient satisfaction program. That is, the pharmacist must determine the objectives, or main outcomes of the program, determine the number of patients involved, or who could be involved, and appropriate parameters for estimating its impact.

Many health care organizations make the mistake of focusing on only one area of care when trying to reduce the overall cost of health care. Evaluating the cost of a single aspect of patient care in isolation from all other aspects is commonly known as component management. Examples of component management include focusing on the costs of drugs or laboratory tests without looking at the impact of these costs on overall health care costs or outcomes. Unfortunately, many studies have focused on just one type of cost without considering the overall care of patients. Studies have suggested that while organizations can reduce drug use and costs used for chronic conditions, these reductions may be accompanied by an increased use of other health care services (Austvoll-Dahlgren et al., 2008). For instance, Kaiser-Permanente of Northern California compared the clinical and economic outcomes of beneficiaries whose annual drug benefit was capped at $1,000 with those with unlimited drug benefits. A cap on drug benefits was associated with lower drug consumption but poorer adherence and control of blood pressure, lipid levels, and glucose levels. Savings in drug cost was offset by increases in costs of emergency department visits and hospitalizations (Hsu et al., 2006).

Disease management strives to consider all individual components of health care for a specific disease with a view toward an outcome that is important for successful management of that disease. Instead of dealing with one element at a time (such as prescription costs, only), disease management attempts to control the disease by integrating the components of health care to provide the best total patient outcomes at the most reasonable cost. As a result, the number of variables measured in this paradigm is greater and more global. When making changes to one component of health care, other components are considered as well (Armstrong and Langley, 1996; Patterson, 1995; Zitter, 1994).

When identifying economic, clinical, and humanistic variables to be measured, it is important to integrate elements from all components of health care. In our example, Cynthia is trying to manage her patients' heart failure medications. Because she is a pharmacist, she may identify only medication-related variables to measure, such as cost of the drug or number of prescriptions filled. When evaluating this project with an eye toward disease management, one could also measure hospitalizations, emergency department visits, laboratory costs, physician visits, and other factors related to the care of these patients if those variables are available to her.

Because the heart failure clinic may require more work and personnel hours, and because other pharmacy staff may be impacted by the clinic, Cynthia could also include variables that would measure employee productivity and satisfaction. For example, all pharmacists working in the pharmacy may feel that a heart failure clinic would improve patient care, but if pharmacists perceive that their workload is high, they may not be able to effectively or happily help Cynthia recruit patients or cover her while she is working in the clinic. Cynthia, not wanting to assume that current services and programs (such as a refill request program) may be working as originally conceptualized, could also assess the effectiveness of current services and programs to determine if both patients and pharmacy staff have responded well to their implementation, and if they are achieving their goals (e.g., increase patient satisfaction and loyalty, increase prescription count, add to the variety of challenging work that pharmacists do) and if they might positive or negatively impacts the success of her clinic.

It is also important to be cognizant of the quality indicators that might be related to your service of interest. In this case, the HMO that Cynthia works for is interested in improving their heart failure indicator scores. The indicators that they would like to see improved are:

• Greater proportion of heart failure patients taking an angiotensin-converting enzyme inhibitor (ACEI)
• Evidence of disease management education for all heart failure patients
• Greater proportion of heart failure patients taking a beta-blocker
• Reduced proportion of heart failure patients taking a calcium-channel blocker
• Reduced proportion of heart failure patients taking a Type I antiarrhythmic agent
• Regular monitoring for digoxin toxicity
• Appropriate anticoagulation for patients with heart failure and atrial fibrillation (Fonarow, 2001).

At the point where one is determining which outcomes will be measured, one also should be keeping in mind the perspective of those to whom the outcomes of the intervention will be of value. While practically any disease management project can be evaluated from the perspective of the patient (e.g., Did the patient's health and quality of life improve? Did the patient save money?), there are other important perspectives whose outcomes could be considered as well (Drummond et al., 1986). These perspectives include that of the health care provider (e.g., Did the job satisfaction of the pharmacist improve as a result of providing the service?), the department (e.g., Did the program result in lower drug costs to the pharmacy?), the organization (e.g., Did the program result in decreased costs of caring for heart failure patients for the HMO?), and even society (e.g., Did the program reduce the prevalence of heart failure in the community?). Not every outcome from every perspective needs to be considered in each evaluation. Pharmacists should revisit their original question, objectives, and framework when determining which variables from which perspectives should be measured.

After identifying a question, objectives, and variables to evaluate, pharmacists also need to think about other variables that may affect the results of the study. For example, Cynthia Marshall may wish to evaluate the impact of her monitoring service on the cost of caring for these patients from the perspective of the HMO, her employer. The results may show that patients who used her service had much lower costs of care compared with those who did not. Based just on this finding, one might jump to the conclusion that the service is an effective way to reduce the cost of caring for heart failure patients. However, on further examination, it may be discovered that many of the patients receiving the service had mild-to-moderate heart failure and only had one to two other disease states. In comparison, those who declined to participate may have had moderate-to-severe heart failure and multiple disease states.

Does this mean that the service did not have a positive effect? Actually, it means that the results are unclear. The severity of heart failure and number of disease states may have clouded the results of the study. For example, the service may not have saved the HMO any money, but the fact that those who used the service (less severely ill patients) generally had lower costs of care than those who did not use the service (more severely ill patients) may have made it look that way.

Cynthia would like to implement a new service that would manage the medications taken by the HMO's heart failure patients. The questions that the HMO is asking include, “How much will the service cost?”, “Will hospitalizations and emergency department visits be reduced?”, and “Will this improve our rating/score on related quality indicators?” Cynthia has additional questions: “Will I be able to increase my patients' health-related quality of life?” and “Will my patients be better satisfied with the quality of pharmacist care than with the care they currently receive?”

Cynthia's objectives are to reduce hospitalizations and emergency department visits by increasing adherence to heart failure medications and educating patients about their disease state and how to make diet and behavioral modifications, as well as improve patients' health-related quality of life and satisfaction with care.

Her examination of the literature about heart failure has given her some information about what types of information she should collect. Patients with heart failure show symptoms of decreased exercise tolerance, shortness of breath, increased fatigue, and fluid buildup in the lungs and tissues. Patients with heart failure are classified clinically by their level of disability so that they may be followed longitudinally and provide a reference point when compared with other patients (New York Heart Association [NYHA, Criteria Committee, 1973]).

Cynthia also has discovered that heart failure is the most common hospital discharge diagnosis in individuals over age 65. Median survival rate from the time of diagnosis is 1.7 years in men and 3.2 years in women (Ho et al., 1993). Cynthia knows that she also will have to collect some demographic data that could impact the results of her program, such as age, gender, marital status, ethnicity, income level, and mental status.

Outcomes evaluations attempt to identify, measure, and evaluate the end results of health care services. Outcomes evaluation examines not only the clinical effects of health care services but also the economic and humanistic impact of these services, such as quality of life and satisfaction (Bootman et al., 1996). From an increasing consumerism perspective, and with regard to resources becoming more scarce, evaluation of these types of outcomes are increasingly important. The economic, clinical, and humanistic outcomes (ECHO) model assumes that the outcomes of medical care can be classified along the three dimensions of economic, clinical, and humanistic outcomes (Kozma et al., 1993). Clinical outcomes are defined as medical events that occur as a result of disease or treatment. Economic outcomes are defined as the direct, indirect, and intangible costs compared with the consequences of medical treatment alternatives. Humanistic outcomes are defined as the consequences of disease or treatment on patient functional status or quality of life. All three of these outcomes need to be balanced simultaneously to assess value.

Economic Outcomes

Traditional cost-containment measures are not always consistent with improved patient care. Thus attention has turned toward demonstrating the value of health care, overall. A full evaluation of relevant costs and consequences would provide significantly more information that simply looking at traditional cost-containment strategies. Costs are defined as the value of the resources consumed by a program or treatment alternative. These include direct costs, or the resources consumed in prevention, detection, or treatment of a disease, nonmedical costs such as transportation to medical facilities or special diet considerations. Indirect costs include loss of productivity at work (such as when a patient is absent from work because of a migraine, or working but is suffering from depression), and intangible costs are things like pain, suffering, and grief. Consequences are defined as the effects, outputs, and outcomes of the program or treatment alternative (Eisenberg, 1989).

The economic evaluation of medical care may occur from many perspectives. These perspectives, or viewpoints, will influence the costs and consequences identified, measured, and compared for a program or treatment alternative. An economic evaluation can be conducted from a single perspective or multiple perspectives. Common perspectives are those of the patient, health care professional, insurer, and society. There are subtle differences between perspectives. A patient may be extremely concerned with his or her quality of life, level of pain, ability to see health care professionals, or how much time is spent in the hospital or nursing home, in addition to the actual direct costs of health care. However, an insurer may be interested only in the direct costs of health care to its organization. In fact, while a patient who is at the end of his or her life may be interested in extending life as long as possible, an insurer may see this as an expensive prospect.

There are also differences between the insurer's and societal perspectives. Many insurers assume that patients will not be covered on their plans throughout their lifetime, so expensive interventions that provide benefits in the very long term tend to be discouraged (i.e., screening programs for diseases that manifest in the elderly). However, from a societal perspective, these screenings may be appropriate because they identify patients at risk earlier. In the long run, such screenings are beneficial to society.

Types of Economic Evaluations

The basic task of economic evaluations is to identify, measure, value, and compare the costs and consequences of the alternatives being considered. Therefore, a full economic evaluation must have two features: a comparison of two or more alternatives and both the costs and consequences of the alternatives being examined.

Cost-minimization analysis is a tool used to compare two or more treatment alternatives that are assumed to be equal in efficacy. It simply compares the direct and indirect costs of treatment alternatives in dollars and does not consider the outcomes of the treatments (because they are assumed to be the same). Examples could be to compare the costs of intravenous versus oral dosage forms of the same drug or hospital versus home administration of intravenous pain management.

Cost-benefit analysis is used when outcomes can be expressed in monetary terms, such as in the amount of money an HMO might save if it implements a new program. The direct and indirect costs of a program or intervention are measured as they occur, and the benefits accrued from the program or intervention are identified and converted into dollars in the year in which they occur. These two values are then expressed as a ratio of the dollar value of the costs of the program or intervention to the dollar value of the outcomes that arose as a result of the program or intervention. For example, if the cost associated with new heart failure monitoring service were $100 per patient and, as a result of the service, the HMO saved $1,000 in hospital care per patient, the cost-benefit ratio would be expressed as the benefit, $1,000, divided by the cost, $100, or as 10:1. This ratio would be interpreted such that the service produces $10 of benefits for every $1 that is spent. The results may also be interpreted by subtracting the costs from the benefits or, in our example, as a $900 net benefit per patient.

Cost-effectiveness analysis is used when the treatment alternatives are not therapeutically equivalent or when outcomes cannot be expressed in monetary units. Cost is measured in monetary units, and outcomes are expressed in terms of obtaining a specific therapeutic objective, such as cases cured, lives saved, or millimeters of mercury drop in blood pressure. Cost-effectiveness analysis allows researchers to summarize the health benefit and resources used by two or more competing programs so that policymakers can choose among them (Detsky and Nagiie, 1990; Doubilet, 1986). For example, pharmacy director, anticipating an increase in prescription count, may want to consider adding another technician or purchasing an automatic dispensing system (i.e., dispensing robot). The pharmacy director can estimate the number of prescriptions, and other duties that a technician can perform and compare those duties to what a dispensing robot can perform. Clearly there are more upfront costs to a robot (i.e., purchasing the equipment and training the staff), whereas the costs for a technician are variable and can change (i.e., the director can hire the technician part-time with the potential for more hours, if needed). Using a cost-effective analysis, the pharmacy director can create a ratio of the prescriptions filled to the hourly cost of the technician or the extrapolated hourly cost of the robot to determine which would be more cost effective for the pharmacy. Another example that Cynthia might consider is whether hospitalizations and emergency department visits would be reduced more with a heart failure clinic or an asthma clinic. The therapeutic objective in this example would be a decrease in hospitalizations or emergency department visits, and the costs would primarily be Cynthia's personnel time at work.

Cost-utility analysis is used when quality of life is the most important outcome being examined. This is common in disease states in which how one feels or what one can do is more important than a clinical laboratory value or economic outcome (e.g., chronic diseases such as heart disease, diabetes, arthritis, cancer, or HIV/AIDS). Cost-utility analyses compare the direct and indirect costs of an intervention with some measurable level of humanistic outcome, such quality of life or level of satisfaction. Quality of life often is measured in quality-adjusted life years (QALYs).

To determine what type of analysis to conduct, Cynthia must identify all the costs related to her project. She created a table to identify all possible costs from every perspective (see Table 9-1).

When determining which method of economic analysis she should use, she eliminated cost-minimization analysis because the treatment alternatives (service versus no service) will not result in equivalent outcomes. A cost-effectiveness analysis would not be appropriate because she is interested only in one particular program. A cost-utility analysis is also not appropriate because quality of life, while included in the project, is not the focus of her project. A cost-benefit analysis could be appropriate. A cost-benefit analysis requires that both the interventions and outcomes be valued in monetary units. She can determine the direct medical and/or nonmedical costs for each patient from data captured by her HMO. The HMO is very interested in costs, both those to implement the service and those it may save as a result. Cynthia decides that the most understandable analysis to present to the HMO is a cost-benefit analysis.

Uses of Economic Outcomes

Many benefits can be realized by applying pharmacoeconomic principles and methods to evaluating pharmacy services and programs. Economic assessments can assist in balancing cost and outcome when determining the most efficient use of health care goods and services. Economic assessments are helpful in optimizing clinical decision making. Decisions involving formulary management, practice guidelines, drug policy, individual patient treatment, and resource allocation are all amenable to economic assessments.

Decision analysis is a technique used in economic evaluations to structure the analysis. It is a systematic way of describing decision problems, identifying possible courses of action, assessing the probability and value of outcomes, and making a calculation to select the optimal course of action. A tool used in decision analysis is a decision table or decision tree. A decision tree, shown in Fig. 9-1, allows you to display graphically all treatment alternatives being compared, the relevant outcomes associated with these alternatives, and the probabilities of these outcomes occurring in a patient population. It can allow for the algebraic conversion of all these variables into one summary measurement, often a cost-effectiveness ratio, to allow for a meaningful comparison of two or more treatment alternatives (Barr and Schumacher, 1994; Crane, 1988).

In economic evaluations, there is a need to make assumptions about the variables in the analysis. For instance, assumptions that are made commonly include the incidence of adverse effects, the drug's efficacy (in clinical trials) and effectiveness (use in actual practice), and the costs of drugs or other direct medical costs. Assumptions that you might make about a current pharmacy service (such as a prescription delivery service, or an adherence program in which you put a patients' prescription in pill boxes) might be how many patients will use a service, what patients might be willing to pay for a service, or how much pharmacist time will need to be allocated to the service. It is important to keep in mind that assumptions are simply predictions about what might happen as a result of a program or intervention. To account for the variety of outcomes that may arise in any intervention, one might use a technique known as sensitivity analysis. a sensitivity analysis, identifies one parameter, hold other evaluation parameters constant, and recalculate the results using a range of the chosen parameter. The results of a sensitivity analysis can determine if the outcome of the study is “sensitive” to ranges in the chosen parameter. If changing the values of specific variables does not substantially alter the results, there is more confidence in the original findings.

For example, if the pharmacist decided to promote and add food tours in a grocery store pharmacy to patients with diabetes, he or she might want to measure if patients transfer their prescriptions to your pharmacy and continue to use your pharmacy. It is estimated that 80 percent of the individuals given food tours will start filling their prescriptions in the pharmacy. That would increase prescription count, which would be used to justify the amount of time required to provide the food tours. In a sensitivity analysis, one might test whether your service would still be cost effective if only 60 percent of food tour recipients start coming to your pharmacy.

Clinical Outcomes

To health care professionals, the most obvious variables that should be addressed when evaluating any medical intervention are clinical outcomes. Any program or service can assert that patients will receive better care if given that service. Clinical outcomes vary with each disease state but should include any pertinent medications [e.g., prescription, over-the-counter (OTC), and herbal], any side effects or adverse effects from the disease state or medications, any laboratory values used, the documented physical state, and any use of other health care professionals or services.

At Cynthia's pharmacy, patients with heart disease are usually on several medications for their heart disease: a diuretic, a beta-blocker, and an angiotensin-converting enzyme (ACE) inhibitor. Patients with more severe disease may also be on an aldosterone inhibitor such as spironolactone, digoxin, hydralazine nitrate, and/or an angiotensin receptor-blocker (ARB). Some patients are also on oxygen. Patients are also required to submit to frequent laboratory testing for potassium and serum creatinine levels. They are assessed for blood pressure, weight, level of edema, and shortness of breath and then are assigned a NYHA classification (I, II, III, or IV). For the pharmacy's heart failure service, each of these pieces of clinical data would be recorded at baseline and at each subsequent visit by the pharmacist. Other clinical markers that Cynthia would like to measure include the incidence of medication side effects and adverse drug reactions.

Humanistic Outcomes

Quality of life refers to an evaluation of all aspects of our lives, including where we live, how we live, how we play, and how we work. Health-related quality of life (HRQoL) encompasses only those aspects of life which are dominated or influenced significantly by personal health or activities performed to maintain or improve health. HRQoL is defined as a multidimensional concept referring to a person's total well-being, including physical symptoms, functional ability, and perceptions. Advances in medical science have encouraged attention to HRQoL because medical care is no longer limited to providing only death-averting treatments (Coons and Kaplan, 1993). Indeed, if a pharmacist can provide information to a physician regarding an improvement in quality of life for his or her patients, the pharmacist could earn trust and loyalty, and build a collaborative relationship with a physician, which could then lead to further collaborations to improve patient care.

To evaluate aspects of HRQoL that are specific to a particular disease or condition, specific measures also may be used. Specific measures include only important aspects of HRQoL that are relevant to the patients being studied, such as the loss of function in asthma patients or the amount of pain in arthritis patients. Disadvantages of using specific measures are that they are not comprehensive and cannot be used to compare across conditions. They also cannot measure unforeseen side effects or conditions.

Another area of humanistic outcomes that may be even more important when measuring outcomes of pharmacy services is patient satisfaction. Satisfaction is defined as the extent to which individuals' needs and wants are met. This is linked with attitudes toward the health care system, as well as expectations and perceptions regarding the quantity and quality of care received.

During the last decade, the health care industry has begun to recognize the importance of satisfaction with services as patients became increasingly recognized as consumers. Consumer or patient satisfaction surveys have evolved from marketing tools to measures of quality. In fact, the level of satisfaction with medical care has been shown to be associated directly with willingness to adhere to medical recommendations, rates of switching medical care providers, and disenrollment from prepaid health plans (Cvengros et al., 2007). In addition, the degree of patient satisfaction has been negatively correlated with a willingness to initiate malpractice lawsuits, meaning that people who are less satisfied with their health care are more likely to sue their health care professionals (Stelfox et al., 2005).

Surveys specifically measuring both traditional pharmacy dispensing and counseling services, as well as disease state and medication management services provided in community pharmacies also have been developed. These surveys can be adapted for one or more specific programs that a pharmacist might want to implement and evaluate (MacKeigan and Larson, 1989; Tinelli et al., 2011; Van Geffen et al., 2011; Moczygemba at al., 2010).

Although economic and humanistic analyses usually have been conducted using the perspective of the insurance company, the perspective of patients' employers also have become more prevalent, particularly if an employer is working directly with a pharmacy to manage his employees' medications or providing immunizations or other wellness programs. To meet the evaluation needs of employers, work functioning can be measured as a humanistic outcome. Traditionally, work functioning has been measured as lost days from work (absenteeism). However, this measurement has become more specific and can be measured as reductions in productive capacity (“presenteeism”). Diseases that may cause reductions in work productivity include migraines, allergic rhinitis, and depression.

Cynthia has found that patients with heart failure report a number of physical symptoms that affect HRQoL, such as dyspnea, fatigue, anorexia, cachexia, edema, sleeplessness, attention deficit, and memory impairment. Functional abilities such as problems with social interactions, depression, and poor psychosocial adjustment also may affect this patient population. Furthermore, perceived health status has been found to be lower in the heart failure population when compared with other patient populations with chronic disease states (Juenger et al., 2002).

In addition to the analysis requested by the HMO, Cynthia is interested in measuring humanistic outcomes and thinks that it would be important to collect information on her patients' HRQoL, as well as clinical and economic measures. She chooses to use the SF-12 and the Chronic Heart Failure Questionnaire found in the literature to assess HRQoL (Guyatt et al., 1989). She also chooses to adapt some of the questions from a patient satisfaction survey developed by Tinelli et al. (2011) that measured satisfaction for a community pharmacy medication therapy management service in patients with coronary artery disease.

Cynthia is interested in measuring actual absenteeism from work as well as decreases in productivity owing to shortness of breath and fatigue. However, she acknowledges that these data would be difficult to obtain from patients and their employers, so she chooses not to collect data on these outcomes at this time.

Once the question, objectives, variables, and methods of analysis have been identified, one must determine how data will be collected for each of their variables. While researchers can use a variety of sources and methods of data collection, health care professionals often can obtain most of their data from records they already maintain.

Partnerships with Other Health Care Entities

As pertinent variables are identified, it will become clear that some data will be easily obtainable, whereas information on other variables will be extremely difficult to acquire. Pharmacy-related data often are easily retrievable because much of the information can be obtained directly from the pharmacy's prescription computer program (and the patient profiles). Patient demographic information such as gender and age generally is also stored on the pharmacy computer. However, depending on the question, data from other areas of the health care system may need to be collected as well. For instance, one hypothesis could be that a pharmacy service may result in a reduction in emergency department visits. Unless pharmacists have access to an electronic medical record, they often do not have access to data on emergency department visits.

To obtain disease-related health data, it is helpful to develop partnerships with physician groups and hospitals that serve patients participating in a clinical pharmacy service. A partnership with physicians may allow pharmacists access to patient charts to collect disease-related health data (with patients' permission). A patient chart contains progress notes, laboratory values, referrals to specialists, and any other pertinent information about the patient. A patient chart could provide a complete picture of the patient and serve to show how a patient has improved under a pharmacist's care.

Another partnership that would be helpful is with insurers for health services. Insurance companies, HMOs, pharmacy benefits managers (PBMs), and government agencies such as Medicaid and the Veterans Administration generally maintain health care expenditure data for the patients for whom they pay for health services. Pharmacist interventions can be correlated with these health care expenditure data to determine the effectiveness of the intervention. For example, the numbers and costs of hospitalizations and emergency department visits for patients enrolled in a pharmacist-managed health program can be examined prior to the start of the program and again 1 year later.

Costs of goods and services seem intuitively simple to quantify. However, a number of potentially complicated issues must be considered. When identifying the goods or services, will the actual cost (wholesale cost) to the pharmacy or health care organization be measured, or will the charge to the patient or payer (retail cost) be measured? Sometimes access to cost data may be limited, or the data may be entirely unavailable. For instance, if Cynthia Marshall was using a hospital database to collect hospitalization and emergency department information for the HMO's heart failure patients, it may be impossible to isolate the actual costs related to heart failure, particularly if a patient was admitted for multiple reasons.

There are advantages and disadvantages to using different data sources. Primary data collection, in which data are collected solely for purposes of the project, are the best sources of data because one can request and generate exactly the information that is needed for the program. This is also helpful when it is important to see how a particular patient behaves or uses the health care system. For instance, if Cynthia changes a patient's drug regimen, she will want to collect data to determine if the patient becomes more compliant with his or her medications or has a better quality of life as a result. The disadvantage of primary data collection is that the data have to be collected manually as opposed to simply retrieving them from a database that already houses the information. Primary data collection can be time consuming and expensive.

A secondary data source, such as a hospital or HMO database, can be a useful resource. Secondary data sources are particularly useful when gathering economic data because there is usually a direct correlation with how much an insurance company is charged and how much the product or service actually costs to provide. A researcher or clinician also can examine the data on a large number of patients to see if an intervention actually affects a population of patients. A disadvantage of data such as these is that they are completely anonymous. There are no patient identifiers, and the data cannot be linked to individual patients. As a result, if one sample of patients patronizing one pharmacy is provided an intervention and another sample of patients is not, there is no way to detect economic change through the secondary data source.

Cynthia is using a secondary data source to conduct her cost-benefit analysis. After she has identified her variables of interest, she asks the HMO to provide her with baseline information on all heart failure patients prior to implementing her service. The HMO is able to stratify the data by diagnosis, so Cynthia asks for annual numbers and costs of hospitalizations, emergency room visits, and medications for their heart failure patients (see Table 9-2). She also asks for the data used to calculate the HMO's quality indicators on heart failure.

Cynthia forges ahead and begins her heart failure clinic. She successfully enrolls 250 patients in her service over the course of 1 year. At the end of the year, she conducts her cost-benefit analysis and asks the HMO for the same information that she collected at baseline (see Table 9-3).

Cynthia also must determine how much the clinic costs the HMO so that she can factor those costs into her analysis. She includes the salary and benefits of a part-time pharmacist to manage the service ($50,000) and the laboratory tests, supplies, and space costs necessary to monitor the patients ($72,000). Therefore, the direct cost of the clinic was $122,000 that year. The benefits of this service included decreases in the number of hospitalizations and average length of stay. Taken together, the total number of hospital days was reduced by an average of 7.2 days [(5 × 5.6) – (4 × 5.2)] for each of the 250 patients enrolled in the HMO and monitored by the service. If hospitalizations are assumed to cost the HMO $500 per day, the service saved the HMO $900,000 in hospitalizations (7.2 days × 250 patients × $500/day). The service also decreased the number of emergency department visits, resulting in savings of $250,000 ($1,000 × 1 visit × 250 patients). However, when Cynthia started to properly manage her heart failure patients, she found that they typically needed more medications to better manage their disease. The average number of prescription drugs for her patient population rose from three to four prescriptions per month, resulting in an increased expenditure of $10,000 ($40 × 1 prescription × 250 patients). When the costs and benefits are summed, the annual total cost of providing the service is $122,000, and the total benefits are $1,140,000 ($900,000 + $250,000 – $10,000). These results can be expressed as either a benefit-cost ratio of 9.3:1 ($1,140,000:$122,000) or a net benefit (cost savings) of $1,018,000. From this analysis, it appears that this service is valuable to the HMO.

Cynthia realizes that her analysis is only as strong as the variables that she has chosen to include. As a result, she decides to conduct a sensitivity analysis to ensure that her conclusions are sound. Some of the variables that she thinks may change the results of the project are the decreased number of hospitalizations, decreased length of stay, and decreased number of emergency department visits. She reexamines the data by taking half the difference in the number of hospital days and half the number of emergency department visits to see if the results change. The average number of hospital days per patient was decreased from 7.2 to 3.6 days, and the average of emergency department visits per patient was decreased from 1 to 0.5. As a result, hospitalizations cost savings were $450,000 (3.6 days × 250 patients × $500/day), and emergency department visits cost savings were $125,000 (0.5 visits × 250 patients × $1,000). This sensitivity analysis shows the total benefit to equal $565,000 ($450,000 + $125,000 − $10,000). The benefit-cost ratio drops to 4.6:1, and the net benefit is $443,000, but the service still clearly appears to be valuable to the HMO.

Pharmacists have the ability to assess patients for general signs and symptoms of many disease states. In many states, pharmacists are also able to check blood glucose and lipid levels. They are able to measure peak flows in pulmonary patients and can assess minor skin abnormalities and wound care. All these clinical variables may be recorded at baseline and at each subsequent visit to chart changes and/or improvements in the patient's condition.

Cynthia has decided to collect information on each patient's level of edema, blood pressure, weight, NYHA classification, and shortness of breath. She also will record laboratory values where appropriate. When patients are enrolled in her clinic, she records baseline values of the variables listed. She sees each patient every month for the first 3 months. Patients continue to be seen monthly until both Cynthia and the patient agree that the patient is stable, and then they will see Cynthia every 3 months thereafter. At each clinic visit, Cynthia recorded the same values in the electronic medical record. At the end of the year, she compiled the data to show that her medication management and education program have provided clinical benefits to her patients (see Table 9-4). She also asks the HMO for the same data on all heart failure patients (not just the ones that enrolled in her clinic) in order to calculate the quality indicators for heart failure.

Humanistic outcomes generally are obtained directly from patients in the form of standardized questionnaires. There are two steps when collecting humanistic information. First is to identify the best and most appropriate survey or surveys to use (general or specific), and second is to decide how to administer the HRQoL instruments or surveys.

The four main ways to administer surveys are self-administered, face-to-face, mail, and telephone. When patients self-administer the surveys, they can answer the questions on their own. They are sometimes more willing to be truthful with sensitive information, such as income. This method is the least expensive method. The downside of self-administered surveys is that patients sometimes do not finish the surveys and that some disease states make it difficult for patients to self-administer (e.g., glaucoma, rheumatoid arthritis, and Parkinson disease).

Mail surveys generally have the same advantages as self-administered surveys and can effectively reach a large number of people. However, it is often difficult to motivate people to complete the survey and return it without some form of incentive, and it is also more logistically complicated for the pharmacist who needs the data. Face-to-face interviews are effective in eliciting narrative responses and can be used with patients who lack good reading or writing skills, whose first language is not English, or who have difficulty seeing. The interviewer can gauge how the survey is going and can determine if the respondent is having difficulty understanding the items. On the other hand interviews take more time and the process can be very costly.

Telephone surveys have the same advantages of face-to-face interviews but tend to be less expensive. Researchers can survey individuals from a large geographic area, and data can be linked to a record management system that can track respondents. Unfortunately, researchers cannot reach respondents who do not have a telephone and whose first language is not English, and they cannot verify the identity of the respondent. Multiple calls may be needed to contact the individual, and visual aids cannot be used unless they are mailed to respondents in advance.

The advantages and disadvantages of each method of administration are important as the study is designed and the patient population is identified. If an indigent population is being studied, the telephone format may not be appropriate because that population may not have access to telephones. If a predominately Spanish-speaking population is being surveyed, face-to-face interviews with bilingual interviewers may be appropriate.

Cynthia feels that health-related quality of life and patient satisfaction are important aspects of her clinic. She thinks that these outcomes can serve to complement the economic and clinical outcomes that she is reporting to the HMO and can further prove her case that the clinic is a valuable service to the heart failure patients.

When patients enroll in the clinic, Cynthia assists them in completing the surveys. Most patients prefer to read the surveys and complete them at their own pace. Cynthia reads and records the survey information for those who are uncomfortable filling out the surveys individually. Through statistical analysis, she determines that there has been statistically significant change in all three quality-of-life measures over the course of the year (i.e., the SF-12, the Heart Failure Questionnaire, and the PSQ) and concludes that the clinic provides benefits from the patients' perspective.

After the data collection is completed, the data must be analyzed and assessed to interpret the results and reach meaningful conclusions. There are several ways to compile data so that they are easy to read and understand.

First, the service needs to be evaluated to determine if it provided a positive intervention for the patients. Prior to implementing the intervention, demographic, clinical, and humanistic information should have been collected. These are called baseline data because collection is done at the beginning of the intervention, prior to any changes. Collecting baseline data is extremely important, a task that health care professionals routinely forget to do prior to starting new programs or services.

After the program ends or has reached a predetermined point, the same pieces of information that were collected at baseline should be collected again. The data then are analyzed to determine if patient outcomes changed after enrolling in the clinic. Again, if baseline data were not collected, it becomes impossible to determine if changes in the values collected after the intervention were due to the intervention or to some other factor. For example, assume that a patient comes into a pharmacy and suggests that a new herbal remedy has caused her diabetes to get better. She says that she knows this because her blood sugar level is below 140 mg/dL. However, on further examination, she admits that she is not aware of what her blood sugar was prior to starting the herbal remedy. As a result, it is unclear whether the patient may have had blood sugar levels below 140 mg/dL prior to the remedy and to what extent, if any, the remedy provided any benefit.

Second, the financial impact of the service must be determined, keeping in mind the perspective of the analysis. For example, were the dollars saved from averted emergency department visit and hospitalization costs greater than the cost of implementing the clinic? Was the financial benefit to the patient worth the extra money he or she spent on the clinic? To perform these analyses, a sample of patients who participated in the clinic (intervention group) and a similar sample of patients (control group) who did not participate are needed. The total costs of health care between the two groups are compared to determine if there are differences. Total health care costs would include all hospitalizations, emergency department visits, physician visits, prescription medications, laboratory costs, and visits to the new clinic. To determine the benefit to the patient, various outcome variables are identified that measure change in patient status (e.g., humanistic outcomes such as HRQoL and decreases in side effects) and increases in patient satisfaction and/or clinical outcomes (e.g., blood pressure, edema, and blood sugar levels). These variables can be evaluated separately or compiled to form an outcome measure.

Assessment of Success or Failure of a Program or Service

After conducting an evaluation, the results may not be strongly in favor of a service. This does not necessarily mean that the service will be, or was a failure. There are many reasons why the results may be equivocal.

First, there may not have been enough patients for a complete evaluation. If the changes that are sought are small but clinically and economically significant, a larger number of patients may have been needed to detect statistically, or even clinically significant changes. The number of subjects needed to detect the impact of the interventions should be determined prior to starting the clinic. In biostatistics, this concept is commonly referred to as power.

Second, many outcomes depend on how much time passes since the intervention. Many changes, particularly for clinical outcomes, take time to occur. A literature review of similar interventions may help to identify how much time may be needed to detect a clinically significant change. For instance, patients with asthma can see almost immediate relief when they are provided appropriate education and long- and short-acting medications to treat exacerbations. Conversely, comparisons between patients being educated with smoking-cessation classes and those not receiving education will be much more long term. Decades may be required to determine if some interventions provide economic, clinical, and humanistic benefits.

If the clinic was implemented and the analysis was conducted only to find that the outcomes were not as anticipated, what should a pharmacist do? While some outcomes can be disappointing, it is important to seek explanations for what was found in the results. Was the appropriate amount of time spent developing the clinic? Was there appropriate marketing to the patients? Was there buy-in by other employees in the pharmacy? Even though these objective results are important, other intangible benefits may be equally or even more important. Developing a trusting relationship with physicians, promoting goodwill among patients or patient organizations, and encouraging patient loyalty to the pharmacy are equally important. Individuals in management, although focused on the numbers, should recognize that some benefits are not seen right away or directly. It is incumbent upon the pharmacist to be able to convey these benefits to management so that they recognize the value that your services and programs provide.

Contributing to the Quality Improvement Programs

Health care organizations, insurance companies, and health care professionals are now being rated for the quality of health care that they are providing. These ratings are made public so that patient can make informed choices, and are used by corporations and employers to choose which health care organization they want to use to provide health care to their employees. Many of the quality indicators that are being used are related to appropriate medication use and medication safety. So, pharmacists are in a unique position to provide value to a health care organization and their physicians seeking to improve their scores.

Development of Additional Services

Once a successful service has been developed, it becomes easier to develop a new service in the future. The skills necessary to evaluate the outcomes of a service can be leveraged to start a new service. Perhaps a similar service with a different disease state would be a good next step. Be careful when thinking of starting new projects that are not directly similar to the one already implemented. For instance, the skills gained with the heart failure clinic may not apply directly to starting a compounding specialty or wound care clinic.

To develop a new service, identify additional resources that will be required. More personnel, both for dispensing and for clinical functions, may be needed. More space for counseling, compounding, conducting classes, or placing diagnostic instruments or durable medical equipment also may be needed. Lastly, the way the pharmacy is perceived and marketed to the public may need some rethinking. If the pharmacy previously has dispensed only medications, the new services should be showcased, and the pharmacy can be marketed as a pharmacy with expanded patient care services.

A pharmacist will need to gain buy-in from management, fellow pharmacists, and physicians (see Chapters 4 and 5). Including the feedback and perspectives from colleagues and one's supervisor is essential to communicating the value of existing services and developing new ones. These individuals will have opinions about what went well, and what could use improvement. They also may have their own ideas of new services that could be offered. It is important to remember that those individuals with decision-making authority (i.e., pharmacy corporation, district manager, pharmacy owner) may prioritize ideas in terms of how consistent the idea might be to their overall market plan, organizational mission, and potential for profitability, rather than strictly improvement in patient care. It is incumbent upon the pharmacist to frame the importance of the program in terms that make sense to these decision makers and to speak to the performance indicators by which they are judged by (e.g., profitability of one store in a chain organization)

Reassessing the Service Objectives

As pharmacists who implement programs and services evaluate all they have accomplished, it is wise to do a little continuous quality improvement. They should identify the mistakes that they made along the way and think about how they can avoid making them again. They can also identify how they could have streamlined the process and created a better workflow for the service. Lastly, they may have conducted the project as a pilot project. These pharmacists will need to think about how they will expand the project in a seamless manner.

Pharmacists should seek the advice of those around them—their employees, supervisors, patients, and others affected by their services—to see if they have different viewpoints to share. Their input and constructive criticism may be as valuable as the economic, clinical, and humanistic results that were calculated.

After performing her analyses, Cynthia shared her results with her administrators at the HMO. They were very happy to see these results and asked Cynthia about other types of pharmacy services she might be able to develop. Cynthia said that she would look into expanding her heart failure clinic to become a more generalized cardiology clinic that included lipid management, anticoagulation management, and blood pressure management.

The objectives and goals of any service should guide the outcomes data that are collected and how those data are analyzed. A common goal of any evaluation is to show the value of the program or service. It must be remembered that value can have numerous meanings (e.g., clinical, economic, and humanistic) and can be measured from a number of perspectives (e.g., patient, provider, organization, insurer, and society). The desired outcomes and perspective of the analysis should drive the data collection process and how the outcomes are assessed.

1. What are some outcome measures that you can collect at your current practice?

2. How can you develop a relationship with a payer or physician group for the purpose of conducting economic analyses on your patients?

3. Based on Cynthia's objectives, what variables might she add to her evaluation?

4. Consider a pharmacy with which you are familiar, in any setting. List direct and indirect and medical and nonmedical costs and outcome measures for both an existing service and one that might be considered for addition to that pharmacy's repertoire.

5. Given what you have learned in this chapter, what are some existing services, or programs at your pharmacy you might take into a more formal evaluation to determine effectiveness? How might these concepts apply to current processes and technologies at this pharmacy? What are some new programs or services you might consider for their potential?