5 Things to Consider when Starting Your Impact Measurement Journey
The following is a guest blog post written by David Goodman Ph.D., Head of Data Trusts at BrightHive, a public benefit company that helps organizations, networks, & communities securely and responsibly link their data to enhance their impact, empower individual and collective decision making, and increase equity of opportunity. David uses his extensive knowledge and experience public, private, philanthropic, and academic sectors to identify opportunities to create new shared data resources for public, private, philanthropic, and academic sectors. Before BrightHive, David has worked in data, impact, and research with Houston Endowment, Teach For America, Edvance Research, Texas Education Agency. David earned a Ph.D. in political science from Rice University.
I came to BrightHive two years ago with the goal of helping organizations and networks share their data to achieve collective goals through collaborative data trusts. In that time, BrightHive has helped to design and develop secure and sustainable data governance frameworks for numerous state, regional, and community-based data trusts. A key part of the data trust process is working with the organization’s staff to better understand their data and technical capacities, but the conversation inevitably touches on their organization’s approach to “impact,” or, in many cases, the organization’s uncomfortability, confusion, and uncertainty around “impact.”
The confusion and uncertainty has been a stark reminder that we still have a long way to go when it comes to achieving impact. But it is also an opportunity to help organizations that are looking to start their journey to impact. Here are a few things for organizations to consider as they contemplate their first steps on the impact journey.
Impact is what you believe it is
Despite what the impact pied pipers tell you, there is no right or wrong answer to what impact is. Impact is what your organization believes it is and based on your preferences and priorities related to answering a specific question and the scope of that question. If your organization believes student achievement is accomplished by teaching students to weave baskets, then so be it. I suspect that you will have a tough time making that causal argument, proving how your impact compares to others, or getting funding to support it, but if that is what you believe, then that is your impact. Don’t let others tell you what your impact should be or what framework you should follow or what metrics you should capture. Remember why you started this work in the first place and go from there.
The best part: your impact can change over time. The more you learn or if your priorities change, so too can your impact. This is especially true for new initiatives and programs that you are testing out. Your impact journey is your own. Don’t let the pied pipers stress you out or pressure you to do something or adopt a framework that you aren’t ready for or that isn’t consistent with your preferences and priorities.
The journey to impact starts with a question, not a solution
Starting an impact journey should not be so difficult. Unfortunately, this is not the case for many organizations who are bombarded with pied pipers promising success and clarity as long as you hire their consultants, use their platform, or adopt their framework. Sure there are some really good folks out there that can assist you on your journey to impact, but please be leary of those that try to impose a framework or ask you to purchase a system or platform without first sitting down with your staff to figure out what you want to do, what your priorities are, and - in some cases most importantly - what capacities you possess (or don’t possess) to do what you want to do. Good consultants will bring their impact tool kit with them and pull out the right tool or framework based on what you share with them and what question you want to answer. If you are just starting out and implementing a new program that has not been tested and your consultant proposes a randomized controlled trial (and they will!), you likely have the wrong consultant.
As I mentioned above, start with the question that you want to address. Identify the goal that you want to achieve with your work and fill in the details about the who, what, where, how, and when. You will be surprised at how your answers to these very questions will inform your journey to impact, including what framework you should use, what metrics will provide the appropriate evidence that you are or have achieved your goal, and which partners can provide you with the data, expertise, and insights you’ll need to achieve your goal. You can even do some research to see who may have done this work in the past or credible studies that can provide information that you should consider in your work. There are plenty of folks out there that can help you with this work, but it's also a really great exercise to do with staff as well. The internal discussions about your work and goals creates alignment and buy-in across teams and departments and often raises issues and preferences for data and results that will need to be incorporated into the design and process.
Technology is not a shortcut to impact
One of the biggest questions for organizations just starting their journey to impact is around the use of technology. With the growing chorus of voices touting the newest app, platform, or dashboard that will lead you to impact, it is hard not to be intrigued. But the reality is that technology is only as good as the information that you put in it. If you don’t have clarity and appropriate data and buy-in across funders, grantees, and partners, then bad, inconsistent, inappropriate data going into any technology will result in bad, inconsistent, and inappropriate analyses or visualizations.
The other issue with technology is that most either impose a specific structure or relationship between data or simply ignore the relationships altogether. The structured technology doesn’t always align with the organization’s approach, preferences, or approach to impact and often requires them to adopt the structure to use the technology. Unstructured technology, on the other hand, requires strict adherence by partners to use it the same way and staff to ensure consistency. Both situations pose significant hurdles for organizations just starting their impact journey - not to mention significant costs and burden on staff to change, adopt, re-train, etc. This is why identifying your impact question and detailing the what, who, how, and when is so critical. The choice of technology should only be after your organization - and I mean all departments within your organization - and your partners are aligned and bought into what you plan to do.
One final thought on technology is that more and more organizations are seeing the value of not choosing an “out of the box” or “off the shelf” technical solution. The days of the “one technology to rule them all” seems to be (thankfully) on its way out. Organizations no longer have to make a substantial investment in technology for a technical solution that will address their data, measurement, and impact challenges today, but will likely be obsolete in a few years when learning happens, priorities change, or new partners come into the picture. More and more organizations are seeing the value and flexibility of interoperable systems that can integrate CRM with CMS with GMS and then with a front end analytic and visualizations platform.
It’s all about the interoperability
The beauty of interoperability is that through application programming interfaces (APIs) and extraction, transform, and load (ETL) algorithms, connected systems can meet organizations “where they are.” This means that they can often use their existing systems, not have to make substantial investments into a new system that would impose tremendous burden on their own staff, as well as their partners. Probably most important, is that interoperability does not constrain the organization: if changes to data, outcomes, priorities, or strategies arise, these changes can be incorporated in the integrations. This is especially important for organizations that are just starting their impact journey and may need to change or pivot as they begin to learn. One caveat to interoperability, however, is that it requires strong and collaborative data governance - the agreement on what data will be shared, how it will be used, and what data privacy and security measures must be taken in order to carry these out, among other things. But, you’ve got friends like BrightHive to help with that tedious stuff!
True “impact” requires collaboration & coordination
One final thing to discuss for organizations looking to start their impact journey is that no matter what question your organization chooses to guide their impact journey, it is very likely that they are not the only ones contributing to it. What I mean is that while your organization’s work in a specific area - whether student achievement, helping workers find jobs, or helping parents find child care - there are other organizations attempting to do the same thing - or something that potentially will impact your work. This is why a 2-hour a week afterschool program cannot take sole credit for an increase in student achievement or reduction in student truancies or disciplinary actions (a real conversation I have had with a large nonprofit organization).
While this may seem obvious to some, I bring it up because it is both a valuable reminder about the work that we do, but also because of the opportunities it creates. If we want to achieve “true” impact, we need to collaborate and coordinate with other organizations that may do similar work or work that, if sequenced or coordinated, could reach more, do more, or sustain the work over time. Collaboration and coordination is also important in terms of leveraging existing knowledge, expertise, relationships and lessons - not to mention data. Each of these represent time and resources for organizations. And for organizations just starting out this can be very costly. This is why the impact question should always incorporate what others are doing and potential partnerships and collaborations to minimize redundancies, improve efficiencies, and expand and sustain impact whenever possible.
At the end of the day, this journey to impact is, in fact, a journey and not a destination. Things will change, (thank you COVID), strategies will work and fail, and the impact question you started with will inevitably change over time. That is part of the process and you should try to enjoy each step. And while the thoughts I have shared here are things to consider as you start on your impact journey, remember that this is your journey and there are good folks and organizations out there who are eager to help you along the way, including myself. Please feel free to reach out to me if I can help in any way.
Join us for a virtual fireside chat on July 9 with David Goodman, Head of Data Trusts at BrightHive and Mariana Catz, Managing Partner at Gen3 as they address the importance of impact measurement in your organization and how you can get started.