10x FY21 Impact Report

FY21 in Review

Where we succeeded

For most of our history, we’ve used what we call the “everything bagel approach” to source the types of ideas we want to invest in. This approach meant we left the market wide open and accepted ideas across every topic in government technology you can think of, including contracting, cybersecurity, robotic process automation (RPA), etc.

In FY21, we did something different. We moved away from the “everything bagel approach” and defined a few specific areas of the government technology ecosystem where we were particularly interested in exploring. We called these areas investment themes. Defining the types of projects where we wanted to invest allowed us to set up parameters for the ideas we wanted to target and center our outreach accordingly. The first themes we chose were Public Trust and Civic Life, Equity in Delivery, and Public Lands and the Environment.

Why these themes? In order to serve the public, we need to understand what matters to the public. And one of the great privileges of being in the civil service is that we are able to bring our whole selves to our work: both as civil servants and as members of the public. After all, we live in communities, we talk with friends and neighbors, and we read the news just like everyone else. In other words, we get to wear two hats at once. We hear what people discuss and what matters to them, and we feel that our themes address topics (like low trust in government and concerns about our changing natural environment) that the public will recognize as timely and directly relevant to their lives.

That change succeeded and worked in our favor for a few reasons:

First, defining themes made outreach easier. For example, when we made Public Lands and the Environment a theme, we immediately knew where to look for great ideas: federal agencies that work to protect and study the environment.

Second, defining themes allowed us to develop important contacts and map the who’s who of certain problem spaces in government technology. One of our challenges on new 10x projects is figuring out where to start and who to talk with first. By moving towards this structure, we’ve built strong contacts with subject-matter experts who we can easily consult on projects relevant to their expertise.

Third, having themes made it easier to compare different ideas during our evaluation periods. For example, it’s easier to debate the merits and opportunities of two ideas that take different angles on climate challenges than it is to compare the merits of a cybersecurity idea to an acquisitions idea.

Lastly, and most importantly, having defined themes allowed us to align our program’s investment portfolio with topics that matter to the public.

Where we shined

For government program managers, one of the best confidence boosters is when our work happens to align well with administration-level priorities.

One area where we really excelled last year was finding areas of significant overlap between what our projects are doing and what the new Administration wants to do. Many of our FY21 projects directly supported legislation and/or government-wide priorities, such as

• CASES Act,
• 21st Century IDEA,
• and the Executive Order on racial justice.

Another success was advancing projects that build products that the public can use. In the past, many projects delivered back-end software for agencies or guidance for federal employees.

But this past year, as a program within GSA’s Technology Transformation Services (TTS), we’ve been focusing more on public-facing work that helps drive TTS’ vision to create trusted modern government experiences for all. If we’re delivering products that support legislation and other priorities — while building things that the public will use directly — we know we’re on the right track.

One example is our Benefits Eligibility Awareness Resource Service (BEARS) team work on Benefit Finder. The BEARS app we’re building will not only help people find and apply for benefits more easily, but it will also add compassion and empathy to public service by centering benefits information around the life events that trigger people to search for benefits, such as a death in the family or losing a home in a natural disaster. Centering benefits around life events rather than agency silos demonstrates that the government understands the needs of the public, which is critical to building trust between public and government. We believe the best time for compassionate and empathetic public service is when people are grieving, or trying to rebuild their lives after a tragedy.

We’ve been working with new technologies in ways we haven’t before. For years, APIs, data products, playbooks, and websites were our bread and butter.

No longer. Keep reading to see how we’re engaging with emerging technologies, like privacy engineering and the Internet of Things (IoT).

Challenges we faced

At 10x, we’re lucky to have flexibility in deciding what projects we take on. Our uniquely iterative investment approach identifies risks and allows us to end projects when it’s clear they won’t deliver impact. The challenge this model presents to us is that sometimes we need to close down projects we love. Certain projects pull at our heartstrings, but if the evidence does not point to their successful exit, then the responsible decision is to end the project.

Other times, projects do have a viable path towards success, but because we don’t have unlimited funding, we are forced to make hard decisions about which projects should move forward from one phase to another. We’re trained to make the most of what we have and to work within these constraints. Having a limited number of projects we can invest in forces us to think more critically about funding decisions. We always have to ask ourselves: will this project deliver value?

Bringing passion into our projects is key to their successful outcomes. But when it comes to making objective funding decisions, we have to be, well, objective. And that requires asking tough questions and making hard choices. Some of the most common reasons we end projects we love is because our partner agencies may not be equipped – or ready – to take them on and maintain them sustainably.

In government, budgets are often planned out two years in advance. But 10x develops products more quickly than that. The problem this presents is that when we are working with an agency that wants to own one of our solutions, they may not have access to the financial liquidity that it will take to own the product long-term. Another challenge we face is that hiring in government often takes time, and many agencies have not yet built a large pool of engineers, product managers, and other technical talent to leverage on new initiatives. These resource constraints make it difficult to transition our projects to partner agencies. While we do face these staffing challenges regularly, there is increased momentum in government to prioritize bringing on top notch technical talent into the civil workforce. A good example is the US Digital Corps program – hosted by GSA - that matches early career technologists with government agencies. We believe this and other initiatives focusing on developing a digital civil service will help a great deal with these types of challenges.

To mitigate these risks, we try to be as up front and transparent with our partners as possible when it comes to the resources that will be required to own a solution. Recognizing the critical importance of meeting people where they are and working with (not for) our partners means we can channel a project into a direction that fully aligns with our agency partners’ capabilities. This is a challenge, but it’s one that 10x is mindful of and committed to solving for, as soon as we see it on the horizon.

But occasionally, even when our teams and our partner agencies are excited about a project and see how much impact it could deliver, we need to end the project early when the hand-off feels too risky. And when we do that, it is always with a heavy heart.

FY21 Project Showcase

We had a lot of projects that really inspired us in FY21 – projects we feel can – and do – deliver true impact for the public. For this year’s Impact Report, we’re focusing on one of the most exciting projects that blossomed during FY21: Combating Bias in Artificial Intelligence and Machine Learning Implementations (AI/ML). To check out some of our other work, please head over to the [projects section of our site](../../projects/). It’s got the scoop on what we learned and the impact we’ve created through our individual projects.

How a 10x investment enshrines democratic principles in a changing govtech landscape

Artificial Intelligence and Machine Learning (AI/ML) technologies will affect how the government serves the public. Public sector spending on AI is projected to reach well into the billions before long.¹ Federal CIOs have encouraged agencies to start experimenting with AI and two previous presidential administrations have all issued guidance, memoranda, or executive orders specific to AI adoption in government.

Despite all of this momentum, AI/ML technologies are fairly new to many government agencies. Many are still figuring out how these exciting new capabilities can support their business needs and how they can start adopting them.

We’ve funded several AI-related projects over the years and made some observations. We believe that AI/ML use in government is going to accelerate and it’s going to come with challenges. The stakes are high and the government needs to get it right the first time.

Stakes are high because AI/ML implementations risk harm if not done thoughtfully and ethically. These technologies help make real decisions that affect people’s lives, so fairness is a paramount concern. Enshrining democratic values like fairness, equity, and equality in AI/ML is a tall order. So where do we begin? How can we deliver impact here?

Bias Toolkit

To meet this challenge, we placed a bet on the table: combating bias in upstream datasets — before they go through an AI pipeline that can make important decisions — is a good place to start.

Throughout FY21, the 10x team worked with the U.S. Census Bureau (Census) to develop the bias toolkit. This project along with our other AI-related projects, is how we’re helping the government get AI/ML right during these early days.

What we delivered

The toolkit includes three functioning de-biasing tools that federal employees can use to lay a more equitable foundation for their AI-enhanced workflows. Together, these three tools provide a great foundation for future de-biasing initiatives.

• The first tool creates carbon copies of datasets with placeholder (or ‘dummy’ data) that allows a federal employee to generate and test multiple, similar datasets through the AI model algorithm. Testing AI models with both real and placeholder data helps reveal sources of bias that might remain hidden if the model were trained only on the target dataset.

• The second tool uses AI to detect ableist language in federal job postings, which is language that may be offensive to people with disabilities. The tool automatically suggests more appropriate, inclusive alternatives that hiring managers can use to create better, more equitable job descriptions. Visit the Ableist Language Detector for Job Descriptions Github Repo to learn more.

• The third tool offers a standard language format, or model card, that describes the AI/ML model’s characteristics. Similar to a nutrition label on a can of soup that tells you the ingredients, nutritional profile, and allergen warnings, the model card shows the characteristics — including errors that could lead to biased outputs if left unaddressed – within that AI model. Model cards provide transparency about the AI model’s limitations and can help people re-use these models equitably and transparently.

The toolkit is currently in beta and awaiting a full launch in FY22. Census will own, maintain, and expand the toolkit. As the U.S. government’s premier statistical agency, Census works with the types of eye-poppingly large, complex datasets that power advanced AI, making them a great fit for championing this work into the future.

Why we’re proud of this

Everyone wins when a project is moved to where it can continue to thrive after exiting the 10x funding pipeline. An example of how two agencies should work together, it’s a great project transition that will have a huge impact.

We showcase this de-biasing project because we believe it resonates with communities outside of our core user base of federal employees. This project lives at the intersection of two of the most important conversations that we, as a country, are having and highlights how the government is responding. The first conversation focuses on reckoning with injustice in our society as experienced by minority and vulnerable populations. The second is the rapidly evolving field of artificial intelligence and the cultural angst around how AI and other emerging technologies will affect society (for better or worse) in the years to come.

We’re proud of this project and we’re proud that we helped ensure that the government adopts rapidly advancing, powerful technologies in a way that shows a serious effort to protect democratic values and delivers positive, meaningful impact for the public. We are excited to see it grow and mature under the stewardship of our U.S. Census Bureau partners. Be on the lookout for the full roll-out in 2022, we’ll keep our site updated. In the meantime, the initial toolkit is open-source and available freely on the web. Anyone in the world can find, use, and contribute to our tools.

Summary

10x is a small program within our own agency and we are only one of many initiatives across the government that are also thinking about the future of AI/ML in public service delivery. Naturally, the private sector and other groups will play a central role in the future of public sector AI/ML adoption. But we believe the government has a special role to play.

Our ambition for this project was to make small, positive interventions that reinforced democratic values of justice and equity as the government adopts AI/ML. We aim to create positive ripple effects that will carry into the future and influence how the government uses AI to serve the public.

Look Ahead to FY22

We’re 100% convinced that FY22 is going to be busy, exciting, and challenging. We have projects advancing into technology we’ve never worked with before. We have a once-in-a-generation opportunity to multiply our impact by funding more work than ever. And new doors have opened that can help scale and support our projects post-10x when the projects complete Phase Four. Here are a few things we’re expecting to come across over the next 365 days and how we’re thinking about them:

Scaling and Multiplying

Two things will be important for us in FY22:

• First, the American Rescue Plan (ARP) will be significant for 10x in FY22.
• Second, we’re expecting to clone our own model and help other agencies replicate something similar to 10x within their organizations.

Our organization — the Technology Transformation Services — received ARP funding specifically to recover, rebuild, and reimagine digital public service delivery in the face of the COVID-19 pandemic. 10x will use some of these funds to reimagine digital public service delivery, which happens to be something we’re particularly good at.

With this TTS ARP support, we’ll fund more work across phases to experiment with and uncover new opportunities for how small improvements can provide a huge impact for the government’s response to COVID-19.

• our projects align specifically to the spirit of the TTS ARP, and/or
• require significant financial advancement to deliver impact beyond what 10x can provide in Phase Four.

Joining forces with the TTS ARP PMO will make our work even more impactful. If they are considering work that may have potential, but is unproven, we will take the lead, putting it through our investment phases. We can identify this work’s potential and viability, before our parent organization considers this work for more substantial support.

We want to formally pursue a goal we’ve had for years: helping other organizations in government spin-up initiatives that harness what we’ve learned and use some of our methods. To do that, we’re actively talking to multiple federal agencies who want to work with us to recreate our model in their organization. We will help these other agencies, but also recognize that the agency partners will have to do the real work:

• Finding funding,
• Finding a safe space to operate within their agency’s bureaucracy, and
• Getting buy-in from leadership for standing up a 10x-like process.

These steps won’t be easy, but we'll help all we can. We also recognize that there is always room for improvement. We know there are opportunities to do things differently while keeping our basic risk-mitigation, iterative model intact.

We fully expect these initiatives in other agencies will differ from our program, since they will need to be tailored to fit the funding, culture, and IT environments in other agencies.