How to cross the chasm in the coding assistant tool space?

These thoughts are based on Erin Yepis' post from the Stack Overflow blog: Developers get by with a little help from AI: Stack Overflow Knows code assistant pulse survey results and my experience leading the Technical Advisory team at Sourcegraph.

Stack Overflow is renowned for running its yearly developer surveys that have been uncovering super interesting insights about trends, preferences, issues, and struggles in the dev community. This time around, SO ran a pulse survey (part of their Stack Overflow Knows series) about Code Assistants and got 1,700 responses. They asked:

• When and how often CodeGen tools fall short
• What challenges do developers face with these tools
• What are devs doing with the free time the tools purport to offer
• What code assistant tools are devs using and how do they feel about them
• Are devs more productive using CodeGen tools?

Here are the most interesting findings:

1. Where adoption is high (more than 50% of devs in a team are using code assistant tools), most devs think the tools are satisfying and easy to use.
2. Use cases that require a lot of contextual understanding of the codebase are producing the lowest quality responses.
3. Even with low-quality responses, most devs expressed they are mostly reclaiming time wasted on repetitive and mundane tasks (toil).
4. Devs feel the current state of the underlying LLMs is better suited for public-facing code such as JS and TypeScript for frontend and not so much for proprietary code as most LLMs are trained on publicly available code.
5. For devs learning how to code who use a coding assistant, ChatGPT is the most compelling option as it offers a popular free option vs. other paid tools like GitHub Copilot; this is also the case with IDEs with the free Visual Studio Code.
6. Dev satisfaction with coding assistants is closely linked to ease of use with Codeium at the top of the list, followed by Copilot and ChatGPT.
7. One of the most interesting findings, in my opinion, is that most companies don’t have a good grasp of productivity metrics; this was especially more prevalent in smaller organizations.
8. The adoption of code assistant tools seems to be hindered by a lack of knowledge if there is a company AI policy. 73% of devs surveyed say “they don’t know or are unsure if their companies have an AI policy”.

After reading and analyzing the findings from the survey, here are my three recommendations to cross the chasm in the coding assistant tool space:

1. A clear enterprise AI strategy to establish a company policy will help remove uncertainty about permission to use coding assistant tools and help with widespread adoption.
2. Defining clear and measurable dev productivity metrics is key to understanding the impact and benefit of AI in software development. Without these, it'll be impossible to measure the ROI objectively rather than relying on CSAT surveys.
3. For AI coding assistant tools to become powerful and versatile for any type of codebase and use case, they need to get much better at understanding the codebase context. Devs are wasting a lot of time today manually feeding more context in the prompts which in the end defeats the productivity purpose. There will be a magical moment in dev satisfaction when the tools just work.