2024 Machine Learning Review (For Technical Writers)#
2024 Dec 31
Back in March 2023 I published GenAI Outlook 2023. With less than 12 hours remaining in 2024 I have managed to keep my yearly streak going. This post recaps how much (or little) the ideas mentioned in GenAI Outlook 2023 have panned out, and then discusses potential future trends for 2025.
Caution: Nothing here is backed up with hard data which means anything and everything could be wildly wrong. These are just my general impressions, based off of anecdotal conversations with other technical writers.
“GenAI Outlook” => “ML Review”#
This year’s post is called Machine Learning Review (For Technical Writers)
rather than GenAI Outlook to reflect the widening scope of discussion that
I want to have. Generative AI (GenAI) is a subset of machine learning (ML), and
ML is a subset of artificial intelligence (AI). Over the past year I’ve realized
that there are many ways that ML and technical writing (TW) might potentially
interact beyond the relatively narrow subfield of GenAI. Maybe next year I’ll
become aware of other AI fields outside of ML (e.g. expert systems) and
I’ll have to update the title again to AI Review (For Technical Writers),
but for now the only field on my radar is machine learning.
Review of 2023 outlook#
First, status updates on the ideas mentioned in GenAI Outlook 2023.
Job loss#
When I wrote the 2023 outlook, a lot of technical writers were worried that GenAI would automate our profession out of existence. This has not objectively happened at scale so far. I am aware of only one case where a technical writer maybe lost their job because of GenAI.
Automation#
In the early days of the GenAI explosion, remember how seemingly every blog post included verbatim Q&A discussion with ChatGPT? “Here’s what ChatGPT has to say on the matter.” My 2023 outlook was a victim of that unfortunate trend. I asked GPT-4 to list out what parts of technical writing are potentially automatable with LLMs. Here’s a quick summary of how much each of those ideas has been adopted to date.
Basic content generation#
ChatGPT can generate paragraphs or sections based on given topics or outlines, providing a starting point for technical writers. This can speed up the content creation process and help maintain consistency in writing.
There’s a lot of this happening. Tom Johnson has been using a prompt engineering approach to automate release notes authoring. I have also automated some of my changelog process with moderate success. Manny Silva is extensively automating first draft work. I can recall many more anecdotes like this.
Data analysis and interpretation#
AI can analyze large datasets and generate summaries, trends, or insights that can be incorporated into technical documents.
Summarization is covered later.
Regarding trends, I’m not aware of anyone using LLMs for this task and I actually don’t even know what “generating trends” would look like.
Regarding insights, I can recall some one-off instances of technical writers using large context window models to help think through some particular problems in their docs. E.g. they would provide all of their docs as input and then ask the LLM pointed questions related to those issues.
Formatting and template creation#
AI can automatically apply formatting and styling rules to documents, ensuring they adhere to specific guidelines or templates.
I personally worked on automated style guide editing a lot in 2023. My current opinion is that it’s feasible but requires a fine-tuned model, which means a lot of feature engineering, which means a lot of upfront toil and careful design. Also, it’s tough to get the UX right.
Grammar and spell-checking#
ChatGPT can identify and correct grammatical errors, spelling mistakes, and other language inconsistencies, leading to higher-quality content.
I have heard of technical writers using LLMs for one-off editing tasks. E.g. they were given the first draft of a new doc written by a software engineer (or product manager, or whatever) and were told that the doc must be published in a couple hours. The first draft had a lot of errors and typos. To meet the ridiculous deadline1 the writers fed the first draft through an LLM to quickly fix the major issues.
1 Pro tip: don’t do this
Terminology consistency#
AI can help maintain the use of consistent terminology and phrases throughout a document, reducing confusion for readers.
This still sounds feasible, but I haven’t heard of anyone using LLMs for this task. It may require a lot of upfront work around defining the preferred terms and phrases.2
2 On the other hand, it would be pretty trivial for me to provide each section of my docs to a model and ask it to extract terms and create a concise definition for each term. I’ll try it later today. It’s moments like these that keep me motivated to keep blogging. When I blog, new ideas just float up to the surface in a really natural and effortless way. The writing itself is hard, as always. But it’s amazing how new ideas just naturally float to the surface as a byproduct of the writing.
Content summarization#
ChatGPT can create concise summaries or abstracts of longer, more complex documents, making them more accessible to a wider audience.
I’m surprised that there hasn’t been more adoption here. LLMs reliably generate high-quality summaries when given the content-to-summarize as input. It’s one of the few use cases where there’s very little risk of hallucination in my experience. Yet I don’t see many docs sites offering LLM-generated summaries and I’m not aware of many teams using LLMs to systematically generate summary-like content behind-the-scenes, such as the opening or closing paragraphs of docs.
Content translation#
AI language models can translate technical content into multiple languages, helping to disseminate information globally.
I haven’t seen a big uptick in more docs sites being translated into multiple languages. I do think that LLMs have made it more feasible but I imagine that the main constraint now is engineering resources. E.g. you need to dedicate engineers to building out the automated translation pipeline for your docs site. Maybe the static site generators and content management systems will start solving this for us. E.g. just give Sphinx an API key to your favorite GenAI service, and it will take care of the end-to-end translation pipeline: determining what docs need to be translated, using the GenAI service to translate the doc, etc.
FAQ generation#
AI can identify common questions related to a topic and generate clear, concise answers.
Not aware of anyone doing this. I still think that Q&A will become increasingly important over time. More on that below.
Metadata generation#
AI can automatically generate metadata for technical documents, such as keywords, tags, and descriptions, improving searchability and discoverability.
Ditto, haven’t heard of anyone doing this.
Plagiarism detection#
AI can identify potential plagiarism cases in technical writing and suggest alternative content to maintain originality.
Ditto again, not aware of anyone doing this in corporate technical writing. I have heard about stuff like this in academia.
Review of other trends#
My initial 2023 outlook left out some important stuff. I want to provide status updates on those things now.
RAG chatbots have not taken over the docs world#
Gather a list of 1000 docs sites from any domain (or a mix of domains). You will find that a supermajority (+80%) of them have not shipped a companion retrieval-augmented generation (RAG) chatbot to supplement the traditional web-based docs experience. Even the OpenAI docs don’t have one.
I actually think that RAG chatbots can be very valuable, and I have heard a few stories of companies enjoying significant productivity boosts thanks to their internal RAG chatbots. But the objective fact remains: most docs sites have not shipped a RAG chatbot.
Policy is a nightmare#
For the minority of technical writers that are interested in seriously adopting GenAI into their workflows, confusing policy seems to be a significant obstacle to adoption for everyone, across companies and across industries. Questions like these are the current blockers:
“What GenAI services are we even approved to use?”
“Can we really trust GenAI service XYZ with our non-public data?”
“Are we setting our company up for legal issues in the future?”
2025 forecasts#
Continued lack of interest in GenAI#
It seems that most (~60%) technical writers (TWs) are not interested in integrating GenAI into their work practices for a variety of reasons:
Fear of accidentally automating themselves out of a job
Environmental concerns
Copyright issues
A deep disdain for hallucination
I expect adoption of GenAI in technical writing to continue to be slow in 2025 because I don’t think these issues will be solved in 2025.
Jobs still safe for another year#
I’m not seeing the type of massive, systematic automation that would be needed to eliminate the role of technical writer. There are faint hints of it in Basic content generation but this is only 1 of like 10 or more things that would need to be extensively and reliably automated in many different products. This extensive automation (and therefore job loss) is still possible for 2026 and beyond.
Progress on the intractable challenges#
I think combining these ML technologies and approaches will help us tangible progress in 2025 on the intractable challenges of technical writing:
Supervised learning (fine-tuning is a form of supervised learning)
Generation models (Gemini 1.5 Pro, Claude 3.5 Sonnet, etc.)
I have a defensive publication in the works that demonstrates how we can combine embeddings and generation models to make progress on the correctness problem.
Q&A renaissance#
This is a primordial soup of an idea. I have a hunch that Q&A (questions & answers) will become more and more important. Q&A is everywhere:
When language models are trained or fine-tuned, the data is often structured as Q&A.
When I interact with Gemini, Claude, etc. through a chat UI, the conversation is often Q&A-style.
Stack Overflow was an invaluable resource for human developers in the 2010s, and it’s all about Q&A.
Reddit threads often take the form of Q&A, where the OP provides a prompt (the question) and the follow-up questions are basically answers.
The theme of Q&A keeps coming up.
Translation pipelines solved for us#
As mentioned in Content translation, I think static site generators (SSG) and content management systems (CMS) should solve machine translation for us. E.g. just provide an API key to a GenAI service and the SSG takes care of translating each doc, updating the translation when the doc changes, etc. This seems like it should be solved at the level of the SSG or CMS provider.