Large Language Models: A Deep Dive
Welcome to the second part of our exploration into large language models. In this episode, we delve deeper into the intricacies of these models, discussing everything from the formulation of effective prompts to the phenomenon of hallucinations with our host David Blank-Edelman and our armchair architects Uli Homann and Eric Charran.
Crafting Effective Prompts
One of the key aspects of working with large language models is the ability to craft effective prompts. These prompts need to be suitably constrained to elicit useful responses. For instance, an architect might want to ask for a solution architecture perspective response that meets specific functional and non-functional requirements.
Eric used an example where he prompted “from a software architect perspective come up and recommend a solution architecture that accomplishes all of these functional and non-functional requirements and then write it as if I’m creating a specification for a developer. The architecture requires investments from the organization in terms of CapEx and OpEx, new services, new cloud subscriptions.” In another prompt, he took the output and prompted “Then take this and write it as an e-mail to the CIO.”
It took the outputs, raised it up a level and created a good foundation, it wasn’t perfect, but a good foundation for executive messaging as to why the CIO would lobby the CFO to invest in these particular technologies.
Then Eric asked it to switch personas. “Assume that I’m an SRE or platform engineering team lead and I need to support this thing that I just created. Write me a quick spec for the SRE or platform engineering team lead who will support the architecture.”
This process involves a form of ‘code switching’, where the language and level of detail are adjusted based on the audience. It provided a great starting point for refinement.
Understanding Hallucinations
As we delve deeper into the workings of large language models, we encounter the phenomenon of ‘hallucinations’. These occur when the model makes assumptions based on the patterns it has seen so far. For example, if the model sees the sequence 1, 2, 3, it might assume that 4 should naturally follow.
While this extrapolation can work in many scenarios, it can also lead to inaccurate or even dangerous assumptions, especially in sensitive domains like healthcare. It’s crucial to remember that these models cannot make assumptions or extrapolations when dealing with diagnostic information.
Hallucinations were quite prevalent in large language models at the beginning of the year. However, thanks to the concerted efforts of the research community, their occurrence has decreased dramatically. Techniques like fine-tuning allow users to constrain and limit the number of hallucinations.
The Mystery of AI Outputs
In the world of artificial intelligence, large language models have emerged as a fascinating area of study. However, their workings often remain a mystery to the users, leading to a myriad of questions and concerns.
One of the intriguing aspects of these models is the generation of outputs. Users often find themselves puzzled by the responses they receive, unsure of the rationale behind them. This lack of understanding can be problematic, especially for professionals like architects who rely on these models for their work.
The key here is to understand how these models function. While reading the response, it’s crucial to fact-check the information to ensure its accuracy. There’s got to be a voice in the back of your head saying, “all right, let me just factually check this thing to make sure it just didn’t make this up because it wants to.” The models are designed to link concepts together and generate a response based on the input. However, they might sometimes fabricate links between concepts, leading to inaccurate outputs.
Understand how the process works and then as you’re reading the output just quality check it to make sure it makes sense before you proffer it as the as the answer.
The Role of the User
Large language models are tools designed to assist users in creating artifacts more efficiently and in greater depth. They provide proposals based on the input given by the user. It’s important to remember that these proposals need to be validated by the user before they can be accepted as the final output.
The user plays a vital role in this process. They need to understand what they’re asking the model to do and validate the output once it’s produced, then it becomes your proposal. The responsibility of the final output lies with the user, not the AI. The user cannot simply blame the AI if something goes wrong.
The World Beyond Natural Language
While much of the discussion around large language models revolves around natural language text, these models are capable of much more. They can understand and generate code, making them useful for tasks beyond generating human language text.
For instance, OpenAI has three model families: GPT for language, DALL-E for images, and Codex for code. These models can express anything that can be represented in code, including schemas. This capability opens up a whole new realm of possibilities for users, allowing them to leverage these models in a variety of ways.
Type Chat: Prompt Engineering with JSON
In the realm of artificial intelligence, large language models have emerged as powerful tools capable of generating a wide array of outputs. From creating JSON schemas to documenting legacy code, these models are revolutionizing the way we approach problem-solving.
One innovative application of large language models is ‘TypeChat’, an application developed by Anders Hejlsberg, and his colleagues. TypeChat leverages the power of prompt engineering to generate outputs in the form of a JSON schema.
Users can instruct the large language model to generate a specific output and format it using JSON. By providing a JSON schema as part of the prompt, the system can automatically respond in a JSON schema. This approach offers an elegant way to create programmable outputs, as parsing a JSON schema is much easier than parsing text.
Large language models can also generate code in various forms. One area where this capability has proven particularly useful is in the documentation of legacy code bases.
For instance, many Cobalt code bases are quite old and lack proper documentation. Generative AI can be used to document these code bases and explain what the code does. This is especially useful when the original purpose or functionality of the code is no longer known.
Knocking Their Socks Off: Impressive Prompts for Architects
When it comes to impressing architects with the capabilities of large language models, one effective approach is to use these models to analyze problematic code. For example, a piece of legacy code or code that’s leaking memory can be plugged into ChatGPT with the prompt “find the memory leak” or “find another way to write this optimally”.
However, it’s important to remember that the outputs generated by these models need to be quality checked to ensure their correctness. It’s also crucial to ensure that your organization is comfortable with the data being fed into the model, especially when using the consumer version of ChatGPT.
ChatGPT Enterprise and Bing Chat Enterprise.
For those concerned about data privacy, there are private versions of ChatGPT available, such as ChatGPT Enterprise and Bing Chat Enterprise. These versions ensure that the data fed into them stays within your organizational boundaries, offering an added layer of security.
Persona-Based Modeling and Prompt Engineering
Another effective strategy when working with large language models is persona-based modeling. This involves framing prompts as if the model is a specific persona, such as a software architect or a support person for a specific technology. This approach helps the model better understand the problem scenario and generate more relevant responses.
As we continue to explore the capabilities of large language models, it’s clear that these tools offer immense potential in a variety of fields. From prompt engineering to code generation, these models are paving the way for innovative solutions to complex problems. Stay tuned for more insights into the fascinating world of AI in our upcoming discussions.
Recommended Next Steps
If you’d like to learn more about the general principles prescribed by Microsoft, we recommend Microsoft Cloud Adoption Framework for platform and environment-level guidance and Azure Well-Architected Framework. You can also register for an upcoming workshop led by Azure partners on cloud migration and adoption topics and incorporate click-through labs to ensure effective, pragmatic training.
You can read Part 1 of this blog if you just read Part 2.
You can view the whole video below and check our more videos from the Azure Enablement Show,
