It’s no surprise in this day and age that organisations need to leverage AI to keep up with competition and ever increasing. Companies see real improvements from applying AI in their businesses. McKinsey estimates, that AI leaders show over 2 time better financial performance, 5-10% increase of revenue and 10-20% reduced cost [1].

This framework has 5 pillars that every organisation needs to adopt to realise value of artificial intelligence: strategy, data, technology, people and process. Let me go through each one of them, and provide an intuition with making a race car example.

Effective AI started to build a product based on this framework that allows companies to fast track their AI journey and become ultra competitive within months, instead of spending years building capabilities. Please reach out to me if you’re interested in participating in the trial of this product.

1. Digital strategy

The first and most important dimension is having a clear strategic vision, plan and roadmap for leveraging AI. Organisational leadership needs to determine how analytics can support key business objectives and priorities. This could include enhancing customer experience, optimising operations, gaining insights into new markets and more. Aligning analytics initiatives to strategic goals ensures efforts are focused and value is maximised.

In the race car example, strategy is the map and the route or routes that you planning to take.

2. Utilising high-quality data

High-quality, relevant data is the lifeblood for any AI initiative. Organisations must establish processes to capture both internal and external data from diverse sources. Ensuring data is cleansed, integrated and governed properly sets the foundation for powerful insights. Investments may also be required to fill data gaps through alternative sources. With a robust data strategy and architecture, organisations can fuel analytics programs for years. The recent advances in AI show that no matter how good your algorithm and hardware is, it’s the data that is the main driver of progress

In our race car example, data is fuel for the car.

3. Technology

Technology is a critical component in these 5 dimensions. In our road trip analogy, it’s a car itself. It includes decisions on infrastructure, architecture, hardware and software and platforms.

Building a robust analytics architecture is critical to enable AI competitive advantage. Depending on the organisation and its goals they can choose on-premise and/or cloud-based infrastructure. Organisations need also to design modern architectures and platforms to help them to reach their goals.

Finally, having the right software tools and libraries enable people to develop the use cases. Languages, with strong analytics support, like Python or R, allow data exploration while tools like PowerBI and Tableau enable visualization. Machine learning libraries like Scikit-learn, TensorFlow power predictive modelling. Natural language processing libraries help with text and language data. Proper model management systems govern deployed algorithms.

4. People and multidisciplinary teams

While technology is critical, people remain the most important asset for making AI a competitive advantage. Organisations need multidisciplinary teams including data scientists, data engineers, machine learning engineers, business analysts and experts to help build models, collaborate with business and ensure value delivery. Training existing staff and strategic hiring allows building these specialised analytical skills over time. Nurturing a culture of experimentation and innovation further unleashes the potential of people.

People - is a car driver. The car will not move or will not get to the end point if the driver doesn’t have relevant skills.

5. Process

Well defined processes in the company make the results of AI efforts optimal, effective, repeatable, scalable and allows for continuous improvement.

A version of Agile methodology allows analytical teams to work iteratively and deliver value every sprint via minimum viable products. Classic agile methodology can also be scaled across multiple levels of the company via frameworks like SAFe.

Additionally, there should be sufficient software engineering best practices, product development best practices, UI/UX and security processes in place to build a robust, scalable and extensible products.

Finally, organisations need a way to measure the results on regular basis to help pivot and understand if the products are going in the right directions.

While well-defined processes provide numerous benefits, organisations need to be cognizant of not going overboard and remain lean.

Conculsion

A holistic set of capabilities across these 5 dimensions allows organisations to maximise competitive advantage with artificial intelligence. With the right strategy, data, technology, skills and processes in place, AI becomes a strategic weapon to outperform competition.

Effective AI started to build a product based on this framework that allows companies to fast track their AI journey and become ultra competitive within months, instead of spending years building capabilities. Please reach out to me if you’re interested in participating in the trial of this product.

References:

[1] McKinsey Tech Trends Outlook 2022 link

While product management and project management roles may seem similar on the surface, they require very different skills and focus on distinct aspects of bringing new offerings to market.

Product management is responsible for the overall success of a product from concept to end of life. Project management focuses on successful planning and execution of projects according to constraints like scope, budget, timeline and resources.

To make it easy to compare, I put together a table that compares product and project management from 4 dimensions: purpose, scope, responsibilities and metrics.

In summary, while product management is about the ‘what’ and ‘why’ (what product to build and why), project management is about the ‘how’ and ‘when’ (how to deliver the project and when). Both disciplines are critical to the success of any organization, but they focus on different aspects of the business.

Across industries, McKinsey estimates highest impact from generative AI in sales, marketing, software engineering and customer operations. (see figure below)

Based on my experience, majority of use cases fall into one of 4 categories:

1. Synthesis of large text data
2. Content generation
3. Chatbots with domain knowledge
4. Code generation

Let’s explore them in detail.

1. Synthesis of large text data

Dealing with vast amounts of text data has always been a challenge for businesses. Generative AI presents a game-changing opportunity to quickly synthesize large documents, saving valuable time and effort for knowledge workers. By employing generative AI algorithms, businesses can efficiently extract specific attributes from text data, enabling them to gain meaningful insights from dialogues, correspondence, and vast textual resources.

Specific use cases in this category include:

• Quick synthesis of large documents: research papers, legal documents.
• Extraction of particular attributes from the text data: sentiment, pricing.
• Insights from dialogues and correspondence: actions for each team member from conversation

2. Content generation

Personalization has become a critical aspect of modern marketing and customer engagement strategies. Generative AI enables businesses to create personalized content tailored to each customer’s unique circumstances and preferences. Whether it’s generating personalized product recommendations, crafting individualized marketing campaigns, or tailoring content for specific customer segments, generative AI proves to be a powerful tool in enhancing the customer experience and driving brand loyalty.

Specific use cases in this category include:

• Personalised marketing content for each customer based on their own circumstances.
• Writing articles
• Creating legal or other documents

3. Chatbots with domain knowledge

Chatbots have become increasingly prevalent in various industries, offering real-time support and seamless interactions with customers. Generative AI takes chatbot capabilities to a whole new level by providing domain knowledge expertise. These intelligent chatbots can serve as advisors in specialized fields such as legal, medical, engineering, and more, offering personalized and expert guidance to users. Moreover, businesses can deploy chatbots as sales agents, enhancing customer engagement and driving sales conversion rates. Additionally, chatbots can provide valuable support to customers, resolving queries efficiently and enhancing customer satisfaction.

Specific use cases in this category include:

• Chatbot as an internal or external advisor for legal, medical, engineering, etc
• Chatbot as sales agent
• Chatbot as support agent

4. Code generation

Software development is a complex and time-consuming process. However, generative AI can act as an invaluable co-pilot for developers, dramatically speeding up the code generation process. This technology can facilitate no-code app generation, empowering individuals with limited coding knowledge to create applications effortlessly. Additionally, generative AI can aid in template generation, automating the creation of standardized code structures, further streamlining development workflows.

Specific use cases in this category include:

• Co-pilot for developers
• No-code app generation
• Template generation

Final thoughts

As businesses continue to explore and adopt this technology, it is crucial to address ethical considerations and ensure responsible AI usage. Transparent and ethical deployment of generative AI will foster trust among customers and stakeholders, promoting the long-term sustainability and success of organizations.

In conclusion, embracing generative AI in the right manner will undoubtedly unlock new possibilities and propel businesses into a future of innovation and prosperity. As this technology continues to evolve, enterprises must remain at the forefront of its adoption to remain competitive and reap the full benefits of generative AI.

The tradeoff between these archetypes is the cost and the effort required vs how domain specific the model is. If mostly general knowledge is required first archetype can be sufficient, but for more specific domains archetype number 3 is more relevant.

Here are the three archetypes:

1. Use out of the box LLM API with the power of prompt engineering to pass context of your problem. In this case, you take a question and ask your model to give an answer. You may provide some context and a few examples as a context, and this is proven to improve the model quite a bit. This is a cost effective, easy to implement and requires careful prompt engineering. The model is limited mostly to general knowledge and the knowledge included in the limited context.

2. Fine tune pre-trained LLM with your proprietary data. In this archetype, you take existing pre-trained model and train it further on proprietary data. Transfer learning can be another option in this case. This will still require GPU compute, probably on a cluster but not as expensive as training the model from scratch. The model will have a lot more domain specific knowledge than in the first archetype.

3. Train your own LLM from scratch. This is the most expensive and hardest to implement archetype. The cost of training such a model is in hundreds of millions of dollars and training can last for months. Sam Altman estimated that the cost to train GPT-4 was about $100 million. An example of such model was trained by Bloomberg - BloombergGPT, which was trained on Bloomberg financial data.

In this article, I will cover model evaluation of LLM models that are already producing good results by themselves (like GPT-3.5 or GPT-4) on a generic standard metrics (like superglue) and we want to evaluate performance for a specific task we are building the model for. I.e, we are not evaluating general performance of LLM but performance of GPT-4 for a task like chatbot for a bank.

Three main points to decide on for model evaluation is to choose the main metric, create a test set and decide how to compare the outputs of the model to the ground truth.

1. Choose the main metric. There might be multiple metrics to get an intuition of what is happening in the system but it’s always easier to compare a single metric rather than multiple. Often simple metric like accuracy works, where we count example as true if the meaning matches one or multiple expected results. Point 3 covers how to compare output to the expected result by meaning.

2. Create a test set, either manually, by generating it with LLM or to use both. My preference is to use the last option because manual examples allow to build an intuition about the queries and answers that we want. The rest can be quicky generated by the LLM.

   LangChain has a very nice chain QAGenerateChain to help to automatically generate the questions and answers from examples.

3. Decide how to compare results of the model with the ground truth. It’s easy to compare numbers but with the fuzzy nature of text inputs and outputs it can become a trickier question. The answer, again, is to use LLM itself to do the comparison.

Additional metrics to keep track of:

• Bias and fairness
• Drifts. Data and model drifts
• Data quality

If there is enough interest, I can explore these additional concepts and relevant metrics in future articles.

Overall approach

Since LLM models are stateless, the most common method of keeping track of memory of previous conversations, is to include it at every new call to LLM as a context. In the future, there may be new models that can keep track of memory naturally but with ChatGPT and GPT-4 it this is the most common way.

Frameworks like LangChain, however, allow developers to abstract out of this limitation and make it seamless. The framework manages the memory and include it every time “automatically”.

In this article I’ll explore examples of the basic LLM memory abstractions:

• ChatMessageHistory
• ConversationBufferMemory
• ConversationBufferWindowMemory
• ConversationSummaryBufferMemory
• ConversationEntityMemory

LangChain abstractions

Let’s review several classes that I used in the last few weeks.

The core class that is behind all the abstractions is a ChatMessageHistory that is just a collection of past user and AI messages.

from langchain.memory import ChatMessageHistory

history = ChatMessageHistory()
history.add_user_message("hello, chatgpt")
history.add_ai_message("hello")

print(history.messages)

Output is:

[HumanMessage(content='hello, chatgpt', additional_kwargs={}, example=False),
 AIMessage(content='hello', additional_kwargs={}, example=False)]

ConversationBufferMemory

ConversationBufferMemory is one abstraction above ChatMessageHistory and can be used in OpenAI models directly. It automatically records all the conversations with the model.

from langchain.llms import OpenAI
from langchain.chains import ConversationChain
from langchain.memory import ConversationBufferMemory


memory = ConversationBufferMemory()
memory.chat_memory.add_user_message("My name is Maksud") 

conversation_with_memory = ConversationChain(
    llm=OpenAI(temperature=0), 
    verbose=True, 
    memory=ConversationBufferMemory()
)

conversation_without_memory =  = ConversationChain(
    llm=OpenAI(temperature=0), 
    verbose=True
)

print(conversation_without_memory.predict(input="What is my name?"))

print(conversation_with_memory.predict(input="What is my name?"))

The output, first with memoryless model and second is with memory. Nice touch GPT-4 reminding us that you don’t store personal information :)

> I apologize, but as an AI language model,
I don't have access to personal information...


> Your name is Maksud.

ConversationBufferWindowMemory

This class is the same as ConversationBufferMemory but it has a window of last N messages to pass to the LLM.

Let’s take a look at example:

memory = ConversationBufferWindowMemory( k=2)
memory.save_context({"input": "a=1"}, {"output": "The value of the variable a is 1."})
memory.save_context({"input": "b=5"}, {"output": "The value of the variable b is 5. "})
memory.save_context({"input": "c=11"}, {"output": "The value of the variable c is 11. "})

print(conversation_without_memory.predict(input="what is value of c?"))
print(conversation_without_memory.predict(input="what is value of a?"))

In the first case LLM knows the value of c because it was recent but in the second case it “forgot” the value of a since it’s outside of buffer window of 2.

> The value of the variable "c" is 11.
Is there anything specific you would like to know or do with this information?


> Based on the information provided so far,
there is no mention of the variable "a" or its value.

ConversationSummaryBufferMemory

Storing all the message history with LLM can accumulate over time and can be expensive tokenwise. To overcome this, a nice trick of summarising the previous chats retains most of information and compresses the content passed to LLM every time.

LangChain’s ConversationSummaryBufferMemory helps to summarise the previous chats and uses LLM itself to create these summaries.

ConversationEntityMemory

More advanced version of summarisation and LLM memory is ConversationEntityMemory. This class helps the model to keep track of multiple entities in the conversation.

memory = ConversationEntityMemory(llm=llm, return_messages=True)
_input = {"input": "Alice and Bob talk over the phone about their gym class"}
memory.load_memory_variables(_input)
memory.save_context(
    _input,
    {"output": " Great to know about Alice and Bob. Sounds like exicting class"}
)

print(memory.load_memory_variables({"input": 'what is Bob doing?'}))

LangChain uses LLM to extract entities from this conversation, in this case Alice and Bob and stores it.

{'history':
[HumanMessage(content='Alice and Bob talk over the phone about their gym class', additional_kwargs={}),
  AIMessage(content=' Great to know about Alice and Bob. Sounds like exicting class', additional_kwargs={})],
 'entities': {
   'Alice': 'Alice is talking to Bob about their gym class.',
   'Bob': 'Bob is talking to Alice about their gym class.',
   }}

Other LangChain memory classes

In this article I covered some of the basic LangChain clasess to emulate memory in LLMs. Some other classes worth considering are:

• VectorStore-Backed Memory
• DynamoDB, Cassandra, Zep, etc and other databases backed memory
• Motörhead Memory

If you are interested, I am happy to connect with you to talk about the right memory class for your LLM project.

Special thanks to my friend Emil Pastor who invited me to the summit and shed the light on the modern graph technology.

Here’s a recap of some of the notable sessions:

• Dr. Jim Webber, Chief Scientist at Neo4j, captivated the audience with his opening keynote on “The Art of the Possible with Graph Technology.” Dr. Webber shed light on the boundless potential of graphs and their applications across various domains.

• Karthick Thanigaimani, Engineering Chapter Lead at Payments Reliability Engineering and Support Services, shared insights into ANZ Payment Ops using Neo4j. His presentation demonstrated how Neo4j’s capabilities could revolutionize the payment operations landscape.

• Matthew Campbell, Manager of Bioinformatics at InterVenn Biosciences, presented “Building knowledge graphs for clinical glycomics and molecular pathway analysis.” He showcased the power of knowledge graphs in advancing clinical research and analysis in glycomics and molecular pathways.

• Michela Ledwidge, Founder and CEO of Mod, addressed the challenge of overcoming adoption barriers in spatial visualization. Her talk highlighted innovative solutions to seamlessly integrate spatial data within graph databases.

• Emil Pastor, Solution Architect Manager at Neo4j ANZ, discussed “Connected Data in the Age of AI/ML: The Path to Success using Graph Database and Data Science.” Attendees gained valuable insights into leveraging the synergies between graph databases and AI/ML techniques to unlock new opportunities.

• Joshua Yu, Director of Pre-Sales and Field Engineer APAC at Neo4j, explored “LLMs using Neo4j.” His session provided valuable insights into how organizations can harness the power of Neo4j to manage large-scale learning and knowledge management systems.

• Ryan Jeffery and Peter Dart delivered an enlightening presentation on “Using Neo4j for Modeling Complex Telco Network.” Their talk delved into the intricacies of modelling telco networks using Neo4j, showcasing the advantages of graph databases in this domain.

In addition to the presentations, attendees had the opportunity to participate in the GraphWorkshop titled “Building a Knowledge Centric Fraud Detection Graph”, conducted by Andrew Conacher and Muddassir Zaidi from Neo4j. The workshop offered hands-on experience in constructing a fraud detection graph and provided practical insights into leveraging knowledge-centric approaches.

Overall, Graph Summit Melbourne 2023 proved invaluable, fostering knowledge sharing and paving the way for future advancements in the field. The GitHub link for the GraphWorkshop materials is available for those who want to dive deeper into the concepts covered during the event.

The goal of this article is not to provide comprehensive documentation of parameters but to point to the ones I found most important. Then readers can refer to the official documentation to get the details.

1. Temperature

Temperature is the parameter that I tune the most. It corresponds to randomness of output or how diverse the output can be. It ranges from 0 to 2. When it’s 0, the output will be deterministic, as the model always chooses the most likely token. At temperatures closer to 2, the output will be diverse and more “creative”.

2. Top_p - nucleus sampling

This is an alternative to temperature parameter. Instead of considering temperature, this parameter considers only outputs that are more likely with cumulative probability adding up to top_p. For example, if top_p = 0.2, it considers tokens that fall into 20% of the probability mass.

3. N - number of completions

This is number of completions, if you want to generate more than one response. Be careful, as it eats into your token budget.

4. Frequency and presence penalties

Frequency penalty penalises multiple use of the same word. Presence penalty does not consider number of times the word used but wheather it is being used or not, so it’s boolean in nature.

They are used directly in calculating logits. Here’s the formula:

mu[j] -> mu[j] - c[j] * alpha_frequency - float(c[j] > 0) * alpha_presence

There are more parameters in the official documentation but these ones made the biggest difference in my experiments. If you think there is a missing one, please drop me a note, I’ll be happy to include it here :).

Here’s what I learned during the last 2 months:

1. Events drive the engagement. When I look at my subscribers graph, the two events that I ran, generated big bumps of new subscribers. Another source of subscribers, although smaller but more consistent, is publishing to my LinkedIn network, where I have 1,500 followers.

2. Organic discovery is slow. The goal for me was not to grow it as quick as possible, but rather to learn the skills required to run the newsletter and a promote it and just to have fun sharing my knowledge. I also realised how important the discovery is and how slow is organic growth, without social media.

3. Even at small scale, publishing online can drive a lot of networking opportunities, sourcing great ideas and finding leads. I’ve never had so many people, who I know and strangers, reaching out with ideas, leads and interesting connections.

4. Social media consumes a lot of time, so prioritising it is vital when you don’t have a team to rely on.

5. Ask for feedback proactively. As with any product or service it is almost impossible to get it right the first time. On the other hand, over-planning is also counterproductive. So, from the beginning I decided to prioritise action over planning, just give it a crack and just fix based on user feedback.

6. I learned heaps by just writing a newsletter every week

   • Writing with the help of GPT-4 is much easier but a bit of an art. You do need to provide your original ideas first.

   • Optimising my time to write weekly edition of newsletter based on my learning routine is important

   • Refreshed my skills on running and optimising a website in the cloud, as well as automating my workflow through scripts and storing content in git

   • SEO optimisation is an art

   • Audio editing skills and working with professional microphones is an entire new world for me

   • Voice training. By hearing your own voice gives a lot of room for improvement. I also discovered that voice training is a thing not only for actors but also for business professionals.

   • Understanding the social media ecosystem that I have never tapped into in the past

   • Overall structuring of my thoughts and knowledge

Next milestone is 500 subscribers!

Here are my 5 best practices for prompt engineering:

1. Write instructions clearly, as you would do for an intern
2. Add structure to your prompt
3. Develop prompts iteratively
4. Reduce the risk of prompt injection by using prompt delimiters
5. Never trust the model

In the rest of the article, I will expand on each of these practices.

Write instructions clearly, as you would do for an intern

The number one skill for prompt engineering, is to give clear, unambiguous instructions to what you want to produce. I find it a bit like coaching interns, where you want to lead them to get the right answer, without letting them going into wrong direction. Clear instructions also mean they may not be short.

If I know what is required to solve the problem, I often also provide the steps that the model should follow, and it really helps to nail the answer faster.

This also includes checking for failure conditions. For example, if I want to extract all the dates from the text and the text doesn’t have any dates, I ask the model to output “no dates found”, to avoid hallucinations.

Add structure to your prompt

The anatomy of an LLM prompt that I usually follow is this:

1. context
2. request
3. text to analyse
4. output specification.

Each one of them, except for probably 3, can be improved with the structure.

For the context, if the problem is nontrivial, I tend to favour few-shots learning, where I provide a couple of examples, how to solve it. The context can also include some assumptions, that model need to know before tackling the problem. When writing a chatbot, context can also include roles, such as system, user, assistant.

In request, if the problem is nontrivial, I tend to specify steps to solve it, failure modes, and even steps to check itself. This can also include specifications such as tone, translation language and references.

I often need to use outputs of the model in other parts of the problem and parsing arbitrary output can be painful. When I write my prompts, I specify exact format that I want the output to be. For example, JSON with the specific structure.

Develop prompts iteratively

Prompts never work from the first time, just like developing any machine learning models. The key here is to build them iteratively starting from simple prompt and growing details. All the best practices that I described in the previous section can and need to be added iteratively, otherwise you are risking of overengineering your prompts.

The process that I use is:

Reduce the risk of prompt injection by using prompt delimiters

When sending LLM model a prompt one part will be some sort of context and instructions and another one will be the actual text to analyse. After reading a couple of articles on prompt injection, I started wrapping text to analyse into prompt delimiters, such as triple backquote. This also flags the model that text between these delimiters is the main text for it to analyse and it’s not context or instructions.

Never trust the model

As good as the LLM model is, it still hallucinates quite a lot. When you give clear instructions and structure, it usually reduces the hallucinations but it still happens. Another issue that I found with these models is forgetting - context that I’ve given a few prompts ago is lost. So, for any final output, I carefully check every sentence that the model produce.

Obviously, this is not a complete list of best practices, just the ones I developed for myself. If you have any best practices that you find useful, I’d love to hear about them, and I can share it in the future articles.