Inside the Mind of a Machine:The Secret Journey of Your Word

Introduction

Do you ever think about what happens inside the application when you type something. Then you get your answer? Let us consider a question like "Why does the moon follow me?". This question seems simple to us. The application has to do a lot of work to give us the right answer. The application has to understand what our question means. Then it has to use computer programs to find the answer.

The application starts by looking at our question "Why does the moon follow me?". It tries to figure out what we are really asking.

This blog is going to explain how the machine takes our question "Why does the moon follow me?". Makes it really complicated. The machine uses a lot of technologies to do this including computer programs. Your Language is Converted Into Answers by Artificial Intelligence. This is what the machine does with Your Language.

1. From Human Languages to Machine-Readable Form

Human beings operate in human languages that is language made up of words, syntax, semantics and so forth. On the hand neural networks perform calculations in numbers. Thus when text is provided by a person there is a need to convert it into numbers. This process enables the machine to understand languages and thus is efficient. Languages of people function through words, sentences, meanings and other similar units. Neural networks function in numbers. In order to transform the text into something by the neural network there is a need for transformation. This kind of transformation is called tokenization.

2. Tokenization: Token has been broken into texts 

Tokenization refers to the process of breaking texts down into components known as tokens. For instance the following sentence "Why does the moon follow me?" can be broken down into tokens such as "Why" "does" "the" "moon" "follow" "me" and "?". A token doesn’t have to be a word as it could either be an entire word or a portion of a word.The format of tokens their length and meaning can differ widely because everything depends on the tokenizer. Each token has its ID that is a numerical representation of a token.

3. Token ID and Embeddings: Converting Tokens into Numbers

Token IDs offer identifiers. The model requires richer numeric representation to enable effective processing of language. An embedding is the representation learnt by the model for a token. When learning the model learns the representations which make it possible to learn from language patterns and relationships. The representations are then processed via the layers of the model. Token ID offers an identifier. An embedding is a representation. Embeddings have numeric dimensions. Representation relationships enable the processing of language patterns.

4. Positional Information: Word Order Importance

It is not enough to know what tokens are present in the sentence. In addition their order influences the meaning of the sentence. As an illustration both "The dog chased the boy". The boy chased the dog" use the same words but mean something different. That is why positional mechanisms are employed in transformer designs to convey information about the order of tokens. Sinusoidal positional encoding was introduced by the transformer and nowadays many language models employ techniques like RoPE.

Word order may influence the meaning of the sentence. Positional mechanisms provide information about the order of tokens. Sinusoidal positional encoding was introduced in the transformer design. Many LLMs employ RoPE

5. Self-Attention: Linking Tokens with Context

It’s also necessary to account for how tokens interact with each other. For instance in the sentence "The animal didn’t cross the street since it was tired" the meaning of "it”'s defined by other information in the sentence. Self-attention allows the token representation to include information coming from parts of the sentence. This is done through Query, Key and Value (Q/K/V) projections to compute attention scores and blend information from tokens. Multi-head attention refers to the performance of attention calculations at the same time.

• Query: The information to be retrieved from that position. 

• Key: The information needed to decide on the relevance of another position. 

• Value: The information that should be blended. 

• Multi-head attention: Several attention calculations made at the time.

6. Transformers and Next-Token Prediction:

Generation of the Responses  Self-attention is one of the components of the Transformer architecture. A transformer layer usually comprises attention along with feed- neural networks, normalization and residual connections which are stacked several times in a language model. After going through the input a causal language model generates logits that serve as scores for next tokens. It is possible to convert these scores into probabilities and a certain decoding method helps choose the token. The chosen token is then added to the sequence of tokens. This procedure is repeated until the response is formed.

The decoding methods are:

• Greedy decoding. 

• Selecting the token with the score. 

• Temperature. 

• Adjusting the probability distribution when sampling. 

• Top-k sampling. 

• Sampling from the k tokens with the highest scores. 

• Top-p sampling. 

• Sampling from the tokens within the probability mass comprising the most probable ones.

This process is known as generation or next-token prediction

7. Context, RAG, and Tools:

A language model can utilize the information in the context but actual AI applications usually require more than just what the model is capable of doing internally. The context given to a language model can consist of the system prompt, conversation history, user input, retrieved documents or outputs of tools. With RAG an AI application can use information from the outside world and give it to the model as context. Additionally tools can be used by an AI application to interact with the world.

The following can be combined in AI applications:

• Context window with information during the current interaction. 

• RAG with retrieval of information from the outside world. 

• Vector databases to store and search for representations. 

• Tools and APIs to interact with systems. 

• Application-controlled memory with information across interactions.

8.  The Whole Process of Your Words

For the user it appears to be easy to work with AI: ask a question and get an answer.However internally there is a series of operations performed: tokenization, numerical representations, positional data, self-attention, Transformer layers, probability computations, token generation. In addition other systems like RAG or tools may offer data or functionality if required by the application.

The whole journey may be described as:

This knowledge will serve as a base for delving into contemporary AI technologies, including Large Language Models, RAG systems, vector databases AI agents and AI applications. A simple question-and-response operation turns out to be the end product of a much more complex computational journey which starts from the words typed by humans. 

Conclusion

It appears that making a question and receiving an answer is quite a thing – however there is a whole lot that happens inside the computer. To begin with your question is split into tokens each of them is turned into a value that allows the computer to make conclusions and at last the context and self-attention mechanism enable every word to interact with one another in the sentence. This continues until those numeric probabilities result in tokens thus forming the sentence. With the help of technologies as RAG the computer makes use of its worldly knowledge in order to give you an answer based on its reasoning.

The time you ask a question and get an immediate answer keep in mind that thousands of computations, attentions and probabilities were performed in just a few seconds. - "From words to numbers to meaning – The invisible journey of artificial intelligence."