Resources:
- https://deci.ai/blog/ggml-vs-gguf-comparing-formats-amp-top-5-methods-for-running-gguf-files/
- https://medium.com/@kyodo-tech/large-language-models-for-consumer-hardware-through-quantization-facb1fc4ab34
blk.0.ffn_up.weight
blk.0.ffn_gate.weight
blk.0.ffn_down.weight
blk.0.attn_norm.weight
blk.0.attn_v.weight
blk.0.attn_q.weight
blk.0.attn_output.weight
blk.0.attn_k.weight
token_embd.weightFrom GPT-4:
Tokenization
Before your input text interacts with the token_embd.weight layer, it undergoes tokenization. Tokenization is the process of splitting the text into a sequence of tokens. Depending on the model, a token can represent a whole word, a part of a word (subword), or even a single character.
Lookup
Each token is then matched with a unique identifier or index. The token_embd.weight layer functions as an embedding table where each row corresponds to the vector representation of a token, and the row's index corresponds to the token's unique identifier. The process of converting a token to its vector representation is essentially a lookup operation in this embedding table.
Vector Representation
The output of the token_embd.weight layer is a sequence of vectors, where each vector is the dense representation of an input token. These vectors capture semantic and syntactic information about the tokens to some extent, although initially, before training, these embeddings might not be very informative. Through the training process, the model adjusts the weights of these embeddings to capture meaningful linguistic properties.