LLAMA is done
My code: https://github.com/krsh-37/diy-ml-stuffs/blob/main/llama/model.py
X: https://x.com/gaussian_noise_/status/1955337465385861450
References
Llama published paper, page 3 especially: https://arxiv.org/pdf/2302.13971
Meta's Llama inference code: https://github.com/meta-llama/llama/blob/llama_v1/llama/model.py
Karpathy's GPT implementation: https://github.com/karpathy/ng-video-lecture/blob/master/gpt.py
RoPE, nice explanation and this channel is goated too: https://www.youtube.com/watch?v=a17DlNxkv2k
Another RoPE video: https://www.youtube.com/watch?v=Kv90HQY9lZA
Llama from scratch blog: https://blog.briankitano.com/llama-from-scratch/
Blocks explanation: https://www.youtube.com/watch?v=Mn_9W1nCFLo
LLAMA inference coding: https://www.youtube.com/watch?v=oM4VmoabDAI
Finally my code :)
https://github.com/krsh-37/diy-ml-stuffs/blob/main/llama/model.py