{"id":4575,"date":"2024-08-14T09:08:51","date_gmt":"2024-08-14T09:08:51","guid":{"rendered":"https:\/\/truelogic.org\/wordpress\/?p=4575"},"modified":"2024-08-14T09:08:51","modified_gmt":"2024-08-14T09:08:51","slug":"installing-llama-cpp-on-ubuntu-with-an-nvidia-gpu","status":"publish","type":"post","link":"https:\/\/truelogic.org\/wordpress\/2024\/08\/14\/installing-llama-cpp-on-ubuntu-with-an-nvidia-gpu\/","title":{"rendered":"Installing Llama.cpp on Ubuntu with an NVIDIA GPU"},"content":{"rendered":"\n

1.Run sudo apt update<\/em> to make sure all packages are updated to the latest versions<\/p>\n\n\n\n

2.Run sudo apt install build-essential<\/em> to install the toolchain for building applications using C++<\/p>\n\n\n\n

3.Create a directory to setup llama.cpp:
mkdir \/var\/projects<\/em>
cd \/var\/projects<\/em><\/p>\n\n\n\n


4.Get the llama.cpp code from Github:
git clone https:\/\/github.com\/ggerganov\/llama.cpp<\/em>
cd llama.cpp<\/em><\/p>\n\n\n\n


5.Verify that nvidia drivers are present in the system by typing the command:
sudo ubuntu-drivers list<\/em> OR sudo ubuntu-drivers list –gpgpu
<\/em> You should get a listing similar to the output below:<\/p>\n\n\n

\nnvidia-driver-390, (kernel modules provided by linux-modules-nvidia-390-generic)\nnvidia-driver-535, (kernel modules provided by linux-modules-nvidia-535-generic)\nnvidia-driver-470, (kernel modules provided by linux-modules-nvidia-470-generic)\nnvidia-driver-418-server, (kernel modules provided by nvidia-dkms-418-server)\nnvidia-driver-450-server, (kernel modules provided by linux-modules-nvidia-450-server-generic)\nnvidia-driver-535-server, (kernel modules provided by linux-modules-nvidia-535-server-generic)\nnvidia-driver-470-server, (kernel modules provided by linux-modules-nvidia-470-server-generic)\nnvidia-driver-545, (kernel modules provided by nvidia-dkms-545)\n<\/pre><\/div>\n\n\n
6.Update the nvidia drivers in the current Ubuntu installation:
    sudo ubuntu-drivers install<\/em><\/p>\n\n\n\n
7.Reboot the system for the drivers to take effect
    sudo shutdown -r now<\/em><\/p>\n\n\n\n
8.After rebooting, verify that nvidia is active by the command:
     nvidia-smi<\/em>
     You should get a similar output as below:
<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
9.Install gpustat utility:
     sudo apt install gpustat<\/em><\/p>\n\n\n\n
10.Run gpustat. You should get a similar output as below:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n

11.Install the NVCC compiler with the command:
    sudo apt install nvidia-cuda-toolkit<\/em><\/p>\n\n\n\n
12.Before we can build llama.cpp we need to know the Compute Capability of the GPU:
     nvidia-smi –query-gpu=compute_cap –format=csv<\/em>
     This will give a single score eg 3.0, 5.2 etc.<\/p>\n\n\n\n
13.Set the Compute Capability score in the shell by typing:
      export CUDA_DOCKER_ARCH=compute_XX<\/em> where XX will be the score (without the decimal point)      eg. export CUDA_DOCKER_ARCH=compute_35<\/em> if the score is 3.5<\/p>\n\n\n\n
14.Next step is to build llama.cpp:
       cd \/var\/projects\/llama.cpp
       make GGML_CUDA=1<\/em><\/p>\n\n\n\n
15.This completes the building of llama.cpp. Next we will run a quick test to see if its working
16.We download a small gguf into the models folder in llama.cpp:
       cd models<\/em>
       wget https:\/\/huggingface.co\/afrideva\/Tiny-Vicuna-1B-GGUF\/resolve\/main\/tiny-vicuna-1b.q5_k_m.gguf<\/em><\/p>\n\n\n\n
17.We run a test query from the llama.cpp root folder
      .\/llama-cli -m models\/tiny-vicuna-1b.q5_k_m.gguf -p “I believe the meaning of life is” -n 128 –n-gpu-layers 6<\/em><\/p>\n\n\n\n
       You should get an output similar to the output below:<\/p>\n\n\n\n
\"\"<\/figure>\n","protected":false},"excerpt":{"rendered":"
1.Run sudo apt update to make sure all packages are updated to the latest versions 2.Run sudo apt install build-essential to install the toolchain for […]<\/a><\/p>\n<\/div>","protected":false},"author":1,"featured_media":4584,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[368],"tags":[],"class_list":["post-4575","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-gpu-and-ai"],"_links":{"self":[{"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/posts\/4575","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/comments?post=4575"}],"version-history":[{"count":5,"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/posts\/4575\/revisions"}],"predecessor-version":[{"id":4583,"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/posts\/4575\/revisions\/4583"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/media\/4584"}],"wp:attachment":[{"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/media?parent=4575"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/categories?post=4575"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/truelogic.org\/wordpress\/wp-json\/wp\/v2\/tags?post=4575"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}