Cuda installed, but nvcc not found

 Problem: nvcc not found, already installed cuda and driver

# local conda env fix

conda install -c nvidia cuda-toolkit=12.8

conda install -c nvidia cuda=12.8 (optional if cuda already installed)

which nvcc

nvcc --version

pip install flash-attn --no-build-isolation

export CUDA_HOME=$CONDA_PREFIX

export PATH=$CUDA_HOME/bin:$PATH

export LD_LIBRARY_PATH=$CUDA_HOME/lib64:$LD_LIBRARY_PATH

# system-wise fix

sudo apt-get update

sudo apt-get install -y nvidia-cuda-toolkit=12.8

which nvcc

nvcc --version

echo 'export PATH=/usr/local/cuda/bin:$PATH' >> ~/.bashrc

echo 'export CUDA_HOME=/usr/local/cuda' >> ~/.bashrc

source ~/.bashrc

# after nvcc is installed, then we can install flash attention

pip install flash-attn --no-build-isolation

Change docker root directory

 From https://www.ibm.com/docs/en/z-logdata-analytics/5.1.0?topic=software-relocating-docker-root-directory

Relocating the Docker root directory

If the space in the file system where the Docker root directory is located is not adequate and cannot be increased, you must relocate the directory.

Procedure

To relocate the Docker root directory, complete the following steps as root or a user with “sudo all” authority:

1. Stop the Docker services:

   sudo systemctl stop docker
   sudo systemctl stop docker.socket
   sudo systemctl stop containerd
2. Create the necessary directory structure into which to move Docker root by running the following command. This directory structure must reside on a file system with at least 50 GB free disk space. Significantly more disk space might be required depending on your daily ingestion volumes and data retention policy.

   sudo mkdir -p /new_dir_structure
3. Move Docker root to the new directory structure:

   sudo mv /var/lib/docker /new_dir_structure
4. Edit the file /etc/docker/daemon.json. If the file does not exist, create the file by running the following command:

   sudo vim /etc/docker/daemon.json

   Add the following information to this file:

   {
     "data-root": "/new_dir_structure/docker"
   }
5. After the /etc/docker/daemon.json file is saved and closed, restart the Docker services:

   sudo systemctl start docker

   After you run the command, all Docker services through dependency management will restart.
6. Validate the new Docker root location:

   docker info -f '{{ .DockerRootDir}}'

Conda & Pip cmd

 

Conda environment export requirements.txt.

conda activate <env>
conda install pip
pip freeze > requirements.txt

For those getting odd path references in requirements.txt, use:

pip list --format=freeze > requirements.txt