File read error when using multithreading (workers)

Detectoron2, pythorch 1.5
panda.read_csv() error
"
pandas.errors.ParserError: Error tokenizing data. C error: Calling read(nbytes) on source failed. Try engine='python'.
"

Need to use with to keep file is closed after reading.

Solution

with open(filename) as f:
  df = panda.read_csv(f)

Install cuda 10.2

1. Uninstall existing nvidia driver

nvidia-installer --uninstall

or

The correct way to uninstall just cuda and keep your nvidia drivers would be:

sudo apt purge "libcublas*" "cuda-*" cuda

Possible, because they were installed as requirements, you can also purge,

sudo apt purge "nsight-*" nvidia-modprobe

After that you can, if you want, also remove the nvidia drivers with:

sudo apt purge "*nvidia*"

Of course, if you installed cuda using nvidia's .run file then this won't work; in that case you probably have some uninstall script.

2. Close X server

# To stop:
sudo init 3
# To resume:
sudo init 5

3. Download & install

https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=runfilelocal

wget http://developer.download.nvidia.com/compute/cuda/10.2/Prod/local_installers/cuda_10.2.89_440.33.01_linux.runsudo sh cuda_10.2.89_440.33.01_linux.run

Screen permission denied

Problem

Cannot make directory '/var/run/screen': Permission denied

Solution

sudo /etc/init.d/screen-cleanup start

Change ssh port for ubuntu

1. Edit the file and set Port option

   Type the following command:
   $ sudo vi /etc/ssh/sshd_config
   Locate line that read as follows:
   Port 22
   OR
   #Port 22
   To set the port to 2222, enter:
   Port 2222

2. Updating your firewall to accept the ssh port 2222 in Linux

   If you are using UFW on a Ubuntu/Debian Linux, type:
   $ sudo ufw allow 2222/tcp
   The syntax for iptables is as follows
   $ sudo /sbin/iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 2222 -j ACCEPT
3. 
OR if you are using Ubuntu/Debian/Mint Linux:
$ sudo service ssh restart

Test

ssh xxxx.xxxx.xxxx.xxxx -p 2222

Windows install pip

1. Open cmd as administrator
2. Install python
3. curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

4. Then run the following command in the folder where you have downloaded get-pip.py:

python get-pip.py

5. Add pip installation path to environment PATH
6. check
pip help

Build tensorflow (1.8) from source

Environment
Target: Tensorflow 1.8
python3.6 (3.5)
cuda 10.0 (9.0)
cudnn 7.6 (7.5, 7.0)
bazel 0.15.0

1. install python3, pip

sudo apt install python-dev python-pip python3 python3-pip

2. install cudnn
Download cudnn
https://developer.nvidia.com/rdp/cudnn-download

sudo cp include/* /usr/local/cuda/include/

sudo cp lib64/* /usr/local/cuda/lib64/

sudo ln -s /usr/local/cuda/lib64/libcudnn.so.7.6.1 /usr/local/cuda/lib64/libcudnn.so.7


sudo ln -s /usr/local/cuda/lib64/libcudnn.so.7.6 /usr/local/cuda/lib64/libcudnn.so.7
sudo ln -s /usr/local/cuda/lib64/libcudnn.so.7 /usr/local/cuda/lib64/libcudnn.so

If cannot find libcudnn.*

 sudo ldconfig /usr/local/cuda/lib64

3. install bazel
https://github.com/bazelbuild/bazel/releases/tag/0.15.0

wget https://github.com/bazelbuild/bazel/releases/download/0.15.0/bazel-0.15.0-installer-linux-x86_64.sh 

chmod +x bazel-0.15.0-installer-linux-x86_64.sh

./bazel-0.15.0-installer-linux-x86_64.sh --user

4. install numpy

pip3 install numpy

5. ./configure
set python path (whereis python3, which python3)
set cuda version & path (10.0, /usr/local/cuda)
set cudnn version & path (7.6, /usr/local/cuda)

6. build

bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_package --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0"  

7. install

bazel-bin/tensorflow/tools/pip_package/build_pip_package ~/tmp/tensorflow_pkg

virtualenv -p python3 tensorflow

(tensorflow)$ source ~/tensorflow/bin/activate

(tensorflow)$ pip install ~/tmp/tensorflow_pkg/tensorflow-1.8-cp36-cp36mu-linux_x86_64.whl

Git push failed with the error LFS upload failed

Git push error

Git LFS: (0 of 3 files) 0 B / 525.39 KB

LFS upload failed:

  (missing) xxx 

error: failed to push some refs to 'https://xxx'

Solution

 git lfs fetch --all