Mount a hard disk in ubuntu + reboot (may have different filesystem type)

1. $ sudo fdisk -l
display device
Disk /dev/sdc: 2 TiB, 2197949513728 bytes, 4292870144 sectors

2. $ mkdir /media/DiskC
Better use /media instead of /mnt, because /mnt is usually the automatic mount point, may be replaced when restarting.

3 $ mount /dev/sdc /media/DiskC
Then error occured
mount: wrong fs type, bad option, bad superblock on /dev/sdc,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

(for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount. helper program)

This is relevant given that you are trying to mount NFS. The /sbin/mount.nfs helper program is provided by nfs-common. You can install it with:

sudo apt install nfs-common

On the other hand, if you are trying to mount CIFS, the helper program is provided by cifs-utils. You can install it with:

sudo apt install cifs-utils

4. convert file type

$ sudo mkfs.ext4 /dev/sdc

goto step 3.

Done!

Auto mount on start up.

[IMPORTANT] sudo cp /etc/fstab /etc/fstab.old - Create a backup of the fstab file just in case something unwanted happens.

Auto-mount at boot

We want the drive to auto-mount at boot.  This usually means editing /etc/fstab.

Firstly, it's always best to use the drives UUID.  To find the drive's UUID do

ls -al /dev/disk/by-uuid/

Copy the resultant UUID (for your disk) and then open fstab for editing (note I'm using vim here but use whatever editor you prefer):

sudo vim /etc/fstab

You want to add an entry for the UUID and mount point.  Below is an example of an fstab file with an entry added for the mount above:

# /etc/fstab: static file system information. # # Use 'blkid' to print the universally unique identifier for a # device; this may be used with UUID= as a more robust way to name devices # that works even if disks are added and removed. See fstab(5). # # <file system> <mount point>   <type>  <options>       <dump>  <pass> # / was on /dev/sdb1 during installation UUID=63a46dce-b895-4c1f-9034-b1104694a956 /               ext4    errors=remount-ro 0       1 # swap was on /dev/sdb5 during installation UUID=b9b9ee49-c69c-475b-894b-1279d44034ae none            swap    sw              0       0 # data drive UUID=19fa40a3-fd17-412f-9063-a29ca0e75f93 /media/data   ext4    defaults        0       0

Note: the entry added is the last line.

Test fstab

We always want to test the fstab before rebooting (an incorrect fstab can render a disk unbootable).  To test do:

sudo findmnt --verify

check the last line for errors.  Warnings can help in improving your fstab.

Use lsblk -o NAME,FSTYPE,UUID to find out the UUIDs and filesystems of the partition you want to mount. For example:

$ lsblk -o NAME,FSTYPE,UUID
NAME   FSTYPE UUID
sda
├─sda2
├─sda5 swap   498d24e5-7755-422f-be45-1b78d50b44e8
└─sda1 ext4   d4873b63-0956-42a7-9dcf-bd64e495a9ff

NTFS

UUID=<uuid> <pathtomount> ntfs uid=<userid>,gid=<groupid>,umask=0022,sync,auto,rw 0 0

Examples for the <> variables:

• <uuid>=3087106951D2FA7E
• <pathtomount>=/home/data/
• <userid>=1000
• <groupid>=1000

Use id -u <username> to get the userid and id -g <username> to get the groupid.

# Mount all disks

sudo mount -a

Parallel compression and decompression command

There is option for tar program:

option c for compression, x for extraction.

-I, --use-compress-program PROG
      filter through PROG (must accept -d)

You can use multithread version of archiver or compressor utility.

Most popular multithread archivers are pigz (instead of gzip) and pbzip2 (instead of bzip2). For instance:

$ tar -I pbzip2 -cf OUTPUT_FILE.tar.bz2 paths_to_archive
$ tar --use-compress-program=pigz -cf OUTPUT_FILE.tar.gz paths_to_archive

Archiver must accept -d. If your replacement utility hasn't this parameter and/or you need specify additional parameters, then use pipes (add parameters if necessary):

$ tar cf - paths_to_archive | pbzip2 > OUTPUT_FILE.tar.gz
$ tar cf - paths_to_archive | pigz > OUTPUT_FILE.tar.gz

Use lbzip2

$ time tar cf tmp.tar.bz2 tmp --use-compress-program=lbzip2

$ time tar xf tmp.tar.bz2 --use-compress-program=lbzip2

Python locale error: unsupported locale setting

When using pip install, the local error problem occur.

export LC_ALL="en_US.UTF-8"
export LC_CTYPE="en_US.UTF-8"
sudo dpkg-reconfigure locales

Then select en_US.UTF-8 to install

Occurred when install virtualenv.

Common command for screen

Reference https://www.rackaid.com/blog/linux-screen-tutorial-and-how-to/

Linux Screen allows you to:

• Use multiple shell windows from a single SSH session.
• Keep a shell active even through network disruptions.
• Disconnect and re-connect to a shell sessions from multiple locations.
• Run a long running process without maintaining an active shell session.

1. Start screen
$ screen

2 Create a screen
$ Ctr+a, c

3. Switch next screen
$ Ctr+a, n

4. List screen
$ screen -ls

5. Deattach screen
$ Ctr+a, d
or terminate
$ Ctr+a, k

6. Reattach screen
$ screen -r <screenid>

ssh forward port to solve denied server port

Sometimes, some port is denied from servers. When you want to access webpage set in server, you can forward remote port to local.

$ ssh -L16006:localhost:6006 user1@remote_server

Set port 6006 at remote server, and you can access at client via port 16006.

Nvidia driver version mismatch (which cause tensorflow gpu not work)

Problem:
When using tensorflow-gpu, get the following error:

Solved in the environment Ubuntu 16.04.

tensorflow/stream_executor/cuda/cuda_driver.cc:406] failed call to cuInit: CUDA_ERROR_NO_DEVICE

1. May be the nvidia driver version problem. Check the installed driver.

$ nvidia-smi

Failed to initialize NVML: Driver/library version mismatch

Show installed nvidia driver
$ dpkg --get-selections | grep nvidia
nvidia-375 install
nvidia-384 install
nvidia-opencl-icd-375 deinstall
nvidia-opencl-icd-384 install
nvidia-prime install
nvidia-settings install

$ dpkg -l | grep -i nvidia

ii  bbswitch-dkms                              0.8-3ubuntu1                                  amd64        Interface for toggling the power on NVIDIA Optimus video cards
ii  libcuda1-375                               375.82-0ubuntu0~gpu16.04.1                    amd64        NVIDIA CUDA runtime library
ii  nvidia-375                                 375.82-0ubuntu0~gpu16.04.1                    amd64        NVIDIA binary driver - version 375.82
ii  nvidia-opencl-icd-375                      375.82-0ubuntu0~gpu16.04.1                    amd64        NVIDIA OpenCL ICD
ii  nvidia-prime                               0.8.2                                         amd64        Tools to enable NVIDIA's Prime
ii  nvidia-settings                            384.90-0ubuntu0~gpu16.04.1                    amd64        Tool for configuring the NVIDIA graphics driver

2. Uninstall current driver and reinstall nvidia-375

$nvidia-uninstall

If there is no nvidia-uninstall, you should remove all nvidia driver

1. Run sudo apt-get purge nvidia-*
2. Run sudo add-apt-repository ppa:graphics-drivers/ppa and then sudo apt-get update.
3. Run sudo apt-get install nvidia-375.
4. Reboot and your graphics issue should be fixed.

You can check your installation status with the following command

lsmod | grep nvidia

Done! Then the "nvidia-smi" should work.

3. May turn off the ubuntu automatical updates.

Other helpful commands.
to list the devices

ubuntu-drivers devices




4. Important. Prevent driver auto-update
Sometimes, only this step is enough!!!

$ sudo apt-mark hold nvidia-375
$ dpkg --get-selections | grep nvidia
nvidia-375 hold
nvidia-384 install
nvidia-opencl-icd-384 install
nvidia-prime install

nvidia-settings install

5. Check tensorflow

import tensorflow as tf
# Creates a graph.
a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a')
b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b')
c = tf.matmul(a, b)
# Creates a session with log_device_placement set to True.
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
# Runs the op.
print(sess.run(c))

I tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 0 with properties: 
name: GeForce GTX TITAN X

Batch file process in linux bash

Batch process *.ext files

Template

$ find /path/ -name *.ext -exec sh -c 'command "$1"' _ {} \;

Example

print file names without extension
$ find /path/ -name *.ext -exec sh -c 'echo "${1%.*}"' _ {} \;
print base name (dirname), directory (dirname)
$ find /path/ -name *.ext -exec sh -c 'echo $(basename "$1")' _ {} \;

Batch convert *.ext to *.ext_bak

Template

$ find /path/ -name *.ext -exec sh -c 'convert "$1" "${1%.*}".ext_bak' _ {} \;

Example

convert egg to bam file
$ find /path/ -name *.egg -exec sh -c 'egg2bam "$1" -o "${1%.*}".bam' _ {} \;