modify tensorflow checkpoint path

The saved checkpoint stored the model absolute path. If you changed some path, you need to update the stored paths in checkpoint.

Two functions in
https://github.com/tensorflow/tensorflow/blob/r1.6/tensorflow/python/training/saver.py

1. get original state
get_checkpoint_state(checkpoint_dir, latest_filename=None)

2. Update the paths, similar in the got paths

def update_checkpoint_state(save_dir,model_checkpoint_path,
all_model_checkpoint_paths=None,
latest_filename=None)

# all_model_checkpoint_paths is a list

tensorflow c++ compiling problems

Problem 1

fatal error: third_party/eigen3/unsupported/Eigen/CXX11/Tensor: No such file or directory

if you pip install using virtual environment, the include files are not copied.

Solution

1. Dirty one. Install tensorflow globally.
2. The including files will be copied to
 /usr/local/lib/python2.7/dist-packages/tensorflow/
Thus, we only need to copy them to ~/tensorflow/include python2.7.
The source files are stored in
tensorflow-1.5.0-rc0/third_party/eigen3/unsupported/Eigen/CXX11

Problem 2

fatal error: nsync_cv.h: No such file or directory

fatal error: nsync_mu.h: No such file or directory

Solution

1. modify source c++ file

find /usr/local/lib/python2.7/dist-packages/tensorflow/ -name "nsync_mu.h"

/usr/local/lib/python2.7/dist-packages/tensorflow/include/external/nsync/public/nsync_mu.h

modify line 25 and 26 to include right path in /usr/local/lib/python2.7/dist-packages/tensorflow/include/tensorflow/core/platform/default/mutex.h

2. 

or modify makefile to add include path

add nsync_cv.h include：-I$TF_INC/external/nsync/public

maybe help you
I found it from https://www.tensorflow.org/extend/adding_an_op#build_the_op_library

below is official demo

TF_INC=$(python -c 'import tensorflow as tf; print(tf.sysconfig.get_include())')
TF_LIB=$(python -c 'import tensorflow as tf; print(tf.sysconfig.get_lib())')
g++ -std=c++11 -shared zero_out.cc -o zero_out.so -fPIC -I$TF_INC -I$TF_INC/external/nsync/public -L$TF_LIB -ltensorflow_framework -O2

Problem 3

Using tensorboard
  File "/home/jinglu/tensorflow/bin/tensorboard", line 7, in <module>
    from tensorboard.main import run_main

Solution

pip install tb-nightly

Problem 4

Undefined symbol: _ZTIN10tensorflow8OpKernelE

Solution

add the flags in g++
-L $TF_LIB -ltensorflow_framework 

Add/Delete sudo user

Original https://www.digitalocean.com/community/tutorials/how-to-create-a-sudo-user-on-ubuntu-quickstart

Steps to Create a New Sudo User

1. Log in to your server as the root user.

   
   ssh root@server_ip_address
   
   
   

2. Use the adduser command to add a new user to your system.
   Be sure to replace username with the user that you want to create.

   
   adduser username
   
   
   

   • Set and confirm the new user's password at the prompt. A strong password is highly recommended!

     
     Set password prompts:
     Enter new UNIX password:
     Retype new UNIX password:
     passwd: password updated successfully
     

   • Follow the prompts to set the new user's information. It is fine to accept the defaults to leave all of this information blank.

     
     User information prompts:
     Changing the user information for username
     Enter the new value, or press ENTER for the default
         Full Name []:
         Room Number []:
         Work Phone []:
         Home Phone []:
         Other []:
     Is the information correct? [Y/n]
     

3. Use the usermod command to add the user to the sudo group.

   
   usermod -aG sudo username
   
   
   

   By default, on Ubuntu, members of the sudo group have sudo privileges.
4. Test sudo access on new user account
• Use the su command to switch to the new user account.

  
  su - username
  
  
  

• As the new user, verify that you can use sudo by prepending "sudo" to the command that you want to run with superuser privileges.

  
  sudo command_to_run
  
  
  

• For example, you can list the contents of the /root directory, which is normally only accessible to the root user.

  
  sudo ls -la /root
  
  
  

• The first time you use sudo in a session, you will be prompted for the password of the user account. Enter the password to proceed.

  
  Output:
  [sudo] password for username:
  

  If your user is in the proper group and you entered the password correctly, the command that you issued with sudo should run with root privileges.

How To Delete a User

In the event that you no longer need a user, it is best to delete the old account.

You can delete the user itself, without deleting any of his or her files by typing this as root:

deluser newuser

If you are signed in as another non-root user with sudo privileges, you could instead type:

sudo deluser newuser

If, instead, you want to delete the user's home directory when the user is deleted, you can issue the following command as root:

deluser --remove-home newuser

If you're running this as a non-root user with sudo privileges, you would instead type:

sudo deluser --remove-home newuser

If you had previously configured sudo privileges for the user you deleted, you may want to remove the relevant line again by typing:

visudo

Or use this if you are a non-root user with sudo privileges:

sudo visudo

root    ALL=(ALL:ALL) ALL
newuser ALL=(ALL:ALL) ALL   # DELETE THIS LINE

This will prevent a new user created with the same name from being accidentally given sudo privileges.

ssh login timeout but scp works

Problem:
ssh login timeout
scp and sftp work

Analysis:

scp and sftp working means port 22 works.

This was due to my router blocking TCP keepalive messages when I connected wirelessly (go figure).

Solution:

ssh my_server -o TCPKeepAlive=no 

From the documentation:

TCPKeepAlive
  Specifies whether the system should send TCP keepalive messages
  to the other side. If they are sent, death of the connection or
  crash of one of the machines will be properly noticed.  However,
  this means that connections will die if the route is down tem-
  porarily, and some people find it annoying.  On the other hand,
  if TCP keepalives are not sent, sessions may hang indefinitely on
  the server, leaving "ghost" users and consuming server resources.

  The default is "yes" (to send TCP keepalive messages), and the
  server will notice if the network goes down or the client host
  crashes.  This avoids infinitely hanging sessions.

  To disable TCP keepalive messages, the value should be set to
  "no".

Mount a hard disk in ubuntu + reboot (may have different filesystem type)

1. $ sudo fdisk -l
display device
Disk /dev/sdc: 2 TiB, 2197949513728 bytes, 4292870144 sectors

2. $ mkdir /media/DiskC
Better use /media instead of /mnt, because /mnt is usually the automatic mount point, may be replaced when restarting.

3 $ mount /dev/sdc /media/DiskC
Then error occured
mount: wrong fs type, bad option, bad superblock on /dev/sdc,
       missing codepage or helper program, or other error

       In some cases useful info is found in syslog - try
       dmesg | tail or so.

(for several filesystems (e.g. nfs, cifs) you might need a /sbin/mount. helper program)

This is relevant given that you are trying to mount NFS. The /sbin/mount.nfs helper program is provided by nfs-common. You can install it with:

sudo apt install nfs-common

On the other hand, if you are trying to mount CIFS, the helper program is provided by cifs-utils. You can install it with:

sudo apt install cifs-utils

4. convert file type

$ sudo mkfs.ext4 /dev/sdc

goto step 3.

Done!

Auto mount on start up.

[IMPORTANT] sudo cp /etc/fstab /etc/fstab.old - Create a backup of the fstab file just in case something unwanted happens.

Auto-mount at boot

We want the drive to auto-mount at boot.  This usually means editing /etc/fstab.

Firstly, it's always best to use the drives UUID.  To find the drive's UUID do

ls -al /dev/disk/by-uuid/

Copy the resultant UUID (for your disk) and then open fstab for editing (note I'm using vim here but use whatever editor you prefer):

sudo vim /etc/fstab

You want to add an entry for the UUID and mount point.  Below is an example of an fstab file with an entry added for the mount above:

# /etc/fstab: static file system information. # # Use 'blkid' to print the universally unique identifier for a # device; this may be used with UUID= as a more robust way to name devices # that works even if disks are added and removed. See fstab(5). # # <file system> <mount point>   <type>  <options>       <dump>  <pass> # / was on /dev/sdb1 during installation UUID=63a46dce-b895-4c1f-9034-b1104694a956 /               ext4    errors=remount-ro 0       1 # swap was on /dev/sdb5 during installation UUID=b9b9ee49-c69c-475b-894b-1279d44034ae none            swap    sw              0       0 # data drive UUID=19fa40a3-fd17-412f-9063-a29ca0e75f93 /media/data   ext4    defaults        0       0

Note: the entry added is the last line.

Test fstab

We always want to test the fstab before rebooting (an incorrect fstab can render a disk unbootable).  To test do:

sudo findmnt --verify

check the last line for errors.  Warnings can help in improving your fstab.

Use lsblk -o NAME,FSTYPE,UUID to find out the UUIDs and filesystems of the partition you want to mount. For example:

$ lsblk -o NAME,FSTYPE,UUID
NAME   FSTYPE UUID
sda
├─sda2
├─sda5 swap   498d24e5-7755-422f-be45-1b78d50b44e8
└─sda1 ext4   d4873b63-0956-42a7-9dcf-bd64e495a9ff

NTFS

UUID=<uuid> <pathtomount> ntfs uid=<userid>,gid=<groupid>,umask=0022,sync,auto,rw 0 0

Examples for the <> variables:

• <uuid>=3087106951D2FA7E
• <pathtomount>=/home/data/
• <userid>=1000
• <groupid>=1000

Use id -u <username> to get the userid and id -g <username> to get the groupid.

# Mount all disks

sudo mount -a

Parallel compression and decompression command

There is option for tar program:

option c for compression, x for extraction.

-I, --use-compress-program PROG
      filter through PROG (must accept -d)

You can use multithread version of archiver or compressor utility.

Most popular multithread archivers are pigz (instead of gzip) and pbzip2 (instead of bzip2). For instance:

$ tar -I pbzip2 -cf OUTPUT_FILE.tar.bz2 paths_to_archive
$ tar --use-compress-program=pigz -cf OUTPUT_FILE.tar.gz paths_to_archive

Archiver must accept -d. If your replacement utility hasn't this parameter and/or you need specify additional parameters, then use pipes (add parameters if necessary):

$ tar cf - paths_to_archive | pbzip2 > OUTPUT_FILE.tar.gz
$ tar cf - paths_to_archive | pigz > OUTPUT_FILE.tar.gz

Use lbzip2

$ time tar cf tmp.tar.bz2 tmp --use-compress-program=lbzip2

$ time tar xf tmp.tar.bz2 --use-compress-program=lbzip2

Python locale error: unsupported locale setting

When using pip install, the local error problem occur.

export LC_ALL="en_US.UTF-8"
export LC_CTYPE="en_US.UTF-8"
sudo dpkg-reconfigure locales

Then select en_US.UTF-8 to install

Occurred when install virtualenv.