As the doctor gone rogue

February 11, 2017

Bitvise X11 forwarding problem

Filed under: bash, SysAdmin, ubuntu — Tags: , , — hypotheses @ 12:00 am

You may have a problem trying to forward X11 from your Linux server through a terminal in Windows. I use Bitvise as the ssh client as it also provides the interface for sftp to download/upload the files. Not that using command line scp or sftp is not efficient, but in Windows you may have to install those tools separately.

Since I was trying to avoid installing Cygwin initially, Cygwin terminal wasn’t what I initially consider. Therefore, I went ahead and install Bitvise ssh client (instead of putty), which actually turns out to be quite good. The only problem (at least now) is when trying to forward GUI from the server with X11 forwarding, it failed miserably.

  • I’ve tried setting up bitvise in the X11 forward.
  • Installed X11-server from Cygwin (finally T_T).
  • But X11 forwarding still did not work. I got the error message below.

Failed to open channel for X11 forwarding from [::1]:47396 to 127.0.0.1:6000. Error connecting to X11 server: FlowSocketConnector: Failed to connect to target address. Windows error 10061: No connection could be made because the target machine actively refused it.

I kind of have an idea that this is something to do with Windows Firewall, which does not allow a specific port to listen to an incoming connection. But I haven’t quite figured out what to do. It is actually described on the bitvise page, but I guess I haven’t quite read the whole thing.

In short, after install cygwin X11 server, make sure to run the command below to fix the firewall permission. 

C:\cygwin64\bin\XWin -listen tcp -multiwindow

Ref: https://www.bitvise.com/ssh-x11-forwarding

February 6, 2017

Backing up FreeNAS to Amazon S3

Filed under: bash — hypotheses @ 7:59 am

Going through the AWS tutorial about creating File GateWay on my own is not exactly a walk in the park. Although I managed to install awscli and configured the basic requirement, there’s a missing gap that I haven’t quite figured out. This saves me lots of time trying to figure out how I can back up data already live on ZFS FreeNas to AmazonS3.


aws s3 sync /mnt/data/ s3://mybucket

What’s left to figure out next would be how to get all the files out when I need it, and how much it would cost, how to save those cost, etc. For the later part, I guess having AWS free credits does help a lot.

Take a look at the full guide following the link below.

ZFS is excellent at storing all your data safely but “there are still lots of potential ways for your data to die, and you still need to back up your pool. Period. PERIOD!” The process …

Source: Backing up FreeNAS to Amazon S3

January 22, 2017

Running analysis in docker container

Filed under: bash, docker — Tags: , , , — hypotheses @ 6:41 pm

There are generally 4 steps to create an analytics environment on your server that will be separate from the rest of the system. By running your analysis within a container, it might reduce the risk of crashing the server because you might have used up all the resouce and cause the server to freeze up.

  1. Install virtualbox (to create a docker machine)

  2. Install docker-machine

  3. Create a docker-machine this will be the machine to run your container

  4. Map your command to run in the container. Following the kaggle/python tutorial.

Install virtualbox-qt

sudo apt-get install virtualbox-qt

Install docker-machine

curl -L https://github.com/docker/machine/releases/download/v0.8.0/docker-machine-`uname -s-uname -m` > /usr/local/bin/docker-machine && \ chmod +x /usr/local/bin/docker-machine

create a docker-machine

This will create a separate docker machine called docker2

docker-machine create -d virtualbox --virtualbox-disk-size "100000" --virtualbox-cpu-count "8" --virtualbox-memory "32092" docker2
docker-machine start docker2

We then need to specify a new destination where docker container will run, i.e. on docker2

eval $(docker-machine env docker2)

See <https://docs.docker.com/machine/install-machine/&gt; for more info

run kaggle/python

You’re now at a point where you can run stuff in the container. Here’s an extra step that will make it super easy: put these lines in your .bashrc file (or the Windows equivalent)

kpython(){
docker run -v $PWD:/tmp/working -w=/tmp/working --rm -it kaggle/python python "$@"
}
ikpython() {
docker run -v $PWD:/tmp/working -w=/tmp/working --rm -it kaggle/python ipython
}
kjupyter() {
(sleep 3 && open "http://$(docker-machine ip docker2):8888")&
docker run -v $PWD:/tmp/working -w=/tmp/working -p 8888:8888 --rm -it kaggle/python jupyter notebook --no-browser --ip="*" --notebook-dir=/tmp/working
}

Reference

<http://blog.kaggle.com/2016/02/05/how-to-get-started-with-data-science-in-containers/&gt;

December 2, 2016

Installing vcftools on MacOS with homebrew

Filed under: bash — hypotheses @ 8:49 am

Life just get a lost easier if you want to install vcftools on MacOS.

Once homebrew is installed (see https://bhoom.wordpress.com/tag/brew/), you can simply install vcftools in one line.

brew install homebrew/science/vcftools

October 21, 2016

Installing Oracle Java8 in 3 lines

Filed under: bash — hypotheses @ 10:37 am

Assumptions

  1. apt-get is installed
  2. You have sudo authority

#add repository from webup8team & install oracle java and set as default

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-set-default

If you have other java installed previously, you might want to complete remove them first.

sudo apt-get purge openjdk-\*

There's a long description on how to install Oracle Java on WikiHow if you want to install them from scratch.

October 18, 2016

Add/Remove user from a group

Filed under: bash, SysAdmin, ubuntu — hypotheses @ 3:15 am

  • Add a user bhoom to an existing group superman: sudo usermod -A -g superman bhoom
  • remove bhoom from a group superhero: sudo gpasswd -d superhero bhoom
  • Create a new group legends:sudo groupadd legends

October 15, 2016

Find files and listing details on Linux

Filed under: bash, miscellaneous, ubuntu — Tags: , , , , , — hypotheses @ 12:01 pm

One common task that we always have to do relatively often is to find the location of some specific files. Although you may have some clues that those files are located in your current working directory, “.” symbolic link used in the command below, you may have no clue to which sub-folder your files are.

(more…)

September 8, 2016

value too great for base ?!

Filed under: bash — hypotheses @ 11:11 pm

So, I wrote a script trying to get it to output the previous month as a number with a leading “0”. The problem arrives as bash complains of some errors.

Initially, my script looked like this

MO=$(date '+%m')
MO2=$((MO-1))
P_MONTH=$(printf "%0*d\n" 2 $MO2);

The last line is there so that I still get a numeric month with a leading zero. However, this for some reasons, gave the error “value too great for base”.

After trying to figure out what it actually means, the base here is the numeric base, which bash thinks number with leading “0” is octal.
So, the solution to fix this is to reassure that the ${MO} variable, which is the numeric Month is a base-10 number.

You can achieve this as follow

MO=$(date '+%m');MO=$((10#$MO))
MO2=$((MO-1))
P_MONTH=$(printf "%0*d\n" 2 $MO2);

April 27, 2016

Encrypt a few files or folder on OsX

Filed under: bash, miscellaneous, OsX, SysAdmin — hypotheses @ 4:00 pm

May be you are working on a sensitive data, or just want to secure your personal data on your personal computers. Mac offers the whole hard disk encryption when you install the system, as well as, encryption of your backup through “Time Machine”. However, there are times that you might want to add an extra security to your file. May be you save some of the files to Dropbox or Google Drive, and the file might contain your personal health information that you would not want just anyone to be able to simply read it.

There’s a simple tool in OsX that can help you do this.

zip -e [output.zip] [file-to-be-encrypt]

zip utility will ask you to create a password. The stronger the password. The harder it will be to crack it. However, make sure you will remember the password, too. For this reason, I recommend you to use a password manager, e.g. LastPass is one of an excellent and easy to use tool with several browser integration for both mobile device and personal computer.

Just for the peace of mind. The encryption algorithm in zip might not be very strong. But at least, you save some disk space, and there’s one more extra-security of a password protection that you have to go through before accessing the content of the file.

October 30, 2015

apt-get error with … for package ‘linux-headers-3.2.0-60’ is missing final newline

Filed under: bash, ubuntu — Tags: , , — hypotheses @ 2:17 am

linux-headers-3.2.0-60-virtual.list

caused you the problem, do the same! Just delete them (I actually just moved them to “linux-headers-3.2.0-60.list.bak” file).

As the doctor gone rogue

I recently clone a virtual machine image and setting up locally. When trying to install a new package through <code>apt-get</code> in Ubuntu. I ran into a strange problem of “missing final newline” in linux-headers.

Trying to google for solution, I found a page in 2004 mentioning this problem. It suggested that a file in /var/lib/dpkg/info/smbf.list is the cause of the error. So, I tried to locate this file but without any success.

I notice that there is “linux-headers-3.2.0-60.list” and a few other files with similar names in this folder. So, well, why not trying to delete them.

It seems like deleting these files fix the problem and allow apt-get to install the package without any problems. So far the system seems normal. We haven’t allowed any additional access through the server, only deleting a few files that don’t look essential for the OS to run.

Further remove other apt-get unused…

View original post 9 more words

Older Posts »

Create a free website or blog at WordPress.com.