Introduction to linux at Introductory Bioinformatics Workshop

Editor's Notes

• #3: Introduction – What is GNU/Linx?, GNU/Linux distributionsTerminal & Virtual Consoles – What is a console; Commands, stdin, stdout, stderr; Typing shortcuts for BASHPopular commands – Directories, Files, File Compressions; Manual and Help; Networking and Monitoring Resources
• #4: The Unix philosophy emphasizes building short, simple, clear, modular, and extendable code that can be easily maintained and repurposed by developers other than its creators. This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface (Doug McIlroy). The most common version of Unix (bearing certification) is Apple's OS X, while Linux is the most popular non-certified workalike. Linus Torvalds is a Finnish software engineer best known as the chief architect of the Linux Kernel. More on the shell later.Linux is also considered a variant of the GNU operating system, initiated in 1983 by Richard Stallman. Therefore, the Free Software Foundation prefers the name GNU/Linux when referring to the operating system as a whole (see GNU/Linux naming controversy).Most operating systems can be grouped into two different families. Aside from Microsoft’s Windows NT-based operating systems, nearly everything else traces its heritage back to Unix.Linux, Mac OS X, Android, iOS, Chrome OS, Orbis OS used on the PlayStation 4, whatever firmware is running on your router — all of these operating systems are often called “Unix-like” operating systems.
• #5: In computing, the kernel is a computer program that manages input/output requests from software and translates them into data processing instructions for the central processing unit and other electronic components of a computer. The kernel is a fundamental part of a modern computer's operating system.[1]
• #6: The operating system will consist of the Linux kernel and, usually, a set of libraries and utilities from the GNU Project, with graphics support from the X Window System. In software, a package management system, also called package manager, is a collection of software tools to automate the process of installing, upgrading, configuring, and removing software packages for a computer's operating system in a consistent manner.Although all Linux distros have the Linux kernel in common, the graphical user interface, system, file structure, and desktop and server applications vary significantly.Unlike most operating systems resembling Unix, Torvalds did not use any of the original Unix source code and chose to release his code under the GNU (Gnu's Not Unix) general public license. To this day, the GNU license allows the free distribution of Linux and its derivatives as long as copies are released under the same license and include the source code.
• #7: Supporting packages includes libraries and tools to automate the process of installing, upgrading, configuring, and removing software packages for a computer's operating system in a consistent manner. The kernel of UNIX is the hub of the operating system: it allocates time and memory to programs and handles the filestore and communications in response to system calls, interacts with hardware etc. Linux distributions include the Linux kernel, supporting utilities and libraries and usually a large amount of application software to fulfil the distribution's intended use.Proprietary software or closed source software is computer software licensed under exclusive legal right of the copyright holder with the intent that the licensee is given the right to use the software only under certain conditions, and restricted from other uses, such as modification, sharing, studying, redistribution, or reverse engineering
• #8: The large number of distributions available, especially those that are still in active development is testament to the diversity of appearance and purpose that can be obtained when software is free and open-source. The Linux kernel has benefited from the contributions of thousands of programmers over the years. The philosophy is to have the choice of several exchangeable components to customize your experience. Linux distros differ in desktop environment and file managers etc.
• #9: All the so-called “Linux” distributions are really distributions of GNU/Linux. GNU is usually the first layer of user interaction. Some distributions, notably Debian, use GNU/Linux when referring to the operating system as a whole.[30] The naming issue remains controversial.As of May 2011, about 8% of a modern Linux distribution is made of GNU components, as determined by counting lines of source code making up Ubuntu's "Natty" release; meanwhile, about 9% is taken by the Linux kernel.[31]GNU = GNU is Not Unix. Gnu – a large dark antelope with a long head
• #10: In Ubuntu Linux, the default web browser is Firefox. In Debian the default web browser is Iceweasel (a rebranding of Mozilla Firefox). Although all Linux distros have the Linux kernel in common, the graphical user interface, system, file structure, and desktop and server applications vary significantly.Distributions (often called distros for short) are Operating Systems including a large collection of software applications such as word processors,spreadsheets, media players, and database applications.
• #11: Ubuntu is a Nguni Bantu term (literally, "human-ness") roughly translating to "human kindness"; in Southern Africa (South Africa and Zimbabwe).  Linux Ubuntu is a Debian-based Linux operating system, with Unity as its default desktop environment. The goal of linux is to be as invisible as possible, doing theheavy lifting on the background. This GNU/Linux operating system is a solid core for a lot of computers and devices.
• #13: Quite often people new to another operating system than Microsoft Windows are confronted with the terms CLI (Command Line Interface) and GUI (Graphical User Interface). Pretty soon they get a notion about what those two are but at this stage they are still far away from being able to tell what is the "better" one. Well, there is no better -- it depends on the tasks that need be done, how experienced a user is and his personal likings.
• #14: This interaction between a computer operating system or application software and user is facilitated by the Shell. The shell includes both command-line and graphical elements for interacting with OS and apps. Graphical user interfaces (GUIs) are helpful for many tasks, but they are not good for all tasks.
• #15: standard streams are preconnected input and output channels between a computer program and its environment (typically a text terminal) when it begins execution. The three I/O connections are called standard input (stdin), standard output (stdout) and standard error (stderr). Stdin, stdout, stderr:These are standard streams for input, output, and error output. By default, standard input is read from the keyboard, while standard output and standard error are printed to the screen. BASH is the default shell for most Linux distros and Mac OS X. The Bourne-Again shell is a clone of the Bourne shell developed by the free software foundation. BASH is the Bourne shell, born again.
• #16: There may be several Options, or none at all.
• #24: The easiest way to check from the Unix command line whether the internet connection works, is to send a request to a known server (e.g. www.google.com) using the ping <web address> command. The command reports, how long a message takes back and forth to the given server. It sends an ECHO_REQUEST datagram to elicit an ICMP ECHO_RESPONSE from a host or gateway.
• #25: Pipes - curl is more in the traditional unix-style, it sends more stuff to stdout, and reads more from stdin in a "everything is a pipe" manner.Curl vs wget: http://daniel.haxx.se/docs/curl-vs-wget.htmlRecursive!Wget's major strong side compared to curl is its ability to download recursively, or even just download everything that is referred to from a remote resource, be it a HTML page or a FTP directory listing.
• #26: Ssh encrypts all data that travels across its connection including your username and password (which you’ll use to login to the remote machine). When the node’s server arrives user’s will be able to login remotely to run jobs on the server. This is common for bioinformatics facilities, and enables users to use the existing (greater) resources such as storage, processing capacity of the server. Other commands such as sftp and scp allow users to securely copy a file to/ from remote hosts.
• #27: Some commands have additional arguments
• #29: To extract id lines from a fasta file, remove the id and # redirect the output to a id.txt file
• #31: Linux has inherited from UNIX the concept of ownerships and permissions for files. This is basically because it was conceived as a networked system where different people would be using a variety of programs, files, etc. Linux is also multi-tasking, meaning one user can use the same computer to do multiple jobs. UNIX everything is a file.
• #33: The Unix operating system (and likewise, Linux) differs from other computing environments in that it is not only a multitasking system but it is also a multi-user system as well. Each file has assigned 9 different permissions, 3 for the file user-owner (u), 3 for the group-owner (g) and 3 for everyone else (o). Ls -alshows something like this for each file/dir: drwxrwxrwx
• #34: The chmod (change mode) command protects files and directories from unauthorized users on the same system, by setting access permissions.
• #35: Does Linux need antivirus software? All computer systems can suffer from malware and viruses, including Linux. Thankfully, very few viruses exist for Linux, so users typically do not install antivirus software. It is still recommended that Linux users have antivirus software installed. Some users may argue that antivirus software uses up too much resources. Thankfully, low-footprint software exists for Linux.
• #36: Path contains directories separated by colons, and tells the shell where to look for programs. $PATH is a colon-separated list of directories in which the shell looks for commands.
• #37: Standard users, by default, cannot install applications on a Linux machine. In order to successfully install an application on a Linux machine you have to have super user privileges. So, to change a command so that you can successfully run an installation you have to prefix it with “sudo”, for example: sudodpkg -isoftware.deb. To add a user to list of sudoers: # adduser foo sudo.
• #41: PHYLIP : Phylogeny Inference Package, computer programs for inferring phylogenies
• #44: A compiler is a computer program (or set of programs) that transforms source code written in a programming language (the source language) into another computer language (the target language, often having a binary form known as object code).[1] The most common reason for wanting to transform source code is to create an executable program.Check from program author’s website for detailed instructions on how to build program from source.
• #45: The standard output of command is connected via a pipe to the standard input of command2. This connection is performed before any redirections specified by the command (see REDIRECTION below). If |& is used, the standard error of command is connected to command2's standard input through the pipe; it is shorthand for 2>&1|. This implicit redirection of the standard error is performed after any redirections specified by the command.
• #46: A pipeline is a sequence of one or more commands separated by one of the control operators: | or |&.
• #47: A shell program, called a script, is an easy-to-use tool for building applications by "gluing together" system calls, tools, utilities, and compiled binaries.scripts, programs written for a special run-time environment that can interpret (rather than compile) and automate the execution of tasks which could alternatively be executed one-by-one by a human operator.More than just the insulating layer between the operating system kernel and the user, it's also a fairly powerful programming language. Bash has become a de facto standard for shell scripting on most flavors of UNIX. Bourne shell compliant scripts are created with a .sh extension.
• #48: A variable is nothing more than a label, a name assigned to a location or set of locations in computer memory holding an item of data.PATH – Your shell search path: directories separated by colonsHOME – Your home directory, such as /home/Smith
• #49: BASH variable is the full path name used to invoke the current instance of BASH
• #50: You can use env or printenv <variable_name> to print shell variable.The scope of a variable (i.e., which programs know about it) is by default, the shell in which it is defined. To make a variable and its value available to other programs your shell invokes (i.e., subshells), use the export command. This variable then becomes an environment variable since it’s available to other programs in your shell’s “environment”. The configuration of your bash shell is found in the hidden file.bashrc Usually the SYNTAX for setting an environment variable in your .bashrc (which is found in your home directory) is the following:export VARIABLE=value1.2NOTE: There shouldn't be any space between the variable and the equals sign ("=") and the value. If your value has spaces then the whole value should be put in quotesFor the case of the environement variable PATH it is a good practice to prepend additional paths to it using colons ":" since it is a system defined environment variable. For example: export PATH=$HOME/bin/perl:$PATH1.3NOTE: That environment variables are accessed by prepending the dollar sign ("$"), but when defined the dollar sign is ommittedPATH is a variable that contains the directories from which your shell (BASH) looks for commands. These directories are separated by colons.
• #52: Pattern matching and data extraction of Linux command line tools like grep, sort, cut, etc. Enable handling of large text files which would otherwise consume large chunks of memory on other platforms such as Windows (or Linux) GUI for example.
• #53: The BLAST programs are widely used tools for searching DNA and protein databases for sequence similarity to identify homologs to a query sequence. The BLAST command line interface offers additional features such as querying a custom database eg. Chromosome of your organism of interest. More advanced usage generally involves taking the output of BLAST as a first step in some kind of script. For example, Torsten's "prokka" tool uses BLAST (amongst other things) to automatically annotate a sequence. Which can best be achieved from the command-line.
• #55: A number of cutting edge programs (Bowtie, Velvet, Trinity, Stampy, etc.) do not come with an web interface, because the developers neither have time nor computing resources to provide web services for everyone. As a rule of thumb, easier an website is to use, more difficult it is to develop. Furthermore, it costs a lot of money to maintain data-intensive web services.
• #56: Freedoms of the OS fosters/ encourages the development of more software to add to the already large existing ,command line, bioinformatics tools. Therefore, it will become common for primarily, wet lab, biomedical researchers to have some command-line knowledge and skill. While learning to use these tools may be challenging, at first, the rewards of UNIX/ Linux command-line proficiency is worth the effort. Therefore, in the tutorial to follow later today we are going to guide you through using some commands and we hope that you have fun doing it.