Linux Admin Interview questions

Hiring good linux people can be tough. Lots of buzzwords, and only high level understanding are thrown around. I've composed a list of questions that should hopefully be able to dig through the buzzword interviewee and test their real knowledge. I've included the questions and not the answers... if you're asking the questions, you should already know the answers :)

Below is the list of questions I print out and take with me to help me remember what to ask, and note the answers they give. Feel free to post questions you ask, and I can add them to the list.

Personality

• Tell me about the last major Linux project you finished. What were some of the obstacles and how did you overcome them. Sometimes asking these open questions will reveal much more then small questions that are easily answered by Google. A great senior admin doesn't need to know everything, but they should be able to come up with amazing solutions to impossible projects.
• Do you contribute to any Open Source projects? Doesn't matter if it's just documentation, it shows a certain dedication to the craft
• Tell me about the biggest mistake you've made in [some recent time period] and how you would do it differently today. What did you learn from this experience?

Admin - Unix/Linux

• How do you check an executable for shared library dependencies?
• You routinely compress old log files. You now need to examine a log from two months ago. In order to view its contents without first having to decompress it ?
• What are the different run levels, and what happens at each?
• What is required to be running for a client to mount a NFS share?
• What's your favorite scripting language and why?
• How do you check for an installed package?
• How would you make a change on 100 hosts? 1000?
• How would you see what group/groups someone is a member of?
• If a client cannot connect to port 80 on a server, what would you do to troubleshoot? How would you be able to tell if the server was allowing connections to that port?
• How would you check if a file system was mounted readonly?
• What's the difference between UDP and TCP?
• What advantages does epoch time have over local time (zones or GMT/UTC).
• Describe the difference between static linked and dynamically linked binaries.
• Describe the difference between TCP and UDP protocols.
• Describe a TCP handshake.
• What protocol does ping use?
• Is ping reliable?
• Your network is getting DDoS'd, what do you do?
• Describe the difference between raid 0 1 and 5.
• What raid level would you typically use for the OS/root disk of a system.
• What command would you use to capture network traffic for analysis?
• How do you find out who owns a domain?
• What kind of storage devices do you have experience with?
• What kind of load balancers are you familiar with?
• Can you describe a VIP and what it does?
• What program would you use to copy entire directories from one machine to anothe r?
• What would you use to only copy files that have changed?
• What does it mean if you see a "connection refused" error in your logs and how i s that different from a "connection timed out" error?
• What does a "connection reset by peer" error indicate?
• What would you do to diagnose an Apache server that has stopped responding. You have shell access to the system and can't connect to Apache, even from localhost , but the process is running..
• How do you split strings on a command line? What commands would you pipe the out put to?
• How do you match any character in regex?
• How do you match the beginning of a line?
• How do you match the end of a line?
• In HTTP protocol, what does HEAD, GET and PUT do?
• What port does Oracle run on?
• What is the name of Oracle hosts file? The file where you define oracle DB names , IP addresses, and port numbers..
• You are getting errors saying the filesystem is full, but df shows plenty of spa ce available, what do you check?
• How do you check inode usage on a filesystem?
• In a shell, how do you redirect the output of a command?
• How do you redirect error (STDERR) output on a command line?
• What's the difference between SAN and NAS?
• You have a system that has been hacked in your datacenter.. You have physical ac cess to the system, what do you do?
• Describe the difference between FTPS and SFTP.

DNS TRIVIA:

• Describe how a DNS query works.
• Describe the parts of a DNS zone file SOA.
• What do you change after modifying a zone file, to make sure your changes take e ffect.
• Where are the root DNS servers defined?
• What is the TTL in DNS and what does it control?
• A domain does not have an MX record, where does the mail go?
• What protocols does DNS use?
• Why would DNS respond with TCP instead of UDP?

Installing Pidgin in CentOS 7

There isn't much to this, but I just wanted to make an easy copy/paste for all the required packages.

Pidgin Version pidgin-2.10.11

Download pidgin from https://www.pidgin.im/download/source/

Install pre-req for the build:

(You may need to install the development tools: sudo yum groupinstall 'Development Tools')

 sudo yum install -y libpur* gtk2-devel libXScrnSaver-devel libSM-devel gtkspell-devel libxml2-devel gstreamer-devel farstream-devel libidn-devel meanwhile-devel avahi-devel avahi-glib-devel NetworkManager-devel gnutls-devel tcl-devel tk-devel  

Installation:

Unzip the source tar -jxf <pidgin.tar.bz2>

Cd <new dir from untar>

./configure

make

sudo make install

File based disk image with encryption

I was thinking this could be used for cloud storage where security is a concern. Once created you can save the disk image to a place like Copy, Dropbox, or Box

Creating the disk image

200M disk image
dd if=/dev/zero of=./disk_image.img bs=1MiB count=200

Set up encryption on the disk

sudo cryptsetup luksFormat disk_image.img sudo cryptsetup open disk_image.img crypt
sudo fdisk -l
Output:

Disk /dev/mapper/crypt: 198 MiB, 207618048 bytes, 405504 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Formatting the encrypted disk

sudo mkfs.xfs /dev/mapper/crypt
Output:

meta-data=/dev/mapper/crypt      isize=256    agcount=4, agsize=12672 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0
data     =                       bsize=4096   blocks=50688, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal log           bsize=4096   blocks=853, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Mount the disk

sudo mount /dev/mapper/crypt /mnt/crypt

Closing everything out

sudo umount /mnt/crypt sudo cryptsetup close /dev/mapper/crypt

Bypassing Firewalls Part 2: stunnel

In my last post I covered a ssh tunnel, and hopefully it made sense. In this post I'm going to expand on that method by adding an SSL tunnel. We'll use the SSL tunnel to get through web proxies, like WebSense.

What you'll need:
Linux server running something like OpenSSH Server.
Stunnel installed on the server and the client.
A client machine

A quick note about stunnel. I know that it's broken in Ubuntu 12.04 LTS, as of today. So you'll either have to compile it yourself or download a good version.


We'll start with the server configuration. Make sure you have openssl installed, 'which openssl' should show you if you have it.

Create a certificate for your stunnel server:
openssl req -new -x509 -days 365 -nodes -out stunnel.pem -keyout stunnel.pem

Create a config file for stunnel, here is what mine looks like:


cert=stunnel.pem
sslVersion = SSLv3
pid=/tmp/stunnel.pid
[stunnel443]
accept = 10.1.1.5:443
connect = 10.1.1.5:22

The IP address in connect and accept is the IP of eth0 on the server, which is inside a NAT'd network. Next we'll run stunnel using this new config (which was named server443). This needs to be run with root privileges:

stunnel server443

If there are no errors it will just return you to a new line, and won't display any info. You can test the connection with openssl:
openssl s_client -connect localhost:443


It should spit out a bunch of stuff, and then the SSH banner at the end. Just hit enter and it'll say protocol mismatch, and spit you back out to the CLI.

Next we'll configure the client.

Create another config file:


sslVersion = SSLv3
pid=/tmp/stunnelclient.pid
#foreground=yes
client=yes
[21222]
accept=21222
connect=<remote ip>:443

Edit the <remote ip> to be your ip or hostname. You can change the accept port, if you want, you just have to remember what it is. Save the file as client443

Next run this command with root privileges:
stunnel client443

Now try to ssh:
ssh -p 21222 localhost

If all goes well, you should now be connecting to your remote server. Please review my previous post on configuring a proxy with ssh, and how to set up firefox to use it.

Bypassing Firewalls Part 1: SSH

This problem has been bothering me for a while, and I just recently put in the effort to solve it. This is the first installment of ways to bypass a firewall. In this method we're going to use SSH to create a tunnel to a system that is outside of the firewall. We will need two systems for this, one is your client machine, and the other is a Linux server.

NOTE: Sometimes a company will use a proxy (like websense) along side a firewall. If that is the case, this method will most likely fail. I will cover how to solve that problem in my next post.

Step one.

You need an external system to connect to for this to work. You can build your own Linux server from old systems, or run a VM on existing hardware. You can even purchase a RaspberryPi for around $30. You can also run a cloud server using Amazon EC2 (which is free right now), or one from Rackspace. In the end you'll need a simple Linux server running sshd.

Step two.

If you're running your server from a home network, you will probably need to enable port forwarding from your home router. Depending on the firewall rules that you are trying to bypass, you'll probably want to set the incoming port to 443 (https), and have it forward to port 22 (ssh) on your server. The reason why I say 443 is because that is usually not blocked by most firewalls because it's used for https. You can also use port 80 (http).

Step three.

Try out your setup. Using either PuTTY for windows, or a terminal in Linux/Unix/OSX ssh to your server on your external IP using the port you set up in your router for forwarding (443 in our example). A quick and easy way to find your external IP is to search google for "whats my ip", and it should give it to you. If you can, good, you're ready to try ssh'ing from behind the firewall. If it doesn't work you'll have to do some troubleshooting. I would first try to ssh to your server on your internal network instead of the external IP.  If you can connect, then you probably did something wrong with port forwarding. If you can't then you probably set up your server wrong.

Step four.

Try to SSh from your firewalled network. If all goes well, and you can connect, then you'll need to use the ssh application to set up a SOCKS 5 proxy.

Windows PuTTY proxy config.
This is taken from http://www.techrepublic.com/blog/security/use-putty-as-a-secure-proxy-on-windows/421

First, configure a PuTTY session to connect to the UNIX/Linux system you will use as your proxy server. Fill in the following data:

1. Host Name (Or IP Address): This may be the domain name for your network, if you have domain name resolution via a dynamic DNS service or other means set up to allow access to your network via a domain name, or it may be the IP address for your router/firewall.
2. Port: SSH normally uses port 22, but this may be different, depending on how port forwarding may be set up on your trusted network.
3. Protocol: Select the SSH option.

In order to facilitate creating these encrypted proxy sessions quickly in the future, you may wish to give the session a name under the Saved Sessions heading and save it for future use. Do not click the Open button to connect yet, though.

Here’s a screenshot to help:

Second, configure an SSH encrypted tunnel through which your HTTP requests can be forwarded to the system you’re using as a proxy server. Open theConnection > SSH > Tunnelsinterface using the hierarchical Categories pane on the left-hand side of the PuTTY dialog box, and fill in this data:

1. Source Port: Fill in a port number that will be used locally, on the laptop, for this connection. For instance, you might use port 8080 for forwarded HTTP requests.
2. Destination: Leave the text field empty. Select the Dynamic and Auto options.

Click the Add button to commit these encrypted tunnel settings. When that happens, you will see a character string appear in the Forwarded Ports field, as shown in the following screenshot:

Linux/Unix/OSX Config.

To connect in a *nix system, you simply run 'ssh -D 8080 <server ip> -p 443'

Now, you'll need to configure your browser to use a proxy. I usually use Firefox, just because I'm more familiar with it. You'll need to configure firefox to use the proxy for DNS requests. This is done by typing about:config into the address bar, and accepting the warning. Next, search for proxy and find network.proxy.socks_remote_dns and set it to true.

You can also configure the proxy settings directly here, or you can go to Preferences->Advanced->Connection settings

I could have missed something. Feel free to comment if you're having issues, and I can try to help.

How to lock the screen saver (or any feature) in gnome.

This one was a pretty big pain to find for some reason so I decided to put it here. If you use gconf-editor this is really easy. Just run gconf-editor with root privileges (sudo, or as root) and navigate to the setting you would like to change. For our example, we'll change the gnome-screensaver settings. This was tested in RHEL 5.6, 5.7, and 6.2.

If you don't have gconf-editor, you can install it with yum (yum install gconf-editor).

mybox $ sudo gconf-editor

Then expand apps->gnome-screensaver

Change the appropriate settings and then right click on them. You can select “Set as Mandatory” to lock this setting for all users.

If you have many systems you need to lock down you can use the command line tool called gconftool-2. Below is an example of the command line usage to change the screen saver settings to for screen lock on users after 15 minutes of inactivity.

mybox $ gconftool-2 --direct \

--config-source xml:readwrite:/etc/gconf/gconf.xml.mandatory \

--type bool \

--set /apps/gnome-screensaver/idle_activation_enabled true

mybox $ gconftool-2 --direct \

--config-source xml:readwrite:/etc/gconf/gconf.xml.mandatory \

--type bool \

--set /apps/gnome-screensaver/lock_enabled true

mybox $ gconftool-2 --direct \

--config-source xml:readwrite:/etc/gconf/gconf.xml.mandatory \

--type string \

--set /apps/gnome-screensaver/mode blank-only

mybox $ gconftool-2 --direct \

--config-source xml:readwrite:/etc/gconf/gconf.xml.mandatory \

--type int \

--set /apps/gnome-screensaver/idle_delay 15

Stop That Banner!

I do a lot of repetitive tasks on many different machines, and I use ssh to get me there. One thing that annoyed me the most was the long banner message taking up most of my console window. The banner is a requirement for us, so I cannot simply remove it on the server side. We use Red Hat Enterprise Linux and Solaris 10, and the following is what I use to stop the banner from displaying:

If you are on Linux or Solaris you first need to have the .hushlogin file in your home directory on the machine that you are ssh'ing into: by running the following command, "touch .hushlogin", you will banner displaying on Solaris, but not on Linux. On Linux, this will stop the "Last login..." message. To stop the banner from displaying in Linux, you need to issue the ssh command with the -q switch, "ssh -q [hostname]".