Handling FirewallD logs with Rsyslog

Lately I'm working on a project for work that's involving standing up Syslog servers for various sources such as Cisco FTD's, WLC's and other Linux servers. We have standardized on FirewallD and Ubuntu (cue the lols) I know, I know.. an uncommon pairing. But it works fine and while I'll admit not quite as easy to manage as UFW it's pretty solid.

Anyway, one problem I've had with FirewallD is how to handle it's own logging functions. By default it's conf file only supports the enabling or disabling of logging denied packets. Everything else seems to be split between kern.log and syslog however it seems the facility on all of them is 'kernel'. I want all of it to be sent to a specific log file, in this case /var/log/firewalld.log. 

Today I finally figured out how:

if ($programname == 'kernel') and ($msg contains "FwD ") then { Action (type="omfile" file="/var/log/firewalld") stop }

Message contains "FwD" is included because everyone one of my rich rules in FirewallD has a 'log prefix' of "FwD" then usually I tag on a bit extra like "SSH In", or "SNMP In" so my logs are easier to read.

Determine IP address from MAC address, in Linux

Found a switch in a closet.. not documented and I'm too lazy to haul a laptop over and try to hit the console so I took a picture of the MAC label and stumbled across this literal GEM of a one-liner:

nmap -sP <subnet>/24 >/dev/null && arp -an | grep xx:xx:xx:xx:xx:xx | awk '{print $2}' | sed 's/[()]//g'

 It literally just works.

Grep returning if results were found or not

Today I needed a one-liner like this to help with with an Ansible playbook, storing here for posterity:

rpm -qa | grep -q "rhn" && echo 'is installed' || echo 'not installed'

SCP files thru multiple proxy hosts

I realize this might be old-hat for quite a few but I just learned it and this will save me so bloody much time!

While there are multiple ways of doing this.. including editing your .ssh/config file to include a similar reference I am choosing the full CLI method.

So first off make sure netcat (nc) is installed on all hosts.  Then craft a string as such:

scp -o ProxyCommand="ssh first_host nc second_host 22" /local/path/to/file user@target_host:/dest/path


So what I have done here is to ssh proxy through host1, to host2 then scp'd from host2 to target_host.  Which in my case was exactly what I needed since this new Java environment requires me to jump through two proxies to reach my target.

Like I said I know this is old stuff to a large percentage of you Unix/Linux admins but hopefully this will help someone.

DavMail, Thunderbird and Systemd..

So I can't recall if I posted about this or not but since switching full time to Fedora I've had to get creative with email clients.  Since Evolution is clunky and rather ugly I decided to brave the DavMail world with Thunderbird.

Now for quite a while I've been suffering through DavMail just quitting on me.  Nothing gets logged anywhere, the pid just dies. However today in #systemadmins I tossed out a question and it was fielded by a guy who I'd consider quite the senior in the whole *nix/SysAdmin world.  Dude took less than 20 minutes and had me a working systemd.service file that will run automagically and restart if it happens to die, oh and get this.. it will LOG too! *giggle*

Here's the file for those interested:

[Unit]
Description=DavMail Exchange Gateway
Requires=network.target
After=network.target
[Service]
Type=simple
RemainAfterExit=no
#ExecStart=/usr/bin/davmail /etc/conf.d/davmail.properties
#ExecStop=killall davmail
Environment=BASE=/path/to/davmail-linux-x86_64-4.8.0-2479/
Environment=PROPERTIES=/path/to/.davmail.properties
ExecStartPre=/usr/bin/bash -c "for i in $BASE/lib/*; do /usr/bin/systemctl set-environment CLASSPATH=${CLASSPATH}:$i ; done"
ExecStart=/usr/bin/java -Xms1G -Xmx1G -Djava.awt.headless=true -XX:ErrorFile=/tmp/davmail_errors.log -XX:-HeapDumpOnOutOfMemoryError -Dsun.net.inetaddr.ttl=60 -cp ${BASE}/davmail.jar:${CLASSPATH} davmail.DavGateway ${PROPERTIES}
Restart=always
RestartSec=3
[Install]
WantedBy=multi-user.target

GitKraken....Linux/Windows/MAC GUI interface for your git needs. Very cool!

So after fighting git on the command line for a solid two weeks, the idiot I am never thought to search for a gui to this madness.  Google respectfully greeted me with a top spot: GitKraken.

Man it's neat, and VERY attractive to the eyeball. Granted I've been using it for all of 10 minutes but I already resolved my merge conflict!  Yeah!

Ok so being truthful, I'm positive anyone experienced enough with git could have not had the issues I did.. but trying to visualize all that is my git branches in my head did become tough.  Yeah purists will call me a wiener but I really don't care.  I'm loving me some GitKraken and I recommend it to all.. experienced or not this is a great tool for helping you work with your repo.

BASH: change ownership of subfolders based on parent folder name

Sounds convoluted, and honestly I felt overwhelmed trying to figure out how to handle this programmatically.

So for the past few days I've been working on a new Cent7 server to replace an aged 6.6 vm.  This box sits in the DMZ and is used by various entities around the state to sftp certain txt files to.  So there are roughly 50 home directories.. each with sub folders and an ssh key.  So in an effort to *not* force the users to create new ssh keys I rsync'd the entire parent folder over.  First time I lacked the proper permissions and wound up creating the home folders but nothing underneath.

So long story short, I eventually was able to get the correct syntax down to rsync the /chroot/parent and all the individual home sub-folders to the new server WITH the two sub folders and the .ssh(which houses the authorized_key file).  EUREKA!

However now all the permissions were borked up.  My own local ssh user had taken ownership of the parents, and children in both user and group. DOH!  I was able to clean up the group side of things easily enough via: 'chown -R :group /*'  However the user side was trickier because the user side on the two sub folders needed to be owned by the user in question.  (Obviously) but luckily enough for me the home folder's names was the username entirely.  Lucked into this absolute gem:

Fix the path in Line 1, and you are golden!

for dir in /home/*/; do
    # strip trailing slash
    homedir="${dir%/}"
    # strip all chars up to and including the last slash
    username="${homedir##*/}"

    case $username in
    *.*) continue ;; # skip name with a dot in it
    esac

    chown -R "$username" "$dir"
done

Fedora 25, Another Shot at a Linux Workstation - UPDATE #2

So a while back I posted complaints about DisplayLink and their (so far) unwillingness to offer up Linux drivers for distros other than Ubuntu.
Since then I have discovered that USB-C docks that utilize DisplayLink seem to work right out of the box.  Display, Ethernet, and usb-passthrough all work perfectly.

So the current plans are, once I'm satisfied that I can perform all my normal functions at work using Fedora and a Win10 vm, connect to all three VPN tunnels in both OS's and access all the files I need then it's time to acquire a dock.

:happy dance:

Rescan Linux VM guest for new vmdk w/o reboot

So recently at $work I've been handling a project to migrate production and test Oracle environments to new hardware on top of new vSphere as well.

So my job is corralling the DBA contractor, keeping him on par and whatnot.  Anyway this post isn't about that really.  The servers themselves are CentOS 7 with Oracle 12c, and in the name of Best Practices I've created 5 hdd's: OS, U01, U02, tmp and Backups.  So last night DBA requests another named U03 to house teh redo log files.  Ok sounds good, but this time I did not want to have to bounce the server to see the hew 'hardware'. So credit goes to: Vivek Gite @ here

echo "- - -" > /sys/class/scsi_host/host#/scan

Works like a friggin champ!  Of course this string returns nothing but a quick Fdisk -l shows that my sdf was found.

*drops mike*

Linux One-Liner Config Backup

So here at $newjob we've ran into Linux server, in all three environments, with huge uptimes and little documentation.  My co-workers already had one such beast get a forced reboot and things did not work correctly when it came back up.
So this morning I lucked into a nice one-liner that will pull all the infos.  Seriously it's damn good.  Props go out to /u/WOLF3D_exe

history >> "$HOSTNAME"_"$USER"_`date +%F`.log && last  -n 666 -a -d -x -F >> "$HOSTNAME"_"$USER"_`date +%F`.log && who >> "$HOSTNAME"_"$USER"_`date +%F`.log && uptime  >> "$HOSTNAME"_"$USER"_`date +%F`.log && uptime -p  >> "$HOSTNAME"_"$USER"_`date +%F`.log && cat /etc/redhat-release  >> "$HOSTNAME"_"$USER"_`date +%F`.log && uname -a >> "$HOSTNAME"_"$USER"_`date +%F`.log && iptables-save  >> "$HOSTNAME"_"$USER"_`date +%F`.log && netstat -tunap >> "$HOSTNAME"_"$USER"_`date +%F`.log && ps aufx >> "$HOSTNAME"_"$USER"_`date +%F`.log && yum list installed >>  "$HOSTNAME"_"$USER"_`date +%F`.log && chkconfig --list >> "$HOSTNAME"_"$USER"_`date +%F`.log  && chkconfig --list |grep on >> "$HOSTNAME"_"$USER"_`date +%F`.log  && >> "$HOSTNAME"_"$USER"_`date +%F`.log && cat /etc/passwd >> "$HOSTNAME"_"$USER"_`date +%F`.log && cat /etc/group >> "$HOSTNAME"_"$USER"_`date +%F`.log && cat /etc/ssh/ssh_config >> "$HOSTNAME"_"$USER"_`date +%F`.log && cat /etc/ssh/sshd_config >> "$HOSTNAME"_"$USER"_`date +%F`.log && cat /root/.ssh/config >> "$HOSTNAME"_"$USER"_`date +%F`.log && cat /root/.ssh/authorized_keys >> "$HOSTNAME"_"$USER"_`date +%F`.log && cat /root/.ssh/known_hosts >> "$HOSTNAME"_"$USER"_`date +%F`.log && cat /home/*/.ssh/config >> "$HOSTNAME"_"$USER"_`date +%F`.log && cat /home/*/.ssh/authorized_keys >> "$HOSTNAME"_"$USER"_`date +%F`.log && cat /home/*/.ssh/known_hosts >> "$HOSTNAME"_"$USER"_`date +%F`.log

Fedora 25, Another Shot at a Linux Workstation - UPDATE

So I posted two weeks about about giving Fedora a shot as my workstation OS.  Well to tell a long story shortly.. It did not work.  I tried Santiagos DisplayLink drivers to no avail.

Nothing I tried worked and to be perfectly honest I lost interest for now.  I want to switch, I really do but I do not want to fight this much to get hardware to work. I realize any change from Windows will result in some work.. but c'mon DisplayLink.  Ubuntu is not the only Linux distribution in widespread use.

Fedora 25, Another Shot at a Linux Workstation

So since I predominately support Linux at the new job, and recent weirdness with Win10 I decided to give Linux another try as a workstation platform.  Granted I do need RDP to hit a few Windows servers but it's nothing like it used to be when I tried switching at work in the past.  Here most of my work is via SSH so yay!

Speaking honestly this is exactly what I've been leaning towards for a while now, moving away from Windows support.  Which I guess, seriously you can't really get fully away from it.. at least IMHO.

So hardware.. what am I running?  A Dell XPS 15 model 9550  pushing a 34" Asus lcd along with a pair of Dell 19" widescreen lcds, along with the laptops builtin screen.  A decent amount of desktop real estate.  Luckily the XPS has a built-in GTX 960m which helps.  The con in this setup is the USB3 DisplayLink "dock".  So far Ubuntu is the only distro with an official driver package for DisplayLink devices.  Usb and ethernet work in a pass through mode but video does not.

So Googling I stumbled across this site: http://nothen.com.ar/en/support-for-displaylink-adapters-on-linux/ So this guy ran into the same issue, just with a slightly older Dell model. I don;t beleive he was trying to drive as many displays as I am, but he made excellent headway in developing an RPM package.
However since I want to run Fedora 25 which is technically still in RC2 until next week I think.. there's too many differences for his version to work with my laptop.  (I know.. I tried)

But this Santiago person on down in the comments looks to be currently developing a package for RPM based distros.  I have not yet had the chance to try his latest release but I have plans on it tonight.  If that fails I will be trying to source a USB-C display adapter and will probably lose one or two of my displays until the DL drivers get to a point of where they work.

In any case I'll report back when I've had a chance to try Santiagos latest.

https://github.com/ssaavedra/displaylink-rpm/releases/

ELK Stack build/issues

So at $newJob I'm setting up an ELK server in the dev environment, for those who don't know ELK stands for Elasticsearch, Logstash and Kibana. And apparently is now called teh Elastic Stack.. lol.

This post is not intended to be a tutorial, but merely things I have run into or noticed during the course of my own time spent on this project.  The tutorial I am using is here: How to install ELK on CentOS 7

One thing right off the bat is that Yaml (yml) files are horribly dependent upon proper spacing.  One wrong space can toss up some damn ugly errors that leave you scratching your head.

When possible use SCP to copy config files across your multiple client servers to help maintain consistency.

Following the instructions to create a self-signed SSL cert based off IP worked.  Now I tried Method 2 using a CN with DNS.  Each client plus the ELK server has hosts defined.  However generating a cert based off CN failed each and every time because the signer could not be authenticated. Now generating off IP using the ssl.conf worked on one client, but not the other.  In the end I scp'd the filebeat.yml from the working client to the non-working one, bounced filebeat's service and it began reporting.

And finally

sudo curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'

returns info! yay.

BTW all of this was built on a CentoS 7 minimal install on ESXI 5.5

Fedora 23 Broken Grub - Boot to Grub Prompt

So today I swapped laptops from a Lenovo W510 with an HP Elitebook 8570p as a test with a Fedora  23 install.

Now for safeties sake I did image the drive with Clonezilla first to a local USB drive for safe keeping.  Then I swapped the drives and booted up.  Boot came directly to the grub prompt.

After multiple Googling's I decided to try the SuperGrubDisk2 iso..  So I booted it up and it found all the entries in my grub.cfg file.  Being relatively new to this level of troubleshooting Linux I decided to have it boot me into my correct kernel.  Once logged in I verified it was indeed the correct kernel (which it was) so then from a terminal I ran the following:


sudo grub2-install /dev/sda

Which completed without error.  Then,

sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Then rebooted.  VIOLA! It works correctly now. And I must say a helluva lot easier and less painful than switching boot drives in Windows.

Linux: A new career goal.

So here lately I've been anxious about the future, for whatever reason.  Windows admins are a dime a dozen and have been for years now.  The only other way to go IMO is Linux.

I'm not a complete nooblet, I've stood up random linux servers in the past.. toyed with various distros when I got bored with Windows. But never aimed for a full on Linux Systems Administrator until now.


So in googling for linux training you get inundated right off the bat.  So I went for the tried and true.. Red Hat.  RHEL training is expensive to be sure, but it's recognized and carries weight.  Then I stumbled across Linux Academy.  LA has decent reviews on Reddit and other various places.  On the other hand die hards don't seem to care for it.  But at $24 per month with access to all of their wares which include Linux Foundation, Red Hat, Amazon AWS, etc it's too cheap to be ignored.

So I decided to go for that initially, at least test the waters for a month.  I started in the Linux Foundation Certified Systems Administrator (LFCSA v2.16) track.  I seems to be almost entirely video based, with a few exercises tossed in. Honestly I don;t know how I feel about that right now.. but I'll keep going.


Also recently I've been spending a lot of time in #systemadmins.  The guys in there are what really spurned me onto linux training.  A dude in that channel wrote up this in response to a question on Reddit:

https://www.reddit.com/r/linuxadmin/comments/2s924h/how_did_you_get_your_start/cnnw1ma


So while also running through this LFCSA course I'm also building the environment he outlines.


eDX is offering the Linux Foundation Intro to Linux course for free too:

https://www.edx.org/course?search_query=linux

https://linuxacademy.com/

Commvault Simpana; Linux Informix Db Jobs failing.

So for the past month I've had a reoccurring issue with one Linux client and backing up the Informix db.

        Error Code: [82:129]
Description: The job is pending because: The remote end has closed network connection unexpectedly
Source: mi-2k2-cv, Process: CVD


Here's some pertinent info from the client logs IFXXBSA.log and bar_act.log:

IFXXBSA.log
7940 b740d940 12/18 16:46:28 527474 BSAQueryObject:  Query.CopyType:3|LGName:|CopyGpName:|ResourceType:ND
7940 b740d940 12/18 16:46:28 527474 BSAQueryObject:  Query.ObjectType=4|ObjectStatus=2
7940 b740d940 12/18 16:46:28 527474 BSAQueryObject:  Query.Description=
7940 b740d940 12/18 16:46:28 527474 ReadFromOnconfig() -  nitems : 2
7940 b740d940 12/18 16:46:28 527474 ReadFromOnconfig() -  Parameter SERVERNUM= 0
7940 b740d940 12/18 16:46:28 527474 BSAQueryObject: Call GetBackupInfoByJob to set/update the ArchFileMap cache
7940 b740d940 12/18 16:46:28 527474 ::GetSubclientDir() - The subclient directory is [/opt/simpana/iDataAgent/jobResults/2/5789].
7940 b740d940 12/18 16:46:28 527474 BSAQueryObject: GetBackupInfoByJob did not succeed.. ignore and continue
7940 b740d940 12/18 16:46:28 527474 BSAQueryObject: /ifmx_online/datadbs01/0 . Try ArchiveManager. size=1|PID=7940
7940 b740d940 12/18 16:46:28 527474 BSAQueryObject: Unable to find object key=/ifmx_online/datadbs01/0 in the map.
7940 b740d940 12/18 16:46:28 527474 BSAQueryObject: Check map(for debugging): key=/ifmx_online/rootdbs/0|val=/ifmx_online/rootdbs/0.1450475164.7940
7940 b740d940 12/18 16:46:28 527474 BSAQueryObject: /ifmx_online/datadbs01/0 not found in CV database
7940 b740d940 12/18 16:46:28 527474 BSAQueryObject: Trying to find /ifmx_online/datadbs01/0 in ixbar..|obj_field2=datadbs01|startrefversion=0
7940 b740d940 12/18 16:46:28 527474 BSAQueryObject: /ifmx_online/datadbs01/0 not found in /opt/IBM/informix/etc/ixbar.0.
7940 b740d940 12/18 16:46:28 527474 BSAQueryObject: Exit, status:26
7940 b740d940 12/18 16:46:28 527474 BSAEndTxn: Enter, vote:2
7940 b740d940 12/18 16:46:28 527474 BSAEndTxn: Error: Transfer Incomplete
7940 b740d940 12/18 16:46:28 527474 BSAEndTxn: Exit, status:3
7940 b740d940 12/18 16:46:28 527474 BSATerminate: Enter - PID:7933, TID:7940, xbsaParams.pid:7933
7940 b740d940 12/18 16:46:28 527474 IfxBackupRestoreBase::ExitHere() - Job[0] thread[7940]: m_totalFilesTrans=0 m_totalFilesTransferred=0
7940 b740d940 12/18 16:46:28 527474 IfxBackupRestoreBase::Finalize() - Job[0] thread[7940]: Exit.
7940 b740d940 12/18 16:46:28 527474 IfxAgentBase::Finalize() - delete m_hostName
7940 b740d940 12/18 16:46:28 527474 BSATerminate: Exit, status=0, exstatus=3


bar_act.log

 2015-12-18 16:04:29 6482  6476 onbar_d complete, returning -1 (0xffffffff)
 2015-12-18 16:24:38 7113  7106 onbar_d -b -L 0
 2015-12-18 16:24:38 7113  7106 Working with cvsm as generic storage manager.
 2015-12-18 16:24:38 7113  7106 Archive started on rootdbs, datadbs01, llogdbs01 (Requested Level 0).
 2015-12-18 16:24:38 7113  7106 (-43078) Open or close failed on file '/opt/IBM/informix/etc/ixbar.0', errno = 2 .
No such file or directory
 2015-12-18 16:24:38 7113  7106 Begin level 0 backup rootdbs.
 2015-12-18 16:24:38 7113  7106 Starting Filter /bin/gzip.
 2015-12-18 16:24:38 7116  7113 Successfully connected to Storage Manager.
 2015-12-18 16:24:58 7116  7113 The child process for the backup and restore filter is terminating
with exit code 0.
 2015-12-18 16:24:58 7113  7106 Successfully connected to Storage Manager.
 2015-12-18 16:25:01 7113  7106 Completed level 0 backup rootdbs (Storage Manager copy ID: 7113 1450473878).
 2015-12-18 16:25:01 7113  7106 Begin level 0 backup datadbs01.
 2015-12-18 16:25:01 7113  7106 Starting Filter /bin/gzip.
 2015-12-18 16:25:01 7139  7113 ASSERT: file bar_unix.c line 1448 - contact product support
 2015-12-18 16:25:01 7139  7113 See also: /storage/informixtmp//core
 2015-12-18 16:25:01 7113  7106 (-43082) Writing to backup and restore filter failed with error 136103679.
 2015-12-18 16:25:12 7113  7106 Begin backup of critical file '/opt/IBM/informix/etc/ixbar.0'.
 2015-12-18 16:25:12 7113  7106 (-43078) Open or close failed on file '/opt/IBM/informix/etc/ixbar.0', errno = 2 .
No such file or directory
 2015-12-18 16:25:12 7113  7106 onbar_d complete, returning -1 (0xffffffff)


The resolution for this particular issue was removing a pair of functions from an onconfig file on the client, specifically:

BACKUP_FILTER /bin/gzip
RESTORE_FILTER /bin/gunzip

Restarted Informix via onmode -ky then oninit -v then re-tried the incremental backup job and EUREKA! much success.

CommVault - Simpana iData Agent install on LInux

So push installations from the Simpana console sometimes don't work on *nix hosts, so I resorted to installing interactively.  Problem was I was getting an error concerning an incorrect version of KSH.

My resolution was a 'chmod -R 755 .'  Running that from within the directory where the cvpkgadd file is.  Fixed me right up!