Logstash; Important Note about Config Files...

If using Logstash, either alone or in cahoots with Elasticsearch, Kibana, Greylog,etc.  make *damn* sure your syntax in /etc/logstash/conf.d/* is correct.  See Logstash, when started, takes every file in that directory (whether a config file or not) and combines them into one large file to be processed.  So you can't keep .old files in there like I tried :(

You might get an error either in the log or when starting manually that references a line # that makes you scratch your head.  Mine mentioned an error on line 26.  Well none of my files had that many lines alone.. but obviously when combined via cat /etc/logstash/conf.d/* > /tmp/total.cnf things made more sense.

Elastic Webinars

Just found this, https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html some good 'get -off-the-ground' material for Elastic users.

Totally pasted the wrong URL above.. I should proofread more.  Here's the correct one:

https://www.elastic.co/webinars/introduction-elk-stack

ELK Stack build/issues

So at $newJob I'm setting up an ELK server in the dev environment, for those who don't know ELK stands for Elasticsearch, Logstash and Kibana. And apparently is now called teh Elastic Stack.. lol.

This post is not intended to be a tutorial, but merely things I have run into or noticed during the course of my own time spent on this project.  The tutorial I am using is here: How to install ELK on CentOS 7

One thing right off the bat is that Yaml (yml) files are horribly dependent upon proper spacing.  One wrong space can toss up some damn ugly errors that leave you scratching your head.

When possible use SCP to copy config files across your multiple client servers to help maintain consistency.

Following the instructions to create a self-signed SSL cert based off IP worked.  Now I tried Method 2 using a CN with DNS.  Each client plus the ELK server has hosts defined.  However generating a cert based off CN failed each and every time because the signer could not be authenticated. Now generating off IP using the ssl.conf worked on one client, but not the other.  In the end I scp'd the filebeat.yml from the working client to the non-working one, bounced filebeat's service and it began reporting.

And finally

sudo curl -XGET 'http://localhost:9200/filebeat-*/_search?pretty'

returns info! yay.

BTW all of this was built on a CentoS 7 minimal install on ESXI 5.5

Long time, no post.

So there's been an extended hiatus here for me.  Had quite a lot of changes in my life.. new jobs, new home state.  It's been exciting, stressful, fun, scary, etc but wholly worth it. IMHO at least.

I can't go into a lot of specifics yet for the general masses but suffice to say we're about 1800 miles from where we used to be.