Trainings

Thursday, February 8, 2018

sending Log file / CSV data to Elasticsearch without header row - PART II

Lets take my previous example , i am having below sample csv data


stock,open,close,date

icici,240,350,05-02-2014

sbi,140,250,05-02-2014

infy,950,1150,05-02-2014

tcs,2400,3500,05-02-2014

and when using below logstash config file "header row" is going to Standard output( we can replace this with elasticsearch . for demonstration purpose using stdout plugin)

logstash.conf

input {
file {
        path => "C:/java-developper-softwares/prashant/ELK/data/stock.csv"
        start_position => "beginning"
sincedb_path => "/dev/null"
    }
}
filter{
csv {
columns => ["stock", "open", "close","date"]
         separator => ","
    }

#if [message] =~ /^stock/
#{
# drop {}
#}
}
output {
  stdout {
codec => rubydebug
   }
}
when running command "logstash -f logstash.conf" getting below output.



















Now our requirement that we don't want to send "header row" i.e. "stock,open,close,date"
since now we parse csv and sending reading each column separately so we can compare stock column value and if it is equal to stock string then we can drop the row.
here in my case i compare stock column however logically we can compare any column mentioned in our csv file.

So lets modify our logstash.conf to below
input {
file {
        path => "C:/java-developper-softwares/prashant/ELK/data/stock.csv"
        start_position => "beginning"
sincedb_path => "/dev/null"
    }
}
filter{
csv {
columns => ["stock", "open", "close","date"]
         separator => ","
    }

if ([stock] == "stock")
{
drop {}
}
}
output {
  stdout {
codec => rubydebug
   }
}


Wednesday, February 7, 2018

Change your credential in eclipse git (EGIT) if your password expires/change

We all know that password expires after a certain time period. So how to set the new password for your Eclipse git client. Here are the steps given below:

open git repository by clicking on open perspective button in eclipse and select git


now expand your Remotes >> origins >> 
and right click on the green node and select "change credentials".














and now try to push the data to stream and your authorization issue will get resolved.


Tuesday, February 6, 2018

sending Log file / CSV data to Elasticsearch without header row - PART I

Lets take an example , i am having below sample csv data

stock,open,close,date
icici,240,350,05-02-2014
sbi,140,250,05-02-2014
infy,950,1150,05-02-2014
tcs,2400,3500,05-02-2014

and when using below logstash config file "header row" is going to Standard output( we can replace this with elasticsearch . for demonstration purpose using stdout plugin)

logstash.conf

input {
file {
        path => "C:/prashant/ELK/data/stock.csv"
        start_position => "beginning"
sincedb_path => "/dev/null"
    }
}
output {
  stdout {
codec => rubydebug
   }
}

when running command "logstash -f logstash.conf" getting below output.

Now our requirement that we don't want to send "header row" i.e. "stock,open,close,date"
So lets modify our logstash.conf to below
input {
file {
        path => "C:/prashant/ELK/data/stock.csv"
        start_position => "beginning"
sincedb_path => "/dev/null"
    }
}
filter{
if [message] =~ /^stock/
{
drop {}
}
}
output {
  stdout {
codec => rubydebug
   }
}

so here we are using filter plugin where we are checking that if message contains "stock" string then drop that row. and boila see below output

So crux of this post is use filter plugin and check any of the column value and drop that row.
filter{
if [message] =~ /^stock/
{
drop {}
}
}

In my next post , i will walk through if i read this file as a csv i.e. comma delimited list of column in that case how we will ignore first row.

Monday, February 5, 2018

Use multiple logstash config file at a time on single command line or as a service

Problem Statement- having multiple logstash config file(As there is differet data configured in each file) for posting data from different machines in cluster which requires to open as many command line instances as number of config files. So is it possible to run all the config files from single instance or anything similar.

Answer: Either put all files in a directory and run Logstash with -f path/to/directory
or use multiple -f options that each point to one of the files i.e.
Command- logstash -f Sample1.conf -f Sample2.conf

Note: Keep in mind that Logstash has a single event pipeline (i.e. internally all configuration files concatenated and treated as a single big file.) and apply to all filters and outputs events unless you use conditionals to select how they apply..

need to use conditionals to select which filters and outputs to apply to which events.

filter {

if [type] == "event" {

# do stuff

}

else if [type] == "data" {

# do other stuff

}

}

OR

input {
file {
path => "BatchData\Batch_Raw_Data.csv"
tags => [ "batchdata" ]
start_position => "beginning"
}
}
output {
if "batchdata" in [tags]{
elasticsearch {
action => "index"
index => "IndexName"
}
}}

If you really need multiple event pipelines that can be stopped and restarted individually you will indeed have to run multiple instances of Logstash.

Sunday, October 9, 2016

Monday, January 19, 2015

two time zone in outlook 2013 calendar

You can have two time zone displayed in your calendar. e.g. I have set up for IST and  CST (circled in red).

It automatically takes care of day light saving as well and so becomes very handy while setting up meeting. 




Steps to add a second time zone

Two separate time zones can be displayed in the Calendar. The two time zones can be, for example, your local time zone and the time zone of a city that you often travel to. The second time zone is used only to show a second time bar in Calendar view, and does not affect the way in which Calendar items are stored or displayed.

1. Click the File tab.

2. Click Options.

3. On the Calendar tab, under Time Zones, select the Show a second time zone check box.

4. In the Label box, type a name for the additional time zone.

5. In the Time zone list, click the time zone that you want to add.

6. If you want your computer clock to automatically adjust for daylight saving time (DST) changes, select theAdjust for daylight saving time check box. This option is available only in time zones that use daylight saving time (DST).

Saturday, November 29, 2014

Different places to select a View in MS Project 2013

There are several built-in views MS Project Provides from which we can pick up as and when required.

For example : Gantt chart, Task usage, Team planner, resource sheet

Also there are number of places from where we can pick up the view
i.e. Task tab , Resource tab ,view tab, Status Bar , Quick access toolbar

From status Bar we can choose Gantt chart, Task usage, Team planner, resource sheet

Below picture showing view selection from Task tab.

If the view that you want do not appear in the list, then click on “More views” and a dialog box appear . And from here you can select whatever view you want .


We can also select the views from Quick access toolbar: turn on view




We can also set a view as favorite view [File-> options-> general -> Default view]


Along with built in views we can also customize view and save it so that we can also reuse this customize view. How to create a custom view , I will cover in my next posts.