If you have been managing Splunk for some time, you might have come across situations where you want to filter away unnecessary data from being ingested.<\/p>\n\n\n\n
The two most common ways of doing this in Splunk is to filter out data by “blacklisting”<\/a> at the input stage or to “nullQueue<\/a>” at the index stage. If you use rsyslog as a syslog collector in front of your Splunk indexers, I will show you a way to filter out data in rsyslog, to avoid spending valuable indexer or forwarder resources.<\/p>\n\n\n\n The first thing you should do is to define a template, aptly named “template_nullQueue”:<\/p>\n\n\n\n For testing purposes you may of course also make the data be routed to another destination than \/dev\/null.<\/p>\n\n\n\n Next, you should define a new ruleset for this template:<\/p>\n\n\n\n Finally, the business end. You may now drop an if-statement in any existing ruleset definition. The following example shows how any syslog message containing the string “Teardown ICMP connection for faddr”, the message will be directly sent to \/dev\/null.<\/p>\n\n\n\n <\/p>\n","protected":false},"excerpt":{"rendered":" If you use rsyslog as a syslog collector in front of your Splunk indexers, I will show you a way to filter out data in rsyslog, to avoid spending valuable indexer or forwarder resources.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":true,"template":"","format":"standard","meta":{"_price":"","_stock":"","_tribe_ticket_header":"","_tribe_default_ticket_provider":"","_ticket_start_date":"","_ticket_end_date":"","_tribe_ticket_show_description":"","_tribe_ticket_show_not_going":false,"_tribe_ticket_use_global_stock":"","_tribe_ticket_global_stock_level":"","_global_stock_mode":"","_global_stock_cap":"","_tribe_rsvp_for_event":"","_tribe_ticket_going_count":"","_tribe_ticket_not_going_count":"","_tribe_tickets_list":"[]","_tribe_ticket_has_attendee_info_fields":false,"_EventAllDay":false,"_EventTimezone":"","_EventStartDate":"","_EventEndDate":"","_EventStartDateUTC":"","_EventEndDateUTC":"","_EventShowMap":false,"_EventShowMapLink":false,"_EventURL":"","_EventCost":"","_EventCostDescription":"","_EventCurrencySymbol":"","_EventCurrencyCode":"","_EventCurrencyPosition":"","_EventDateTimeSeparator":"","_EventTimeRangeSeparator":"","_EventOrganizerID":[],"_EventVenueID":[],"_OrganizerEmail":"","_OrganizerPhone":"","_OrganizerWebsite":"","_VenueAddress":"","_VenueCity":"","_VenueCountry":"","_VenueProvince":"","_VenueState":"","_VenueZip":"","_VenuePhone":"","_VenueURL":"","_VenueStateProvince":"","_VenueLat":"","_VenueLng":"","_VenueShowMap":false,"_VenueShowMapLink":false,"footnotes":""},"categories":[12,1],"tags":[14],"class_list":["post-67","post","type-post","status-publish","format-standard","hentry","category-splunk","category-uncategorized","tag-rsyslog"],"_links":{"self":[{"href":"https:\/\/blog.vaglid.net\/index.php\/wp-json\/wp\/v2\/posts\/67","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.vaglid.net\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.vaglid.net\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.vaglid.net\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.vaglid.net\/index.php\/wp-json\/wp\/v2\/comments?post=67"}],"version-history":[{"count":12,"href":"https:\/\/blog.vaglid.net\/index.php\/wp-json\/wp\/v2\/posts\/67\/revisions"}],"predecessor-version":[{"id":160,"href":"https:\/\/blog.vaglid.net\/index.php\/wp-json\/wp\/v2\/posts\/67\/revisions\/160"}],"wp:attachment":[{"href":"https:\/\/blog.vaglid.net\/index.php\/wp-json\/wp\/v2\/media?parent=67"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.vaglid.net\/index.php\/wp-json\/wp\/v2\/categories?post=67"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.vaglid.net\/index.php\/wp-json\/wp\/v2\/tags?post=67"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
Both of these ways of filtering has the benefit that the data will not be forwarded, and does not count against your Splunk license.<\/p>\n\n\n\n$template template_nullqueue,\"\/dev\/null\"<\/code><\/code><\/pre>\n\n\n\nruleset(name=\"ruleset_nullqueue\") <\/code>{\n action(type=\"omfile\" dynafile=\"template_nullqueue\" )\n}<\/code><\/code><\/pre>\n\n\n\n[...]\nruleset(name=\"ruleset_10601\") {\n if($msg contains 'Teardown ICMP connection for faddr') then <\/code>{\n call ruleset_nullqueue\nstop\n\n}\n\naction(type=\"omfile\" dynafile=\"template_10601\" dirCreateMode=\"0750\" FileCreateMode=\"0640\" fileOwner=\"splunk\" fileGroup=\"adm\" dirOwner=\"splunk\" dirGroup=\"adm\")\n\n}\n\nruleset(name=\"ruleset_10602\") {\n\naction(type=\"omfile\" dynafile=\"template_10601\" dirCreateMode=\"0750\" FileCreateMode=\"0640\" fileOwner=\"splunk\" fileGroup=\"adm\" dirOwner=\"splunk\" dirGroup=\"adm\")\n\n}\n\n[...]\n\n<\/code><\/code><\/pre>\n\n\n\n
Note that the if-statement needs to be above the action-statement<\/p>\n\n\n\n