Most of Splunk users especially Admin knows it is a must to optimize the way Splunk handles Scheduler searches especially in big environments, where there is 1000s of saved searches running every hour.

Regardless the search is a scheduled search or an Ad-hoc search, indexers are involved since your data is stored in indexers. Your search is sent to the indexers and look for data based on your search criteria and pulls results. The retrieved data is further processed based on the subsequent search commands.

Have a look at this .conf presentation to have a better understanding How search works

For Clustered Enviroment , please refer to How search works in an indexer cluster

Also The anatomy of a search will give you more information

This post will describe some important Splunk quires used by Administrators in order to check the load on their Splunk Cluster.

First , You simply need to run the below in order to see a list of all scheduler searches running

| rest /services/saved/searches | where is_scheduled=1

To get a history of scheduled search , check the internal logs

index=_internal sourcetype=scheduler | table _time user savedsearch_name status scheduled_time run_time result_count

To get Scheduler searches by runtime , below I am filtering where run_time is greater than 300 Sec

index=_internal sourcetype=scheduler host=<Splunk_SearchHeads>* | table _time user savedsearch_name status priority scheduled_time run_time result_count window_time alert_actions | where run_time>300 AND window_time=0 | stats max(run_time) by savedsearch_name,user

Get more details our of the Scheduler Searches using rest

| rest /servicesNS/-/-/saved/searches splunk_server=* | table title,cron_schedule,actions,eai:acl.app,eai:acl.owner,splunk_server,status

I will try to dedicate an individual post for discussing how to write an optimized search in Splunk.

Happy Splunking :D