Splunk Architect exam

95 flashcards · Other · by Nicky Carlisle

1. The splunk btool command displays current config values on disk. The btool command syntax is: ./splunk btool [conf] list [options]

   Multiple questions on the test where btool is brought up.

2. The default bandwidth for a universal forwarder is 256KB and can be changed in the limits.conf file. Avoid using a HF if at all possible because sending data from a HF reduces throughput performance on the splunk deployment.

3. The search scheduler is responsible for the following: If when and how search alerts can be triggered. Determines if a search should be skipped or deferred when splunk is under full load. A search will be skipped if its marked as real-time or deferred if its marked as continuous . prioritizes real-time scheduling over continuous scheduling. The realtime_schedule option in the savedsearches.conf file shows weather a search is real-time or continuous. Records of why a search was skipped or deferr…

4. Splunk search scheduling priority: Ad-hoc > Real-time scheduled > continuous schedule > Automatic

5. Real-time schedule - Used when data recency is priority. Will be skipped if no resources to run it are available. Realtime_schedule = 1 Continuous schedule - Used when data availability is priority. Will be deferred if no resources to run it are available. Realtime_schedule = 0

6. The .tsidx file ranges from 10% to 110% of the size of the rawdata

7. Common causes of a splunk crash: Splunk/search runs out of RAM - Too many events from the join, append or transaction commands. You can check a diag or the MC for RAM usage. Not enough disk space because of too many search artifacts. Use (index=mydiag source="*systeminfo.txt" "DF output")

8. A fresh forwarder install only needs the deploymentclient.conf file, as it will pull the rest of the apps from the deployment server.

9. When sending data to integrated systems splunk uses the REST api. It can forward all or a subset of data over TCP to other systems, from any point in the logging process without affecting license usage.

10. Notes on migrating from single site clusters to multisite clusters: Multsite policies will only effect new data being indexed. Existing non-clustered buckets will not be replicated and will be phased out over time Existing single site buckets follow existing policies until they age out.

    Multiple questions were on migrating from single instance to multi-site and clustered environments. This slide covers most of it.

11. Compressed Syslog data occupies 50% of its original footprint. 15% for the Rawdata data and 35% for .tsidx files.

12. A deployer uses both the push and pull method. It will push apps to the search heads and can be polled by a restarted SH. KO's are not sent to the deployer, but are replicated.

13. The splunk troubleshooting approach is to Clarify the problem, confirm the problem and submit a case

14. Virtualization reduces performance by 10% - 15% and can be done to all splunk components as long minimum specs are met. Data is sent to splunk cloud by using forwarders and HEC

15. The 2 key benefits to Multisite indexer clustering are: Disaster recovery Stores copies of indexes at multiple sites Search affinity Reduces WAN network traffic.

    Both blanks are 2 words.

16. You can enable a reduction in the size of .tsidx files by adding the timePeriodInSecBeforeTsidxReduction option to indexes.conf. This removes the merged_lexicon.lex file and replaces some index files with mini-index files called .mini.tsidx files which take much longer to search.

    Blanks in the first paragraph share the same answer Final answer is 2 words separated by a dash

17. You can use the monitoring console to get baseline compression rates per index.

18. NAS stands for network attached storage and SAN stands for Storage area networks and is best used in high speed environments.

19. When de-commissioning a site cluster that is no longer in use be aware that all previous single site buckets will be lost.

20. An index file uses the file extension .tsidx Metadata files use the extension .data

21. Splunk uses an "inverted index" structure for the .tsidx files

    2 words

22. Bloom filters are applied to buckets to efficiently structure the data in them and rule out search terms. The have a very fast IO of 1-2

23. Full index files are normally kept with the rawdata until the bucket is frozen.

24. The 3 phases of the splunk deployment process are as follows: Infrastructure planning and building phase: Splunk deployment and data enrichment User planning and roll out

25. Clustering allows the architect to balance growth, speed of recovery, and overall disk usage

26. Accelerations use summaries to speed up reports and data models. By default they use an unlimited amount of disk space and can managed from Settings > Report Acceleration Summaries

27. In the _internal index, splunkd records each child processor event and assigns a component field to enable the ability to filter by component when troubleshooting.

28. Splunk collects resource usage logs and stores them in the _introspection index. If you opt into sharing telemetry data with splunk all collected data will be anonymized.

    There was a lot of questions where knowing what is stored in the _introspection and _internal index was essential.

29. A summary index is created on a search head by running a populating search and storing the results as statistics. The new summary index must be forwarded to an indexer by configuring outputs.conf on the SH, otherwise the other SH's will not have access to them.

30. To disable search affinity you must edit the server.conf file on the search head. You must set site = site0.

    Be familiar with site affinity. Learn the the syntax for site_replication_factor and site_search_factor.

31. Automatic detention happens when the available disk space is less than outlined in the server.conf file option minFreeSpace. It stops all indexing and and replication but allows searches to continue.

32. When a SH becomes clustered changes happen to the members web menu. The Search head clustering menu is enabled on all members and you must now use the deployer to push changes to hidden options.

33. Data source inventory: What data is needed by all Users How much data is generated per day?

34. Data Policy: How long should the data be Stored? What Data needs to be protected from tampering? What can be used for proof of integrity? Will splunk be used as the primary storage for the data?

35. When making changes to a KO, splunk will create a new file called .delta. This file represents changes to an already existing KO. Splunk uses this file for more efficient replication but will replicate the full .bundle file if it fails.

36. Never store configurations in $SPLUNK_HOME/etc/system/local, as the deployment server cannot override system level configurations.

37. The license_usage.log file can only be found on the license master.

38. A search artifact is the results and metadata of a search.

39. Splunk internal logs can be found in SPLUNK_HOME/var/log And have a default size of 25mb before being rolled.

40. By default, the indexer discovery setting is turned off. This means that the site flag will look like "site = site0" The polling_rate is a setting that the master node uses to determine the polling rate. Can be set between 1 and 10.

41. The default phone home interval on a forwarder is 60 seconds and can be configured from the phoneHomeIntervalInSecs attribute found in deploymentclient.conf

    This was a question on the test.

42. The command to initialize a SH cluster member is: Splunk init shcluster-config -mgmt_uri https://SH_IP:8089 -replication_port 9200 –secret shcluster The command to tell the cluster to elect a captain: Splunk bootstrap shcluster-captain –servers_list https://SH2:8089,https://SH3:8089,https://SH4:8089

43. A job is the individual search process kicked off by a search, alert, etc.

44. In a distributed environment a Search head should have 12 GB of ram, a 16 core CPU and 2 10K RPM 300GB SAS drives in RAID 1

45. What command is used when a SH cluster cant elect a captain or a new member can't join? Splunk clean raft

    This is on the test

46. Some advantages of clustering on the indexing tier are: Single sites can have Index replication(HA) Multi-sites can withstand an entire site failing(DR)

47. Data model acceleration summaries are stored in buckets next to the raw data. "maxVolumeDataSizeMB =" is used in indexes.conf to determine retention policies and override the default value of unlimited.

48. Key things to look for when appraising the pre-splunk IT environment: Overall topology Data centers Network zones Number and types of servers Location of users Obtain a network diagram What is in place for Authentication

49. Can you mix clustered and non clustered indexers and search heads? Yes

50. Steps to estimate splunk data volume: Check raw log sizes Check Daily, peak, retained, future volume Number of data sources Number of hosts producing data Add above estimates to a data source spreadsheet. Use metrics from the _internal index to confirm estimates.

51. Replication factor (RF) determines how many copies of rawdata should be available. Search factor (SF) determines how copies of data will be searchable.

52. Some things that should be done in the Infrastructure planning and building phase: Hardware procurement and setup Network configuration install splunk Train Admins Identify data sources forwarder allocation High availability(HA) and Disaster recovery(DR) plan

53. In a distributed environment an indexer should have 12 GB of ram, a 12 core CPU and a disk capable of 800 IOPS. Indexing can use 4 full cores at full load and can index 20MB per second of raw data..

54. Key things to look for when appraising the pre-splunk logging environment: Is there any thing that is already storing logs from multiple sources? SAN/NAS devices Centralized Syslog devices SQL databases What logging tools are already in place? Log parsers or Scripts Query tools? Who uses them? are logs a source for a monitoring system? Is there a ticketing system in place?

55. A deployment plan includes: Deployment goals User roles current topology, both the physical environment and the logging environment. Splunk deployment topology Data sources Data policy Suggested apps Education plan for users Deployment schedule

    This was a question on the test

56. A Master Node is responsible for the following things: Coordinates the replicating activities of the peer nodes points SH's to data Configures peer nodes Tries to fix peer nodes that go offline

    This question was on the test.

57. Networking standards to be in place before spunk deployment: Low latency network 1GB minimum bandwidth under 200ms between SH cluster members under 100ms between indexers NTP in place for time synchronization A DNS service capable of handling the increased load from splunk. THP turned off and increased ulimit settings on all *NIX systems

58. Horizontal scaling happens by adding additional indexers. Vertical Scaling happens by improving indexers and other hardware that is already in place. Horizontal scaling is almost always better/recommended.

    The highlight question is on the test.

59. Mid range specifications on an indexer add 12 more CPU cores for a total of 24 cores. An additional 52GB of ram is also added for a total of 64GB.

    This question is on the test.

60. Additional components of a splunk installation: License master - All instances of splunk are slaved to this component and it determines licensing capacity and usage of all instances. Deployment server - This component "Pulls" (Phones home) clients over the network and sends them updates to their conf files. Master node - manages indexers in its cluster. Deployer - Pushes configurations to Search heads in its cluster.

    Multiple questions on the test regarding this information.

61. You can blacklist large files contained in a KO that do not need to be shared by the indexer.

62. many small buckets are best for speed but can cause problems with stability. In order to prevent high volume indexes from having to many buckets we can increase the size from 750MB to 10GB. We can keep the number of buckets manageable with the stanza maxDataSize = auto_high_volume

63. All search heads in a cluster share: Configurations and apps search artifacts job scheduling This means all cluster members must share identical hardware to prevent crashing. A search head cluster must have a minimum 3 search heads and a deployer.

64. Splunkd components: Processor - Performs small but discrete tasks in a pipeline. Pipeline - A single thread of processors. Pipeline set - a series of pipelines used to process data from input to index.

65. the 5 pipelines are: Input pipeline - parsing pipeline - the data runs through Utf8, linebreaker and header merging pipeline - Uses the aggregator process to identify event boundaries and extract time stamps. Typing pipeline - preforms transformation tasks. index pipeline -

66. Roles created with splunk web are stored in the authorize.conf file. This is not to be confused with the file used by LDAP, authentication.conf, which is case sensitive.

67. Common Monitor Input Errors: Permission issues - Splunk running as a user that doesn't have access to files in the input settings. Network problem - Splunk is unable to monitor files because of a network blockage. Status of monitored files - File confusion or missing data. Lagging input

68. Tools splunk uses to monitor health of files: CRC or cyclical redundancy check - checks the first and last 256 kbs of a file and stores it in the fishbucket index to make sure it is not re-indexed. Tailreader - Shows the number of failed attempts splunk was unable to add data to Queue. Indication of full queues.

69. You can edit advanced configurations for your forwarders in the serverclass.conf file on the deployment server.

70. You can monitor data quality and look for bad event breaking with the Data quality dashboard on the MC.

71. Logs that track pipeline activity can be found in the metrics.log file, along with service loads and job activity. splunk samples introspection data every 30 seconds and Reports the top 10 results per each type to the metrics.log file.

    The second paragraph was a question on the test.

72. Some components have levels of collection filters when running a diag command --exclude "*/dispatch/*" --include-lookups <what> --all-dumps=<bool> --log-age=<days> --index-files=[manifests|full] --index-listing=[light|full] --etc-filesize-limit=<size>

    Wasn't relevant on the test I took, still might want to know it.

73. Notable logs in a distributed search: Splunkd_access.log - located on search peers. Shows splunkd and REST calls. Shows time to response and artifact size. remote_searches.log - Also located on search peers. Contains information on searches run on the Search peers that were requested by a search head. scheduler.log - Stored on the Search head. A record of scheduled activity by the alert scheduler and splunkd.

74. 5 logs to check in the event of a crash: audit.log crash*log (Doesn't always exist) metrics.log splunkd.log web*.log

    Ordered alphabetically. All of these logs are referenced on the test. Know what they hold and where to find them.

75. The Job inspector gives details on a search like: Execution cost - Shows search processor performance Search Job properties - A list of of details on the search e.g. Disk usage, drop count.

    This is a question on the test.

76. The license master should always be forwarding its internal logs to the indexing layer. This does not count against the license and enables us to use splunk to search through them for trouble shooting. License masters can be configured through CLI commands: To add a license - splunk add licenses (License file path) To check the license slaves - splunk list licenser-slaves To switch from standalone to slave - splunk edit licenser-localslave -master_uri https://{LM:Mport}

    The first paragraph is a question on the test and you are asked about the linux commands frequently.

77. It is best practice to have a standby master node to switch to the same master_uri in case the primary master node fails. After a master node fails and is brought back online the buckets in the cluster are re-balanced When a master node restarts it will block indexing if the cluster does not meet replication factors. To fix this, always run the command "splunk set indexing-ready" every time the master node is restarted.

    First paragraph is on the test. Second paragraph is good to know.

78. A master node can only be 3 minor versions later than search heads and peers at most but should be the same if possible. It can't run an earlier version. The same rule applies to the search heads. All peers must be on the same version.

    This question is on the test.

79. When converting a standalone indexer to a cluster the following things happen: Apps in the master-apps folder on the master node are redistributed. Only the new data coming in is replicated while existing buckets are not.

    This info is referenced multiple times on the test.

80. In clustering, all indexes.conf changes must be made through the master node by placing the new conf file in the splunk_home/etc/master-apps folder. Ensure that each index stanza in the indexes.conf file has repFactor option set to auto to make the index replicate.

81. During a rolling restart, all searches that take longer than 180 seconds are retried so the indexer can do a proper restart. The "splunk offline --enforce-counts" is the recommended command used to bring down an indexer peer. The indexer will only shutdown if doing so will maintain RF and SF.

    The highlighted cmd option is a question on the test.

82. In order to connect a SHC to an indexer cluster you must run the following command: splunk edit cluster-config -mode searchhead -master_uri https://10.0.1.3:8089 -secret idxcluster For multi-site clusters you must specify all sites.

83. A diag file gathers OS settings, internal logs and conf files and produces a diag.log file. No customer data is retrieved and it can be used as a backup of conf files.

    Multiple questions on diags, Make sure you know this slide.

84. An ES deployment requires its own dedicated SH. The ES SH and Indexer must have the minimum specs of a 16 core CPU and 32 GB of ram. One indexer can only index 100 GB of ES data daily and that is assuming it is running 15 correlation searches.

85. In a scenario where an index houses High volume and low volume data, searching the low volume data will take longer than necessary. You can create a separate index for the high volume data to improve search performance on the low volume data.

    This is almost word for word on the test.

86. Artifact reaping takes place when a search artifact expires and is deleted by the member that created it. The captain is notified and deletes all copies.

87. In order to change SHC captains to a preferred SH run: splunk transfer shcluster-captain -mgmt_uri <new_captain>

    This is a question on the test.

88. When configuring a search head to be in a multisite cluster the following must be done: Enable the SH to be in a cluster by using the splunk edit cluster-config command. Add the SH to a cluster using the splunk add cluster-master command. You can edit configuration setting on the SH using the splunk edit cluster-master command

    Know these commands.

89. The second half of the cluster admin PDF is not on these slides. Additional things to study: Know the splunk.d process inside and out.

91. 3 things cause primary re-balancing in a clustered site: A peer node joining the cluster The master node rejoins the cluster After a rolling restart completes

    This is a question on the test.

92. Helpful searches when trouble shooting errors in splunk: Where is the forwarder trying to send data to? index=_internal host=<uf> sourcetype=splunkd StatusMgr destHost What output Queues are set up? index=_internal host=<uf> source=*metrics.log group=queue tcpout | stats count by name

    These searches are featured on the test.

93. The license master should always be forwarding its internal logs to the indexing layer. This does not count against the license and enables us to use splunk to search through them for trouble shooting. License masters can be configured through CLI commands: To add a license - splunk add licenses (License file path) To check the license slaves - splunk list licenser-slaves To switch from standalone to slave - splunk edit licenser-localslave -master_uri https://{LM:Mport}

    The first paragraph is a question on the test and you are asked about the linux commands frequently.

94. To add a search head to an already existing cluster you can use the following CLI command on the search head: splunk add cluster-master

    This command is referenced on the test.

95. In a indexer cluster, the number of peer nodes should be equal to the Replication factor at minimum. Best practice is to have equal number of peer nodes plus 1. When calculating storage for disk storage, always multiply rawdata total by RF and index data total by SF