Introduction
WatchEye - Line Graph is a free product that brings true 2D and 3D rendering to the Splunk platform.
This is an initial proof of concept and will be updated in future with new features, including support for SQL Server queries, maybe other data sources too, and perhaps some other small tweaks to enhance the chart experience.
This product is best suited to be displayed on large screens and will support resolutions up to 4K with crystal clear quality. The software is extremely simple, it runs uninterrupted, very smoothly, and recovers automatically from errors.
We hope you have a great time experiencing something that has not been achieved before, and we hope that you benefit from the innovation this realtime chart has to offer.
Download & Installation
Go to the downloads page to obtain the installer. Apologies this is not code signed at the moment, it will be soon!
Once installed, do not run the software, jump to the next section.
Quick Start for Ninja's
- Open up the config.txt file. (root application install path under \LineGraphRT_Data\config.txt)
- Edit SPL_QUERY_TEXT and set to your stats query that will return your label in the left column, and value in the right column:
- Add "search" before this in your config, e.g. "SPL_QUERY_TEXT=search index=_internal | stats count by log_level"
- Ensire that SPL_REST_PROTOCOL=https
- Set SPL_REST_HOST_ADDR=your.search.head.fqdn
- Set SPL_REST_USERNAME=your.username
- Set SPL_REST_PASSWORD=your.password
- Optionally, set SPL_UTC_TZ_OFFSET_SECONDS to be the timezone offset in your current location, in seconds. If you are in the UK, set this to -3600 now to match BST. Otherwise, set to 0 to use your local system time for querying Splunk.
- Run the program.
- If any issues, check the logFile.txt. (root application install path under \LineGraphRT_Data\logFile.txt)
- Read-on for further information on how to tune the chart.
SPL Query Examples
The format of the Splunk query must follow a specific route which is fully explained here.
An example that will work on all splunk instances, where we query the _internal index for activity based on log_level:
search index=_internal sourcetype=splunkd | search log_level="*" | stats count by log_level
Note how the above query starts with "search ".
Here are some more examples, the first one will work if your user has very limited permissions (See below about this in the Recommendations section) as this does not require access to any indexed data:
| SPL |
Description |
| | makeresults count=10 | streamstats count as value | eval value = case(value=1, 1, value=2, 2, value=3, 3, value=4, 4, value=5, 5,value=6, 6,value=7, 7,value=8, 8,value=9, 9,value=10, 10) | eval Code = case(value=1, "Line1", value=2, "Line2", value=3, "Line3", value=4, "Line4", value=5, "Line5",value=6, "Line6",value=7, "Line7",value=8, "Line8", value=9, "Line9", value=10, "Line10") | table Code, value |
This query will draw 10 lines with a value of "1" apart. This query does not rely on access to any indices. Note that because it is not a standard search, it does not required the "search" prefix. |
| | tstats summariesonly=true allow_old_summaries=true count from datamodel=Authentication.Authentication by Authentication.action |
This is using a tstats query. Note that because it is not a standard search, it does not required the "search" prefix. |
| search index=windows | stats count by EventCode |
Note how the usual searches "index=" contain the word "search" at the beginning, this is required. After developing such searches in Splunk, add "search " when using with this app. |
Populate your SPL_QUERY_TEXT key in the config with your search. Ideally, all searches results should be in a table format with your key/series name to the left and your values to the right...
The table headers / field names do not have to be strictly named, keep it simple however and name them as you wish.
Configuration & Running
The configuration file is located in the root application install path under \LineGraphRT_Data\config.txt
The default values in the config are to run over a 60 second chart length and update (plot points) every 5 seconds, using http aimed at the localhost, and a simple date & time mask of HH:MM:ss.
All of the configuration items are detailed below with their in-line defaults:
| Key |
Default Value |
Description |
| SPL_QUERY_TEXT |
[empty] |
This is your SPL Query |
| SPL_TIME_WINDOW_SECONDS |
5 |
This is the time window, in seconds, that each SPL query dispatched to Splunk, will abide to.
This should match CHART_ANIMATION_TIME.
|
| SPL_REST_PROTOCOL |
http |
When querying Splunk, use http or https scheme.
This is very important, if you are using self-signed certificates or splunk free edition, then this value will likely need setting to http.
See later in this section for more details on this situation.
|
| SPL_REST_HOST_ADDR |
localhost |
This is the FQDN, IP address or host name of the Splunk instance which is acting as a search head.
If using https as protocol, this MUST match the endpoint name. For example
|
| SPL_REST_USERNAME |
admin |
The splunk username of the account used for the API communications.
This needs to be "admin" when using Splunk Free license.
|
| SPL_REST_PASSWORD |
[empty] |
The password of the Splunk account.
This needs to be empty when using Splunk Free license.
When this value is populated, upon program start, the password will be re-written in to the file as an encrypted password, this will keep this secure.
Note: As of now, we are not supporting tokens, but given requests or concerns, we will in future!
|
| SPL_UTC_TZ_OFFSET_SECONDS |
0 |
Your TZ Offset value in seconds. The chart operates in the UTC zone. This is explained later in this document.
Example: at time of writing (August, 16:30) , our location is in the UK which is currently under BST.
This value therefore has to be set to -3600 as the UTC time is currently 15:30 (1 hour behind). The chart shows the UTC zone, not the BST zone
You can also use this value for event lag and time skewing - a use case where this might apply is where your data models aren't immediately up to date, or events are arriving later than expected! In such a scenario, you would want to negate this value.
For Data Models - negate an extra 300 seconds (Splunk default is 5 minutes). In the case for BST, the value would be -3900.
|
| CHART_SLIDING_WINDOW_SECONDS |
60 |
This is the "length" of the chart in seconds.
To play this safe, ensure that this value is a multiple of SPL_TIME_WINDOW_SECONDS.
Example: CHART_SLIDING_WINDOW_SECONDS=120 & SPL_TIME_WINDOW_SECONDS=12 means that a factor of 10 exists.
But if CHART_SLIDING_WINDOW_SECONDS was 111 then we would have remainders when dividing by 12 (9.25 is not an integer).
Please ensure that CHART_SLIDING_WINDOW_SECONDS / SPL_TIME_WINDOW_SECONDS == a solid integer.
|
| CHART_ANIMATION_TIME |
5 |
This is the time in seconds, that the chart will spend animating the drawing of lines between points.
This value and SPL_TIME_WINDOW_SECONDS should be the same, if they are not then the chart will start to appear to behave differently.
If you choose to mis-match this value with SPL_TIME_WINDOW_SECONDS, then do so on the basis you understand this value and what it does.
|
| CHART_LINE_SIZE |
4 |
The thickness of the line drawn between the points. |
| CHART_POINT_SIZE |
5 |
The radius of the points drawn between the lines. |
| CHART_SMALL_TITLE |
default small title text |
This will set the "top left corner" interactive title.
When points are hovered over on the chart with the mouse, this value will be over-written with the value of that point.
|
| CHART_BIG_TITLE |
default big title text |
This will set the main title of the chart. |
| CHART_DATE_FORMAT |
H:mm:ss |
Advanced setting, strict requirements laid out here.
This setting will change the format of the date & time on the horizontal axis and should not need to be changed.
|
| DEBUG_LOGGING |
false |
This will enable debug logging in the log file located at \LineGraphRT_Data\logFile.txt.
Do not keep this enabled for long, only use this when instructed by us on a support basis, or if you feel that you are able to tackle a problem.
When debug logging is not enabled, sufficient information is provided in the default logs to be able to ascertain an issue.
|
If you're using free splunk, skip this part and go to "With Splunk Free License".
To get started with configuration, ensure that you have created a username and password in Splunk for this usage only, this will need to be splunk native and this is what will go in to your configuration file. An existing (and default) configuration file also exists in \LineGraphRT_Data\config.sample.txt.
To create a user that has access only to the Splunk HTTP API, create a bare role with no inherited roles, give it the "search" capability only and allow it access only to the indexes you want it to:
Give your new user this role. Remove the default "user" role.
For Splunk Cloud
This is much easier and in most cases, works straight out of the box.
All your need to do is ensure that SPL_REST_PROTOCOL=https and SPL_REST_HOST_ADDR=your-search-head.address.splunkcloud.com.
Everything else is done for you, Splunk maintain the certificates on the server and they are valid, issued by a public CA!
If this does not work then contact support, there may be a setting you need them to lift, it can be the case that the HTTP REST API does not work immediately out of the box, or it has been bolted down. Click here to know more.
For On-premises Splunk Enterprise (Or Trial Version)
Your on-premises Splunk instance must be using a certificate that is trusted by the client machine where you will run this software from, and this is the Splunkd port (8089) - not the web, HEC or data ports.
We are not offering any support here to ensure that this is in place and we do not recommend disabling SSL on the splunkd port for a live production cluster.
It may be possible to install the Root CA used by the search heads under your Trusted Root Certificate Authorities, but we can not comment further on that as we do not know what PKI you have in place - we also do not offer bad practice advice and strongly oppose the default certificates shipped with Splunk.
Ensure that you configure SPL_REST_PROTOCOL=https and SPL_REST_HOST_ADDR=your-search-head.internal.address. (The DNS name of the server and Subject Alternate Name of the cert need to match - do not use the IP address)
| Configuration Key |
Value |
| SPL_REST_PROTOCOL |
https |
| SPL_REST_HOST_ADDR |
your-search-head.internal.address |
| SPL_REST_USERNAME |
[your rest username] |
| SPL_REST_PASSWORD |
[the password]
|
You're now good to go!
With Splunk Free License
It is not typical to configure an SSL certificate on splunk free edition - though some people do, we do here!
An instance of splunk free will no doubt be running on a box somewhere in the network, or most likely on a desktop/laptop, and is limited to 500MB/day, has no authentication, and does not allow for remote access or any of the fancy bells and whistles you find with a full enterprise license (DMA / Alerting).
Therefore, we recommend that SSL/TLS is disabled on the Splunkd port. How to do this ...
We will want to change a setting in server.conf under the [sslConfig] stanza.
Your server.conf will be located in either C:\Program Files\Splunk\etc\system\local\server.conf OR /opt/splunk/etc/system/local/server.conf. You may want to back this up first before making any changes.
We need to add the key value pair "enableSplunkdSSL=false" - and comment out anything else under the [sslConfig] stanza.
It may contain references to either "serverCert" and/or "sslPassword". This is similar to how it would look (in notepad) when you've made your changes:
Restart splunk. After re-starting, "sslPassword" will likely be re-created & re-written under this stanza to the default splunk one. Leave this as it is.
To revert these changes, either restore the backup, or simply delete the new "sslPassword" line, set enableSplunkSSL=true, uncomment/restore your original lines, and restart.
You are now ready to configure the client, use the following settings to do this:
| Configuration Key |
Value |
| SPL_REST_PROTOCOL |
http |
| SPL_REST_HOST_ADDR |
IP, hostname or FQDN of instance, 127.0.0.1 or localhost if this is running locally. |
| SPL_REST_USERNAME |
admin |
| SPL_REST_PASSWORD |
[leave blank] |
There is very little that can go wrong here and things should start working straight away, the reason for this is that the free edition is watered down and permissions and accounts are not used.
You can run the app once you have decided upon your query!
Strong Recommendations & Objections (Best Practices)
We do not advise the following:
- Using self-signed certificates.
- Disabling TLS/SSL on production clusters or any remotely accessible platform.
- Using plain HTTP over the internet.
- Creating admin accounts/roles for HTTP REST API purposes.
- Testing/Trialing this software in production - use a free license and follow our guide to doing this.
- Making a free instance available to non-local nodes.
We recommend the following:
- For all editions of Splunk, use TLS for all ports, there are no restrictions between them, certificates can be obtained for free from Public CA's such as ZeroSSL, these are the easiest.
- Accounts created have minimal privileges to conduct their operations.
- Accounts are regularly audited, especially API accounts.
- The use of strong passwords, and password rotation.
- Optimal SPL code.
These best practices are simply to cover our backs - we do not want to be held responsible for a security breach at your organisation.
The password in the configuration file is stored encrypted when the program starts.
Splunk can not be directly queried from this application.
Issues, Limitations & Notable Information
DECLARATION
There are no identified issues right now - however we are always interested in hearing about bugs.
The software has been thoroughly tested against a V9 free license, a trial license, splunk cloud and also a developer license.
We've provided as much information as we can in this manual, Contact Us if you have any questions, requests or comments.
The software does not auto-update or report back to us, it is safe to place this software behind a blocking egress firewall.
New additions, tweaks, bug fixes etc, are all provided ad-hoc and not to any schedule, keep in tune with our blog to see work in progress, announcements or anything else related.
This product "WatchEye - Line Graph" is free now and will remain free throughout it's lifetime. If a chargeable product comes to market, we will likely give it another name.
We have not (yet) code signed the installer or the executables - we have provided the SHA of the installer on the download page, and this is sufficient for now. If this software has been obtained from anywhere else, this is a breach of the terms, please report this to us immediately.
As things can heat up quickly in the SIEM & Cyber space since a number of new players have appeared, both big and small, fallouts have happened and criticism is contagious in tech. We strongly defend our reputation, we ask that you assist us with this - please report to us any concerns you have with any negative rumours, bad reviews etc about this software or our business. Your report will be valued greatly and handled with strict confidence.
Limitations
The line graph has the following limits (*some of which will be resolved in future updates):
- Does not support proxy configuration.*
- Does not support negative values.*
- Does not support decimal places.*
- Behaves only in 1 way - plot points in near-real-time on an auto-scrolling basis.
- Does not scroll by user interaction.*
- Is a fixed dimensional space that only adapts horizontally based on what the user specifies before launch (the time window), and vertically based on the value maximum.
- Upon launch, the chart will not move at all if there is no data, and seem as if it is stuck.*
- Maximum of 10 axes/series.*
- The labels for the axes are able to over-write each other if too long.
- Consideration has to be made when performing tstats searches, as these are not as real-time as ingested data and are delayed for up to 5 minutes by default.
- It will only run on windows.*
- To get the most out of this you may need a farily decent graphics card and DirectX.
- If 10 axes (labels/lines) are exposed over time, the current behaviour is to keep them visible and in memory, therefore not remove them once the series has not been seen for a certain amount of time.*
- The application can run multiple instances if it is copied locally, due to the configuration file being singular.*
Some extra information: The software notifies-of and auto recovers from errors, like communications outages. Search delays etc would be obvious but the chart just keeps going and never stops.
FAQ
Questions of interest, or that people often ask, will end up here.
Troubleshooting
DEBUG_LOGGING is a key that can be set in the config.txt, to true or false. Default is false.
Even without this variable set, there is enough info provided to a user in order to determine the issue - which will mostly be API permission/account login issue or network connectviity problems.
We've yet to hear from anyone about any issues.
Obtaining Additional Support
Contact Us.
Please be aware - if the problem you are asking about is resolvable by this page, we may not respond.
If the problem is resolvable via the FAQ list then do not expect a response.
You are welcome to make feature requests or suggestions.