Splunk ♥ API’s

API stands for Application Programming Interface. It is a way of manipulating an application through other applications. This manipulation could involve changing, creating or deleting items in the application, but it can also be used as a way of extracting data from an application. This makes it possible to interface applications with each other and have them use each other’s data.

Perhaps needless to say, Splunk loves APIs. Splunk is based on the central collection of data and APIs form a large part of this data. Splunk itself is built up of APIs; Splunk Web is built on the APIs of Splunk Core.

Splunk Core is an interesting item for a blog post in itself, but today I’d like to talk about connecting APIs to Splunk.

API flavours

APIs come in many different flavours.

  • Web Service API’s (SOAP, XML-RPC, REST)
  • WebSocket API’s
  • Library-based API’s (JavaScript)
  • Class-based API’s (Java, Android, .Net)
  • Object Remoting API’s (CORBA, .Net Remoting, RMI)

Today we will be concentrating on WebService APIs because they help us to send data from application A to application B.

SOAP and REST

The most popular WebService APIs by far are SOAP and REST. The strength of these APIs is found in the fact that applications can communicate with each other in various languages on a protocol (HTTP(S)) that has been broadly rolled out and makes it possible to communicate online.

SOAP was designed in 1998 and stands for Simple Object Access Protocol. This is a way of communicating with applications via XML over HTTP(S). All applications that allow SOAP specify which services and methods other applications can call on in their WSDL (Web Service Description Language).

REST is the trendy newcomer, and stands for Representational State Transfer. Although REST dates back as an architecture to 1999, it has undergone huge growth since the emergence of cloud-based services. REST also works over HTTP(S) and can return data via XML or JSON.

The big difference between SOAP and REST is that with SOAP application A indicates in the body of the HTTP(S) message which service, methods and parameters are to be carried out by application B. With REST we use an HTTP GET or HTTP POST, which means that the information about the service, methods and parameters is stored in the HTTP URI or HTTP Headers. This makes it possible for application B to determine on the basis of the application how the data is to be returned, after which application A can process this data itself.

Splunk and Web Service API’s

As I mentioned in the introduction, Splunk is the best solution for centrally collecting data from several sources. Connecting Splunk to a Web Service API is therefore a frequently used way of making this possible.

65037606Python all-the-things

Many splunkbase add-ons make use of APIs to collect and present data for specific solutions. These add-ons provide input configuration and Python scripts to communicate with the APIs.

Python can be extended with a REST client or a SOAP client to communicate with Web Service APIs. These clients make it very easy to quickly write an add-on that can communicate with other solutions.

Modular Inputs

Modular Inputs are a way of creating custom Data Inputs in Splunk. Splunk treats these custom inputs as native inputs so that users can easily configure inputs via Splunk Web.

The benefits of Modular Inputs:

  • Management via Splunk Web
  • Management via Splunk REST API 🙂
  • Validation on your inputs
  • Platform-specific versions in your add-on are possible
  • Stream data as XML so you can carry out advanced data processing
  • Permissions on inputs possible

Splunk has already made its own Modular Input for REST-based APIs. You can download this add-on from splunkbase.

Splunk-REST-modular-input

This add-on makes it possible to connect applications in your IT landscape to Splunk if no specific add-on is available for them. This makes everything a good deal easier for Splunk managers without any knowledge of Python.

splunk-json-fieldsWeb Service data processing

Splunk likes data whether it is formatted in XML or JSON. It is easy to parse the structure in such a way that the right fields are automatically displayed in Splunk. Set the KV_MODE in your props.conf to XML or JSON and Splunk will automatically extract the fields.

 

 

 

Conclusion

It will be clear from above that the Web Service APIs have made the world a good deal easier. Certainly when it comes to Splunk. It is much easier to collect and process JSON or XML data than raw log files.

I expect the integration with SOAP based applications to decline in the favour of REST APIs. This is a good development since SOAP applications in Splunk generally still run via Python scripts, which are harder to manage than REST Data Inputs.

Many of my clients start by considering splunkbase for integration with another solution. If no add-on is available they ask me to build one for them. If there is a REST API I can configure the right Data Inputs in consultation with the client and the client has access to the data within a single day. It takes roughly between 5 to 10 days to design a SOAP interface, depending on the number of services and the methods to be called on.

If you would like to know more about Splunk and API integration feel free to contact me or leave a message. Either way, I hope that I have been able to add to your knowledge about the options and the easy method you can use to onboard Splunk data.

About Coen Meerbeek

Splunk consultant @ Blue Factory, eigenaar en oprichter @ BuzzardLabs, basketbalspeler en Xbox-gamer. Lees meer van Coen op Launchers.nl en Twitter.

Speak Your Mind

*