I wish to collect data from different endpoints every five minutes, this is an easy way to collect near live data to our Splunk environment. I just have a simple script that looks for changes within the last period, with a three minute buffer.
The problem is that the modificationDate-timestamp in the endpoints (which is the latest timestamp available to filter by), is not added when the event is made available at the enpoint. In many cases, the events are made available in the API a long time after the modificationDate. This results in a lag where I have to have a big buffer on my time periods, and in the worst case loose data because events was added retroactively (according to the latest timestamp available).
My suggestion is that you add another timestamp which describes when the event was written to the API endpoint. This way you can retrive data based on when it was available and know that you have collected all the data.
Guest
Dear customer,
Thank you for posting this idea and sorry for my late reply.
Unfortunately, we will not be able to implement your idea. Today our system is asynchrones, that means we have a small delay between the data reception and registration. Implementing your idea would necessitate to update the event while being processed and that would impact consequently the performances of our system along with engaging a lot of ressources for development.
We are all sorry for the inconvenience it may cause.
Best,
Emmanuelle
Still hoping this can be implemented :)
Any status on the request?