converting string to timestamp...

[dss010101]

Im using the following to parse log times: %{TIMESTAMP_ISO8601:log_time}

The result looks like this: "2023-05-16 17:50:49,565"

I have two questions here...i am tryng to convert this value to timestamp to compare it with the ingestion time (.timestamp). The following code is not working for me...

    if .log_time != null{
        .log_time = replace!(.log_time, ",", ".")
        .log_time, err = to_timestamp(.log_time)
        if err != null {
            log("ERROR converting log_time to time: " + err, level: "error")
        } else {
            .timestamp = .log_time
        }
    }

Exception:
function call error for "to_timestamp" at (655:678): No matching timestamp format found for "2023-05-16 17:50:49.565" internal_log_rate_secs=1 vrl_position=716

How would i convert this string to timestamp?
Also, how can i do a diff on two timestamps? i would like to assign .timestamp to .log_time if the difference is more than say a a few minutes.

[dss010101]

came across parse_timestamp and trying this modification...

    if .log_time != null{
        t , err = parse_timestamp(.log_time, "%Y-%m-%d %H:%M:S%,%.3f")
        if err != null {
            log("ERROR converting log_time to time: " + err, level: "error")
        } else {
            .timestamp = t
        }
    }

but getting this error:
[ERROR converting log_time to time: function call error for "parse_timestamp" at (603:655): Invalid timestamp "2023-05-16 17:50:49,565": input contains invalid characters]

[dss010101]

had a typo in my format string, should be: %Y-%m-%d %H:%M:%S,%.3f

after that change, seems to works...but only label that shows up in loki/grafana is .log_time now, not the other labels im parsing. Also, the ingestion time showing is still the same, and does not match the log_time. Im guessing then .timestamp is not the value that is displayed as the ingestion time in loki+grafana? how would i change that time if i wanted to?

[dss010101]

anyone have any idea why , without the line below that simply re-assigns the timestamp field, i get all my parsed fields showing in loki/grafana, but with it, ONLY the log_time field apparently is shown.
If i do a 'log(.)' before and after, i see all the fields in both cases. so frustrating...

source = '''
    . |= parse_groks!(
            .message, patterns:[
                s'\[%{TIMESTAMP_ISO8601:log_time}\]\[%{INT:pid}:%{INT:tid}\]\[%{LOGLEVEL:log_level}\]\[%{WORD:module}.%{NOTSPACE:func}\] run_id:%{NOTSPACE:run_id} %{GREEDYDATA:msg}',
                s'\[%{TIMESTAMP_ISO8601:log_time}\]\[%{INT:pid}:%{INT:tid}\]\[%{LOGLEVEL:log_level}\]\[%{WORD:module}.%{NOTSPACE:func}\] %{GREEDYDATA:msg}',          
                s'\[%{TIMESTAMP_ISO8601:log_time}\]\[%{INT:pid}:%{INT:tid}\]%{LOGLEVEL:log_level}:%{WORD:module}:%{GREEDYDATA:msg}'
            ]
    )
    .timestamp = parse_timestamp!(.log_time, "%Y-%m-%d %H:%M:%S,%3f")
'''

[jszwedko]

My guess is that the parse_timestamp! call is failing and so you are receiving the original event. I'd suggest something like:

timestamp, err = parse_timestamp(.log_time, "%Y-%m-%d %H:%M:%S,%3f")
if err != null {
  log("could not parse timestamp + " " + .log_time + " using format: " + err)
} else {
  .timestamp = timestamp
}

This would let you see which timestamp values are invalid.

[dss010101]

My guess is that the parse_timestamp! call is failing and so you are receiving the original event. I'd suggest something like:

timestamp, err = parse_timestamp(.log_time, "%Y-%m-%d %H:%M:%S,%3f")
if err != null {
  log("could not parse timestamp + " " + .log_time + " using format: " + err)
} else {
  .timestamp = timestamp
}

This would let you see which timestamp values are invalid.

getting a couple of errors:

| 12 │         .timestamp = timestamp
|    │                      ^^^^^^^^^
|    │                      │
|    │                      this identifier name is reserved for future use in the language
|    │                      use a different name instead

So i changed it to just 't': then getting the below:

| error[E630]: fallible argument
|    ┌─ :10:13
|    │
| 10 │         log("could not parse t" + " " + .log_time + " using format: " + err)
|    │             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|    │             │
|    │             this expression can fail
|    │             handle the error before passing it in as an argument

guessing because .log_time may be null, so finally changed the condition to this:

    t, err = parse_timestamp(.log_time, "%Y-%m-%d %H:%M:%S,%3f")
    if err != null {
        log("could not parse t" + " " + err )
    } else {
        .timestamp = t
    }

But i still see the behavior - with the condition, only .log_time shows in list of labels. without it, i see my other parsed labels