http.Request doesn't allow URL with spaces inside of it

[wrussell1999]

Describe the issue

This user wanted to make a HTTP Request using the variable api_url which contains spaces.

id: nyc-ems-calls-ingestion
namespace: nyc-ems-calls-namespace

variables:
  start_date: "{{ now() | dateAdd(-10, 'YEARS') | date('yyyy-MM-dd') }}"
  api_url: "https://data.cityofnewyork.us/resource/76xm-jjuj?$where=incident_datetime between '{{ render(vars.start_date) }}T00:00:00' and '{{ render(vars.start_date)}}T12:00:00'&$limit=50000" 

tasks:
  - id: check_url
    type: io.kestra.plugin.core.log.Log
    message:
      - "{{render(vars.api_url)}}"
  
  - id: python
    type: io.kestra.plugin.scripts.python.Script
    script: |
      import pandas as pd
      import requests
      import json 

      url = "{{render(vars.api_url)}}"
      r = requests.get(url)
      df = pd.DataFrame(json.loads(r.text))
      print(df.shape)
    beforeCommands:
      - pip install requests pandas
  
  - id: api
    type: io.kestra.plugin.core.http.Request
    uri: "{{ render(vars.api_url) }}"

Now this is a valid request when put into a browser and works with the python task however Kestra threw an error because of the space:

Illegal character in query at index 73: https://data.cityofnewyork.us/resource/76xm-jjuj?$where=incident_datetime between '2015-01-24T00:00:00' and '2015-01-24T12:00:00'&$limit=50000

It works if you replace the spaces with %20 instead. Can we have the plugin do something like this automatically if the spaces exist like a browser does?

Environment

• Kestra Version: latest

[deepPublicGit]

Which plugin does this belong to?

[wrussell1999]

Which plugin does this belong to?

The core plugins

[Ben8t]

Now this is a valid request when put into a browser and works with the python task

I think both browser engine and Python request lib encode the url properly on the backend. In Kestra you can use the urlencode Pebble function to manage such case properly

[anna-geller]

let's explore during the cooldown if we can do that under the hood

[anna-geller]

@loicmathieu would be cool if you could have a look during the cooldown. the question here is: could we check within the HTTP plugin if the URL contains spaces and if so, replace each space with %20

[loicmathieu]

It's a bug, URL must be URL encoded, not a cooldown investiguation.