Using approle for Vault secret source in JCasC results in 403

[imalobster]

Jenkins and plugins versions report

Environment

Jenkins: 2.416
OS: Linux - 5.15.46-Unraid
Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)

ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5
bootstrap5-api:5.3.0-1
bouncycastle-api:2.29
branch-api:2.1122.v09cb_8ea_8a_724
build-timeout:1.31
caffeine-api:3.1.6-115.vb_8b_b_328e59d8
checks-api:2.0.0
cloudbees-folder:6.815.v0dd5a_cb_40e0e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.10.0-68.v0d0b_c439292b_
config-file-provider:952.va_544a_6234b_46
configuration-as-code:1670.v564dc8b_982d0
credentials:1271.v54b_1c2c6388a_
credentials-binding:631.v861c06d062b_4
display-url-api:2.3.8
durable-task:513.vc48a_a_075a_d93
echarts-api:5.4.0-5
email-ext:2.100
font-awesome-api:6.4.0-2
git:5.2.0
git-client:4.4.0
git-parameter:0.9.19
github:1.37.2
github-api:1.314-431.v78d72a_3fe4c3
github-branch-source:1732.v3f1889a_c475b_
gradle:2.8.2
hashicorp-vault-plugin:360.v0a_1c04cf807d
htmlpublisher:1.32
instance-identity:173.va_37c494ec4e5
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.15.2-350.v0c2f3f8fc595
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.8-1
jjwt-api:0.11.5-77.v646c772fddb_0
jquery3-api:3.7.0-1
junit:1217.v4297208a_a_b_ce
ldap:682.v7b_544c9d1512
mailer:463.vedf8358e006b_
matrix-auth:3.1.10
matrix-project:802.v8013b_40c7edc
mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_
mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_
okhttp-api:4.11.0-157.v6852a_a_fa_ec11
pam-auth:1.10
pipeline-build-step:505.v5f0844d8d126
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:671.v07c339c842e8
pipeline-input-step:477.v339683a_8d55e
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2144.v077a_d1928a_40
pipeline-model-definition:2.2144.v077a_d1928a_40
pipeline-model-extensions:2.2144.v077a_d1928a_40
pipeline-rest-api:2.33
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40
pipeline-stage-view:2.33
pipeline-utility-steps:2.16.0
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.3.0
resource-disposer:0.23
scm-api:676.v886669a_199a_a_
script-security:1264.vecf66020eb_7d
snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4
ssh-credentials:308.ve4497b_ccd8f4
ssh-slaves:2.916.vd17b_43357ce4
sshd:3.312.v1c601b_c83b_0e
structs:324.va_f5d6774f3a_d
timestamper:1.26
token-macro:359.vb_cde11682e0c
trilead-api:2.84.v72119de229b_7
variant:59.vf075fe829ccb
workflow-aggregator:596.v8c21c963d92d
workflow-api:1251.vd4889a_b_0a_065
workflow-basic-steps:1042.ve7b_140c4a_e0c
workflow-cps:3731.ve4b_5b_857b_a_d3
workflow-durable-task-step:1278.v94b_dc2b_50c6f
workflow-job:1316.vd2290d3341a_f
workflow-multibranch:756.v891d88f2cd46
workflow-scm-step:415.v434365564324
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:848.v5a_383b_d14921
ws-cleanup:0.45

What Operating System are you using (both controller, and any agents involved in the problem)?

Unraid v6.10.3
Jenkins running as container built from jenkins/jenkins:latest

Reproduction steps

1. Create (or use existing) AppRole in Vault instance that can read a KV secret for testing (in my case using KV1 engine)
2. Write a JCasC file that relies on values to be substituted by a secret source (see attached jenkins,yml for example)
3. Write a Dockerfile to build custom Jenkins container image that includes Hashicorp Vault Plugin (see attached Dockerfile for example)
4. Include the following ENV values in the Dockerfile so that the Hashicorp Vault secret source is used:
   ENV CASC_VAULT_PATHS <path_to_kv_secret>
ENV CASC_VAULT_ENGINE_VERSION 1
ENV CASC_VAULT_APPROLE <approle_role_id>' 'ENV CASC_VAULT_APPROLE_SECRET <approle_secret_id>
ENV CASC_VAULT_URL <vault_url>
5. Build the container image
6. Run the new container image
7. Observe Jenkins start up log
   Dockerfile.txt
   jenkins.yml.txt

Expected Results

JCasC uses the CASC_VAULT_APPROLE and CASC_VAULT_APPROLE_SECRET environment variables to successfully authenticate to HC Vault and retrieve the key-value secret pairs at the path(s) specified. If successful, Jenkins start up logs do not report any details.

Actual Results

2023-08-08 10:18:14.416+0000 [id=35] WARNING c.d.j.v.j.s.VaultSecretSource#init: Could not authenticate with vault client
com.bettercloud.vault.VaultException: Vault responded with HTTP status code: 403
Response body: {"errors":["permission denied"]}

    at com.bettercloud.vault.api.Auth.loginByAppRole(Auth.java:524)
    at com.datapipe.jenkins.vault.jcasc.secrets.VaultAppRoleAuthenticator.authenticate(VaultAppRoleAuthenticator.java:25)
    at com.datapipe.jenkins.vault.jcasc.secrets.VaultSecretSource.init(VaultSecretSource.java:253)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
    at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:778)
    at io.jenkins.plugins.casc.ConfigurationAsCode.configureWith(ConfigurationAsCode.java:652)
    at io.jenkins.plugins.casc.ConfigurationAsCode.configure(ConfigurationAsCode.java:314)
    at io.jenkins.plugins.casc.ConfigurationAsCode.init(ConfigurationAsCode.java:306)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:109)
    at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:185)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
    at jenkins.model.Jenkins$5.runTask(Jenkins.java:1166)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)

2023-08-08 10:18:14.497+0000 [id=35] WARNING i.j.p.c.SecretSourceResolver$UnresolvedLookup#lookup: Configuration import: Found unresolved variable 'PAT'. Will default to empty string

Anything else?

If I exec into the running Jenkins container and manually call Vault to generate a token using the CASC_VAULT_APPROLE and CASC_VAULT_APPROLE_SECRET environment variables, I am able to successfully read the secrets at CASC_VAULT_PATHS. Therefore it doesn't seem to be blocked by the AppRole permissions to secrets at this path nor blocked by any bound CIDRs.

Additionally, if I generate a token using the AppRole role-id and secret-id values, and instead use this generated token in my Dockerfile via CASC_VAULT_TOKEN (instead of CASC_VAULT_APPROLE and CASC_VAULT_APPROLE_SECRET), then it works as expected.

[paul-e-allen]

I am also seeing similar behavior. I use CASC_VAULT_APPROLE and CASC_VAULT_APPROLE_SECRET and get the same 403 error. I too can login to the running container and use curl to login to Vault using the exact values from those environment variables.

I am running:

• Jenkins 2.414.3 from jenkins/jenkins:lts-jdk17.
• hashicorp-vault-plugin version 361.v44fea_4fc08d9