feat: docs automation for website

[AnimeshKumar923]

Description

• This PR introduces an automated workflow to push community files onto the website.
• We can have several other PRs which modifies the structure of the docs present on this repo if required.
• Bringing your kind attention on this PR and corresponding issue @alequetzalli @TRohit20 @akshatnema @derberg @sambhavgupta0705 @anshgoyalevil

Related issue(s)

Resolves #1028
Resolves #773

[sambhavgupta0705]

https://github.com/asyncapi/extensions-catalog/blob/master/.github%2Fworkflows%2Fupdate-extensions-in-website.yml
Take inspiration from this file

[AnimeshKumar923]

https://github.com/asyncapi/extensions-catalog/blob/master/.github%2Fworkflows%2Fupdate-extensions-in-website.yml Take inspiration from this file

I'll take a look at it 👍 @sambhavgupta0705

---

btw, have a look at the following PR which is of similar nature and was used to push Glee docs to website:

• ci: glee docs automation workflow glee#548

I took initial reference from here. Can you please explain me what's the difference between Lukasz's reference and your reference?

[sambhavgupta0705]

btw, have a look at the following PR which is of similar nature and was used to push Glee docs to website:

• ci: glee docs automation workflow glee#548

I took initial reference from here. Can you please explain me what's the difference between Lukasz's reference and your reference?

the workflow I shared directly adds the link of the community to the edit-on-github.json config file

[sambhavgupta0705]

we need to see if we need the increment of weights workflow for this workflow

[AnimeshKumar923]

btw, have a look at the following PR which is of similar nature and was used to push Glee docs to website:

• ci: glee docs automation workflow glee#548

I took initial reference from here. Can you please explain me what's the difference between Lukasz's reference and your reference?

the workflow I shared directly adds the link of the community to the edit-on-github.json config file

Got it. Let me work on that...

[TRohit20]

@AnimeshKumar923 From what I understand the PR is setting up a workflow such that, on push to community docs in the community repo, we want the changes to sync on the website(website repo) which means the workflow should create a PR and get auto-merged.

I have already worked on similar automation workflow previously, you can check out the workflow here for reference.

cc @sambhavgupta0705 @alequetzalli

[AnimeshKumar923]

@AnimeshKumar923 From what I understand the PR is setting up a workflow such that, on push to community docs in the community repo, we want the changes to sync on the website(website repo) which means the workflow should create a PR and get auto-merged.

I have already worked on similar automation workflow previously, you can check out the workflow here for reference.

cc @sambhavgupta0705 @alequetzalli

Thanks for the reference! 👍 @TRohit20

[AnimeshKumar923]

/au

[AnimeshKumar923]

also, what docs will we be pushing onto the website? @alequetzalli @TRohit20
I assume the onboarding-guide will definitely be pushed as far as I know... What else? Can we have a list of it?

[TRohit20]

I'd suggest not to hard code the files we'd push on, use paths instead. Makes your file easier haha

[quetzalliwrites]

Question, @AnimeshKumar923, is this PR to replace the one I tried to make via #1064? I don't mind at all, if that is the case, I honestly still don't understand fully how to automate pushing our community docs to the website. If someone else such as yourself is making a PR to show me how to do so, please do share. 😅

Also, sorry but I do not fully understand the code. Can you walk me through it more? 🙏🏻 cc @TRohit20

[AnimeshKumar923]

Question, @AnimeshKumar923, is this PR to replace the one I tried to make via #1064? I don't mind at all, if that is the case, I honestly still don't understand fully how to automate pushing our community docs to the website. If someone else such as yourself is making a PR to show me how to do so, please do share. 😅

Also, sorry but I do not fully understand the code. Can you walk me through it more? 🙏🏻 cc @TRohit20

Hey @alequetzalli sorry about that. I didn't see that PR as it wasn't linked in the original issue, hence I missed #1064 😅

If it's fine with you, I can help with the community docs automation through this PR. 😁

[AnimeshKumar923]

@alequetzalli
(DISCLAIMER: I'll try my best to explain, although I'm still learning so please verify the explanation @sambhavgupta0705 @akshatnema)

Automation Workflow

• Whenever there's a docs PR on the master branch which is of .md type, the workflow gets triggered.

• Then there are number of jobs that takes place.

• Make-PR gets executed. Inside this, there are further sub-jobs which takes place.

• It then checks out the specified repository (community repo here)

  steps:
    - name: Checkout Current repository
      uses: actions/checkout@v4
      with:
        path: community

• Then it checks out another repo (which is website repo here) using the following:

  name: Checkout Another repository
      uses: actions/checkout@v4
      with:
        repository: asyncapi/website
        path: website
        token: ${{ env.GITHUB_TOKEN }}

• After that, it configures the git

  name: Config git
      run: |
        git config --global user.name asyncapi-bot
        git config --global user.email [email protected]

• Then it creates a new branch with commit id

  name: Create branch
      working-directory: ./website
      run: |
        git checkout -b update-community-docs-${{ github.sha }}

• @sambhavgupta0705 please explain this part...

      name: Update edit-page-config.json
          uses: actions/github-script@v4
          with:
              script: |
              const { writeFile } = require('fs').promises;
              const configPath = './website/config/edit-page-config.json';
              const checkSlug = 'reference/extensions/';
              const slug = {
                  "value": checkSlug,
                  "href": "https://github.com/asyncapi/community/tree/master/docs"
              };
              
              const configData = require(configPath);
              const entryExists = configData.some(entry => entry.value === checkSlug);
              if (!entryExists) {
                  configData.push(slug);
                  await writeFile(configPath, JSON.stringify(configData, null, 2))
              }
      ```
  

• After that, it copies the specified files from community repo to the website repo

  name: Copy community folder from Current Repo to Another
      working-directory: ./website
      run: |
        mkdir -p ./pages/docs/community/
        printf "%s\ntitle: Community\nweight: 7\n%s" "---" "---"> ../community/docs/_section.md
        mv ../community/docs/*.md ./pages/docs/community

• After the files are copies, it commits and pushes the changes

  name: Commit and push
      working-directory: ./website
      run: |
        git add .
        git commit -m "docs(community): update latest community docs"
        git push https://${{ env.GITHUB_TOKEN }}@github.com/asyncapi/website

• At last, we create a PR with title docs(community): update latest community documentation with PR body being Updated community documentation is available and this PR introduces update to community folder on the website

  name: Create PR
      working-directory: ./website
      run: |
        gh pr create --title "docs(community): update latest community documentation" --body "Updated community documentation is available and this PR introduces update to community folder on the website" --head "update-community-docs-${{ github.sha }}"

[sambhavgupta0705]

@AnimeshKumar923 I will explain the whole workflow to you soon

[quetzalliwrites]

NOTE: Sharing some notes that @KhudaDad414 shared with me via DM about this task in case it helps @AnimeshKumar923.

---

If I understood the issue correctly then this is how we approached it in glee:

• created this workflow in the glee project. https://github.com/asyncapi/glee/blob/master/.github/workflows/update-docs-in-website.yml
• adding a _section.md file in the Glee project here (https://github.com/asyncapi/glee/blob/master/docs/pages/_section.md)
• made sure all of the .md files can compile as we want them to by putting them in a local instance of the website.
• now any time we update the docs, website is going to be updated automatically.

[quetzalliwrites]

Thank you for that extremely detailed walk-through of your code, @AnimeshKumar923!! I really appreciate it. 😀

also, what docs will we be pushing onto the website? I assume the onboarding-guide will definitely be pushed as far as I know... What else? Can we have a list of it?

Thank you, good question! We should push whatever is in the /docs directory, with the exception of the README. That leaves us with:

1. Onboarding Guide
2. How changes in spec are introduced

Does that make sense? lemme know ✌🏽

[AnimeshKumar923]

Thank you for that extremely detailed walk-through of your code, @AnimeshKumar923!! I really appreciate it. 😀

My pleasure! 😁

also, what docs will we be pushing onto the website? I assume the onboarding-guide will definitely be pushed as far as I know... What else? Can we have a list of it?

Thank you, good question! We should push whatever is in the /docs directory, with the exception of the README. That leaves us with:

1. [Onboarding Guide](https://github.com/asyncapi/community/tree/master/docs/onboarding-guide)

2. [How changes in spec are introduced](https://github.com/asyncapi/community/blob/master/docs/how-changes-in-the-spec-are-introduced.md)

Does that make sense? lemme know ✌🏽

Yes, definitely. Thanks for the clarification! @alequetzalli

[AnimeshKumar923]

NOTE: Sharing some notes that @KhudaDad414 shared with me via DM about this task in case it helps @AnimeshKumar923.

If I understood the issue correctly then this is how we approached it in glee:

* created this workflow in the glee project. https://github.com/asyncapi/glee/blob/master/.github/workflows/update-docs-in-website.yml

* adding a _section.md file in the Glee project here (https://github.com/asyncapi/glee/blob/master/docs/pages/_section.md)

* made sure all of the .md files can compile as we want them to by putting them in a local instance of the website.

* now any time we update the docs, website is going to be updated automatically.

@alequetzalli

Thanks for sharing the notes. It will definitely help in the development process of this feature. 👍

---

I think for the modification in the docs structure, we should have separate PRs for better management and reviews.
cc: @TRohit20

[AnimeshKumar923]

/au

[thulieblack]

Any update on this?

[TRohit20]

@akshatnema Any chance you got to validate the changes being introduced on your system so we can see If it works for project level?

[derberg]

can be removed, I see it was committed directly to repo already https://github.com/asyncapi/community/blob/master/docs/onboarding-guide/_section.md and because there will be multiple folders with different content, each with its own section, I guess better hardcode in repo than dynamically create here

[derberg]

some things changed in website, and also afaik here we will have multiple subdirs in docs

suggested required improvements

[thulieblack]

/au

[derberg]

lgtm @thulieblack
when do you want to merge it? I'd like to observe it's first run

[thulieblack]

lets merge it and see if it works

[derberg]

/rtm

[derberg]

didn't work

https://github.com/asyncapi/community/blob/master/.github/workflows/update-docs-in-website.yml#L70 failed

rm -r ./markdown/docs/community/onboarding-guide

because of course the folder is not there. We could solve it with simple additiona of -f, but I just realized that we will have multiple folders with different topics, so cannot have hardcode like "onboarding-guide" (btw this is wrong right, as this is only for docs, and others will go in another folder?)

we probably need to remove that part of bash script

          rm -r ./markdown/docs/community/onboarding-guide
          mkdir -p ./markdown/docs/community/onboarding-guide

and replace

# Remove all directories inside, so after moving changes from community, we also reflect potential deletions of some files
find "./markdown/docs/community" -mindepth 1 -maxdepth 1 -type d -exec rm -rf {} +

yes, mkdir is not needed as we anyway always move all docs folders from community repo

also add workflow_dispatch: (yes, with : at the end) under events trigger as it should be like below so we can also manually run workflow

on:
  workflow_dispatch:
  push:
    branches:
      - 'master'
    paths:
      - 'docs/*.md'