Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Some character converted to diamond shaped question mark #2387

Open
HanpoomAnj opened this issue Dec 8, 2022 · 10 comments
Open

[BUG] Some character converted to diamond shaped question mark #2387

HanpoomAnj opened this issue Dec 8, 2022 · 10 comments
Labels
bug Something isn't working

Comments

@HanpoomAnj
Copy link

HanpoomAnj commented Dec 8, 2022 •
edited
Loading

Describe the bug
Issue with the Korean character converting to a diamond symbol. It only happens on pages with very long text content such as terms of use etc. The issue doesn't happen on specific text consistently, it will happen from a different text.

Expected behaviour
Korean text should be rendered correctly and should not convert to a diamond question mark symbol

Screenshots
image

image

Additional context
Note that we already apply either of these two in our index.html file

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta charset=UTF-8">

Also, we cannot replicate it in our local branch
@HanpoomAnj HanpoomAnj added the bug Something isn't working label Dec 8, 2022
@RaymondCrandall
Copy link

RaymondCrandall commented Dec 13, 2022 •
edited
Loading

I would start by correcting the html syntax, the attribute and value delimiters are not being declared correctly, avoiding duplicate declarations the character set, and testing with a complete html header. The server header is also able to handle language related metadata declarations.


<!DOCTYPE html>
<html lang="ko">
<title>Example Header Korean Language</title>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<body>
  <p dir="ltr">이 HTML 문서 언어 속성은 한국어입니다.</p>
  <p dir="ltr" lang="en">This is a different example of if 
                                    you wanted to switch languages inside the document</p>
  </body>
</html>

Seems to be working for me, if this does not help feel free to post the characters not showing up in the thread here and I'll see if I can figure out what is happening

@RaymondCrandall
Copy link

The editor and debugging software you are using might not be configured for your language, I would also start by telling us what software you are using and then we can see if the characters are not rendering due to software configurations

@HanpoomAnj
Copy link
Author

Hi @RaymondCrandall
thanks for your help appreciate it! This is still an issue on our site...

Reply to your 1st comment:
This is what we currently have on our header.

<html lang="ko"><head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
    _<more code here>_
  </head>

We've also tried this one <meta charset=UTF-8"> instead of <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> but that didn't fix.

Reply to your 2nd comment:
Software: WebStorm by JetBrains
Software file encoding: UTF8
image

I want to add, I saw a ticket with a similar issue: https://github.com/Shopify/hydrogen/issues/2302
Do you think it might be related to this?

@RaymondCrandall
Copy link

I'm not familiar with this problem, so we may have to try different things to narrow down what the issue is.

Does Korean show up in your browser on most websites without any issues? If so, assuming your http encoding headers are correctly formatted, find the character(s)/symbols(s) which fail to render properly, and use an analysis tool to see if they are supported, and maybe we will find an answer there.

I would definitely run the html syntax through a markup validator tool to make sure it's not just a typing error

the ticket with a similar issue (link above) goes to a no results found page...so I looked up the issue by its ID
It looks like you made a mistake in your markdown formatting.
Here is the corrected link:
Shopify/hydrogen#2302

I also found this which may be worth checking out:

Shopify/hydrogen#2351

I can look into seeing what is being talked about in these links later today, I'm working on a few things currently

hope this helps

@HanpoomAnj
Copy link
Author

@RaymondCrandall Yes Korean characters are showing correctly in my browser on all Korean websites I visited. This is our first time to encounter an issue after switching from Woocommerce to Shopify hydrogen.

@RaymondCrandall
Copy link

@HanpoomAnj did you run the generated html response through a markup validator?

@HanpoomAnj
Copy link
Author

@RaymondCrandall here's the result of the markup validator
screencapture-validator-w3-org-nu-2023-01-09-10_13_41

@HanpoomAnj
Copy link
Author

@RaymondCrandall Let me know if you have check on this again, thank you!! 🙏

@omapon
Copy link

omapon commented Mar 23, 2023

It occurs in Japanese long text too, only when production deploy (at netlfy, vercel, cloudflare worker).
yarn dev works fine.
So, our team deploy by docker with yarn build.

@charumeyn
Copy link

Is there any update on this? It still happens on our site randomly. @RaymondCrandall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@RaymondCrandall @charumeyn @omapon @HanpoomAnj