yahoo.finance broken due to Yahoo's design change

[devzero12]

All yahoo.finance scripts are not working any more because Yahoo decided to roll out a design update.

[changbl]

+1 any fix?

[mauricionr]

+1 any fix or workaround?

[BobAleena]

Where can we find the new design info? I have code using yahoo.finance.keystats which is failing now.

[trafflux]

The problem you are having is that the YQL API end point URL was changed for some reason in the console but not on the server or in the routing. I think the URL discrepancy pertains to the community data tables group only. Anyways you won't be able to use the console with the following fix but it should work in your external code:

https://query.yahooapis.com/v1/yql Current Broken End Point URL

https://query.yahooapis.com/v1/public/yql Correct End Point URL

#477 #476

[mauricionr]

@trafflux i'm using the correct URL but still broken

https://query.yahooapis.com/v1/public/yql

my full url is

//query.yahooapis.com/v1/public/yql?q=select * from json where url="http://finance.yahoo.com/webservice/v1/symbols/TRPL4.SA,TRPL3.SA/quote?format=json&view=detail" and itemPath = "list.resources.resource.fields"&format=json&env=store://datatables.org/alltableswithkeys&callback=?

The response is

{"query":{"count":0,"created":"2016-08-24T14:13:22Z","lang":"pt-BR","results":null}}

[trafflux]

@mauricionr I've never seen YQL formatted that way. Did it ever work or is this just something you are trying for the 1st time and it not working? Anyways, I assumed you were trying to do a detailed quote pull on > 1 stock symbols, so I have included the method I have used in the past. I've included calls to 2 different tables which contain the current price. The only major difference is the number of fields that are returned, ~ 75 vs 15.

Larger pull from "quotes" table:
//query.yahooapis.com/v1/public/yql?q=select * from yahoo.finance.quotes where symbol in ("TRPL4.SA","TRPL3.SA")&env=store://datatables.org/alltableswithkeys&format=json

Smaller pull from "quote" table:
//query.yahooapis.com/v1/public/yql?q=select * from yahoo.finance.quote where symbol in ("TRPL4.SA","TRPL3.SA")&env=store://datatables.org/alltableswithkeys&format=json

Anyways I hope this helps. If not, explain in more detail what you're trying to do and I'm sure I can help. I developed a website a few years ago that relies on yahoo finance API calls and its still running fine without any tweaking of the code.

[mauricionr]

This was working fine until last month, i will test your sugestions

Thanks!

[mauricionr]

@trafflux thanks for the tip, now it's working 😄

[BobAleena]

I'm receiving back empty results. Previously I received all the stats, but it looks like the design change broke the expected json.

query = query + URLEncoder.encode("select * " +
                        "from keystatistics " + //yahoo.finance.keystats " +
                        "where symbol in (" + symbols +")", "UTF-8");

The actual request that is sent is:

http://query.yahooapis.com/v1/public/yql?q=use+%22https%3A%2F%2Fraw.githubusercontent.com%2Fyql%2Fyql-tables%2Fe50ccc07980c7697be987e37a331af253b032a64%2Fyahoo%2Ffinance%2Fyahoo.finance.keystats.xml%22++as+keystatistics%3B+select+*+from+keystatistics+where+symbol+in+%28%22WMT%22%29&format=json&diagnostics=true&env=http%3A%2F%2Fdatatables.org%2Falltables.env

Where can I see the design of what the expected results of that should be?

[BobAleena]

I am current getting:

"results":{"stats":[{"symbol":"WMT"}]}

but previously, I was receiving the actual stats to extract

[BobAleena]

anyone have this working yet?

[mauricionr]

@BobAleena with your url i got this error

{"error":{"lang":"en-US","diagnostics":{"url":[{"execution-start-time":"1","execution-stop-time":"2","execution-time":"1","http-status-code":"400","http-status-message":"Bad Request","content":"http://datatables.org/alltables.env"},{"execution-start-time":"2","execution-stop-time":"3","execution-time":"1","http-status-code":"400","http-status-message":"Bad Request","content":"https://raw.githubusercontent.com/yql/yql-tables/e50ccc07980c7697be987e37a331af253b032a64/yahoo/finance/yahoo.finance.keystats.xml"}],"warning":"Invalid environment specified: http://datatables.org/alltables.env"},"description":"No definition found for Table keystatistics"}}

[BobAleena]

@mauricionr Are you sure you copied it correctly? I just tried it again and it succeeds to executes, but still gives me the same response as I said on Sep 4.

[BobAleena]

I actually tried it again, it works, but just gives me back the symbol but nothing else. tried it in yql console as well.
http://query.yahooapis.com/v1/public/yql?q=use+%22https%3A%2F%2Fraw.githubusercontent.com%2Fyql%2Fyql-tables%2Fd352c5ec239e2aa3d3d1192ddfa9b6271eb3f42e%2Fyahoo%2Ffinance%2Fyahoo.finance.keystats.xml%22++as+keystatistics%3B+select+symbol%2C+PEGRatio%2C+ForwardPE%2C+CurrentRatio%2C+PriceBook%2C+ShortRatio%2C+LeveredFreeCashFlow+from+keystatistics+where+symbol+in+%28%22yhoo%22%2C%22msft%22%29&format=json&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys

[yashkhandelwal]

@trafflux The mentioned urls are not working though they were working earlier, getting following error for both urls :
description: "No definition found for Table yahoo.finance.quotes"
any idea if anything changed ?

[trafflux]

The amount of people that have contacted me from this thread for support with YQL is crazy. I'm all yours for $200/hr... Anyways, @yashkhandelwal I logged into YQL and tested out the quotes query and it worked fine...
The YQL console statement is:
select * from yahoo.finance.quotes where symbol in ("TRPL4.SA","TRPL3.SA")

And the REST query using JSON format is:
https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.quotes%20where%20symbol%20in%20(%22TRPL4.SA%22%2C%22TRPL3.SA%22)&format=json&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys&callback=

Basically just plug in what ticker symbol(s) you need and it will spit out a ton of data points, including last price/quote. I've attached a screen shot that might help you figure out what you may be doing wrong. Be sure to be logged in and have the community table's enabled via the checkbox in the top left.

Back in 2014 I wrote some code to grab quotes, volume, and about 5 other datapoints every 5 min for a couple hundred tickers and log it into a database and serve it up as an API, then I had the website I was creating for this financial company hit the API endpoint, grab the last X data points for each ticker and incorporate it into various widgets/charts/graphs etc that were needed across the entire website. The funny thing is, the system is still working fine despite all the issues people have had with YQL finance over the years since and despite that the company I created it for doesn't have a web developer; they only have 1 guy who manages the site content via a cms. The sad thing is, that company screwed me out of a lot of money and I was hoping the whole system would break down =(

[wonius]

@trafflux
my url is:
https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.quotes%20where%20symbol%20in%20(%22YHOO%22%2C%22AAPL%22%2C%22GOOG%22%2C%22MSFT%22)&diagnostics=true&env=store%3A%2F%2Fdatatables.org%2Falltableswithkeys

what it returned is:
Failed to read from storage: store://datatables.org/alltableswithkeys: Invalid store url: store://datatables.org/alltableswithkeys

your website still work now?

[trafflux]

@wonius I just checked the link you supplied and it looks like its working. I'm assuming yahoo/datatables.org had a brief issue that caused the error.

[wonius]

@trafflux five minutes ago, I tried again and it returned the correct result.
But now...the result error again. So unstable!
Still thank you, anyway.

[BobAleena]

@trafflux Thanks for this! I was able to get it working for yahoo.finance.quotes, but realized it does not have the datapoints I needed. Any suggestions on getting it to work using yahoo.finance.keystats table? That is what I was originally using but seems the scraping is not working due to layout changes... (btw - $200 is not that bad...)

select * from yahoo.finance.keystats where symbol in ("aapl","msft")

[kasimvali786]

Hello,

Can you please help me.

I have used the below YQL query to get exchange rates , it has been working fine up to the last month.

https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20yahoo.finance.xchange%20where%20pair%20in%20(%22USDEUR%22,%20%22USDJPY%22,%20%22USDBGN%22,%20%22USDCZK%22,%20%22USDDKK%22,%20%22USDGBP%22,%20%22USDHUF%22,%20%22USDLTL%22,%20%22USDLVL%22,%20%22USDPLN%22,%20%22USDRON%22,%20%22USDSEK%22,%20%22USDCHF%22,%20%22USDNOK%22,%20%22USDHRK%22,%20%22USDRUB%22,%20%22USDTRY%22,%20%22USDAUD%22,%20%22USDBRL%22,%20%22USDCAD%22,%20%22USDCNY%22,%20%22USDHKD%22,%20%22USDIDR%22,%20%22USDILS%22,%20%22USDINR%22,%20%22USDKRW%22,%20%22USDMXN%22,%20%22USDMYR%22,%20%22USDNZD%22,%20%22USDPHP%22,%20%22USDSGD%22,%20%22USDTHB%22,%20%22USDZAR%22,%20%22USDISK%22)&env=store://datatables.org/alltableswithkeys

now it is getting response like below.
{"query":{"count":0,"created":"2016-08-24T14:13:22Z","lang":"pt-BR","results":null}}

from the above what i understand is the URL has been discontinued.
Could you please suggest me how i can get new URL/ new rest API URL to get exchange rates.

Thanks in advanced.

[trafflux]

@kasimvali786 It looks like they've discontinued the yahoo YQL console and are making you register an app and have a proper API key. I would bet the lack of API key is your issue. See this page: https://developer.yahoo.com/yql/
I hope it helps and I hope their new system is more stable than the old one. As a side note, there's a LOT of stuff that depends on the Yahoo exchange rates for data, specifically Magento defaults to using it to set prices in other currencies besides the site default.

[kasimvali786]

Hi Trafflux,

Thank you for your response.

one more query : I have below URL to get all currencies from Yahoo.
https://finance.yahoo.com/webservice/v1/symbols/allcurrencies/quote

If i want to get particular pair currencies (for instance if i want to know USDINR currencies details), then how i can change the URL.

I have tried in the place of "allcurrencies" from URL(https://finance.yahoo.com/webservice/v1/symbols/USDINR=/quote),I am getting response below.
"Description: Could not process this "GET" request."

any suggestions ?

[trafflux]

@kasimvali786 It does look like they killed the whole service. See this thread https://stackoverflow.com/questions/47072264/https-query-yahooapis-com-v1-public-yqlq-select-from-yahoo-finance-xchange

But that XML you found is updated and kept updated which means you could use XPATH to look it up, just not as easy as YQL. You'd basically want to curl the xml file then apply whatever language-specific XML handling lib to it and use its functions to find the values. If you do use the XPATH method of simply scraping their data, don't do it too frequently and be prepared for them to discontinue the data being available. Sorry I couldn't help much.

EDIT: With some searching, I found this API service which offers a free plan that limits you to 1k requests per month and they have currency conversion. https://currencylayer.com

[kasimvali786]

Hi Trafflux,

I am trying to access the URL to get the all currencies by using XPATH method.

https://finance.yahoo.com/webservice/v1/symbols/allcurrencies/quote?format=json

The it is ends up with below message:
Error : HttpCallException: HTTP GET call to http://finance.yahoo.com/webservice/v1/symbols/allcurrencies/quote?format=json not successful. HTTP/1.1 403 Forbidden

meaning don't have authorization.

Any idea/suggestions how to proceed further.

Thanks,
Kasim,

[trafflux]

@kasimvali786
This is strange because I clicked that link and it showed the data fine a couple times, then I tried a few more times and it gave me a 0-length resource list a few times, but now its back to giving me the complete array. But no 403 errors were given. You may need to either use an API key with your request or spoof a standard browser (possibly they are trying to cut off bot access to it). I'd recommend that you look into the API key 1st. I've really stopped working with YQL and yahoo data sets because of the headaches they cause. My company moved to paying for Quotemedia JSON data sets and its pretty great.

[kasimvali786]

Hi Trafflux,

Even if we use API key in the URL then it is throwing 404 error.

quote

My company moved to paying for Quotemedia JSON data sets and its pretty great.

unquote

from your above comments : can you please provide URL to get exchange rates.

[kasimvali786]

Hi Trafflux,

Thank you for the inputs.

any how yahoo API has dead.
http://www.financial-hacker.com/bye-yahoo-and-thank-you-for-the-fish/