-
Notifications
You must be signed in to change notification settings - Fork 18
/
FB-Crash_blog.html
348 lines (295 loc) · 19.6 KB
/
FB-Crash_blog.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
<!DOCTYPE html>
<html lang="en">
<head>
<title>OSS — IIT Dharwad</title>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<link rel="shortcut icon" href="./images/oss_logo.png" />
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Nunito+Sans:200,300,400,700,900|Roboto+Mono:300,400,500">
<link rel="stylesheet" href="fonts/icomoon/style.css">
<link rel="stylesheet" href="css/bootstrap.min.css">
<link rel="stylesheet" href="css/magnific-popup.css">
<link rel="stylesheet" href="css/jquery-ui.css">
<link rel="stylesheet" href="css/owl.carousel.min.css">
<link rel="stylesheet" href="css/owl.theme.default.min.css">
<link rel="stylesheet" href="css/bootstrap-datepicker.css">
<link rel="stylesheet" href="css/animate.css">
<link rel="stylesheet" href="fonts/flaticon/font/flaticon.css">
<link rel="stylesheet" href="css/fl-bigmug-line.css">
<link rel="stylesheet" href="css/aos.css">
<link rel="stylesheet" href="css/style.css">
<link rel="stylesheet" href="css/custom.css">
</head>
<body>
<div class="site-wrap">
<div class="site-mobile-menu">
<div class="site-mobile-menu-header">
<div class="site-mobile-menu-close mt-3">
<span class="icon-close2 js-menu-toggle"></span>
</div>
</div>
<div class="site-mobile-menu-body"></div>
</div> <!-- .site-mobile-menu -->
<header class="site-navbar py-1" role="banner">
<div class="container">
<div class="row align-items-center">
<div class="col-6 col-xl-2">
<a href="index.html"><img src="images/oss_logo.png" style="max-height: 100px;"></a>
<!-- <h1 class="mb-0"><a href="index.html" class="text-black h2 mb-0">O<strong>SS</strong></a></h1> -->
</div>
<div class="col-10 col-xl-10 d-none d-xl-block">
<nav class="site-navigation text-right" role="navigation">
<ul class="site-menu js-clone-nav mr-auto d-none d-lg-block">
<li><a href="index.html">Home</a></li>
<li><a href="events.html">Events</a></li>
<li><a href="blog.html">Blog</a></li>
<li><a href="team.html">Team</a></li>
<li><a href="previousTeam.html">Previous Team</a></li>
<li><a href="new_post.html"><span class="rounded bg-primary py-2 px-3 text-white"><span class="h5 mr-2">+</span>Propose an Idea</span></a></li>
</ul>
</nav>
</div>
<div class="col-6 col-xl-2 text-right d-block">
<div class="d-inline-block d-xl-none ml-md-0 mr-auto py-3" style="position: relative; top: 3px;"><a href="#" class="site-menu-toggle js-menu-toggle text-black"><span class="icon-menu h3"></span></a></div>
</div>
</div>
</div>
</header>
<div class="unit-5 overlay" style="background:teal">
<div class="container text-center">
<h2 class="mb-0">Blog</h2>
</div>
</div>
<div class="site-section">
<div class="container">
<!-- <div class="row">
<div class="col-lg-6 col-md-12 mb-5" data-aos="fade">
<img src="images/about.jpg" class="img-md-fluid" alt="Placeholder image" style="max-width: 100%">
</div>
<div class="col-lg-6 col-md-12">
<div class="bg-white pl-lg-5 pl-0 pb-lg-5 pb-0">
<h2 class="font-weight-bold text-black" data-aos="fade">About</h2>
<p>Open Student Society(OSS) is a community for students to join together to develop, innovate, share and grow together as a community. It is important to note that OSS is not a club, rather a community that welcomes students, from all branches and walks of life, to participate and contribute. Some of the goals we intend to achieve with you include:</p>
<ul class="site-block-check">
<li><strong>Organize</strong> workshops, hackfests and other great events!</li>
<li><strong>Share</strong> skills, information and resources among students to transform our society.</li>
<li><strong>Support</strong> , create and work on revolutionary Projects for public benefit.</li>
<li><strong>Make</strong> strong relations to build an OSSM('awesome') community </li>
</ul>
</div>
</div>
</div> -->
<div class="container text-center">
<h2 class="mb-0">Story Behind Facebook Crash - October 2021</h2>
</div>
<br>
<div class="col-md-12">
<p align="justify">Simply appears to be a frivolous joke to us every one of us, we hear - <b>"Facebook is down from web"</b>. On 4-5<sup>th</sup> October 2021 at 15:51 UTC, we opened an inside episode named "Facebook DNS query returning SERVFAIL" on the grounds that we were stressed that something was off-base with our DNS resolver <u><a href="https://developers.cloudflare.com/warp-client/">1.1.1.1</a></u>. Yet, as we were going to post on our public status page we understood something different more genuine was going on. Online media immediately burst into flares, announcing what our designers quickly affirmed as well. Facebook and its subsidiary administrations WhatsApp and Instagram were, indeed, all down. Their DNS names quit settling, and their framework IPs were inaccessible. Maybe somebody had "pulled the links" from their server farms at the same time and separated them from the Web. This wasn't a DNS issue itself, however bombing DNS was the primary manifestation we'd seen of a bigger Facebook blackout.</p>
<p align="justify">Online media immediately burst into blazes, revealing what our designers quickly affirmed as well. Facebook and its partnered administrations WhatsApp and Instagram were, indeed, all down. Their DNS names quit settling, and their framework IPs were inaccessible. Maybe somebody had "pulled the links" from their server farms at the same time and separated them from the Web. This wasn't a DNS issue itself, however bombing DNS was the principal indication we had seen of a bigger Facebook blackout.</p>
</div>
<div class="container">
<h2 class="mb-0">What did Facebook say about this issue?</h2>
</div>
<br>
<div class="col-md-12">
<p align="justify">
<b>Official Statement:</b> "Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt."</p>
<p align="justify">Facebook has now published a blog <a href="https://engineering.fb.com/2021/10/04/networking-traffic/outage/">post</a> giving some details of what happened internally. Externally, Cloudfare saw the <a href="https://en.wikipedia.org/wiki/Border_Gateway_Protocol">BGP</a> and <a href="https://en.wikipedia.org/wiki/Border_Gateway_Protocol">DNS</a> problems outlined in this post but the problem actually began with a configuration change that affected the entire internal backbone. That cascaded into Facebook and other properties disappearing and staff internal to Facebook having difficulty getting service going again. Facebook posted a further blog <a href="https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/">post</a> with a lot more detail about what happened. You can read that post for the inside view and this post for the outside view.
</p>
</div>
<div class="container">
<h2 class="mb-0">BGP Issue</h2>
</div>
<br>
<div class="col-md-12">
<p align="justify">
BGP conventions profits a component to trade steering data between Independent Frameworks <a href="https://en.wikipedia.org/wiki/Autonomous_system_(Internet)">Autonomous Systems (AS)</a> on the Web. The large switches that make the Web work have colossal, continually refreshed arrangements of the potential courses that can be utilized to convey each organization parcel to their last objections. Without BGP, the Web switches wouldn't realize what to do, and the Web wouldn't work.
</p>
<p align="justify">
The individual networks each have an ASN: an Autonomous System Number. An Autonomous System (AS) is an individual network with a unified internal routing policy. An AS can begin prefixes (say that they control a gathering of IP addresses), just as travel prefixes (say they realize how to arrive at explicit gatherings of IP addresses). Cloudflare's ASN is <a href="https://www.peeringdb.com/asn/13335">AS13335<a>. Each ASN needs to declare its prefix courses to the Web utilizing BGP; any other way, nobody will realize how to associate and where to discover us.
</p>
<center> <img src="images/blogs/fb_crash/blog_as.png" height="500" width="1000"></center>
<p align="justify">
In the above diagram, 6 autonomous systems on the Internet and two possible routes that one packet can use to go from Start to End. AS1 → AS2 → AS3 being the fastest, and AS1 → AS6 → AS5 → AS4 → AS3 being the slowest, but that can be used if the first fails.
</p>
<p align="justify">
At 15:58 UTC, Cloudfare saw that Facebook had quit declaring the courses to their DNS prefixes. That implied that, in any event, Facebook's DNS servers were inaccessible. As a result of this current Cloudflare's 1.1.1.1 DNS resolver could presently don't react to inquiries requesting the IP address of <a href="https://facebook.com">facebook.com</a>.
<xmp>
route-views>show ip bgp 185.89.218.0/23
% Network not in table
route-views>show ip bgp 129.134.30.0/23
% Network not in table
route-views>
</xmp>
Then again, Facebook IP addresses remained directed yet weren't especially valuable since without DNS Facebook and related administrations were adequately inaccessible:
<xmp>
route-views>show ip bgp 129.134.30.0
BGP routing table entry for 129.134.0.0/17, version 1025798334
Paths: (24 available, best #14, table default)
Not advertised to any peer
Refresh Epoch 2
3303 6453 32934
217.192.89.50 from 217.192.89.50 (138.187.128.158)
Origin IGP, localpref 100, valid, external
Community: 3303:1004 3303:1006 3303:3075 6453:3000 6453:3400 6453:3402
path 7FE1408ED9C8 RPKI State not found
rx pathid: 0, tx pathid: 0
Refresh Epoch 1
route-views>
</xmp>
A BGP UPDATE message notifies a router of any changes to a prefix advertisement or completely withdraws the prefix. When Cloudfare checked their time-series BGP database, they saw a significant increase in the amount of updates they got from Facebook. Normally, this chart is quite quiet: Facebook doesn't make many modifications to its network on a minute-by-minute basis.
But at around 15:40 UTC Cloudfare saw a peak of routing changes from Facebook as shown below. That’s when the trouble commenced.
<center> <img src="images/blogs/fb_crash/blog_bgp_updates.png" height="400" width="1100"></center>
Cloudfare engineers started to worry that there was some fault within their systems and hence 1.1.1.1 was not able to resolve to facebook.com.
</p>
</div>
<div class="container">
<h2 class="mb-0">DNS Cascade</h2>
</div>
<br>
<div class="col-md-12">
<p align="justify">
As a direct consequence of this, DNS resolvers all over the world stopped resolving their domain names. One of the systems check done by Couldfare engineers was as follows:
<xmp>
➜ ~ dig @1.1.1.1 facebook.com
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 31322
;facebook.com. IN A
➜ ~ dig @1.1.1.1 whatsapp.com
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 31322
;whatsapp.com. IN A
➜ ~ dig @8.8.8.8 facebook.com
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 31322
;facebook.com. IN A
➜ ~ dig @8.8.8.8 whatsapp.com
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 31322
;whatsapp.com. IN A
</xmp>
This is due to the fact that DNS, like many other systems on the Internet, has its own routing algorithm. When someone puts <a href="https://facebook.com">https://facebook.com</a> into their browser, the DNS resolver, which is in charge of converting domain names into real IP addresses to connect to, first checks its cache and utilises what it finds. If not, it attempts to obtain the response from the domain nameservers, which are generally hosted by the entity that owns the domain. If the nameservers are unavailable or fail to reply for any other reason, a SERVFAIL is returned, and the browser displays an error message to the user.
</p>
</div>
<div class="container">
<h2 class="mb-0">Impact on other services</h2>
</div>
<br>
<div class="col-md-12">
<p align="justify">
People seek alternatives and want to learn more about or debate what is going on. When Facebook went down, we saw a spike in DNS requests to Twitter, Signal, and other messaging and social media services. The graph below depicts the progressive rise in interaction activity on several social media platforms:
<center> <img src="images/blogs/fb_crash/blog_queries.png" height="400" width="1100"></center>
</p>
<p align="justify">
Another side effect of this unavailability may be seen in our WARP traffic to and from Facebook's impacted ASN 32934. This graph depicts how traffic in each nation changed from 15:45 UTC to 16:45 UTC compared to three hours earlier. WARP communications to and from Facebook's network just vanished throughout the world.
<center> <img src="images/blogs/fb_crash/blog_world.png" height="500" width="1100"></center>
</p>
</div>
<br>
<div class="container">
<h2 class="mb-0">Update from Facebook after reboot</h2>
</div>
<br>
<div class="col-md-12">
<p align="justify">
At around 21:00 UTC Cloudfare saw renewed BGP activity from Facebook's network which peaked at 21:17 UTC.
</p>
<p align="justify">
<b>PS:</b> These incidents serve as a gentle reminder that the Internet is a very complex and interdependent system comprised of millions of interconnected systems and protocols. That trust, uniformity, and collaboration across entities is at the heart of making it function for almost five billion active users globally.
</p>
</div>
<b>Source and References:</b> <u><a href="https://blog.cloudflare.com/october-2021-facebook-outage/">Cloudfare Blog/Post</a></u>, <u><a href="https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/">Facebook Blog</a></u>
</div>
</div>
<!-- END section -->
<!--Start Footer-->
<footer class="site-footer unit-5 overlay footer-color">
<div class="container">
<div class="row">
<div class="col-lg-9">
<div class="row">
<div class="col-6 col-md-6 col-lg-6 mb-5 mb-lg-0">
<img src="images/oss_logo.png" style="max-width: 100%">
</div>
<div class="col-6 col-md-3 col-lg-3 mb-5 mb-lg-0">
</div>
<div class="col-6 col-md-3 col-lg-3 mb-5 mb-lg-0">
<div class="text-center">
<h3 class="footer-heading mb-4">Quick Links</h3>
<ul class="list-unstyled">
<li><a href="index.html">Home</a></li>
<li><a href="about.html">About</a></li>
<li><a href="team.html">Team</a></li>
<li><a href="codeofconduct.html">Terms & Policies</a></li>
</ul>
</div>
<div class="text-center">
<h3 class="footer-heading mb-4">Follow Us</h3>
<span class="unit-3-icon icon" style="margin-left: 5%" >
<a href="https://www.facebook.com/OSS-Open-Student-Society-390468168410308/" target="_blank">
<img src="images/facebook.png" style="max-width: 20px">
</a>
</span>
<span class="unit-3-icon icon" style="margin-left: 5%" >
<a href="https://www.linkedin.com/company/oss2019" target="_blank">
<img src="images/linkedin.png" style="max-width: 20px">
</a>
</span>
<span class="unit-3-icon icon" style="margin-left: 5%" >
<a href="https://github.com/oss2019/oss2019.github.io/" target="_blank">
<img src="images/github.png" style="max-width: 20px">
</a>
</span>
</div>
</div>
</div>
</div>
<div class="col-lg-3">
<h3 class="footer-heading mb-4">Contact Info</h3>
<ul class="list-unstyled">
<li>
<span class="d-block text-white">Address</span>
Indian Institute of Technology, Dharwad,
Karnataka 580011
</li>
<li>
<span class="d-block text-white">Email</span>
</li>
</ul>
</div>
</div>
<div class="row pt-5 mt-5 text-center">
<div class="col-md-12">
<p>
<!-- Link back to Colorlib can't be removed. Template is licensed under CC BY 3.0. -->
Copyright © <script data-cfasync="false" src="/cdn-cgi/scripts/5c5dd728/cloudflare-static/email-decode.min.js"></script><script>document.write(new Date().getFullYear());</script> All Rights Reserved <br> OSS, IIT Dharwad<br>
</p>
<p style="font-size: 10px;">
This template is made by <a href="https://colorlib.com" target="_blank" >Colorlib</a>
<!-- Link back to Colorlib can't be removed. Template is licensed under CC BY 3.0. -->
<br>
Icons made by <a href="https://www.flaticon.com/authors/geotatah" target="_blank" title="geotatah" style="">geotatah</a> from <a href="https://www.flaticon.com/" target="_blank" title="Flaticon">www.flaticon.com </a> is licensed by <a href="http://creativecommons.org/licenses/by/3.0/" title="Creative Commons BY 3.0" target="_blank">CC 3.0 BY</a>
</p>
</div>
</div>
</div>
<div class="footer-up-btn">
<div>
<img src="images/arrow-up.png" alt="up">
</div>
</div>
</footer>
<!--End Footer-->
</div>
<script src="js/custom.js"></script>
<script src="js/jquery-3.3.1.min.js"></script>
<script src="js/jquery-migrate-3.0.1.min.js"></script>
<script src="js/jquery-ui.js"></script>
<script src="js/popper.min.js"></script>
<script src="js/bootstrap.min.js"></script>
<script src="js/owl.carousel.min.js"></script>
<script src="js/jquery.stellar.min.js"></script>
<script src="js/jquery.countdown.min.js"></script>
<script src="js/jquery.magnific-popup.min.js"></script>
<script src="js/bootstrap-datepicker.min.js"></script>
<script src="js/aos.js"></script>
<script src="js/main.js"></script>
</body>
</html>