-
Notifications
You must be signed in to change notification settings - Fork 6
/
buildcop.html
520 lines (401 loc) · 24.9 KB
/
buildcop.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
<!DOCTYPE html>
<html lang="en-US">
<head>
<meta name="msapplication-config" content="/browserconfig.xml"/>
<meta name="viewport" content="width=device-width, initial-scale=1"/>
<meta charset="utf-8"/>
<link rel="apple-touch-icon" type="image/png" href="/apple-touch-icon.png"/>
<link rel="manifest" type="application/manifest+json" href="/site.webmanifest"/>
<link rel="mask-icon" type="image/svg+xml" href="/mask-icon.svg" color="#990000"/>
<link rel="shortcut icon" type="image/png" href="/favicon.png"/>
<title>Drake: Build Cop</title>
<meta
name="description"
content="Drake ("dragon" in Middle English) is a C++ toolbox started by the Robot
Locomotion Group at the MIT Computer Science and Artificial Intelligence
Lab (CSAIL). The development team has now grown significantly, with core
development led by the Toyota Research Institute. It is a collection of
tools for analyzing the dynamics of our robots and building control
systems for them, with a heavy emphasis on optimization-based design/
analysis.
"/>
<!--
The "Work Sans" font is licensed under the SIL Open Font License (OFL). For
more information, see:
- https://fonts.google.com/specimen/Work+Sans?preview.text_type=custom#about
- https://scripts.sil.org/cms/scripts/page.php?site_id=nrsi&id=OFL
-->
<link href="https://fonts.googleapis.com/css2?family=Space+Mono:wght@400;700&family=Work+Sans:wght@300;400;600;700;800&display=swap" rel="stylesheet"/>
<link rel="stylesheet" href="/third_party/github-styling/github-markdown.css"/>
<link rel="stylesheet" href="/third_party/dracula/syntax.css"/>
<link rel="stylesheet" href="/third_party/pylons/pylons.css"/>
<link rel="stylesheet" href="/assets/css/main.css"/>
</head>
<body>
<header class="site-header">
<div class="site-header-inner contain">
<a href="/" class="drake-logo">
<img src="/images/drake-logo-white.svg">
</a>
<div class="menu-mobile-toggle">
<span></span>
</div>
<nav class="site-menu">
<ul>
<li class="site-menu-item site-menu-item-main">
<a href="/" class="site-menu-item">Home</a>
</li>
<li class="site-menu-item site-menu-item-main">
<a href="/installation.html" class="site-menu-item">Installation</a>
</li>
<li class="site-menu-item site-menu-item-main">
<a href="/gallery.html" class="site-menu-item">Gallery</a>
</li>
<li class="site-menu-item site-menu-item-main">
API Documentation
<div class="sub">
<a href="/doxygen_cxx/index.html" class="site-menu-item">C++</a>
<a href="/pydrake/index.html" class="site-menu-item">Python</a>
</div>
</li>
<li class="site-menu-item site-menu-item-main">
Resources
<div class="sub">
<a href="/getting_help.html" class="site-menu-item">Getting Help</a>
<a href="https://deepnote.com/workspace/Drake-0b3b2c53-a7ad-441b-80f8-bf8350752305/project/Tutorials-2b4fc509-aef2-417d-a40d-6071dfed9199/%2Findex.ipynb" class="site-menu-item">Tutorials</a>
<a href="/python_bindings.html" class="site-menu-item">Python Bindings</a>
<a href="/developers.html" class="site-menu-item">For Developers</a>
<a href="/credits.html" class="site-menu-item">Credits</a>
</div>
</li>
<li class="search">
<div class="search-icon">
<!-- This is an inline SVG image of a magnifying glass. -->
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 374.9 374.84">
<path d="M235 270a148.74 148.74 0 1 1 35-35l97.74 97.74a24.37 24.37 0 0 1 0 34.58l-.4.4a24.47 24.47 0 0 1-34.58 0L235 270Zm-86.22-7.47A113.75 113.75 0 1 0 35 148.75 113.75 113.75 0 0 0 148.75 262.5Z"/>
</svg>
</div>
<div class="search-bar">
<form id="search_form" action="https://google.com/search" method="get">
<input type="text" name="q" placeholder="Search all of Drake…" />
<input type="hidden" name="q" value="site:drake.mit.edu OR site:underactuated.csail.mit.edu OR site:manipulation.csail.mit.edu" />
</form>
<div class="search-close">
<!-- This is an inline SVG image of an "X". -->
<svg height="20" width="20" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 373.61 373.57">
<path d="M219.71 186.77 366.71 40a23.43 23.43 0 1 0-33.13-33.13l-146.77 147-146.77-147A23.43 23.43 0 0 0 6.9 40l147 146.77-147 146.77a23.43 23.43 0 1 0 33.14 33.13l146.77-147 146.77 147a23.43 23.43 0 1 0 33.13-33.13Z"/>
</svg>
</div>
</div>
<ul id="results-container"></ul>
</li>
<li class="github-link">
<a href="https://github.com/RobotLocomotion/drake" class="site-menu-item">GitHub <img src="/third_party/images/GitHub-Mark-Light-64px.png" /></a>
</li>
</ul>
</nav>
</div>
</header>
<div class="page">
<div class="content">
<div class="drake-page">
<header class="drake-page-header">
<div class="contain">
<h1>Build Cop</h1>
</div>
</header>
<section class="padding">
<div class="contain">
<article class="markdown-body">
<h1 id="overview">Overview</h1>
<p>The Drake build cop monitors <a href="https://drake-jenkins.csail.mit.edu/view/Continuous%20Production/">continuous</a>,
and <a href="https://drake-jenkins.csail.mit.edu/view/Nightly%20Production/">nightly</a>
production continuous integration failures in the
<a href="https://github.com/RobotLocomotion/drake">RobotLocomotion/drake</a> GitHub
repo.</p>
<p>The build cop also checks in on the builds of the
<a href="https://github.com/RobotLocomotion/drake-external-examples/">drake-external-examples</a>
and
<a href="https://github.com/RobotLocomotion/RobotLocomotion.github.io/commits/master">automatically generated documentation</a>
repositories from time to time.</p>
<p>The build cop will rotate on a weekly basis. The
<a href="https://github.com/RobotLocomotion/drake-ci/wiki/Build-Cop-Rotation">schedule</a>
is maintained on the
<a href="https://github.com/RobotLocomotion/drake-ci">RobotLocomotion/drake-ci</a> wiki.</p>
<h1 id="process">Process</h1>
<p>Two build cops are expected to be on duty Monday through Thursday, holidays excepted. At
least one build cop should be on duty during normal business hours Eastern Time,
approximately 9am to 5pm. Developers are encouraged, but not required, to merge
pull requests during times when the build cop is on duty. Nightly
build failures will be addressed the following weekday morning.</p>
<p>When a CI build failure occurs, the build cop will be notified by email.
Notifications are sent to <code class="language-plaintext highlighter-rouge">[email protected]</code> and
<code class="language-plaintext highlighter-rouge">[email protected]</code>. To receive these notifications,
subscribe to the <code class="language-plaintext highlighter-rouge">drake-alerts</code> Google Group at TRI or the
<code class="language-plaintext highlighter-rouge">drake-developers</code> Google Group at Kitware. The build cop will triage the
failure by identifying the recent commits or other problem that caused the CI
build to fail.</p>
<p>In the case of an issue with the CI infrastructure, Kitware will be responsible
for fixing the issue.</p>
<p>In the case of an issue with recent commits, the build cop will contact the
authors of the commits and prepare a new pull request that reverts those
commits. If the authors are not responsive within 15 minutes, or are unable to
fix the failure within 60 minutes, the build cop will merge the pull request to
revert the commits and verify that the continuous builds triggered by that merge
pass.</p>
<p>In the case of failures in a <code class="language-plaintext highlighter-rouge">dev</code> directory, the build cop should disable the
failing test instead of reverting the entire commit. To disable the test, add a
<code class="language-plaintext highlighter-rouge">tags = []</code> attribute to its BUILD rule. If it only fails in certain
configurations, you can add tags for just those, e.g., <code class="language-plaintext highlighter-rouge">"no_asan"</code>. If it fails in
the default configuration or in too many configurations to list one by one, use
the tag “manual” to disable the test under all configurations.</p>
<p>In the case of intermittent failures of unclear origin or which cannot
reasonably be prevented (for example, network failures of remotely hosted
build machines), the build cop should file a drake issue (or add a comment
to an existing issue) and apply the <code class="language-plaintext highlighter-rouge">component: continuous integration</code> and
<code class="language-plaintext highlighter-rouge">type: bug</code> labels. The first time an unexplained failure occurs, close the
issue immediately – there is not much value in keeping an open issue for a
failure that only ever happened once. If the issue occurs a second time,
reopen it.</p>
<ul>
<li><a href="https://github.com/RobotLocomotion/drake/issues?q=is:issue+is:open+label:%22component:+continuous+integration%22+label:%22type:+bug%22">The list of open buildcop issues</a>
should generally be kept fairly short for quick reference.</li>
<li><a href="https://github.com/RobotLocomotion/drake/issues?q=is:issue+label:%22component:+continuous+integration%22+label:%22type:+bug%22">The list of all buildcop issues</a> may also
be helpful in determining whether to file a new issue or reopen an old one.</li>
</ul>
<p>With the exception of build failures that are quickly fixed without a code
change, every CI failure should result in an issue, comment on an existing
issue, revert PR, or fix-forward PR. There will of course be exceptions to
this rule.</p>
<p>Use the <a href="https://drakedevelopers.slack.com/messages/buildcop/details/">DrakeDevelopers Slack channel #buildcop</a>
to discuss build issues with your partner build cop and other Drake
contributors.</p>
<p>At the end of each rotation, the build cop should
notify the next build cop on the <a href="https://drakedevelopers.slack.com/messages/buildcop/details/">DrakeDevelopers Slack channel #buildcop</a>.</p>
<h1 id="revert-template">Revert Template</h1>
<p>When creating a revert PR, the build cop will assign that PR to the original
author, and include the following template in the PR description.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
Dear $AUTHOR,
The on-call build cop, $BUILD_COP, believes that your PR $NUMBER may have
broken one or more of Drake's continuous integration builds [1]. It is
possible to break a build even if your PR passed continuous integration
pre-merge because additional platforms are tested post-merge.
The specific build failures under investigation are:
$LINK_TO_BROKEN_BUILD_ON_JENKINS
$LINK_TO_BROKEN_BUILD_ON_JENKINS
Therefore, the build cop has created this revert PR and started a complete
post-merge build to determine whether your PR was in fact the cause of the
problem. If that build passes, this revert PR will be merged 60 minutes from
now. You can then fix the problem at your leisure, and send a new PR to
reinstate your change.
If you believe your original PR did not actually break the build, please
explain on this thread.
If you believe you can fix the break promptly in lieu of a revert, please
explain on this thread, and send a PR to the build cop for review ASAP.
If you believe your original PR definitely did break the build and should be
reverted, please review and LGTM this PR. This allows the build cop to merge
without waiting for CI results.
For advice on how to handle a build cop revert, see [2].
Thanks!
Your Friendly On-call Build Cop
[1] CI Production Dashboard: https://drake-jenkins.csail.mit.edu/view/Production/
[2] https://drake.mit.edu/buildcop.html#workflow-for-handling-a-build-cop-revert
</code></pre></div></div>
<h1 id="workflow-for-handling-a-build-cop-revert">Workflow for Handling a Build Cop Revert</h1>
<p>Suppose your merged PR was reverted on the master branch. What do you do?</p>
<p>Here’s one workflow:</p>
<ol>
<li>Create a new development branch based off of the <code class="language-plaintext highlighter-rouge">HEAD</code> of master.</li>
<li><a href="https://git-scm.com/docs/git-revert">Revert</a> the revert of your
originally-merged PR to get your changes back.</li>
<li>Debug the problem. This may require you to
<a href="/jenkins.html#scheduling-an-on-demand-build">run on-demand continuous integration</a> to
ensure the problem that caused your PR to be reverted was actually fixed.</li>
<li>Commit your changes into your new branch.</li>
<li>Issue a new PR containing your fixes. Be sure to link to the build cop revert
PR in your new PR.</li>
</ol>
<h1 id="build-cop-playbook">Build Cop Playbook</h1>
<p>This section is a quick-reference manual for the on-call build cop.</p>
<h2 id="monitor-the-build">Monitor the Build</h2>
<p>Check the <a href="https://drake-jenkins.csail.mit.edu/view/Continuous%20Production/">Continuous Production</a>
build dashboard in Jenkins at least once an hour during on-call hours. These
builds run after every merge to Drake. Also check the
<a href="https://drake-jenkins.csail.mit.edu/view/Nightly%20Production/">Nightly Production</a>
build dashboard every morning. These builds are unusually
resource-intensive, and therefore run at most once per day.</p>
<p>If any Continuous or Nightly Production builds turn yellow or red, you
need to act.</p>
<p>In Jenkins, builds that are in progress (blinking on and off) will show the
color of the previous build.</p>
<p>Note that CDash pages may take a minute to populate.</p>
<p>In addition, check the <a href="https://github.com/RobotLocomotion/RobotLocomotion.github.io/commits/master">automatically generated documentation repo</a>
to confirm that the latest commit has a green circle, not a red x.</p>
<h2 id="monitor-the-cache-servers">Monitor the Cache Servers</h2>
<p>Check once per week that caching is still enabled, there are currently two cache
servers that need to be confirmed. Open each of the following jobs and search
for <code class="language-plaintext highlighter-rouge">REMOTE_CACHE_KEY</code> and confirm it has a value:</p>
<ul>
<li><a href="https://drake-jenkins.csail.mit.edu/job/linux-noble-clang-bazel-continuous-release/lastBuild/consoleFull">https://drake-jenkins.csail.mit.edu/job/linux-noble-clang-bazel-continuous-release/lastBuild/consoleFull</a></li>
<li><a href="https://drake-jenkins.csail.mit.edu/job/mac-arm-sonoma-clang-bazel-continuous-release/lastBuild/consoleFull">https://drake-jenkins.csail.mit.edu/job/mac-arm-sonoma-clang-bazel-continuous-release/lastBuild/consoleFull</a></li>
</ul>
<p>Message indicating a problem:</p>
<p><code class="language-plaintext highlighter-rouge">REMOTE_CACHE_KEY =</code></p>
<p>Message indicating success:</p>
<p><code class="language-plaintext highlighter-rouge">REMOTE_CACHE_KEY = 3a677c9194643f253d3eb3fcd6b09ee370f663da30bf1f43fb547edc6851e339</code></p>
<p>The exact key hash is not important, it’s just important that it’s non-empty.</p>
<p>If there is an issue with either cache server, post the details on the
<a href="https://drakedevelopers.slack.com/messages/buildcop/details/">#buildcop</a>
channel on Slack, ensuring that <code class="language-plaintext highlighter-rouge">@betsymcphail</code> is mentioned in the message.</p>
<h2 id="monitor-the-build-queue">Monitor the Build Queue</h2>
<p>Check once per week that jobs are not stuck on the Build Queue. Occasionally,
jobs are added to the queue but never start. This is likely due to
infrastructure issues that prevent nodes from launching. In this case, as the
jobs never fail, build cops are not notified via email.</p>
<p>If jobs seems to be stuck, report the issue by tagging <code class="language-plaintext highlighter-rouge">@betsymcphail</code> on the
<a href="https://drakedevelopers.slack.com/messages/buildcop/details/">#buildcop</a>
channel on Slack.</p>
<h2 id="respond-to-breakage">Respond to Breakage</h2>
<p>There are various reasons the build might break. Diagnose the failure, and
then take appropriate action. This section lists some common failures and
recommended responses. However, build cops often have to address unexpected
circumstances. Do not expect this list to be complete, and always apply your
own judgment.</p>
<p>In almost any build breakage, the first information-gathering step is to
click on the build that is yellow or red in Jenkins, then click on the first
breaking change in the Build History. You will see a list of the new commits
in that particular run.</p>
<p>Determine if an open GitHub Drake issue describes the situation. For example,
some tests are flaky for reasons that have no known resolution, but are
described by Drake issues. If you find that your broken build is described by
such an issue, consider adding the build information to the issue for future
analysis</p>
<ul>
<li><a href="https://github.com/RobotLocomotion/drake/issues?q=is:issue+is:open+label:%22component:+continuous+integration%22+label:%22type:+bug%22">The list of open buildcop issues</a>
has most recent recurring failures, and should be the first place you look.</li>
<li><a href="https://github.com/RobotLocomotion/drake/issues?q=is:issue+label:%22component:+continuous+integration%22+label:%22type:+bug%22">The list of all buildcop issues</a>
has historical and non-recurring failures, and is a good second line of
inquiry.</li>
</ul>
<h2 id="broken-compile-or-test">Broken Compile or Test</h2>
<p>Sometimes people merge code that doesn’t compile, or that fails a test.
This can happen for several reasons:</p>
<ul>
<li>The platform or test case only runs post-merge.</li>
<li>An administrator performed an override-merge of the culprit PR,
circumventing pre-merge checks.</li>
<li>The failure is an interaction between the culprit PR and some other
recent change to master.</li>
</ul>
<p>Compile failures will be red in Jenkins. Test failures will be yellow.
Consult the list of commits in the breaking change to identify possible culprit
PRs. Try to rule out some of those PRs by comparing their contents to the
specifics of the failure. For any PRs you cannot rule out, create a rollback
by clicking “Revert” in the GitHub UI. Use the
<a href="/buildcop.html#revert-template">template message</a> to communicate with the author, and
proceed as specified in that message.</p>
<p><a href="/jenkins.html#run-specific-build">Manually schedule</a> the failing build as an
experimental build on the rollback PR. If it passes, the odds are good that you
have found the culprit. Proceed as specified in the template message.</p>
<h2 id="flaky-test">Flaky Test</h2>
<p>Sometimes people introduce code that makes a test non-deterministic, failing
on some runs and passing on others. You cannot reliably attribute a flaky test
failure to the first failing build, because it may have passed by chance for
the first few continuous builds after the culprit PR landed.</p>
<p>Test failures will be yellow in Jenkins. If the list of commits in the breaking
change does not include any plausible culprits, you may be looking at a flaky
test. Look through earlier commits one-by-one for plausible culprits.
After you identify one, create a rollback by clicking “Revert” in the
GitHub UI. Use the <a href="/buildcop.html#revert-template">template message</a> to communicate
with the author, and proceed as specified in that message.</p>
<h2 id="restarting-mac-nightly-builds">Restarting Mac Nightly Builds</h2>
<p>Occasionally there will be flaky tests or timeouts in the Mac nightly builds.
While it is tempting to restart these builds to clear the errors, Mac resources
are limited and restarting the long-running nightly builds may tie up resources
needed for continuous builds. In addition, too many simultaneous Mac builds
will increase the chances of timeouts and other flakes. Build cops should use
their best judgement, keeping in mind the following guidelines:</p>
<ul>
<li>If the nightly job is mirrored by a continuous job, don’t re-run.</li>
<li>If the test passed last build, don’t re-run.</li>
<li>If it is a linter only timeout, don’t re-run.</li>
<li>If there are many timeouts, you may consider re-running.</li>
<li>If the timed-out test failed last build (not just timed out), you may consider re-running.</li>
</ul>
<h2 id="broken-ci-script">Broken CI Script</h2>
<p>Sometimes people merge changes to the Drake CI scripts that result in spurious
CI failures. The list of commits in Jenkins for each continuous build includes
the <a href="https://github.com/RobotLocomotion/drake-ci">drake-ci</a> repository as well
as Drake proper. Consider whether those changes are possible culprits.</p>
<p>If you believe a CI script change is the culprit, contact the author.
If they are not responsive, revert the commit yourself and see what happens on
the next continuous build. There are no pre-merge builds you can run that
exercise changes to the CI scripts themselves.</p>
<h2 id="infrastructure-flake">Infrastructure Flake</h2>
<p>The machinery of the CI system itself sometimes fails for reasons unrelated to
any code change. The most common infrastructure flakes include:</p>
<ul>
<li>Unable to obtain a Gurobi license.</li>
<li>Broken connection to a Mac build agent.</li>
</ul>
<p>Infrastructure flakes will be red in Jenkins. If you believe you are looking at
an infrastructure flake, run the build manually at HEAD. If it passes, you are
definitely looking at an infrastructure flake, and no further action is
required. If you believe the rate of a particular infrastructure flake has
increased, alert Kitware by assigning a GitHub issue to <code class="language-plaintext highlighter-rouge">@BetsyMcPhail</code>.</p>
<p>Note that “slow read” warnings during Bazel builds are due to the relative
slowness of the remote storage used by the CI infrastructure when compared to
storage connected to the local bus on a local developer workstation build and
can be safely ignored.</p>
<p>If you see “All nodes of label <label> are offline”, this should disappear
eventually and the build should run, once Jenkins gets a node booted up.</label></p>
<h2 id="infrastructure-collapse">Infrastructure Collapse</h2>
<p>Occasionally, some piece of CI infrastructure completely stops working. For
instance, GitHub, AWS, or MacStadium could have an outage, or our Jenkins server
could crash or become wedged. During infrastructure collapses, lots of builds
will turn red and stay red.</p>
<p>Attempt to figure out what infrastructure collapsed. If it’s under our control,
alert Kitware by assigning a GitHub issue to <code class="language-plaintext highlighter-rouge">@BetsyMcPhail</code>. If it’s under a
vendor’s control, spread the news and simply wait it out.</p>
<h2 id="drake-external-examples">Drake External Examples</h2>
<p>Details of failures in the <a href="https://github.com/RobotLocomotion/drake-external-examples/">drake-external-examples</a>
repository, which may be denoted by red “build failing” icons at the top of the build
dashboard on Jenkins, should be posted to the <a href="https://drakedevelopers.slack.com/messages/buildcop/details/">#buildcop</a>
channel on Slack, ensuring that <code class="language-plaintext highlighter-rouge">@betsymcphail</code> is mentioned in the message.</p>
<h2 id="documentation-repo-failures">Documentation Repo Failures</h2>
<p>If the <a href="https://github.com/RobotLocomotion/RobotLocomotion.github.io/commits/master">automatically generated documentation repo</a>
fails, post in <a href="https://drakedevelopers.slack.com/messages/buildcop/details/">DrakeDevelopers Slack channel #buildcop</a>,
and try to locate the offensive Drake commit. Each commit in the documentation repo is associated with a matching Drake commit.</p>
</article>
</div>
</section>
</div>
<footer class="site-footer padding">
<div class="contain">
<a href="/" class="drake-logo">
<img src="/images/drake-logo.svg">
</a>
<div class="footer-menu">
<ul>
<li>
<a href="/doxygen_cxx/index.html" class="site-menu-item">C++</a>
</li>
<li>
<a href="/pydrake/index.html" class="site-menu-item">Python</a>
</li>
<li class="github-link">
<a href="https://github.com/RobotLocomotion/drake" class="site-menu-item">GitHub <img src="/third_party/images/GitHub-Mark-64px.png" /></a>
</li>
</ul>
</div>
</div>
<!-- TODO(eric.cousineau): Consider placing copyright here. -->
</footer>
</div>
</div>
<script src="/assets/js/mobile.js"></script>
<!-- Search -->
<script src="/assets/js/search.js"></script>
</body>
</html>