Create tags fails during DeployJob because ec2 instance "does not exist"

[landonreed]

I'm not entirely sure if this is easily reproducible, but the following error was thrown during a DeployJob to an ELB server.

It appears to be due to the CreateTagsRequest being initiated too quickly after a new EC2 instance was initialized. Perhaps the right fix is to wait a few seconds before this request is submitted? Thoughts, @evansiroky?

Code ref

datatools-server/src/main/java/com/conveyal/datatools/manager/jobs/DeployJob.java

Lines 732 to 740 in 17ca3c4

	ec2.createTags(new CreateTagsRequest()
	.withTags(new Tag("Name", serverName))
	.withTags(new Tag("projectId", deployment.projectId))
	.withTags(new Tag("deploymentId", deployment.id))
	.withTags(new Tag("jobId", this.jobId))
	.withTags(new Tag("serverId", otpServer.id))
	.withTags(new Tag("routerId", getRouterId()))
	.withTags(new Tag("user", retrieveEmail()))
	.withResources(instance.getInstanceId())

Exception

exceptionDetails: "com.amazonaws.services.ec2.model.AmazonEC2Exception: The instance ID 'i-0d7f80fc99f3dc661' does not exist (Service: AmazonEC2; Status Code: 400; Error Code: InvalidInstanceID.NotFound; Request ID: 89713fd5-845e-4668-a7b5-268c573c91fb)↵ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)↵ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)↵ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)↵ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)↵ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)↵ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)↵ at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)↵ at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)↵ at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)↵ at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)↵ at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:22364)↵ at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:22331)↵ at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:22320)↵ at com.amazonaws.services.ec2.AmazonEC2Client.executeCreateTags(AmazonEC2Client.java:4810)↵ at com.amazonaws.services.ec2.AmazonEC2Client.createTags(AmazonEC2Client.java:4782)↵ at com.conveyal.datatools.manager.jobs.DeployJob.startEC2Instances(DeployJob.java:732)↵ at com.conveyal.datatools.manager.jobs.DeployJob.replaceEC2Servers(DeployJob.java:504)↵ at com.conveyal.datatools.manager.jobs.DeployJob.jobLogic(DeployJob.java:264)↵ at com.conveyal.datatools.common.status.MonitorableJob.run(MonitorableJob.java:142)↵ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)↵ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)↵ at java.lang.Thread.run(Thread.java:748)↵"